Retrieval-Augmented Generation (RAG) systems retrieve relevant text chunks from a knowledge base before generating answers. The chunking strategy—how you split documents—shifts accuracy by over 60% in production benchmarks.

Poor chunking causes fragmented context, retrieval failure, and token waste. This guide compares three strategies based on 2026 benchmarks: recursive character splitting (default), semantic chunking (topic-aware), and LLM-based chunking (highest recall).

Key takeaway: Recursive splitting wins for most cases. Semantic chunking excels in specialized domains with metadata.

Recursively splits text by separators (paragraphs, sentences, words) to create uniform 512-token chunks with 10-20% overlap.

LangChain Implementation

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = splitter.split_text(text)
print(f"Created {len(chunks)} chunks")

Best practices: 512 tokens optimal, 10-20% overlap, customize separators for code/docs, add metadata for filtering.

Analyzes sentence embeddings to group semantically similar content, splitting at topic shifts.

LangChain Implementation

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()
chunker = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95
)

chunks = chunker.split_text(text)

When it wins: Financial reports (0.70 vs 0.55), legal docs (0.68 vs 0.50), technical manuals (0.75 vs 0.60)—but only with metadata filtering.

Hybrid: Recursive + Semantic

Combine coarse recursive split by headers with fine semantic split within sections.

Metadata Enrichment

from langchain.docstore.document import Document

chunks_with_metadata = [
    Document(
        page_content=chunk,
        metadata={'source': 'manual.pdf', 'page': i}
    )
    for i, chunk in enumerate(chunks)
]

Production Checklist

• Benchmark on your domain with real queries
• Monitor chunk stats: avg size, retrieval precision, accuracy
• Version embeddings for A/B testing
• 12-20% overlap boosts recall 20-40%
• Cache embeddings—don't re-embed on every query

Should I use recursive or semantic chunking?

Start with recursive (512 tokens, 20% overlap)—it wins in 2026 benchmarks. Switch to semantic only for dense technical docs with metadata filtering.

What chunk size should I use?

512 tokens is optimal. Smaller (128-256) fragments context; larger (1024+) dilutes relevance. Adjust by domain: code (256-512), legal (768-1024).

How much overlap between chunks?

10-20%. For 512-token chunks, use 50-100 token overlap. This boosts recall by 20-40%.

Why does semantic chunking underperform?

Variable chunk sizes fragment retrieval (more vectors = lower scores). Wins only with metadata filtering and overlap.

Can I use LLM-based chunking in production?

Yes, but resource-heavy (API calls per chunk). Best for high-value use cases where 0.919 recall justifies cost.