What Chunk Size and Overlap Should You Use for RAG Document Chunking?

Mar 22, 2026

Cowrie

Dev @ Bswen

My RAG pipeline was returning garbage answers. Queries about specific code snippets returned completely irrelevant documentation. Queries about “authentication” pulled chunks about “authorization” and mixed them together.

I spent weeks tweaking embedding models, trying different vector databases, and rewriting prompts. Nothing helped.

The problem? My chunk size was 500 tokens with zero overlap.

My terrible initial config

Chunk 47: "...the user must authenticate via the API key which should be stored"
Chunk 48: "securely in the environment variable and never committed to git..."

Query: "How do I handle API keys?"
Result: Chunk 47 (irrelevant partial) + Chunk 52 (completely wrong doc)

The authentication instructions were split across chunks 47 and 48. Neither chunk alone made sense. When I searched for “API key handling,” I got fragments without context.

The Core Problem

Chunk size is the most underrated RAG parameter. It’s also the one nobody gives you straight answers about.

• Too large: You stuff irrelevant context into your LLM, waste tokens, and dilute the answer quality
• Too small: You fragment semantic meaning, losing the thread of ideas
• No overlap: Information at chunk boundaries gets lost forever
• Too much overlap: Redundant storage, redundant retrieval, wasted computation

After months of trial and error in production, here’s what actually works.

The Short Answer

Start with chunk_size=1000 and chunk_overlap=200 (20% overlap) for most documents.

baseline_chunking.py

from langchain.text_splitter import RecursiveCharacterTextSplitter
import tiktoken

def tiktoken_len(text):
    encoder = tiktoken.encoding_for_model("gpt-4")
    return len(encoder.encode(text))

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,        # ~750 words, ~1-2 paragraphs
    chunk_overlap=200,       # 20% overlap - THIS IS CRITICAL
    length_function=tiktoken_len,
    separators=["\n\n", "\n", ". ", " ", ""],
)

Why these numbers?

Parameter	Value	Rationale
chunk_size	1000	Fits ~1-2 paragraphs, good semantic unit
chunk_overlap	200	Ensures no info lost at boundaries
20% ratio	Proven	Balance between redundancy and completeness

Why 1000 Tokens?

1000 tokens ≈ 750 words ≈ 1-2 coherent paragraphs. This is a natural semantic unit for most documents.

Visual representation

┌─────────────────────────────────────────────────────────┐
│ Chunk 1 (1000 tokens)                                   │
│                                                         │
│ ...end of paragraph 1......│┌──start of paragraph 2... │
├─────────────────────────────┼───────────────────────────┤
│                    Overlap (200 tokens)                  │
└─────────────────────────────┴───────────────────────────┘
┌─────────────────────────────┬───────────────────────────┐
│                    Overlap  │ Chunk 2 (1000 tokens)     │
│                             │                           │
│ ...end of paragraph 2......││───start of paragraph 3... │
└─────────────────────────────┴───────────────────────────┘

Without overlap, a sentence like “The authentication token expires after 24 hours and must be refreshed” might get split as:

No overlap disaster

Chunk 5: "The authentication token expires after"
Chunk 6: "24 hours and must be refreshed"

Neither chunk contains the complete thought. Query “token expiration” returns incomplete garbage.

With 200 token overlap:

With overlap - both chunks have the answer

Chunk 5: "...The authentication token expires after 24 hours and must be refreshed..."
Chunk 6: "...The authentication token expires after 24 hours and must be refreshed..."

Both chunks contain the complete answer. Your retrieval now works.

Document Type Matters (A Lot)

One size does not fit all. I learned this the hard way when my technical documentation chunks worked great, but my FAQ retrieval completely failed.

Chunk size by document type

┌─────────────────────┬───────────────┬─────────────────────────┐
│ Document Type       │ chunk_size    │ Why                     │
├─────────────────────┼───────────────┼─────────────────────────┤
│ Technical docs      │ 1000-1500     │ Concepts span paragraphs│
│ Code files          │ 500-1000      │ Function boundaries     │
│ FAQ / Q&A           │ 300-500       │ Self-contained entries  │
│ Legal contracts     │ 1500-2000     │ Long clause spans       │
│ News articles       │ 800-1000      │ Paragraph structure     │
│ Academic papers     │ 1000-1500     │ Abstract concepts       │
└─────────────────────┴───────────────┴─────────────────────────┘

Technical Documentation: 1000-1500

Technical docs have concepts that span multiple paragraphs. A chunk of 500 tokens cuts off explanations mid-stream.

technical_chunker.py

technical_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1200,
    chunk_overlap=240,  # 20%
    length_function=tiktoken_len,
    separators=["\n\n", "\n", ". ", " ", ""],
)

Code Files: 500-1000

Code is different. You want chunks that contain complete functions or classes, not fragments.

code_chunker.py

code_splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=160,
    length_function=tiktoken_len,
    separators=["\nclass ", "\ndef ", "\n\t", "\n", " ", ""],
)

The separators are different too. Split on class/function boundaries, not arbitrary character counts.

FAQ/Q&A: 300-500

FAQ entries are self-contained. A 1000-token chunk would contain multiple Q&As, diluting relevance.

faq_chunker.py

faq_splitter = RecursiveCharacterTextSplitter(
    chunk_size=400,
    chunk_overlap=80,
    length_function=tiktoken_len,
    separators=["\n\n", "\n", " ", ""],
)

The Overlap Formula

Overlap should be 10-25% of chunk size. Here’s when to use each:

Overlap guidelines

┌────────────┬─────────────────────────────────────────────┐
│ 10% overlap │ Large chunks (1500+), minimal boundary loss│
│ 15-20%     │ Standard (1000), RECOMMENDED DEFAULT       │
│ 25%        │ Small chunks (500), critical context       │
└────────────┴─────────────────────────────────────────────┘

The smaller your chunks, the more overlap you need. With small chunks, every boundary is a potential information loss point.

The Real Formula

Your chunk size is constrained by your embedding model’s context window:

Context window math

optimal_chunk_size = embedding_context_window - prompt_overhead - retrieved_docs_buffer

For OpenAI text-embedding-3-small (8191 token limit):

context_calculation.py

EMBEDDING_LIMIT = 8191
PROMPT_OVERHEAD = 500      # System prompt, instructions
RETRIEVED_DOCS_BUFFER = 1000  # Multiple chunks retrieved

MAX_SAFE_CHUNK = EMBEDDING_LIMIT - PROMPT_OVERHEAD - RETRIEVED_DOCS_BUFFER
# = 8191 - 500 - 1000 = 6691 tokens (theoretical max)

# In practice, stay well under:
RECOMMENDED_MAX = 2000  # Safe for multiple retrieved chunks

Don’t push the limit. You need room for multiple retrieved chunks plus your prompt.

Production Configuration

Here’s what I use in production now:

optimized_chunker.py

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.schema import Document
import tiktoken

class OptimizedChunker:
    """Production-ready chunker with document-type awareness."""

    ENCODER = tiktoken.encoding_for_model("gpt-4")

    def __init__(self):
        self.configs = {
            "technical": {"chunk_size": 1200, "overlap": 240},
            "code": {"chunk_size": 800, "overlap": 160},
            "faq": {"chunk_size": 400, "overlap": 80},
            "legal": {"chunk_size": 1800, "overlap": 360},
            "news": {"chunk_size": 900, "overlap": 180},
            "default": {"chunk_size": 1000, "overlap": 200},
        }

    def _token_len(self, text: str) -> int:
        return len(self.ENCODER.encode(text))

    def chunk(
        self, documents: list[Document], doc_type: str = "default"
    ) -> list[Document]:
        config = self.configs.get(doc_type, self.configs["default"])

        splitter = RecursiveCharacterTextSplitter(
            chunk_size=config["chunk_size"],
            chunk_overlap=config["overlap"],
            length_function=self._token_len,
            separators=["\n\n", "\n", ". ", " ", ""],
        )

        return splitter.split_documents(documents)


# Usage
chunker = OptimizedChunker()
technical_chunks = chunker.chunk(docs, doc_type="technical")
faq_chunks = chunker.chunk(faq_docs, doc_type="faq")

Testing Different Sizes

Don’t guess. Test. Here’s how I systematically evaluated chunk sizes:

chunk_tester.py

from ragas import evaluate
from ragas.metrics import context_recall, faithfulness

def test_chunk_sizes(documents, sizes=[500, 1000, 1500, 2000]):
    """Test multiple chunk sizes and measure retrieval quality."""
    results = {}

    for chunk_size in sizes:
        overlap = int(chunk_size * 0.2)  # 20% overlap

        splitter = RecursiveCharacterTextSplitter(
            chunk_size=chunk_size,
            chunk_overlap=overlap,
            length_function=tiktoken_len,
        )

        chunks = splitter.split_documents(documents)

        results[chunk_size] = {
            "num_chunks": len(chunks),
            "overlap": overlap,
            "avg_tokens": sum(
                tiktoken_len(c.page_content) for c in chunks
            ) / len(chunks),
        }

        print(f"chunk_size={chunk_size}: {len(chunks)} chunks created")

    return results

For actual quality measurement, use RAGAS:

quality_evaluation.py

def evaluate_chunk_config(chunk_size: int, documents, test_questions):
    """Evaluate retrieval quality for a specific chunk configuration."""

    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=int(chunk_size * 0.2),
    )

    chunks = splitter.split_documents(documents)
    vectorstore = build_vectorstore(chunks)  # Your vector DB setup

    # Run test queries and evaluate
    dataset = create_ragas_dataset(vectorstore, test_questions)
    scores = evaluate(dataset, metrics=[context_recall, faithfulness])

    return {
        "chunk_size": chunk_size,
        "context_recall": scores["context_recall"],
        "faithfulness": scores["faithfulness"],
    }

Common Mistakes I Made

1. One Size For Everything

My first mistake was using chunk_size=1000 for everything. FAQs were chunked into garbage. Code was cut mid-function.

Fix: Detect document type and apply appropriate settings.

2. Ignoring Embedding Limits

I pushed chunks to 3000 tokens. Then my retrieval failed because combined context exceeded limits.

Fix: Stay under 2000 tokens per chunk for safety.

3. Zero Overlap

I thought overlap was wasteful. It’s not. It’s essential.

My terrible no-overlap config

Chunks created: 500
Retrieval accuracy: 45%
Context completeness: Poor

Fix: Always use at least 15% overlap.

4. Not Measuring

I tweaked settings based on “feel” instead of data.

Fix: Use RAGAS or similar to measure context recall and faithfulness systematically.

When To Adjust

Increase chunk_size when:

• Answers need more context (technical docs, legal)
• Retrieval is too fragmented
• Your embedding model has a large context window

Decrease chunk_size when:

• Answers are too long and unfocused
• Documents are short and self-contained (FAQs)
• You need more granular retrieval

Increase overlap when:

• Information at boundaries is getting lost
• You’re using small chunks
• Precision is more important than efficiency

Decrease overlap when:

• Storage/computation costs are high
• Your chunks are large (1500+)
• Redundancy is causing confusion

Quick Reference

Quick reference card

┌─────────────────────────────────────────────────────────────┐
│                    CHUNK SIZE CHEATSHEET                    │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  DEFAULT START: chunk_size=1000, overlap=200 (20%)         │
│                                                             │
│  Technical docs:  1000-1500 + 20% overlap                  │
│  Code:            500-1000  + 20% overlap                   │
│  FAQ:             300-500   + 20% overlap                  │
│  Legal:           1500-2000 + 20% overlap                  │
│                                                             │
│  FORMULA:                                                   │
│  chunk_size < embedding_limit - 1500                      │
│  overlap = chunk_size * 0.20                               │
│                                                             │
│  ALWAYS:                                                    │
│  - Test with your actual documents                         │
│  - Measure retrieval quality (RAGAS)                       │
│  - Consider document type                                   │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Summary

Stop guessing. Start with chunk_size=1000, overlap=200. Adjust based on document type. Measure retrieval quality. Iterate.

The right chunk size is the one that gives your LLM complete, relevant context without noise. Everything else is optimization.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

• 👨‍💻 LangChain Text Splitters Documentation
• 👨‍💻 RAGAS Evaluation Framework
• 👨‍💻 RecursiveCharacterTextSplitter API Reference

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Comments

Tags:

• ai
• llm
• database

ChatGPT Plus vs Pro: Why Multiple Accounts Beat One Premium Subscription

How to Force Claude to Cite Sources: Citation Requirements in System Prompts