Chunkless RAG vs Traditional Chunking: Why Your RAG Pipeline Quality Suffers
The Problem
I spent weeks debugging a RAG pipeline for technical documentation. The retrieval results were terrible. Queries about “database connection pooling” returned chunks about “connection strings” without the pooling configuration tables. Questions about “performance metrics” fetched random paragraphs mentioning “metrics” but missed the actual benchmark tables.
The root cause was always the same: chunking destroyed document structure.
Query: "What are the database connection pool settings?"
Traditional RAG Retrieved: - Chunk 23: "...connection pool is configured..." - Chunk 45: "...settings include timeout..." - Chunk 67: "...the table below shows..."
Problem: The actual configuration TABLE was in chunk 23-25, split into three pieces. The caption explaining the table was in chunk 22. The query never retrieved all pieces together.I tried every chunking strategy: fixed-size, semantic, recursive, overlapping. Each had trade-offs but none solved the fundamental problem: tables, figures, and sections belong together, but chunking tears them apart.
Why Chunking Breaks Documents
Traditional RAG follows this pipeline:
Document → Chunk → Embed → Vector DB → Query → Cosine Similarity → Top-K Chunks → LLM
Problems at each stage: Chunk: Breaks tables, separates figures from captions, flattens hierarchy Embed: Each chunk loses context of its neighbors Retrieve: Similarity doesn't know that chunk A belongs with chunk BI saw these patterns repeatedly:
Table fragmentation: A 10-row configuration table split across 3 chunks. The header row in one chunk, data rows in others. Queries retrieved individual rows without context.
Figure-caption separation: A diagram explaining the architecture was chunk 15. Its caption describing what it shows was chunk 14. Queries about architecture retrieved the caption but not the diagram.
Section hierarchy loss: “Section 3.2 Performance Tuning” with subsections 3.2.1, 3.2.2, 3.2.3 became flat chunks. Queries about “performance tuning” retrieved 3.2.2 without its parent context.
I tried larger chunks. That helped some queries but hurt others. Longer chunks diluted relevance - the signal got buried in noise.
I tried overlapping chunks. That helped tables sometimes, but doubled storage and still missed the structural relationship.
I tried semantic chunking with boundaries at paragraph breaks. That worked for plain text but failed on documents with tables, code blocks, and nested sections.
The pattern was clear: I was fighting against a flawed abstraction.
The Solution: Chunkless RAG
I found a Reddit discussion about Docling’s Chunkless RAG approach. The core idea hit me immediately:
“Instead of the classic chunk+embed+cosine pipeline, use graph/tree structures that preserve document hierarchy. Sections, tables, figures stay connected.”
This reframed the problem entirely. Instead of asking “how should I chunk?”, Chunkless RAG asks “why am I chunking at all?”
Traditional Approach: Document → Flatten → Chunk → Lose structure → Fight with retrieval
Chunkless RAG Approach: Document → Parse structure → Keep tree → Navigate hierarchy → Better retrievalThe key insight: Documents already have structure. Headings, sections, tables, figures, captions - these relationships exist naturally. Chunking destroys them. Chunkless RAG preserves them.
How Document Structure Looks
Docling (the IBM document parsing library) extracts rich document representation:
Document├── Section 1: Introduction│ ├── Paragraph: "This document describes..."│ └── Figure: architecture_diagram.png│ └── Caption: "Figure 1: System Architecture"│├── Section 2: Configuration│ ├── Paragraph: "Database settings..."│ └── Table: connection_pool_config│ ├── Header: ["Setting", "Default", "Description"]│ ├── Row 1: ["maxPoolSize", "10", "Maximum connections"]│ ├── Row 2: ["timeout", "30s", "Connection timeout"]│ └── Caption: "Table 1: Pool Configuration"│└── Section 3: Performance ├── Subsection 3.1: Benchmarks │ ├── Paragraph: "We tested..." │ └── Table: benchmark_results │ └── Code Block: benchmark_test.py │ └── Subsection 3.2: Tuning ├── Paragraph: "To optimize..." └── List: tuning_stepsInstead of flattening this to [chunk1, chunk2, chunk3...], Chunkless RAG keeps the tree intact. Retrieval navigates the structure, not similarity scores.
Comparison: Chunkless vs Traditional vs Hierarchical
I mapped out the approaches:
┌─────────────────────────────────────────────────────────────────────────────┐│ TRADITIONAL CHUNKING ││ ││ Document → [chunk1] [chunk2] [chunk3] [chunk4] [chunk5] ││ ↓ ↓ ↓ ↓ ↓ ││ [embed1] [embed2] [embed3] [embed4] [embed5] ││ ││ Query → Similarity Match → Retrieve [chunk3] [chunk4] ││ ││ Problem: chunk3 and chunk4 are neighbors by coincidence, not structure ││ Tables split, figures disconnected, hierarchy flattened │└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐│ HIERARCHICAL CHUNKING ││ ││ Document → Parent chunks + Child chunks ││ ↓ ↓ ││ [embed_parent] [embed_child1] [embed_child2] ││ ││ Query → Match child → Retrieve child + parent ││ ││ Improvement: Gets context from parent chunk ││ Problem: Still flattening. Parent-child is artificial, not real structure ││ Tables still split. Figures still separated. │└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐│ CHUNKLESS RAG ││ ││ Document → Tree Structure (preserved) ││ ↓ ││ Navigate: Section → Subsection → Table ││ ││ Query → Structural match → Retrieve [Section + Table + Caption] ││ ││ Advantages: Table stays with caption ││ Figure stays with its description ││ Section context preserved ││ No arbitrary boundaries │└─────────────────────────────────────────────────────────────────────────────┘The comparison shows why hierarchical chunking only partially addresses the problem. It adds parent-child relationships, but those relationships are still chunk-based abstractions, not the document’s natural structure.
When Chunkless RAG Works Best
After understanding the approach, I identified ideal use cases:
✓ Technical documentation with tables and code blocks✓ Research papers with figures and citations✓ Legal contracts with clauses and cross-references✓ API documentation with parameter tables and examples✓ Manuals with step-by-step procedures and diagrams
✗ Simple Q&A over flat text (traditional chunking still fine)✗ Large corpus of unrelated short documents✗ Quick prototyping where quality isn't criticalFor my technical documentation RAG, this was exactly the problem I faced. Tables of configuration settings. Diagrams of architecture. Nested sections explaining features. Chunkless RAG matched my needs.
What I Don’t Know Yet
The Reddit discussion was about an announcement, not detailed documentation. Several questions remain unanswered:
1. Exact API: How do queries traverse the tree structure? - XPath-like queries? - Semantic matching on node content? - Hybrid approach?
2. Hybrid support: Can Chunkless RAG combine structure + embedding? - Structure for navigation - Embeddings for semantic similarity within nodes
3. Performance benchmarks: No published comparison data yet - Retrieval accuracy vs traditional chunking - Latency comparison - Memory/storage requirements
4. Query types: Which queries benefit most? - Point queries (specific table lookup) - Exploratory queries (broad topic coverage) - Relationship queries (figure + explanation together)Docling’s repository exists, but Chunkless RAG implementation details are marked “coming soon.” The concept is sound, but practical guidance awaits official documentation.
Community Debate Points
The Reddit thread raised valid questions about overlap with existing approaches:
┌─────────────────────────────────────────────────────────────────────────────┐│ "Isn't this just hierarchical chunking?" ││ ││ Response: Similar intent, different abstraction ││ Hierarchical chunking: chunks with parent-child metadata ││ Chunkless RAG: actual document tree structure ││ ││ The difference: hierarchical chunking still breaks documents into pieces. ││ Chunkless RAG keeps documents intact and navigates them. │└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐│ "What about Graph-RAG?" ││ ││ Response: Different focus ││ Graph-RAG: Entity extraction, knowledge graph construction ││ Chunkless RAG: Document structure preservation ││ ││ Graph-RAG finds entities and relationships across documents. ││ Chunkless RAG keeps one document's internal structure coherent. ││ They could complement each other. │└─────────────────────────────────────────────────────────────────────────────┘These distinctions matter for choosing the right approach. Chunkless RAG isn’t hierarchical chunking with better marketing - it’s a genuinely different abstraction.
Summary
Chunkless RAG reframes the retrieval problem: stop destroying document structure and start using it. The core insight:
- Chunking is the root cause: Tables split, figures disconnected, hierarchy flattened
- Document structure has value: Sections, captions, relationships contain semantic information
- Navigation over similarity: Use the tree structure for retrieval, not cosine similarity alone
- Docling enables this: Rich document parsing makes structural retrieval possible
For technical documentation RAG, this approach directly addresses the quality issues I struggled with. Tables would stay with their captions. Figures would stay with their descriptions. Section context would be preserved.
The practical implementation details await official documentation. But the concept has changed how I think about RAG design: instead of asking “how should I chunk?”, I now ask “can I avoid chunking at all?”
Final Words + More Resources
My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me
Here are also the most important links from this article along with some further resources that will help you in this scope:
- 👨💻 Docling GitHub Repository
- 👨💻 Reddit: Chunkless RAG Discussion
- 👨💻 Docling Document Parser Documentation
Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!
Comments