Graph-Based Retrieval vs Vector Similarity in RAG: When Chunks Fail to Connect

Apr 16, 2026

The Problem

I built a RAG system for a legal document corpus. Vector similarity worked great for simple queries like “What is the penalty for breach of contract?” But when I asked “How does case X relate to case Y through precedent Z?”, the system returned chunks from case X, chunks from case Y, and chunks mentioning precedent Z—but never connected them.

The chunks were all “similar” to my query, but similarity isn’t connection.

I tried increasing chunk size. I tried overlapping chunks. I tried hybrid retrieval (vector + keyword). None of it worked because the fundamental problem wasn’t retrieval quality—it was that vector similarity has no concept of relationships.

Why Vector Similarity Fails for Multi-Hop Queries

Traditional RAG with vector similarity does this:

Document → Chunk → Embed → Store in Vector DB
Query → Embed → Cosine Similarity → Return Top-K Similar Chunks → LLM Answer

This works when the answer lives in a single chunk. But real questions often need multiple hops:

Query: "Who collaborated with Smith on the AI project?"

Vector RAG returns:
  Chunk A: "Smith worked on AI project"     [SIMILAR]
  Chunk B: "Jones contributed to AI paper"  [SIMILAR]
  Chunk C: "AI project timeline"            [SIMILAR]

But Chunk A and Chunk B are from different documents.
They never mention each other.
The connection "Smith + Jones = collaborators" doesn't exist in any single chunk.

Microsoft’s GraphRAG documentation calls this “connecting disparate information.” Vector similarity cannot do this because it only measures semantic proximity, not structural relationships.

The Three Failure Modes

Connecting Disparate Information: Queries requiring synthesis across multiple concepts that don’t appear in the same chunk
Holistic Semantic Concepts: Questions about themes and patterns across large datasets (not individual facts)
Entity Relationships: Cannot reason about connections between extracted entities

I hit all three. My legal corpus had 500+ cases. Questions about precedent chains, judge reasoning patterns, and jurisdiction connections all failed with vector RAG.

What Graph-Based Retrieval Does Instead

GraphRAG doesn’t retrieve chunks. It navigates a knowledge graph.

Document → Entity Extraction → Relationship Extraction → Knowledge Graph
         → Community Detection → Community Summaries

Query → Local Search (entity lookup) or Global Search (community navigation)
      → Graph Traversal → Context Assembly → LLM Answer

The key difference: instead of asking “what text looks like my query?”, you ask “what entities and relationships are relevant to my query?”

How It Builds the Graph

Document: "Smith collaborated with Jones on Project X.
          Project X was funded by Company Y.
          Jones previously worked at Company Z."

LLM extracts:
  Entities: [Smith, Jones, Project X, Company Y, Company Z]
  Relationships:
    - Smith → collaborated_with → Jones
    - Smith → worked_on → Project X
    - Jones → worked_on → Project X
    - Project X → funded_by → Company Y
    - Jones → previously_at → Company Z

Knowledge Graph:
  Smith ──collab── Jones
    │              │
  worked_on    worked_on
    │              │
    └── Project X ─┘
         │
      funded_by
         │
      Company Y

  Jones ──prev_at── Company Z

Now when I ask “Who collaborated with Smith?”, the system:

Finds entity “Smith” in the graph
Traverses the “collaborated_with” relationship
Returns connected entity “Jones” with full relationship context

Not just similar text—structurally connected information.

Community Detection for Global Queries

For questions like “What are the main themes in this corpus?”, GraphRAG uses community detection:

Knowledge Graph → Leiden Algorithm → Hierarchical Communities

Community 1 (AI Research):
  Entities: [Smith, Jones, AI papers, ML conferences]
  Summary: "Research team focused on machine learning applications"

Community 2 (Corporate Funding):
  Entities: [Company Y, Company Z, funding rounds]
  Summary: "Tech companies investing in AI research"

Community 3 (Legal Framework):
  Entities: [contracts, regulations, compliance]
  Summary: "Legal considerations for AI deployment"

Global Query: "What themes connect this dataset?"
→ Retrieve Community Summaries → Synthesize holistic answer

This is impossible with vector similarity. You’d just get chunks that mention “themes” or “topics”—not actual thematic organization.

Two Search Modes: Local vs Global

GraphRAG provides two fundamentally different search modes:

Mode	What It Does	Best For
Local Search	Find specific entity, traverse relationships, assemble context from connected entities	”Who worked with Smith?” “What companies funded Project X?”
Global Search	Select relevant communities, use pre-generated community summaries	”What are the main themes?” “Summarize insights from this corpus”

Local search is entity-focused. Global search is thematic.

When I Tried GraphRAG

I implemented Microsoft GraphRAG for my legal corpus:

import asyncio
import pandas as pd
import graphrag.api as api
from graphrag.config.load_config import load_config
from pathlib import Path

async def legal_search():
    config = load_config(Path("./legal_corpus"))

    # Load graph-structured data
    entities = pd.read_parquet("./output/entities.parquet")
    relationships = pd.read_parquet("./output/relationships.parquet")
    communities = pd.read_parquet("./output/communities.parquet")
    community_reports = pd.read_parquet("./output/community_reports.parquet")
    text_units = pd.read_parquet("./output/text_units.parquet")

    # Local search: Find precedent chain
    response, context = await api.local_search(
        config=config,
        entities=entities,
        relationships=relationships,
        communities=communities,
        community_reports=community_reports,
        text_units=text_units,
        community_level=2,
        response_type="Multiple Paragraphs",
        query="How does Roe v. Wade connect to subsequent abortion cases?"
    )

    return response, context

asyncio.run(legal_search())

The context returned included:

Entity “Roe v. Wade” with all connected entities
Relationship chains to subsequent cases
Relevant text units from multiple documents
Community context showing the broader legal landscape

For the first time, my system understood that cases connect through precedent—not just that they’re semantically similar.

But Here’s the Catch: Setup Overhead

GraphRAG requires upfront processing:

1. Index your documents (entity + relationship extraction)
   - Uses LLM calls for each document
   - Cost: ~$0.50-$2 per document (depends on length + model)

2. Build knowledge graph
   - Store entities and relationships
   - Parquet files or Neo4j

3. Community detection
   - Run Leiden algorithm
   - Generate community summaries (more LLM calls)

4. Maintain the graph
   - New documents require re-processing
   - Community structure may need updates

For a 500-document corpus, my indexing cost was ~$150. A vector-only approach would have been ~$5.

Is it worth it? Depends on your queries.

Decision Framework

QUERY TYPE ANALYSIS
──────────────────────────────────────────────────────────────

Start: What kind of query?

┌─ Multi-hop reasoning needed?
│  │
│  ├─ YES → Cross-document connections?
│  │        │
│  │        ├─ YES → Use GraphRAG
│  │        │         (entity extraction overhead, best for
│  │        │          thematic analysis, relationship queries)
│  │        │
│  │        └─ NO → Single complex document?
│  │                 │
│  │                 ├─ YES → Use Tree-based/Chunkless
│  │                 │        (preserves tables, figures, sections)
│  │                 │
│  │                 └─ NO → Use GraphRAG anyway
│  │                         (better for entity relationships)
│  │
│  └─ NO → Complex document structure?
│          │
│          ├─ YES (tables, figures matter) → Tree-based/Chunkless
│          │
│          └─ NO → Simple factual query?
│                   │
│                   ├─ YES → Vector Similarity
│                   │        (lowest overhead, fastest)
│                   │
│                   └─ NO → Vector Similarity
│                          (good enough for most cases)

Choose GraphRAG when:

Queries need to connect information across documents
You need thematic analysis or pattern discovery
Entity relationships matter (who, what, where connections)
Questions about holistic concepts across the corpus

Choose Vector Similarity when:

Simple factual queries
Large corpus of unrelated documents (GraphRAG overhead not worth it)
Speed and simplicity are priorities
No complex relationships to reason about

Choose Tree-based/Chunkless when:

Document structure matters (tables connected to captions)
Single complex document (technical specs, research papers)
You want LLM to navigate structure, not search chunks

Note: Tree-based approaches struggle with multi-document systems. Reddit discussion pointed this out: “This doesn’t seem like it would be a RAG system replacement though… it can’t really manage a knowledge base/lots of documents.”

What About Hybrid Approaches?

Neo4j GraphRAG lets you combine graph traversal with vector search:

from neo4j import GraphDatabase
from neo4j_graphrag.retrievers import VectorCypherRetriever
from neo4j_graphrag.generation import GraphRAG
from neo4j_graphrag.llm import OpenAILLM

driver = GraphDatabase.driver("neo4j://localhost:7687")

# Vector search + graph traversal in one query
retriever = VectorCypherRetriever(
    driver=driver,
    index_name="document-embeddings",
    retrieval_query="""
        WITH node, score
        // After vector match, traverse graph
        OPTIONAL MATCH (author:Person)-[:AUTHORED]->(node)
        OPTIONAL MATCH (node)-[:REFERENCES]->(ref:Document)
        RETURN node.title, node.content,
               collect(author.name) AS authors,
               collect(ref.title) AS references,
               score
    """
)

llm = OpenAILLM(model_name="gpt-4o")
rag = GraphRAG(retriever=retriever, llm=llm)

result = rag.search(
    query_text="What themes connect these research papers?"
)

This gives you vector similarity for initial retrieval, then graph traversal for context enrichment. Best of both worlds—but requires maintaining both vector index and knowledge graph.

During my research, I discovered related concepts that overlap with GraphRAG:

Hierarchical Chunking

Keeps parent-child relationships between chunks:

Still uses vector similarity at chunk level
Maintains some structure context
Less overhead than full GraphRAG

Not the same as GraphRAG. GraphRAG extracts entities and builds a true knowledge graph; hierarchical chunking just keeps chunk ancestry.

Chunkless RAG (Docling)

Preserves document tree structure:

Sections, tables, figures stay connected
LLM navigates the tree instead of searching chunks
Good for single complex documents

But Reddit commenters noted it may not scale to knowledge bases: “it can’t really manage a knowledge base/lots of documents.”

PageIndex

Mentioned in Reddit discussion as similar concept. I couldn’t find detailed documentation, but it appears to use document structure for retrieval like Chunkless RAG.

What I Learned

After implementing both approaches:

Vector similarity is fast but blind to relationships. Good for “find similar text”, bad for “find connected information.”
GraphRAG overhead is real but worthwhile for complex queries. My legal corpus indexing cost 30x more than vector-only, but precedent chain queries actually worked.
The right tool depends on query type. Don’t default to vector similarity. Ask: “Does my query need connections or just similarity?”
Hybrid approaches are emerging. Neo4j’s VectorCypherRetriever shows you can combine both—use vector for initial hit, then traverse graph for context.
Document structure approaches have limits. Tree-based/Chunkless works for single documents but struggles with multi-document knowledge bases.

The future isn’t vector-only or graph-only. It’s hybrid retrieval that knows when to search similarity and when to traverse structure.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 Microsoft GraphRAG GitHub
👨‍💻 Microsoft GraphRAG Documentation
👨‍💻 Neo4j GraphRAG Python
👨‍💻 Reddit: Docling Agent + Chunkless RAG Discussion
👨‍💻 Docling Project

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!