How to Debug RAG Systems When Answers Are Wrong: A Step-by-Step Guide

Mar 26, 2026

Problem

When my RAG system returned wrong answers, I had no idea where to start debugging. Was it the retrieval? The chunking? The prompt? The model?

I found myself randomly tweaking parameters—changing chunk sizes, swapping embedding models, adjusting temperature—hoping something would work. Sometimes things improved, but I didn’t know why. Other times, they got worse.

u/Lucky-Duck-2968 on Reddit captured this perfectly: “The hardest part is debugging. You tweak chunk sizes, swap embedding models, adjust prompts, maybe add a reranker… and sometimes things improve, but you don’t really know why.”

Environment

Python 3.11
LangChain for RAG pipeline
Pinecone for vector storage
OpenAI GPT-4 for generation
No evaluation or tracing infrastructure (the problem)

What happened?

My RAG pipeline looked like this:

def answer_question(query):
    # Retrieve
    docs = vector_store.similarity_search(query, k=5)

    # Generate
    context = "\n\n".join([d.page_content for d in docs])
    response = llm.invoke(f"Answer based on context:\n{context}\n\nQuestion: {query}")

    return response

When this returned a wrong answer, I had no visibility into what went wrong. The pipeline was a black box.

Common failure modes I encountered:

Retrieval failure: The right documents weren’t retrieved
Missing context: Retrieved chunks didn’t have complete information
Context ignored: The model didn’t use the provided context
Hallucination: The model made up information not in the context

But without instrumentation, I couldn’t tell which one happened.

How to solve it?

Step 1: Implement Layer-by-Layer Tracing

I added logging at each pipeline stage:

class RAGDebugger:
    def debug_query(self, query: str):
        trace = {
            'query': query,
            'stages': {}
        }

        # Stage 1: Retrieval
        retrieved_docs = self.retriever.search(query, k=20)
        trace['stages']['retrieval'] = {
            'total_candidates': len(retrieved_docs),
            'top_k_scores': [doc.score for doc in retrieved_docs[:5]],
            'all_documents': [{'content': doc.text[:200],
                               'score': doc.score,
                               'metadata': doc.metadata}
                              for doc in retrieved_docs]
        }

        # Stage 2: Reranking (if applicable)
        reranked = self.reranker.rerank(query, retrieved_docs, k=5)
        trace['stages']['reranking'] = {
            'selected_docs': len(reranked),
            'relevance_scores': [doc.rerank_score for doc in reranked]
        }

        # Stage 3: Context assembly
        context = self.assemble_context(reranked)
        trace['stages']['context'] = {
            'total_tokens': self.count_tokens(context),
            'chunk_count': len(reranked),
            'context_preview': context[:500]
        }

        # Stage 4: Generation
        response = self.llm.generate(query, context)
        trace['stages']['generation'] = {
            'model_used': self.llm.model_name,
            'response': response,
            'token_usage': response.usage
        }

        return trace

Now when an answer is wrong, I can inspect the trace to see exactly where things broke.

Step 2: Ask the Right Questions

I changed how I approach debugging:

Old Question	New Question
”Did I retrieve something relevant?"	"Did I retrieve ALL necessary information?"
"Is the answer wrong?"	"Where in the pipeline did correctness break down?"
"Should I increase chunk size?"	"What’s the optimal chunk size for my document structure?”

The shift from vague to specific questions changed everything.

Step 3: Build Evaluation Datasets

I created a test set with known answers:

evaluation_set = [
    {
        'query': 'What is the refund policy?',
        'expected_sources': ['policy.md#refunds'],
        'expected_answer_contains': ['30 days', 'full refund'],
        'critical_facts': ['30-day window', 'original payment method']
    },
    {
        'query': 'Compare the refund policies for premium vs basic users',
        'expected_sources': ['premium_policy.md', 'basic_policy.md'],
        'requires_comparison': True
    }
]

Then I could run systematic evaluation:

def evaluate_rag_system(test_cases):
    results = []
    for case in test_cases:
        trace = rag.debug_query(case['query'])

        # Check retrieval recall
        retrieved_sources = [doc['metadata']['source']
                           for doc in trace['stages']['retrieval']['all_documents']]
        recall = len(set(case['expected_sources']) & set(retrieved_sources)) / len(case['expected_sources'])

        # Check answer quality
        answer = trace['stages']['generation']['response']
        answer_quality = all(phrase in answer
                           for phrase in case['expected_answer_contains'])

        results.append({
            'query': case['query'],
            'retrieval_recall': recall,
            'answer_correctness': answer_quality,
            'trace': trace
        })

    return results

Step 4: Identify Failure Patterns

With traces, I could categorize failures:

Failure Mode	Root Cause	Fix
Low retrieval recall	Query-document mismatch	Query expansion, hybrid search
Missing context	Chunking breaks continuity	Larger chunks, overlapping chunks
Context not utilized	Context too long or irrelevant	Reranking, context compression
Hallucination	Model overrides context	Stronger prompts, citation requirements

Step 5: Measure Before Tweaking

The biggest change—I stopped making random changes:

# Before any changes, establish baseline
baseline_metrics = evaluate_rag_system(test_set)
print(f"Baseline recall: {baseline_metrics['avg_recall']:.2f}")

# Make ONE change
rag.retriever.set_chunk_size(512)

# Measure impact
new_metrics = evaluate_rag_system(test_set)
print(f"Change impact: {new_metrics['avg_recall'] - baseline_metrics['avg_recall']:+.2f}")

The reason

I think the key reason debugging was hard is that RAG pipelines look simple but have many failure points. Each stage—retrieval, reranking, context assembly, generation—can fail independently.

The Reddit discussion noted: “That’s also why there’s been more focus lately on adding evaluation and debugging layers around RAG systems.”

Without instrumentation, you’re flying blind. With it, debugging becomes engineering rather than guesswork.

Summary

In this post, I showed how to systematically debug RAG systems. The key point is implementing tracing at each pipeline stage so you can see exactly where failures occur.

Start by adding logging to your pipeline. Then build a test set of known queries. Finally, measure before making changes—treat RAG debugging as engineering, not trial and error.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 Reddit Discussion: Most RAG tutorials are misleading
👨‍💻 RAGAS - RAG Evaluation Framework
👨‍💻 LangSmith - LLM Observability

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!