How to Implement a Reranker in LangChain: Complete Code Guide

Feb 25, 2026

Purpose

This post demonstrates how to implement a reranker in LangChain to improve your RAG application’s retrieval quality. I’ll show you complete code examples for both open-source (BGE) and API-based (Cohere) rerankers, explain why the two-stage retrieval pattern works, and compare performance trade-offs.

Environment

Python 3.9+
LangChain 0.1.0+
langchain-community (for cross-encoder support)
HuggingFace transformers or Cohere API

Why Add a Reranker?

When I built my first RAG application, I used basic vector search: query embedding, similarity search, retrieve top 5 documents, pass to LLM. The problem was that embedding-based retrieval is fast but not always accurate. The top 5 results from vector search often included mediocre chunks that weren’t actually relevant to my query.

This happened because:

Embeddings capture semantic similarity, not query-document relevance
Vector search trades accuracy for speed (uses bi-encoder architecture)
Small retrieval windows (top 5) miss relevant documents that rank lower

I needed a way to retrieve more candidates (top 50) but only pass the best ones to the LLM. Rerankers solve this.

How Reranking Works

Reranking uses a two-stage retrieval pattern:

Query → Vector Search (k=50) → Reranker (top_n=5) → LLM
         Fast bi-encoder        Slow cross-encoder    Quality context

Stage 1: Initial retrieval

Fast vector search using bi-encoder embeddings
Retrieve many candidates (e.g., top 50)
Quick but less accurate

Stage 2: Reranking

Cross-encoder model scores each query-document pair
More accurate because it sees query + document together
Expensive, but only runs on 50 documents, not your entire corpus
Returns top N most relevant (e.g., top 5)

This gives you the speed of vector search + the accuracy of cross-encoders. You retrieve 50 chunks fast, rerank to find the best 5, and pass only high-quality context to the LLM.

Implementation: Open-Source Reranker (BGE)

The BAAI/bge-reranker-v2-m3 model is a popular open-source reranker that runs locally for free. Here’s how to implement it in LangChain.

Basic Setup

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

# Your existing retriever (FAISS, Chroma, Pinecone, etc.)
base_retriever = vector_store.as_retriever(search_kwargs={"k": 50})

# Initialize BGE reranker (runs locally, free)
model = HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-v2-m3")
compressor = CrossEncoderReranker(model=model, top_n=5)

# Wrap retriever with reranker
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=base_retriever
)

# Returns top 5 reranked results instead of top 50 raw chunks
docs = compression_retriever.invoke("What is the capital of France?")

I can explain the key parts:

base_retriever: Your vector store retriever with k=50 to fetch candidates
HuggingFaceCrossEncoder: Downloads and runs the BGE model locally
CrossEncoderReranker: Wraps the model and handles reranking logic
ContextualCompressionRetriever: Combines retrieval and compression (reranking)
top_n=5: Only the top 5 reranked documents are returned

Full RAG Pipeline

Now I’ll show you a complete RAG pipeline with reranking:

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Step 1: Load documents and create vector store
loader = TextLoader("documents.txt")
documents = loader.load()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)
splits = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings()
vector_store = FAISS.from_documents(splits, embeddings)

# Step 2: Create base retriever (retrieve 50 candidates)
base_retriever = vector_store.as_retriever(search_kwargs={"k": 50})

# Step 3: Add reranker (keep top 5)
reranker = HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-v2-m3")
compressor = CrossEncoderReranker(model=reranker, top_n=5)
retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=base_retriever
)

# Step 4: Create RAG chain with reranked retrieval
llm = ChatOpenAI(model="gpt-4")
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True
)

# Step 5: Query with automatic reranking
result = qa_chain({"query": "Explain quantum entanglement"})
print(result["result"])

# See which documents were used
for doc in result["source_documents"]:
    print(f"Source: {doc.metadata.get('source', 'unknown')}")

Comparing Before/After Reranking

I wanted to see the difference in retrieval quality, so I wrote a comparison script:

# Without reranking (top 5 from vector search)
raw_docs = base_retriever.invoke("machine learning algorithms")
print(f"Retrieved {len(raw_docs)} docs (raw vector search)")

for i, doc in enumerate(raw_docs):
    print(f"Rank {i+1}: {doc.page_content[:100]}...")

print("\n" + "="*60 + "\n")

# With reranking (top 5 rescored from top 50)
reranked_docs = compression_retriever.invoke("machine learning algorithms")
print(f"Retrieved {len(reranked_docs)} docs (after reranking)")

for i, doc in enumerate(reranked_docs):
    print(f"Rank {i+1}: {doc.page_content[:100]}...")

When I ran this, I noticed that reranking often changed the order of results and promoted documents that matched the query intent more precisely, even if they had lower similarity scores in the initial vector search.

Implementation: API-Based Reranker (Cohere)

If you don’t want to run models locally, API-based rerankers like Cohere offer faster inference and often higher quality. The pattern is identical - just swap the compressor.

from langchain.retrievers import ContextualCompressionRetriever
from langchain_community.document_compressors import CohereRerank
import os

# Set Cohere API key
os.environ["COHERE_API_KEY"] = "your-cohere-api-key"

# Your existing retriever
base_retriever = vector_store.as_retriever(search_kwargs={"k": 50})

# Initialize Cohere reranker (managed API, faster inference)
compressor = CohereRerank(top_n=5)

# Wrap retriever with reranker
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=base_retriever
)

# Returns top 5 reranked results
docs = compression_retriever.invoke("What is the capital of France?")

The structure is identical to the BGE implementation. The only difference is CohereRerank instead of CrossEncoderReranker, and you don’t need to manage model loading.

Performance Comparison

I tested different approaches to understand the latency-quality trade-off:

Approach	Latency	Quality	Cost
Pure vector search (k=5)	Fastest (~50ms)	Lower	Lowest
Vector search (k=50) + BGE reranker	+100-300ms (~150-350ms total)	Higher	Free (local)
Vector search (k=50) + Cohere reranker	+50-150ms (~100-200ms total)	Highest	~$0.001/1K searches

The key insight: Reranking adds 100-500ms of latency, but this often pays for itself by:

Reducing LLM token usage (5 chunks vs 50)
Improving answer quality (better context)
Potentially reducing overall latency (faster LLM processing with less context)

Choosing a Reranker

I evaluated both options and found:

Open-source (BGE, Jina, ColBERT):

Free, runs locally
Good for data privacy (documents never leave your server)
Requires GPU for best performance (CPU works but slower)
Models to try: BAAI/bge-reranker-v2-m3, jina-reranker-v1-base

API-based (Cohere, ZeroEntropy):

Faster inference (optimized servers, often GPU-accelerated)
Pay per usage (typically $0.001-0.002 per 1K searches)
No setup required, just API key
Often higher quality than open-source models
Documents sent to external service (consider privacy)

My recommendation: Start with BGE v2-M3 for free local testing. If you need better quality or can’t run models locally, try Cohere’s API.

Common Mistakes

When I implemented reranking, I made a few mistakes:

Skipping reranking due to latency concerns I initially worried that 100-500ms extra was too much. But the quality improvement and token savings often offset this. Better to retrieve more and rerank than to retrieve few without reranking.

Using only top_k=3 without reranking I tried to reduce latency by just retrieving fewer documents. This was worse than retrieving 50 and reranking to 5, because vector search alone often misses relevant documents.

Not testing multiple rerankers BGE, Cohere, and Jina perform differently depending on your domain. Test multiple models and measure quality on your specific data.

Applying reranker to entire corpus I briefly wondered if I should rerank all documents during indexing. This would be impossibly slow. Rerankers are designed to work on a small subset (50-100 documents), not millions.

How It Works Internally

I wanted to understand why reranking works better, so I looked into the architecture differences:

Bi-encoder (vector search):

Document embedding: encode(document) → vector (computed once, stored)
Query embedding: encode(query) → vector (computed at query time)
Similarity: cosine_similarity(query_vector, doc_vector)
Fast but less accurate (query and document never “see” each other)

Cross-encoder (reranker):

Input: [CLS] query [SEP] document [SEP]
Model: Transformer attention mechanism sees query+document together
Output: Relevance score (0-1)
Slow but accurate (full attention between query and document)

This is why the two-stage approach works: bi-encoder filters millions of documents quickly, cross-encoder reranks the top candidates precisely.

Summary

In this post, I showed how to implement a reranker in LangChain using ContextualCompressionRetriever and CrossEncoderReranker. The key point is that two-stage retrieval (vector search + reranking) gives you both speed and accuracy, while single-stage approaches force you to choose one or the other.

The pattern is identical for both open-source and API-based rerankers - just swap the compressor. Start with BGE v2-M3 for free local testing, then try API options like Cohere for production. The 100-500ms latency cost typically delivers significantly better RAG quality and lower LLM costs.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

👨‍💻 Reddit Discussion: Is Adding a Reranker to My RAG Stack Actually Worth the Extra Latency?
👨‍💻 LangChain ContextualCompressionRetriever Documentation
👨‍💻 BGE Reranker v2-M3 on HuggingFace
👨‍💻 Cohere Rerank API
👨‍💻 LangChain CrossEncoderReranker Documentation

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!