Skip to content
BSWEN
Blog

Can Local LLMs Handle Large Codebases? Practical Tips and Model Recommendations

Cowrie
Cowrie
Dev @ Bswen

Problem

I have a codebase with 100K+ lines of code. My local LLM can only see a few thousand tokens at a time. How can it possibly understand my entire project?

I wanted to ask questions like “Where is the authentication logic?” or “What does this obscure hardcoded value mean?” - but my model was blind to 99% of my code.

The Short Answer

Yes, local LLMs can handle large codebases - but not by stuffing everything into the context window. You need RAG (Retrieval-Augmented Generation) with a local vector database.

The key insight from the community: “Cache embeddings or repo context somewhere (simple local vector db) so the model can actually reason over big codebases instead of just the open file.”

The Problem with Context Windows

Even the largest context windows aren’t enough for enterprise codebases:

Context window reality check
| Context Size | What Fits                    |
|--------------|------------------------------|
| 4K tokens    | ~1-2 files                   |
| 32K tokens   | ~10-20 files                 |
| 128K tokens  | ~100 files                   |
| Your codebase| 100K+ lines = thousands of files |

No matter how big your context window, there’s always a bigger codebase.

The Solution: RAG Architecture

RAG lets your model reason over millions of lines of code by:

  1. Chunking - Split code into meaningful pieces (functions, classes, modules)
  2. Embedding - Convert each chunk to a vector using a local embedding model
  3. Indexing - Store vectors in a local database (Chroma, Qdrant, FAISS)
  4. Retrieving - Find relevant chunks based on semantic similarity
  5. Querying - Feed retrieved context to your LLM
RAG workflow
┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│  Your Query  │───►│  Vector DB   │───►│  Top-K Chunks│
└──────────────┘    │  (ChromaDB)  │    └──────────────┘
                    └──────────────┘           │
                                               ▼
┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│   Response   │◄───│  Local LLM   │◄───│ Query+Context│
└──────────────┘    │  (Qwen 27B)  │    └──────────────┘
                    └──────────────┘

Setting Up Basic RAG

Here’s a simple Python implementation:

Basic RAG with ChromaDB
from chromadb import Client
from sentence_transformers import SentenceTransformer


# Initialize embedding model (runs locally)
embedder = SentenceTransformer('all-MiniLM-L6-v2')
chroma = Client()
collection = chroma.create_collection("codebase")


# Index your codebase
def index_codebase(file_paths):
    for path in file_paths:
        with open(path) as f:
            content = f.read()
        # Split into chunks (functions, classes)
        chunks = split_into_chunks(content)
        for i, chunk in enumerate(chunks):
            embedding = embedder.encode(chunk)
            collection.add(
                documents=[chunk],
                embeddings=[embedding.tolist()],
                ids=[f"{path}_{i}"]
            )


# Query with context
def query_codebase(question, top_k=5):
    query_embedding = embedder.encode(question)
    results = collection.query(
        query_embeddings=[query_embedding.tolist()],
        n_results=top_k
    )
    context = "\n\n".join(results['documents'][0])
    # Now send question + context to your LLM
    return context

Model Recommendations by VRAM

For codebase analysis, you need enough VRAM for both the model and the context:

Model recommendations for RAG
| VRAM  | Recommended Models              | Use Case                |
|-------|---------------------------------|-------------------------|
| 24GB  | Qwen2.5-Coder-7B, DeepSeek-Lite | Daily coding, quick     |
| 32GB  | Qwen3.5-27B, Qwen2.5-Coder-14B  | Large codebase analysis |
| 48GB  | Qwen3.5-32B, DeepSeek-33B       | Complex investigation   |
| 80GB+ | Qwen3.5-122B Q3                 | Deep code understanding |

Why Model Size Isn’t Everything

A surprising finding from the community: The 35B A3B model failed where the smaller 27B succeeded.

Why? Architecture and training matter more than raw parameter count.

One user reported that Qwen3.5 122B Q3 successfully found an “obscure hardcoded value” in a complex investigation task, while the 35B A3B “failed to one-shot it.” But Qwen3.5 27B also succeeded at the same task.

Real-world task comparison
| Model          | Task: Find obscure hardcoded value |
|----------------|-----------------------------------|
| Qwen3.5 122B   | ✓ Found                           |
| Qwen3.5 27B    | ✓ Found                           |
| Qwen3.5 35B A3B| ✗ Failed to one-shot              |

The 27B dense model outperformed the larger MoE model for this reasoning task.

Context Window Optimization

Don’t just dump all retrieved chunks into the context. Be smart:

Smart retrieval with token budget
def smart_retrieval(query, max_tokens=8000):
    # 1. Get more candidates than needed
    candidates = collection.query(
        query_embeddings=[embedder.encode(query).tolist()],
        n_results=20  # Over-retrieve
    )


    # 2. Re-rank by relevance (optional but improves quality)
    ranked = rerank_by_relevance(query, candidates)


    # 3. Fit within token budget
    selected = []
    total_tokens = 0
    for chunk in ranked:
        tokens = count_tokens(chunk)
        if total_tokens + tokens <= max_tokens:
            selected.append(chunk)
            total_tokens += tokens
        else:
            break


    return selected

Common Mistakes

  1. Relying solely on context window - Even 128K isn’t enough for enterprise codebases. Use RAG.

  2. Ignoring embeddings - Not caching repo context means re-processing every query. Slow and wasteful.

  3. Over-quantizing large models - A heavily quantized 100B model often underperforms a well-quantized 30B model.

  4. Single-model approach - Use smaller fast models for autocomplete, larger models for complex reasoning.

  5. Not chunking properly - Random chunks are worse than semantic chunks (functions, classes).

Vector Database Options

Local vector database comparison
| Database | Pros                    | Cons                  |
|----------|-------------------------|-----------------------|
| Chroma   | Easy setup, Python-first| Less scalable         |
| Qdrant   | Fast, production-ready  | More complex setup    |
| FAISS    | Very fast, Meta-backed  | Lower-level API       |
| SQLite   | Simple, familiar        | No native vector ops  |

For personal use, ChromaDB is the easiest to start with.

Summary

In this post, I showed how local LLMs can handle large codebases using RAG and vector databases. The key point is caching code embeddings so your model reasons over your entire codebase, not just the visible context window.

For RTX 5090 owners, Qwen3.5 27B offers the best balance of reasoning power and context efficiency. Set up a local vector database, index your codebase once, and query infinitely.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Comments

Tags: