RAG

RAG for Codebases: Building a Chatbot That Understands Your Repo

Ira Menon · May 12, 2026 · 18 min read

Every engineering team has had this moment: a new hire asks "where does auth actually happen?" and three senior engineers give three different answers. Codebases grow faster than institutional memory, and by the time a repo hits a few hundred thousand lines, no single person holds the whole map in their head anymore. This is the exact problem retrieval-augmented generation was built to solve, except most tutorials teach RAG on PDFs and Wikipedia articles, not on .py files with circular imports and half-finished refactors. Code is a different beast. It has structure that plain-text chunking destroys, it has cross-file relationships that a vector search alone can't see, and it changes every day, which means your index goes stale the moment someone merges a PR. Building a chatbot that genuinely understands a repository means rethinking almost every step of the standard RAG pipeline: how you split the code, what you embed, how you retrieve, and how you keep the whole thing in sync with git. This article walks through that rebuild end to end, with working code, so you can stand up a rag for code assistant against your own repo instead of a toy dataset.

Why Generic RAG Breaks on Source Code

The standard RAG playbook says: chunk your documents into 500-token windows with some overlap, embed them, stuff them in a vector database, and retrieve the top-k matches for a query. This works reasonably well for prose because paragraphs are self-contained units of meaning. Code is not prose. A function that spans 40 lines might get sliced in half by a naive chunker, separating the function signature from its return statement. A class definition might get split from the imports it depends on. Worse, code has meaning that isn't visible in the text at all — a function call in checkout.py only makes sense once you know it resolves to a method in payment_gateway.py three directories away.

There's also a scale mismatch. A single repo can have tens of thousands of files, and most of those files are irrelevant to any given question. Naive RAG treats every chunk as equally likely to be relevant, but in code, relevance is heavily structural: if I ask "how does the retry logic work in the Stripe webhook handler," the answer is almost certainly in one file, maybe two, not scattered across fifty semantically-similar-sounding snippets. Pure embedding similarity search tends to retrieve code that "reads similar" (lots of try/except blocks, for instance) rather than code that is actually *related* by call graph or import structure.

Finally, code has a freshness problem that documentation doesn't have to the same degree. Docs get updated occasionally; code gets updated constantly, sometimes dozens of times a day on an active repo. A RAG index that was built once and never refreshed becomes actively misleading — it'll confidently tell a user about a function signature that changed last Tuesday.

There's a subtler issue too: code carries multiple layers of meaning stacked on top of each other. The literal tokens matter (variable names, syntax), but so does the *intent* behind them (what business problem this function solves), and so does the *convention* of the surrounding codebase (is this the "old way" of doing auth that's being phased out, or the current standard). A generic RAG pipeline built for text search has no concept of any of these layers — it treats every chunk as an isolated bag of tokens to be matched against a query. For code, you need retrieval that's aware of at least two of those three layers, or your chatbot ends up confidently citing the deprecated pattern half your team is trying to migrate away from.

None of this means RAG is the wrong tool for codebases. It means rag for code needs a pipeline built around how code is actually structured, not one borrowed wholesale from a text-search demo.

Chunking Strategy: Parse, Don't Slice

The single highest-leverage change you can make is to stop chunking by character count and start chunking by syntax. Instead of a sliding window over raw text, parse the file with a proper parser and chunk along function, method, and class boundaries. tree-sitter is the tool of choice here because it supports dozens of languages with a consistent API and it's fast enough to run over an entire repo in seconds.

Here's a minimal example using tree-sitter to extract function-level chunks from a Python file:

from tree_sitter import Language, Parser
import tree_sitter_python as tspython

PY_LANGUAGE = Language(tspython.language())
parser = Parser(PY_LANGUAGE)

def extract_chunks(file_path: str):
    with open(file_path, "rb") as f:
        source = f.read()

    tree = parser.parse(source)
    root = tree.root_node
    chunks = []

    def walk(node):
        if node.type in ("function_definition", "class_definition"):
            start, end = node.start_byte, node.end_byte
            text = source[start:end].decode("utf-8")
            chunks.append({
                "file": file_path,
                "type": node.type,
                "start_line": node.start_point[0] + 1,
                "end_line": node.end_point[0] + 1,
                "text": text,
            })
            return  # don't descend further into this node's children
        for child in node.children:
            walk(child)

    walk(root)
    return chunks

This gives you chunks that are semantically coherent — a whole function or a whole class, never a jagged half-block. For very large functions (a 300-line God function isn't rare), you'll still want a fallback that splits by logical sub-blocks like nested if/for bodies, but the majority of real-world code chunks cleanly this way.

A second, often-overlooked step: attach metadata to every chunk before embedding. File path, the enclosing class name, the function's docstring, and the git blame author are all cheap to grab at chunk time and expensive to reconstruct later. Store them alongside the vector, not just in a separate lookup table, so your retrieval layer can filter and your generation layer can cite them.

Embedding Code vs Embedding Text

You have two real options for turning chunks into vectors: a code-specialized embedding model, or a general-purpose text embedding model applied to a text *representation* of the code (docstring plus signature plus a short natural-language summary). In practice, the second approach often outperforms raw code embeddings for chatbot-style Q&A, because user questions are asked in English, not in the target programming language, and cross-modal retrieval (English query against raw source tokens) is inherently harder than English-to-English retrieval.

A practical middle ground that works well: embed a synthetic "chunk summary" that concatenates the file path, the function signature, the docstring if one exists, and the raw code — then embed *that combined string*, not just the code alone.

def build_embedding_text(chunk: dict) -> str:
    header = f"File: {chunk['file']}\nType: {chunk['type']}"
    if chunk.get("docstring"):
        header += f"\nDocstring: {chunk['docstring']}"
    return f"{header}\n\nCode:\n{chunk['text']}"

def embed_chunks(chunks, embed_model):
    texts = [build_embedding_text(c) for c in chunks]
    vectors = embed_model.embed_documents(texts)
    for chunk, vector in zip(chunks, vectors):
        chunk["embedding"] = vector
    return chunks

If you're using a model that supports separate "query" and "document" embedding modes (several modern embedding APIs do), use the document mode for chunks and the query mode for user questions — mixing them up quietly degrades retrieval quality and is a common source of "why is this returning garbage" bugs.

Building the Retrieval Layer: Hybrid Search Over Vectors Alone

Pure vector similarity search is the wrong primary signal for code retrieval, full stop. If a user asks "where is calculate_shipping_cost defined," you want an exact symbol match, not a nearest-neighbor lookup that might surface five functions with similar-sounding logic but the wrong name. This is where hybrid search earns its keep: combine a lexical/keyword index (BM25 or even a simple inverted index over identifiers) with your vector index, and merge the results.

from rank_bm25 import BM25Okapi

class HybridRetriever:
    def __init__(self, chunks, vector_store):
        self.chunks = chunks
        self.vector_store = vector_store
        tokenized = [c["text"].split() for c in chunks]
        self.bm25 = BM25Okapi(tokenized)

    def retrieve(self, query: str, k: int = 8):
        # lexical pass
        bm25_scores = self.bm25.get_scores(query.split())
        bm25_ranked = sorted(
            range(len(self.chunks)), key=lambda i: bm25_scores[i], reverse=True
        )[:k]

        # semantic pass
        vector_hits = self.vector_store.similarity_search(query, k=k)

        # merge with reciprocal rank fusion
        scores = {}
        for rank, idx in enumerate(bm25_ranked):
            scores[idx] = scores.get(idx, 0) + 1 / (60 + rank)
        for rank, hit in enumerate(vector_hits):
            idx = hit.metadata["chunk_id"]
            scores[idx] = scores.get(idx, 0) + 1 / (60 + rank)

        ranked_ids = sorted(scores, key=scores.get, reverse=True)[:k]
        return [self.chunks[i] for i in ranked_ids]

Reciprocal rank fusion is a simple, well-understood way to combine two ranked lists without needing to calibrate scores across systems that produce wildly different score distributions. In production, you'll likely add a third signal too: a symbol index that does exact and fuzzy matching on function/class/variable names, since that's often the highest-precision signal available for code. This hybrid setup — lexical plus semantic plus symbol matching — is what separates a usable code assistant from a demo that only works on the exact three example queries someone tested it with.

Making the Retriever Structure-Aware

Chunk-level retrieval alone misses the biggest advantage you have with code: an explicit dependency graph. When you retrieve a function, you almost always want its immediate neighbors too — the functions it calls, the class it belongs to, and the file's imports. Building a lightweight call graph at index time lets you expand retrieval results along real edges instead of hoping the vector search happens to surface the caller and callee together.

import ast

def extract_calls(file_path: str) -> dict:
    with open(file_path) as f:
        tree = ast.parse(f.read())

    call_graph = {}
    for node in ast.walk(tree):
        if isinstance(node, ast.FunctionDef):
            calls = []
            for child in ast.walk(node):
                if isinstance(child, ast.Call) and isinstance(child.func, ast.Name):
                    calls.append(child.func.id)
            call_graph[node.name] = calls
    return call_graph

def expand_with_neighbors(chunk, call_graph, chunk_lookup, depth=1):
    expanded = [chunk]
    fn_name = chunk.get("function_name")
    if fn_name and fn_name in call_graph:
        for callee in call_graph[fn_name]:
            if callee in chunk_lookup:
                expanded.append(chunk_lookup[callee])
    return expanded

This is a simplified single-file version; a real implementation needs cross-file symbol resolution, which is exactly what language servers already do. If your stack has an LSP available (most do), it's often less work to shell out to pyright or gopls for "find references" and "go to definition" than to hand-roll a call graph parser for every language in a polyglot repo. Feed those structural neighbors into the context window alongside the vector/BM25 hits, and you'll notice a real jump in answer quality — especially for "how does X call into Y" style questions, which are the questions engineers actually ask.

Handling Repo Scale and Staleness

A monorepo with 500,000 lines of code and daily commits from forty engineers cannot be treated like a static document collection. You need an indexing pipeline that's incremental, not a one-shot batch job. The pattern that works well:

On initial setup, walk the full repo, chunk, embed, and index everything.
On every merge to the main branch, diff against the last indexed commit and only re-chunk and re-embed the files that changed.
Delete stale chunks for files that were deleted or renamed, using the file path as the deletion key.
Store the last-indexed commit SHA somewhere durable (a small metadata table works fine) so step 2 always knows its starting point.

import subprocess

def get_changed_files(last_sha: str, current_sha: str) -> list[str]:
    result = subprocess.run(
        ["git", "diff", "--name-status", last_sha, current_sha],
        capture_output=True, text=True, check=True,
    )
    changed = []
    for line in result.stdout.strip().splitlines():
        status, *paths = line.split("\t")
        changed.append((status, paths))
    return changed

def sync_index(vector_store, last_sha: str, current_sha: str):
    for status, paths in get_changed_files(last_sha, current_sha):
        if status == "D":
            vector_store.delete(where={"file": paths[0]})
        elif status in ("M", "A"):
            vector_store.delete(where={"file": paths[0]})
            new_chunks = extract_chunks(paths[0])
            embed_chunks(new_chunks, vector_store.embed_model)
            vector_store.add(new_chunks)
        elif status.startswith("R"):
            old_path, new_path = paths
            vector_store.delete(where={"file": old_path})
            new_chunks = extract_chunks(new_path)
            embed_chunks(new_chunks, vector_store.embed_model)
            vector_store.add(new_chunks)

Wiring this into a post-merge git hook or a CI job that fires on push to main keeps the index within minutes of the actual codebase, which matters enormously for developer trust. Nothing kills adoption of an internal tool faster than it confidently describing code that was deleted two sprints ago.

Prompting the Model to Reason Like an Engineer, Not a Search Engine

Retrieval only gets you halfway. The generation step needs a system prompt that pushes the model toward engineering reasoning rather than generic summarization. A few habits make a measurable difference in output quality:

Tell the model explicitly to cite file paths and line numbers for every claim it makes, so a reviewer can verify instantly.
Instruct it to say "I don't have enough context" rather than guessing when the retrieved chunks don't actually answer the question — code chatbots that hallucinate function signatures erode trust fast.
Give it the retrieved call-graph neighbors as clearly labeled "related code" separate from the primary match, so it understands which chunk is the direct answer versus supporting context.
Ask it to note when retrieved code looks like it might be outdated relative to imports or patterns seen elsewhere in the same result set (this catches a surprising number of "we deprecated this six months ago" situations).

SYSTEM_PROMPT = """You are a senior engineer answering questions about this codebase.
Rules:
- Only answer using the provided code context. Do not invent function names,
  signatures, or file paths that are not in the context.
- Always cite the file path and line range for any code you reference.
- If the context does not contain enough information to answer confidently,
  say so explicitly and suggest what to search for instead.
- When multiple versions of similar logic appear, flag the discrepancy rather
  than silently picking one.
"""

This kind of prompt, paired with strong retrieval, turns the chatbot from "plausible-sounding text generator" into something closer to a knowledgeable pair-programmer who happens to have read the whole repo yesterday.

Evaluating Your Code RAG System

You cannot improve what you don't measure, and code Q&A has a nasty failure mode where answers *look* authoritative even when they're wrong (a hallucinated function name reads exactly like a real one). Build a small evaluation set early — even twenty question/answer pairs sourced from real Slack threads where engineers asked "where is X" is enough to start. For each pair, track three things: did retrieval surface the correct file, did the generated answer cite that file, and was the final answer actually correct per a human reviewer.

def evaluate_retrieval(eval_set, retriever):
    hits = 0
    for item in eval_set:
        results = retriever.retrieve(item["question"], k=8)
        retrieved_files = {r["file"] for r in results}
        if item["expected_file"] in retrieved_files:
            hits += 1
    return hits / len(eval_set)

Run this after every meaningful change to chunking or retrieval logic. It's tempting to skip formal eval and just "vibe check" a few queries, but chunking changes in particular have non-obvious side effects — a change that helps retrieval for one file type can quietly hurt it for another, and you'll only catch that with a held-out set that covers multiple parts of the repo.

Deployment Shape: Where This Actually Lives

Most teams land on one of three deployment patterns for a repo-aware chatbot. The simplest is a CLI or IDE plugin that queries a locally-hosted vector index — good for a single team's repo, minimal infra, but doesn't scale to org-wide use. The second is a Slack or Teams bot backed by a shared vector database (Postgres with pgvector is a perfectly reasonable choice at repo scale, no need to reach for a dedicated vector DB unless you're indexing dozens of repos with heavy concurrent traffic). The third, and increasingly common, is embedding this retrieval pipeline as a tool inside a coding agent — the agent calls a search_codebase tool instead of grepping blindly, which is materially faster and more accurate than having the agent read files one at a time hoping to stumble onto the right one.

Whichever shape you pick, keep the retrieval layer as a separate, testable service from the chat interface. You'll want to swap embedding models, tune chunk sizes, and add new signals (test coverage data, recent commit frequency, ownership metadata from a CODEOWNERS file) over time, and that's much easier when retrieval isn't tangled up with the chat UI's request handling.

Common Failure Modes and How to Catch Them

A few problems show up in almost every code RAG deployment, and it's worth watching for them deliberately rather than waiting for a user to complain:

Retrieval returns test files instead of implementation files when a query mentions a function name that also appears in test fixtures. Fix by weighting non-test paths higher, or by giving test and implementation chunks separate namespaces the retriever can filter on.
Duplicate or near-duplicate chunks across vendored dependencies (a node_modules or vendor directory that got indexed by mistake) drown out real answers. Exclude these directories at chunk time, not just at query time.
Answers reference deleted code because the incremental sync job silently failed on a rename. Add alerting on sync job failures, and periodically do a full re-index as a sanity check even if incremental sync is your primary path.
Context window overflow when a question genuinely needs five or six files of context (a common "how does the whole request lifecycle work" style question). Consider a summarization pass over lower-priority chunks so the highest-relevance chunk gets full text while supporting chunks get compressed summaries.

Catching these early saves you from the slow trust erosion that happens when an internal tool is "mostly right" — mostly right is often worse than obviously broken, because people keep using it and occasionally getting burned.

Multi-Repo and Monorepo Considerations

If your organization runs a single monorepo, indexing is conceptually simple even if the volume is large — one git history, one commit SHA to track, one consistent directory layout to exclude vendored code from. Multi-repo setups are messier in practice. You'll usually want a shared vector store with a repo field in every chunk's metadata so queries can be scoped ("only search the payments-service repo") or left open ("search everything the user has access to"). Access control deserves real attention here: if engineers only have read access to certain repos on GitHub, your retrieval layer needs to enforce the same boundaries, or you've built an accidental way to leak private-repo code to anyone with access to the chatbot. The cleanest approach is to tag every chunk with the repo's visibility and the team that owns it at index time, then filter retrieval results against the querying user's actual permissions before they ever reach the language model — not as a prompt-level instruction, which a determined user can route around, but as a hard filter in the retrieval query itself.

Monorepos bring their own wrinkle: a single logical service might span a frontend/, backend/, and shared/ directory, and a good question like "how does checkout work end to end" genuinely needs chunks from all three. This is another place where the call-graph expansion technique pays for itself — pure top-k vector retrieval within a monorepo tends to cluster results from whichever directory has the most files, simply because there's more surface area to match against, and can starve out the smaller shared/ directory that actually holds the interesting logic.

Cost and Latency Tradeoffs Worth Knowing Upfront

Embedding an entire large repo isn't free, and re-embedding it on every commit compounds that cost if you're not careful. The incremental sync approach described earlier is what keeps this sane in practice — you're only ever re-embedding the diff, not the world, so steady-state cost after the initial index build tends to be small even on an active repo. The bigger cost lever is usually the generation step, not embedding: stuffing five or six full files into a prompt for every query adds up quickly if your chatbot gets heavy daily use across a large engineering org.

Two practical levers help here. First, cache retrieval results (not final answers, just the retrieved chunk set) for identical or near-identical queries within a short window — a surprising number of questions in any given week are minor rephrasings of something already asked. Second, be deliberate about how much raw code goes into the prompt versus a compressed summary. A common pattern: send the top one or two chunks as full source, and compress everything past that into a one-line "here's what this function does and where it lives" summary generated once at index time and cached. This keeps prompt size — and therefore latency and cost — roughly constant even as retrieval breadth grows, instead of scaling linearly with however many files a tricky question happens to touch.

Closing Thoughts

A chatbot that actually understands your repo isn't built by pointing an off-the-shelf RAG template at a folder of source files — it's built by respecting what makes code different from prose: syntactic structure that deserves a real parser, cross-file relationships that deserve a call graph, and a rate of change that demands incremental indexing rather than a one-time batch job. Get the chunking right, layer lexical and semantic and structural retrieval together, keep the index fresh against git, and write a system prompt that pushes the model to cite its sources instead of guessing. None of these pieces are exotic on their own, but stacked together they're the difference between a demo that impresses people once and a tool engineers actually reach for every day. Whether you're building rag for code on top of a single service repo or an entire monorepo shared by hundreds of engineers, the underlying discipline is the same: respect the structure, keep it fresh, and measure honestly. If you want the foundational concepts behind everything covered here — chunking, embeddings, retrieval scoring, and evaluation — properly explained from first principles, our course Introduction to RAG walks through the same pipeline in much more depth, with exercises you can run against your own codebase instead of a sample dataset.