CapstonePython (ingestion), TypeScript (API + UI)

Capstone 02 — RAG over Codebase (Cross-Repo Semantic Search)

Serious engineering orgs in 2026 run an internal code search that understands meaning, not just strings. Sourcegraph Amp, Cursor's codebase answers, Augment's enterprise graph, Aider's repomap and Pinterest's internal MCP all share one shape: ingest many repos, parse with tree-sitter, embed function- and class-level chunks, hybrid-search, re-rank, answer with citations. You build one that handles 2M lines of code across 10 repos and survives incremental re-indexing on every git push.

TIME: 30 hoursPREREQUISITES: Phase 5 (NLP foundations), Phase 7 (transformers), Phase 11 (LLM engineering), Phase 13 (tools), Phase 17 (infrastructure)

Learning Objectives

Build AST-aware chunks with tree-sitter and give each chunk a dense embedding, BM25 terms, and a one-sentence summary.
Implement hybrid retrieval that fuses dense and BM25 results, re-ranks them with a cross-encoder, and returns answers with file:line citations.
Keep the index fresh on every git push by re-embedding only changed chunks and recomputing only affected symbol-graph edges.
Evaluate the system with MRR@10, nDCG@10, and citation faithfulness against a labeled 100-question cross-repo set.

Problem

Codebase RAG is retrieval-augmented generation where the corpus is source code spread across many repositories, and every answer has to point back to the exact lines it came from. By 2026 every frontier coding agent ships with a retrieval layer like this, because a bigger context window alone does not solve cross-repo questions. Claude's 1M-token context helps, but it does not remove the need for ranked retrieval.

Naive cosine search over raw chunks fails in three places: generated code, duplication across a monorepo, and the long tail of rarely imported symbols. The production answer is hybrid (dense + BM25) search over AST-aware chunks, followed by a re-ranker and backed by a graph of symbol references.

You learn this by indexing a real fleet, not one tutorial repo, and measuring MRR@10, citation faithfulness, and incremental freshness. The hard failures are infrastructural: a 100k-file monorepo, a push that retouches half the files, a query that has to cross four repos to be answered correctly.

Warm-up0/3

1. A developer searches internal code for "how do we authorize requests", but the relevant function is named `check_permission`. Which search approach is most likely to find it?Warm-up

2. You split source files into fixed 512-token windows before embedding them. What is the main weakness of that approach?Warm-up

3. A service registers a webhook URL with a Git host. When does the host call that URL?Warm-up

Check: Problem0/1

1. Why does a 1M-token context window not remove the need for retrieval over a fleet of repos?Explain

Concept

AST-aware chunking means parsing each file with tree-sitter, extracting function and class nodes, and cutting at node boundaries instead of fixed token windows. Each chunk then gets three representations: a dense embedding (Voyage-code-3 or nomic-embed-code), sparse BM25 terms, and a short natural-language summary. The summary is a third way to find the chunk. Users ask "how is X authorized", and the summary mentions "authz" even when the code only says check_permission.

Retrieval is hybrid. A query fires dense and BM25 searches, merges the top-k, and hands the union to a cross-encoder re-ranker (Cohere rerank-3 or bge-reranker-v2-gemma-2b). The re-ranked list goes to a long-context synthesizer (Claude Sonnet 4.7 with prompt caching, or Llama 3.3 70B self-hosted), which is told to cite every claim by file and line range. A post-filter rejects any answer without citations.

Incremental freshness is the infrastructure problem. A git push triggers a diff of which files and which symbols changed. Only the affected chunks re-embed, and affected cross-file symbol edges (imports, method calls) get recomputed. The index stays consistent without reprocessing 2M lines on every commit.

Check: Concept0/9

1. Which three representations does each chunk get in this pipeline?Define

2. What is a dense embedding of a code chunk?Define

3. What does AST-aware chunking do to a source file?Define

4. What is hybrid search in this pipeline?Define

5. What does the cross-encoder re-ranker do after hybrid retrieval?Define

6. What does prompt caching change about a model call?Define

7. What is incremental re-index time, as this lesson uses the phrase?Define

8. Why can a cross-encoder rank candidates more accurately than comparing precomputed embeddings with cosine?Explain

9. Users ask "how is X authorized", but the code only says `check_permission`. How does the chunk summary help?Explain

Architecture

git push --> webhook --> ingest worker (LlamaIndex Workflow)
                           |
                           v
             tree-sitter parse + AST chunk
                           |
            +--------------+----------------+
            v              v                v
          dense        BM25 index       summary (LLM)
        (Voyage / bge)  (Tantivy)        (Haiku 4.5)
            |              |                |
            +------> Qdrant / pgvector <----+
                            |
                            v
                      symbol graph (Neo4j / kuzu)
                            |
  query --> LangGraph agent (retrieve -> rerank -> synth)
                            |
                            v
                 Claude Sonnet 4.7 1M context
                            |
                            v
                 answer + file:line citations

Stack

Parsing: tree-sitter with 17 language grammars (Python, TS, Rust, Go, Java, C++, etc.)
Dense embeddings: Voyage-code-3 (hosted) or nomic-embed-code-v1.5 (self-host), bge-code-v1 fallback
Sparse index: Tantivy (Rust) with BM25F, field-weighted on symbol name vs body
Vector DB: Qdrant 1.12 with hybrid search, or pgvector + pgvectorscale for teams under 50M vectors
Chunk summary model: Claude Haiku 4.5 or Gemini 2.5 Flash, prompt-cached
Re-ranker: Cohere rerank-3 or bge-reranker-v2-gemma-2b self-hosted
Orchestration: LlamaIndex Workflows for ingestion, LangGraph for query agent
Synthesizer: Claude Sonnet 4.7 (1M context) with prompt caching
Symbol graph: Neo4j (managed) or kuzu (embedded) for import and call edges
Observability: Langfuse spans per retrieval + synthesis step

Check: Stack0/1

1. A team with about 8M vectors already runs Postgres and wants hybrid search with little new infrastructure. Which vector store fits the stack's guidance?Design

Build It

Ingestion walker. Walk git history on every push hook and collect the changed files. Parse each file with tree-sitter and extract function and class nodes with their full source span. Emit chunk records {repo, path, start_line, end_line, symbol, body}.
Chunk summarizer. Batch chunks into Haiku 4.5 calls with prompt caching on the system preamble. Prompt: "Summarize this function in one sentence, naming its public contract and side effects." Store the summary alongside the chunk.
Embedding pool. Run two parallel queues: dense (Voyage-code-3, batch 128) and summary (same model, applied to the summary string). Write vectors to Qdrant with payload {repo, path, start_line, end_line, symbol, kind}.
BM25 index. Build a field-weighted Tantivy index: symbol name weight 4, symbol body weight 1, summary weight 2. This supports "find the function named X" queries alongside "find the function that does X".
Symbol graph. For each chunk, record edges for imports (this file uses symbol Y from repo Z), calls (this function calls method M on class C), and inheritance. Store them in kuzu. At query time the graph expands retrieval across repo boundaries.
Query agent. A LangGraph agent with three nodes. retrieve fires dense and BM25 in parallel, merges them with reciprocal rank fusion (RRF), and deduplicates by (repo, path, symbol). rerank runs the cross-encoder on the top 50 and keeps the top 10 (the bundled main.py demo keeps 5). synth calls Claude Sonnet 4.7 with the reranked chunks in context, caches the system prompt, and requires file:line citations.
Citation enforcement. Parse the model output. Any claim without a (repo/path:start-end) anchor gets flagged for re-ask or dropped, so the user only sees cited answers.
Incremental re-index. On each webhook, compute the symbol-level diff. Re-embed only chunks whose text changed, and recompute symbol edges only for chunks whose imports changed. Target: a 50-file push re-indexed in under 60 seconds for a 2M-LOC fleet.
Eval. Label 100 cross-repo questions with gold file:line answers. Measure MRR@10, nDCG@10, citation faithfulness (the fraction of claims with verifiable anchors), and p50/p99 latency.

Check: Build It0/24

1. What is the symbol graph?Define

2. What does reciprocal rank fusion (RRF) do in the query agent?Define

3. Why does each chunk record carry start_line and end_line?Explain

4. Why does the summarizer put prompt caching on its system preamble across batched Haiku calls?Explain

5. Why does the BM25 index weight symbol name at 4 and body at 1?Explain

6. Why run BM25 alongside dense search for code questions?Explain

7. After merging the dense and BM25 lists, why does the agent deduplicate by (repo, path, symbol)?Explain

8. Why does the pipeline drop or re-ask any answer claim that lacks a (repo/path:start-end) anchor?Explain

9. A question needs a function in repo A that calls a client class defined in repo B, and neither chunk mentions the other by name. Which component lets retrieval cross that boundary?Explain

10. Why does incremental indexing recompute symbol edges only for chunks whose imports changed?Explain

11. Why does each of the 100 evaluation questions carry a gold file:line answer?Explain

12. In main.py a chunk has repo "uploader", path "services/retry.go", start_line 122 and end_line 148. What does its anchor() return?Apply

13. In main.py a term appears once in a chunk's symbol, once in its summary and once in its body. What term frequency does BM25 count for that chunk?Apply

14. In main.py rrf() uses k_rrf = 60. Chunk A is first in the dense list and absent from BM25. Chunk B is second in both lists. Which fused score is higher?Apply

15. Running main.py on "how does rank fusion work" prints one sparse hit but three dense hits. Why?Apply

16. In main.py's stub reranker, X has fused score 0.0325 and two summary-token matches with the query. Y has 0.0160 and one symbol-token match. Which ranks first?Apply

17. A push changes only the body of one function in a file holding 30 function chunks, and imports are untouched. What does the incremental re-index redo?Apply

18. A refactor moves a helper into another repo and updates imports. Chunk embeddings are fresh, yet answers miss the new cross-repo call path. Which step most likely failed?Analyze

19. A team ships dense-only search. Users who type the exact name AbortMultipartOnFail often do not see that function ranked first. What is the flaw?Analyze

20. After you add 20% LLM-generated boilerplate to the corpus, near-duplicate generated chunks crowd the top results. What is the best first fix?Analyze

21. You swap in a new embedding model and MRR@10 improves, but you never measured MRR@10 before the swap. What is the problem?Analyze

22. Your fleet adds a 100k-file monorepo, and a typical push retouches half its files. Which ingest design best keeps push-to-search time low?Design

23. A Python function calls a Go service over gRPC, and neither chunk names the other. How do you let retrieval link them?Design

24. An embedding vendor may update its model without notice. Which design catches a retrieval regression without a human watching?Design

Use It

$ code-rag ask "how is S3 multipart abort wired into our retry budget?"
[retrieve]  12 chunks dense + 7 chunks bm25, 16 unique after dedup
[rerank]    top-5 kept (cohere rerank-3)
[synth]     claude-sonnet-4.7, cache hit rate 68%, 2.1s
answer:
  Multipart aborts are triggered by `AbortMultipartOnFail` in
  services/uploader/retry.go:122-148, which decrements the per-bucket
  retry budget defined in config/budgets.yaml:34-51 ...
  citations: [services/uploader/retry.go:122-148, config/budgets.yaml:34-51,
              libs/s3client/multipart.ts:44-61]

Check: Use It0/2

1. The retrieve step reports 12 dense chunks, 7 BM25 chunks, and 16 unique after dedup. Assuming no repeats within a list, how many chunks did both searches return?Apply

2. In the Use It run, the answer says the abort decrements the per-bucket retry budget. Which cited span holds that budget definition?Apply

Ship It

The deliverable is the skill outputs/skill-codebase-rag.md. Given a corpus of repos, it stands up the ingestion pipeline, the hybrid index, and the query agent, and returns a cited answer for any cross-repo question. The rubric:

Weight	Criterion	How it is measured
25	Retrieval quality	MRR@10 and nDCG@10 on a 100-question held-out set
20	Citation faithfulness	Fraction of answer claims with verifiable file:line anchors
20	Latency and scale	p95 query latency at 10k QPS on the indexed corpus size
20	Incremental indexing correctness	Time from git push to searchable on a 50-file commit
15	UX and answer formatting	Citation clickability, snippet previews, follow-up affordance
100

Check: Ship It0/5

1. Which metric measures whether the claims in an answer can be verified in the source?Define

2. Why does the rubric time a 50-file commit from git push to searchable, rather than timing a full rebuild?Explain

3. An answer makes 5 claims. Four carry (repo/path:start-end) anchors, but one anchored span does not contain what its claim says. What is the citation faithfulness?Apply

4. Reviewers say answers read well, but many clicked anchors lead to spans unrelated to the claims, while MRR@10 is high. Which measure is failing?Analyze

5. You can improve only one rubric criterion before review and want the most points. Which is the best single bet?Design

Author's Note

I'd label the 100 evaluation questions before writing the ingestion walker. Exercises 1, 2 and 4 all depend on an MRR@10 baseline, and a change needs a baseline measured before you start swapping embedding models, flagging generated code, or adding drift alerts. Without one, you can't tell whether the re-ranker or the payload flag moved the number, and the retrieval-quality criterion is the largest line in the rubric at 25 points.

Exercises

Swap Voyage-code-3 for self-hosted nomic-embed-code and measure the MRR@10 delta. Report whether the gap closes once re-ranking is enabled.
Inject 20% generated code (LLM-produced boilerplate) into the corpus and re-evaluate. You should see retrieval poisoning. Add a "generated" flag to the payload and down-weight those hits.
Benchmark Qdrant hybrid search against pgvector + pgvectorscale at your corpus size. Report p99 at batch size 1.
Add a sampling-based drift check: rerun the 100-question eval weekly and alert on an MRR@10 drop of more than 5%.
Extend to cross-language symbol resolution, such as a Python function that calls a Go service over gRPC. Use the symbol graph to link them.

Key Terms

Term	What people say	What it actually means
AST-aware chunking	"Function-level splits"	Cutting code at tree-sitter node boundaries instead of fixed token windows
Hybrid search	"Dense + sparse"	Run BM25 and vector search in parallel, merge top-k, rerank
Cross-encoder rerank	"Second-stage rank"	Model that scores each (query, candidate) pair together, more accurate than cosine
Prompt caching	"Cached system prompt"	2026 Claude / OpenAI feature that discounts repeat prefix tokens up to 90%
Symbol graph	"Code graph"	Edges for imports, calls, inheritance across files and repos
Citation faithfulness	"Grounded answer rate"	Fraction of claims a user can verify by clicking the anchor and reading the referenced span
Incremental re-index	"Push-to-search time"	Wall-clock from git push to the changed symbols being queryable