End-to-End RAG System

An end-to-end RAG system is the whole chain from raw documents to a cited answer: chunk, optionally rewrite the query, retrieve, rerank, generate. Each component can look good alone and still fail inside the chain, so the grade that matters is the one on the assembled pipeline.

Here is how the parts break each other. The chunker can win on recall@5 against the corpus and lose on the system's recall@5 because the retriever cannot rank what the chunker emits. The reranker can lift MRR on a synthetic candidate pool and fail on real bi-encoder candidates because the bi-encoder's recall at the rerank budget is too low. The query rewriter can promote the gold doc on one query and break on the next because the hypothetical document it retrieves with misses the next query's vocabulary.

The integration test runs the whole pipeline against the same fixture qrels, with the same metrics, through one orchestrator file that wires everything together. That file is what gets built here. If the integrated metrics clear the thresholds set in the demo, you have evidence the wiring works. Comparing them against each stage's isolated numbers is left as an exercise.

Wiring choices

The pipeline is a small graph, and each stage is a function with a clear signature.

Composition is easy once the signatures hold still. The Pipeline class holds the chunker, index, rewriter, and reranker, plus a query method that runs them in order and then hands the result to the generator. Swap in a different chunker, index, rewriter, or reranker that keeps the same signature and the pipeline still runs. The generator is the exception: query calls generate_answer directly, so replacing it means editing that call.

Answer generator with citations

The generator is the last stage and the easiest to break. A deterministic mock generator ships here. It:

1. Takes the top-K reranked chunks.
2. Looks at the top three, and selects up to two whose text has the highest content-token overlap with the query (falling back to the rank-1 chunk if none overlap).
3. Emits an answer that concatenates one sentence from each selected chunk, each followed by a [doc_id:chunk_index] anchor.
4. If the rank-1 score is below the refuse threshold, skips all of that and emits "I do not know." with no citation.

In production you swap the mock for a real LLM call with a prompt template like this:

You are answering a question using only the snippets below.
Cite every claim with the anchor in parentheses.
If the snippets do not answer the question, say "I do not know".

Question: {query}

Snippets:
{enumerated chunks with anchors}

Answer:

The refuse-on-low-confidence path reads the rank-1 score, which is the cross-encoder score plus 0.3 times the chunk's content-token overlap with the query. If it sits below REFUSE_THRESHOLD (0.05 on the fixture), the generator refuses. The demo's query trace prints the top-3 blended scores, rank 1 included. That is the safety valve against hallucinated answers when the retrieved context does not address the question.

The self-terminating demo

The demo runs the system end to end. It prints a trace of one query (rewriter strategy, top-3 scores, answer, citations, per-stage latency), runs the eval over the four fixture qrels, prints a metrics table, and exits with status zero when the lesson 68 metrics meet the thresholds set in the demo. If a metric falls below its threshold, it exits non-zero and names the failing metric.

That is the shape of a CI smoke test: offline, fast, deterministic. The thresholds are tight on the fixture, so a regression in any upstream component should fail the demo.

code/main.py implements:

• Chunk - the record carried through all stages (extends lesson 64's shape with a chunk_index and source doc_id).
• Chunker - the recursive split from lesson 64 with a target of 400 characters. There is no strategy switch.
• HybridIndex - bundles BM25 + dense + RRF from lesson 65.
• Rewriter - pick_strategy routes by string checks: a query containing " and " is decomposed, one mentioning api, function, endpoint, or config goes to HyDE, and everything else gets multi-query (lesson 67). The HyDE hypotheticals are a hardcoded dict keyed by the fixture queries, not a model call.
• CrossEncoder and rerank - the small cross-encoder from lesson 66, trained by train_reranker on 17 fixture triples for 80 epochs so it converges in seconds. rerank blends its score with 0.3 times the content-token overlap between query and chunk. This needs torch installed.
• generate_answer - a plain function, not a class: the deterministic mock generator with citations and refuse-on-low-confidence.
• Pipeline - a dataclass holding chunker, index, rewriter, reranker, top_n, and top_k, with a query(question) method that returns Result(answer, citations, top_k, rewriter_strategy, latency_ms). latency_ms is a dict with one entry per stage.
• run_demo() - ingests the corpus, trains the reranker, prints one query trace, runs the eval over four fixture queries, prints results, and sets the exit code by threshold.

Run it:

python3 code/main.py

You get one printed query trace, the full eval table, and a final pass/fail status. On the fixture it returns exit code 0.

Chunker boundary drift. Change the chunker strategy (the shipped chunker has no switch, exercise 4 adds one) between the qrels labeling pass and the demo run, and the chunk anchors, doc_id:chunk_index, stop lining up because the boundaries moved. The qrels and metrics here are keyed by doc_id, so recall, precision, and MRR survive a boundary change, but any label or citation check pinned to an anchor does not. Lock the chunker strategy in the qrels file. Print the chunker strategy at the top of the demo output so a mismatch is visible.

Reranker training set leaks into the eval. The 17 training triples in TRAIN_TRIPLES use corpus documents as positives and hard negatives, including the eval gold documents. The four eval query strings do not appear in the triples, but the documents do, so the rerank numbers are optimistic. In production, hold out the eval queries and their gold documents strictly.

The mock generator hides hallucination risk. It cannot hallucinate because it only emits text from the retrieved chunks. Swap in a real model before you trust the generation-side numbers.

No streaming. The pipeline returns the full answer at the end of each stage. A production system would stream the generator's output. Streaming is out of scope here, and the answer-grade metrics work on the final string either way.

Latency is offline. Nothing here calls an LLM: the generator is a local function and the HyDE hypotheticals are a dictionary lookup. Real LLM calls dominate latency in production. Plan a latency budget at the request level. The per-stage timing here (rewrite, retrieve, rerank, generate) measures CPU work and nothing else.

Production patterns:

• Ship the pipeline file under one orchestrator with explicit stage interfaces. Don't spread the wiring across the repo.
• Run the eval before each merge that touches a stage, and block the merge if the eval drops.
• Persist the metric trace per CI run so you can attribute a regression to a stage swap.
• Keep a smoke set of 20 queries (a subset of the regression set) that runs in under 30 seconds, and run the full regression set nightly.

This pipeline file is the shape the rest of Phase 19's Track F lessons assume. Later lessons would layer ingestion automation, incremental re-index, telemetry, and a serving layer on top. The retrieval, rerank, rewrite, and eval halves are complete here.

1. Extend pick_strategy, which currently checks for " and " and a few keywords, with the other lesson 67 heuristics (length, jargon ratio) when choosing HyDE, multi-query, or decomposition.
2. Add a real LLM call for the generator behind an env flag. Default to the mock. Measure the latency delta.
3. Extend the demo to take a --corpus path flag that loads a real corpus. Re-run the eval and the threshold check.
4. Add a --strategy flag to the chunker. Measure each strategy's contribution to end-to-end recall.
5. Add a streaming generator interface and feed it into the eval. Confirm that faithfulness is computed on the final string and not on the streamed prefix.
6. Run each stage's isolated demo from lessons 64-68 and compare its metrics with the integrated numbers.

• Anthropic, Building search and retrieval
• Pinterest, MCP internal search - reference production architecture
• Ragas: Automated Evaluation of RAG Pipelines
• Phase 11 lesson 06 - RAG fundamentals
• Phase 19 lessons 64-68 - the components composed here