Hybrid Retrieval with BM25 and Dense Embeddings

Hybrid retrieval runs a lexical retriever and a dense retriever on the same query and merges their two ranked lists into one. Each retriever is strong where the other is weak.

Lexical search wins when the query carries a literal identifier the corpus contains verbatim. A query for AbortMultipartOnFail returns the right Go function via BM25 in microseconds. The same query, embedded, sits at the boundary of three similarity clusters, so a real embedding model can rank a neighboring file above the right one. The hashed mock embedding in the demo keeps d1 on top for this query, so the demo understates the weakness.

Dense search wins when the query is paraphrased away from the corpus's literal tokens. A user asking "how do we handle cancelled uploads" never typed the word abort or multipart. BM25 tends to favor the documentation chunk on "uploading large files" because that page contains the word uploads, while a real embedding model can match the abort function whose summary mentions cancellation. The hashed mock embedding in the demo is cruder, so its dense list does not always show this.

Which one wins depends on the query distribution. A production RAG system serves both kinds of query from one endpoint, so retrieval has to handle both at once. The merge step is where this goes right or wrong.

BM25 in one paragraph

BM25 scores a query-document pair by summing, over the query terms, an inverse document frequency factor multiplied by a saturating term-frequency factor that includes a length-normalization correction. It has two knobs. k1 controls term-frequency saturation. The default of 1.5 is the published recommendation, and I wouldn't move it without a benchmark. b controls how much document length matters. The default of 0.75 penalizes longer documents, but not linearly.

The IDF uses the smoothed Robertson and Sparck Jones definition, log((N - df + 0.5) / (df + 0.5) + 1). The plus-one inside the log keeps the IDF positive when a term appears in more than half the corpus, which matters in small corpora where stopwords are technically rare.

Field weighting lets you tell BM25 that a match on the symbol name counts for more than a match in the body. You implement it as a multiplier on the term counts at indexing time, not at scoring time. The math stays identical and you avoid keeping a separate score per field.

Dense retrieval in one paragraph

Dense retrieval embeds each chunk into a fixed-dimension vector with an embedding model. At query time you embed the query, cosine-rank every chunk against it, and return the top-k. The embedding model decides quality. The retrieval algorithm itself is two lines: a dot product and a sort.

Here the embedding is a deterministic hash, so you can read the fusion math without a network call. It sums token-keyed offsets into a 96-dimensional vector and normalizes. The cosine ranks come out the same on every run, which is what the test suite needs.

Reciprocal rank fusion, the published formula

Reciprocal rank fusion (RRF) merges ranked lists by giving each candidate a score of 1 / (k + rank) from every list it appears in, summing those contributions, and sorting by the total. The 2009 paper used k = 60 as the default. That is the whole algorithm.

The constant k = 60 isn't arbitrary. At k = 60 the rank-1 contribution is 1 / 61 and the rank-10 contribution is 1 / 70, so the contribution decays slowly and deep candidates still vote. A smaller k lets the top results dominate, and a larger k flattens the curve.

This implementation exposes two knobs: the k constant, and a pair of per-modality weights for when you have prior evidence that BM25 or dense is better on your corpus. The weight multiplies the rank contribution. That is the simplest principled option, since it preserves the rank-decay shape and stays scale-free.

Why fusion beats score-weighted interpolation

BM25 scores are unbounded and corpus-dependent, while cosine similarities sit between -1 and 1. A linear combination alpha * bm25 + (1 - alpha) * cosine needs alpha tuned per corpus and breaks whenever you reindex. Rank-based fusion avoids that, because two ranks are comparable across modalities. The published RRF baseline beats score-interpolation in every public TREC track since 2010.

Vespa and Weaviate make the same argument in their documentation and reach the same conclusion: stay rank-based unless you have very strong evidence for interpolating scores.

code/main.py implements:

• tokenize(text): a fast regex tokenizer.
• BM25Index: field-weighted, with add and search and tunable k1, b.
• mock_embed, DenseIndex: the same deterministic embedding as lesson 64, so chunks are comparable.
• rrf(rankings, k, weights): the published fusion, with per-modality weights.
• HybridRetriever: combines BM25 and dense.
• A demo main() that loads a small fixture corpus, runs three queries that target each retriever's strength and weakness, and prints the rankings each modality produced plus the fused list.

Run it:

python3 code/main.py

Read the demo output side by side. For the literal query AbortMultipartOnFail, BM25 and dense both rank d1 first, so RRF returns d1 first at 0.0328 and the fusion is invisible. For the paraphrased query, BM25 ranks d2 first while dense ranks d3 first and d2 second. RRF scores both at 0.0325, so d2 and d3 tie at the top and both outrank everything else. For the mixed query, BM25 and dense agree on d4 and RRF keeps it first. The fusion never drops the document either modality ranked first, and it surfaces both when they disagree.

Tune by re-running lesson 68's eval harness on your held-out query set. Don't tune by intuition.

Out-of-vocabulary tokens. BM25's IDF is computed from the corpus, so a term that appears only in the query contributes zero. A dense embedding still produces a vector for that same term, so on out-of-corpus identifiers the dense modality returns plausible-looking but wrong neighbors. The fusion absorbs this because BM25 returns nothing and its rank contribution drops out, but only if you de-duplicate by document, not by chunk.

Stop-token domination. BM25 against the word "the" carries almost no ranking signal. In the demo corpus it appears in six of seven documents, so its IDF is tiny and the six matches score between 0.40 and 0.23, ordered only by term frequency and length normalization. Filter stop tokens in the indexer, or accept that high-IDF terms dominate naturally.

Identical content across modalities. If the corpus is small enough that BM25's top-1 is also dense's top-1, RRF returns the same top-1 with the same neighbors. That is correct behavior, not a failure, but it makes the fusion look invisible. Add an adversarial query pair to your eval to verify the fusion is actually working.

Production patterns:

• Index BM25 in process. The bottleneck is the term-frequency dictionary, not the vectors.
• Index dense vectors in a separate store. This code uses a flat list; in production you would use HNSW.
• Run both queries in parallel. The fusion is a sort over the union of the two candidate lists, which stays cheap because each list holds only the top few dozen hits.
• Persist the modality of each retrieved hit so a downstream reranker can see which modality voted for it.

Lesson 66 takes the fused top-k from here and reranks it with a cross-encoder. Lesson 68 evaluates the entire pipeline with precision, recall, MRR, and nDCG. This hybrid retriever is the first stage of the end-to-end system in lesson 69.

1. Replace mock_embed with a real model from your provider. Re-run the demo and report how the dense-only ranking changes on the paraphrased query.
2. Add a third modality: chunk summaries indexed separately and fused as a third ranked list. Measure the gain.
3. Sweep RRF k across 10, 30, 60, 100, 200. Plot the recall@k curve from lesson 68. Report the value of k where the curve peaks on your corpus.
4. Implement BM25F properly (per-field length normalization rather than the multiplier trick) and compare on a corpus where symbol matches matter most.

• Cormack, Clarke, Buettcher, "Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods", SIGIR 2009
• Robertson, Walker, Beaulieu, Gatford, Payne, "Okapi at TREC-3" (the original BM25 paper)
• Vespa: Hybrid Retrieval with BM25 and Embeddings
• Weaviate: Hybrid Search
• Phase 11 lesson 06 - RAG fundamentals
• Phase 19 lesson 64 - chunkers whose output is indexed here
• Phase 19 lesson 66 - cross-encoder reranker that consumes the fused top-k