BuildPython

ColPali and Vision-Native Document RAG

Traditional RAG parses PDFs into text, splits the text into chunks, embeds the chunks, and stores vectors. Each step loses signal: OCR drops chart data, chunking breaks table rows, and text embeddings ignore figures. ColPali (Faysse et al., July 2024) asked the simpler question of why extract text at all. It embeds the page image directly via PaliGemma and retrieves with ColBERT-style late interaction, so the layout, figures, fonts, and formatting all stay in play. Published benchmarks report 20-40% better end-to-end accuracy than text-RAG on visually-rich documents. ColQwen2, ColSmol, and VisRAG extended the pattern. This lesson reads the vision-native RAG thesis and builds a tiny ColPali-like indexer.

TIME: ~180 minutesPREREQUISITES: Phase 11 (LLM Engineering — RAG basics), Phase 12 · 05 (LLaVA)

Learning Objectives

Explain the difference between bi-encoder retrieval (one vector per document) and late-interaction retrieval (many vectors per document).
Describe ColBERT's MaxSim operation and how ColPali generalizes it from text tokens to image patches.
Build a tiny ColPali-like indexer: page → patch embeddings → MaxSim over query-term embeddings → top-k pages.
Compare ColPali + Qwen2.5-VL generator vs text-RAG + GPT-4 on an invoices / financial reports use case.

The Problem

Text-RAG on PDFs throws away most of what makes a document a document. A financial report's Q3 revenue growth usually lives in a chart. A medical report's findings sit in annotated images. A legal contract's signature block is a layout fact, not a text fact.

The text-RAG pipeline:

PDF → text via OCR / pdftotext.
Text → 300-500 token chunks.
Chunk → bi-encoder embedding (one vector).
User query → embedding → cosine similarity → top-k chunks.
Chunks + query → LLM.

That is five lossy steps. Charts go uncaptured, tables get broken across chunks, multi-column layouts flatten, and figure annotations disappear.

ColPali's fix is to skip OCR and embed the page image directly. Late interaction, the ColBERT way, lets the model attend to fine-grained patches at query time.

Warm-up0/3

1. In a standard text-RAG pipeline, each chunk is embedded as one vector and retrieved by cosine similarity. What is the main limitation of squeezing a whole chunk into a single vector?Warm-up

Typed schematic of one text-RAG chunk and its embedding. The 768 dimension is the text-RAG row of the main.py storage table.

2. Two embedding vectors point in almost the same direction. What does their cosine similarity come out as?Warm-up

Two arrows from the origin of a 2-D plot, a in blue to (4.0, 2.5) and b in orange to (4.2, 2.3), pointing in nearly the same direction. — Typed 2-D stand-in; coordinates chosen for the picture.

3. Why do RAG systems split documents into chunks before embedding them?Warm-up

Typed schematic; five chunks is illustrative, 300-500 tokens is from the lesson.

Check: The Problem0/1

1. What is a bi-encoder retriever?Define

Two lanes: the query Q3 revenue growth and the sentence Revenue growth in Q3 came from Europe each go through an encoder box into a dashed box holding a question mark. — Typed schematic.

The Concept

ColBERT (2020)

ColBERT (Khattab & Zaharia, arXiv:2004.12832) is a text retrieval method. Instead of one vector per document, it produces one vector per token. At query time:

Query tokens get their own embeddings (N_q vectors).
Document tokens get embeddings (N_d vectors, typically cached).
Score = sum over query tokens of max over document tokens of cosine similarity: Σ_i max_j cos(q_i, d_j).

This is the MaxSim operation. Each query token "picks" its best-matching document token, and the final score is the sum.

Pros: strong recall, handles term-level semantics. Cons: N_d vectors per document, so storage is expensive.

ColPali

ColPali (Faysse et al., arXiv:2407.01449) applies the ColBERT pattern to images.

Each page is encoded by PaliGemma (ViT + language) into patch embeddings: N_p vectors per page.
Each user query (text) is encoded into query-token embeddings: N_q vectors.
Score = Σ_i max_j cos(q_i, p_j), i.e., MaxSim over query-text-tokens and page-image-patches.
Retrieve top-k pages by total score.

At ingestion time, embed every page with PaliGemma and store all the patch embeddings. At query time, embed the query tokens, compute MaxSim against every stored page, and return the top-k pages.

Pros: end-to-end accuracy beats text-RAG by 20-40% on visually rich documents, because each patch vector captures local layout and content.

Cons: N_p patches × 4-byte floats × D-dim vectors per page means storage grows fast. PQ / OPQ quantization mitigates it.

ColQwen2 and ColSmol

ColQwen2 (illuin-tech, 2024-2025) swaps PaliGemma for Qwen2-VL. That gives a better base encoder and better retrieval.

ColSmol is the smaller-scale variant for local / edge use. A ColSmol retriever at ~1B params runs on a consumer GPU.

VisRAG

VisRAG (Yu et al., arXiv:2410.10594) takes a different route. Instead of MaxSim on patches, it pools each page into a single vector with a VLM and then does bi-encoder retrieval. Indexing is faster and storage is smaller, but recall is weaker.

The trade-off is quality versus cost: ColPali for quality, VisRAG for scale.

M3DocRAG

M3DocRAG (Cho et al., arXiv:2411.04952) extends multi-modal retrieval to multi-page, multi-document reasoning. It retrieves pages across documents and composes a multi-page context for the VLM.

ViDoRe, the benchmark

ViDoRe (Visual Document Retrieval Evaluation) is ColPali's companion benchmark. Tasks include financial reports, scientific papers, administrative documents, medical records, and manuals. The metric is nDCG@5.

ColPali-v1 scores ~80% nDCG@5 on ViDoRe. Text-RAG on the same documents scores ~50-60%.

The end-to-end RAG pipeline

For a vision-native RAG:

Ingest: PDF → page images → PaliGemma encoding → store all patch embeddings.
Query: user text → query-token embeddings → MaxSim against all indexed pages → top-k pages.
Generate: top-k page images + query → VLM (Qwen2.5-VL or Claude) → answer.

There is no OCR step, so figures, charts, fonts, and layout can all flow into the answer.

Storage math

A 50-page financial report with 729 patches per page and 128-dim embeddings:

ColPali: 50 * 729 * 128 * 4 bytes = ~18 MB raw, ~4 MB after PQ.
Text-RAG: 50 chunks * 768-dim * 4 bytes = ~150 kB.

Raw, ColPali needs roughly 120x the storage of text-RAG for the same document. At scale, OPQ / PQ brings that down to ~5-10x, which is usually tolerable.

When text-RAG still wins

Pure-text documents with no layout signal (wiki articles, chat logs). Text-RAG is simpler and cheaper to store.
Multi-million-page archives where storage dominates cost.
Strict regulatory requirements that demand extractable OCR text alongside the retrieval.

For everything else in 2026, including financial reports, scientific papers, legal contracts, medical records, and UX documentation, vision-native RAG wins.

Check: The Concept0/23

1. What is late interaction in retrieval?Define

Two rows of small chips: three blue query vectors q1 to q3 and eight orange document vectors d1 to d8. — Typed schematic of a ColBERT-style query and document; the counts 3 and 8 are illustrative.

2. What does MaxSim compute for a query and a document?Define

A grid with three rows q1 to q3 and five columns d1 to d5, each cell holding a cosine value between 0.05 and 0.81. — Illustrative cosine values, typed, not captured.

3. What does a multi-vector index store?Define

A page cut into a 4 by 4 grid of 16 patches goes through a page encoder into an index entry whose content is shown as a question mark. — Typed schematic; 16 patches matches the toy pages in main.py.

4. What is a patch embedding in ColPali?Define

Typed schematic; the grid is coarser than a real page's patch grid.

5. What is ViDoRe?Define

A query box reading Q3 revenue growth above five small page thumbnails labelled finance, science, legal, medical and engineering. — Typed example item; page names from main.py.

6. What is product quantization (PQ) used for in a ColPali index?Define

Typed schematic; 512 bytes is 128 x 4.

7. What do ColQwen2 and ColSmol change relative to the original ColPali?Define

A left to right family tree: ColBERT (text, 2020) leads to ColPali (document pages), which leads to two boxes, ColQwen2 (illuin-tech) and ColSmol (variant). — Typed schematic of the lineage named in the lesson.

8. What does M3DocRAG add on top of ColPali-style retrieval?Define

Typed schematic; page counts are illustrative.

9. Why does a bi-encoder lose detail that ColBERT's token-level scoring keeps?Explain

A sentence of seven token chips. Below it, boxes for a bi-encoder and for ColBERT both point to one dashed box labelled how they differ, holding a question mark. — Typed example sentence.

10. How does ColPali apply the ColBERT pattern to document pages?Explain

On the left a box reading ColBERT pattern, text retriever, 2020, and a query reading Q3 revenue growth; on the right a finance page image; a dashed question-mark box sits between them. — Typed schematic.

11. How does VisRAG differ from ColPali, and what is the trade-off?Explain

A finance page image with arrows to ColPali (arXiv:2407.01449) and to VisRAG (arXiv:2410.10594), each followed by a dashed box reading index entry for this page with a question mark. — Typed schematic; arXiv ids from the lesson.

12. Why does ColPali beat text-RAG on visually rich documents such as financial reports?Explain

A financial report page with a title, text lines, a bar chart and a table, annotated with those parts and a query reading Q3 revenue growth. — Typed schematic page.

13. Why does MaxSim take the best match for each query token instead of averaging over all patches?Explain

Two blue query vectors [1.0, 0.1, 0.0] and [0.0, 1.0, 0.1] and four orange patch vectors [0.9, 0.9, 0.0], [0.1, 0.1, 0.1], [0.2, 0.2, 0.2] and [0.0, 0.0, 0.0]. — Vectors typed from main.py.

14. Why does the generation step pass the top-k page images, not extracted text, to the VLM?Explain

Three stacked page thumbnails labelled top-k pages (k = 3) lead through a dashed question-mark box, the input to the VLM, into a VLM box naming Qwen2.5-VL. — Typed schematic.

15. Which case is where text-RAG still wins over vision-native RAG?Explain

Two boxes, text-RAG (OCR text, chunks) and vision-native RAG (page images, patches), both pointing to a box asking which case favors the left one. — Typed schematic; the step names come from the lesson.

16. What nDCG@5 gap on ViDoRe is reported between ColPali-v1 and text-RAG on the same documents?Apply

An empty bar chart with a 0 to 100 nDCG@5 percent axis and two unfilled slots labelled ColPali-v1 and text-RAG. — Axes only; no values drawn.

17. Using the storage math (50 pages, 729 patches, 128-dim, 4-byte floats versus 50 chunks of 768-dim), about how much more raw storage does ColPali need than text-RAG?Apply

Two products of chips: ColPali is 50 pages times 729 patches times 128 dims times 4 bytes, text-RAG is 50 chunks times 768 dims times 4 bytes; both equal a question mark. — Setup from the stem.

18. A 200-page annual report has 729 patches per page, 128-dim embeddings and 4-byte floats. What are the raw and 8x PQ storage sizes?Apply

A grid of 200 page tiles in 20 columns by 10 rows, beside notes that each page has 729 patches of 128-dim 4-byte floats, and a line asking for raw size and size after 8x PQ. — Typed schematic; 200 tiles drawn as a 20 x 10 grid.

19. After moving a financial-report corpus from text-RAG to ColPali, answers about charts improve but the index grows about 100x. Which property explains the growth?Analyze

Two horizontal bars for index size: text-RAG a thin stub labelled about 1x and ColPali a long bar labelled about 100x. A note says answers about charts improve after the move. — Bars from the stem's rough figures, not measured.

20. A team keeps ColPali's patch encoder but averages each page's patch vectors into one vector at indexing time. Recall on chart-based questions drops. What changed?Analyze

A 4 by 4 patch grid labelled before, stored as N_p patch vectors; below it a chip labelled 1 vector under after, patch vectors averaged then stored; and a note that recall on chart-based questions drops. — Typed schematic.

21. A pipeline retrieves pages with ColPali but sends the OCR text of those pages to a text-only LLM. Answers about chart values are still wrong. Which part of the vision-native design was dropped?Analyze

Four boxes in a row: query, ColPali retrieval, OCR text of the top pages, text-only LLM. Beneath them a chart page and a note that chart values in the retrieved page come out wrong in the answers. — Typed schematic.

22. A team indexes 2,000 scanned insurance forms full of checkboxes, stamps and multi-column tables, and its OCR-plus-chunking RAG keeps missing answers. What should they try first, and what should they plan for?Design

A scanned insurance form with checkboxes, a red stamp and tables, labelled with those parts, and a note that the current pipeline is OCR then chunks then one vector per chunk, and that answers keep getting missed. — Typed schematic.

23. You must index a visually rich archive of 1M pages where storage and indexing cost matter more than the last bit of recall. Choosing between ColQwen2 and VisRAG, which fits?Design

A grid of 1,000 small cells (40 by 25), each standing for 1,000 pages, 1M pages in all, then two lines: priority 1 is storage and indexing cost, priority 2 is the last bit of recall, and a line naming the candidates ColQwen2 and VisRAG. — Typed schematic; 40 x 25 = 1,000 cells of 1,000 pages each.

Use It

code/main.py:

Builds 5 toy pages of 16 random 32-dim patch embeddings each, with a per-topic mean shift. There is no real encoder, only random vectors.
MaxSim scorer: computes the ColBERT-style score between a set of query token embeddings and a page's patch set.
Indexes the 5 pages, runs 3 queries of 4 token embeddings each, and prints the top 3 pages with scores.
Also prints MaxSim versus mean similarity on a small hand-built example, and a storage table.

Check: Use It0/5

1. One page has 729 patches of 128 dimensions stored as 4-byte floats. About how large is its raw ColPali storage?Apply

Typed schematic; 27 x 27 = 729.

2. In main.py, a query has 4 token embeddings and a page has 16 patches. How many cosine computations does maxsim run to score that page?Apply

The seven code lines of maxsim from main.py (its docstring is left out), followed by a note: query_tokens has 4 vectors and patches has 16 vectors. — Source lines quoted from main.py (asserted).

3. In the hand-built MaxSim versus mean example, MaxSim is 1.478 and the mean is 0.501. If 100 more irrelevant patches are added, each less similar to both query tokens than the strong patch, what happens?Apply

A 2 by 4 grid of cosines for the hand-built example, a dashed block for 100 more irrelevant patches, and a line giving the before scores MaxSim 1.478 and mean 0.501 with the after scores as a question mark. — Cosines computed with main.py's cosine; before-scores from the stem.

4. main.py calls retrieve(query, pages, k=3) on the 5 indexed pages. What does it return?Apply

Three lines from retrieve in main.py (scoring every page with maxsim, sorting by negative score, slicing to k) and the call retrieve(query, pages, k=3) over five indexed pages named finance, science, legal, medical, engineering. — Source lines quoted from main.py (asserted); the signature line is omitted.

5. In main.py's retrieval output, none of the three queries ranks its matching topic page first. What is the right conclusion?Analyze

Captured retrieval output of main.py for three queries, each listing three pages with scores. The page matching each query's topic is in orange: page_0_finance, page_1_science, page_3_medical. For the first query legal ranks first, for the second medical, for the third science. — Real output of python main.py, retrieval section only. The matching-page colouring is added by the script.

Ship It

This lesson produces outputs/skill-vision-rag-designer.md. Given a document-RAG project, picks ColPali / ColQwen2 / VisRAG / text-RAG and sizes the storage.

Author's Note

I'd run main.py and read the MaxSim versus mean printout first, because it is the whole idea in a dozen lines. On the hand-built example each of the two query tokens lands on the one strong patch, and MaxSim keeps those matches (1.478) while the mean gets diluted by the other patches (0.501).

Be careful with the retrieval half of the output. Pages and queries are random Gaussian vectors with a small per-topic mean shift, and there is no PaliGemma anywhere. In my run, none of the three queries ranked its matching topic page first. Read that output as a check that maxsim and retrieve behave, not as evidence that late interaction retrieves well. The 20-40% and ViDoRe figures come from the published benchmarks, not from this code.

The storage table is hardcoded strings too, not something the script measures. The arithmetic behind it is worth redoing by hand, and Exercise 1 does that for a 200-page report. Storage is what pushes a real project toward VisRAG or text-RAG, so I'd size it before picking a retriever.

Exercises

A 200-page annual report at 729 patches per page, 128-dim emb, 4-byte floats. Compute raw storage and PQ-compressed (8x) storage.
MaxSim is Σ_i max_j cos(q_i, p_j). What does this sum capture that a simple mean similarity does not?
ColPali indexes pages as patch sets. What changes if we instead index at the word level (as ColBERT does)? Trade-offs?
Design the end-to-end pipeline for a 1M-page corpus with a latency budget of 500ms per query. Pick ColQwen2 / VisRAG and justify.
Read M3DocRAG (arXiv:2411.04952). Describe the multi-page attention pattern and how it differs from single-page ColPali retrieval.

Key Terms

Term	What people say	What it actually means
Late interaction	"ColBERT-style"	Retrieval using per-token or per-patch embeddings + MaxSim, not a single doc vector
MaxSim	"Max-over-patches"	For each query token, pick the highest-similarity document token; sum across query
Bi-encoder	"Single-vector"	One vector per document; faster but loses granularity
Multi-vector	"Many-vectors-per-doc"	Store N_p vectors per document / page; storage cost grows but recall improves
Patch embedding	"Page feature"	One vector per image patch from a VLM encoder, cached per page
ViDoRe	"Vision doc bench"	ColPali's benchmark suite for visual document retrieval
PQ quantization	"Product quantization"	Compression that maintains vector similarity while shrinking storage ~8x