TAI BUI
Back to curriculum
CapstonePython (pipeline), TypeScript (viewer UI)

Capstone 04 — Multimodal Document QA (Vision-First PDF, Tables, Charts)

Vision-first document QA treats each PDF page as an image instead of running OCR and searching the text. ColPali, ColQwen2.5, and ColQwen3-omni embed every page with multi-vector late interaction, so the query attends to patches directly. On financial 10-Ks, scientific papers, and handwritten notes, that beats OCR-first by a wide margin. Build the pipeline end to end on 10k pages and publish the side-by-side against OCR-then-text.

TIME: 30 hoursPREREQUISITES: Phase 4 (computer vision), Phase 5 (NLP), Phase 7 (transformers), Phase 11 (LLM engineering), Phase 12 (multimodal), Phase 17 (infrastructure)

Learning Objectives

  • Explain how vision-first late-interaction retrieval (ColPali, ColQwen) scores page images with MaxSim, and why it beats OCR-then-text on tables, charts, and equations.
  • Build the pipeline end to end: render pages, embed with multi-vector ColQwen, prune with DocPruner, and index 10k pages in a multi-vector store.
  • Wire a vision-language answerer that cites pages and evidence regions, with an OCR text channel for equation-dense pages.
  • Evaluate the system with ViDoRe v3 nDCG@5 and M3DocVQA against an OCR-then-text baseline, and report a content-type by approach matrix.

Problem

Vision-first document retrieval means embedding the raw image of each PDF page and searching those embeddings, instead of running OCR and searching the extracted text. The reason to do it: enterprises sit on PDFs that OCR pipelines mangle. Scanned 10-Ks have rotated tables, scientific papers are dense with equations, charts only make sense as images, and handwritten annotations don't survive text extraction. Treat these as text-first and you lose half the signal.

The 2026 answer is late-interaction multi-vector retrieval on page images. ColPali (Illuin Tech) introduced it, and ColQwen2.5-v0.2 and ColQwen3-omni pushed accuracy further. On ViDoRe v3, vision-first retrieval scores above OCR-then-text by meaningful margins, and the gap widens on charts, tables, and handwriting.

The cost is storage and latency. A ColQwen embedding is ~2048 patch vectors per page (each a small 128-dim vector), not a single pooled vector per page, so raw storage balloons. DocPruner (2026) prunes 50% of the patches without measurable accuracy loss. You will index 10k pages, measure ViDoRe v3 nDCG@5, serve answers under 2s, and compare directly against an OCR-then-text baseline.

Warm-up0/3
1. A scanned annual report has rotated tables and charts. What is the main risk of running OCR first and searching only the extracted text?Warm-up
2. A retriever stores one embedding per document and compares it with the query embedding. What does it give up compared with keeping many embeddings per document?Warm-up
3. You build a new search system and also run a simple existing method on the same data. Why?Warm-up
Check: Problem0/4
1. What does the ViDoRe v3 benchmark measure in this capstone?Define
2. What is vision-first document retrieval?Define
3. Roughly what does a ColQwen embedding of one page look like, and what problem does that create?Explain
4. A teammate wants to avoid the multi-vector storage cost by embedding each page as one pooled vector. What is wrong with that fix?Analyze

Concept

Late interaction is a scoring scheme where every query token scores against every patch token, and the maximum score per query token is summed. That sum is called MaxSim. You get fine-grained matching without squeezing a page into one pooled vector. A multi-vector index (Vespa, Qdrant multi-vector, or AstraDB) stores the per-patch embeddings and runs MaxSim at retrieval time.

The answerer is a vision-language model that takes the query plus the top-k retrieved pages as images and writes an answer with evidence regions (bounding boxes or page references). Qwen3-VL-30B, Gemini 2.5 Pro, and InternVL3 are the 2026 frontier choices. For equations and scientific notation, an OCR fallback (Nougat, dots.ocr) can be spliced in as an optional text channel.

Evaluation is a two-dimensional matrix. One axis is content type: plain text paragraphs, dense tables, bar/line charts, handwritten notes, equations. The other is retrieval approach: vision-first late interaction, OCR-then-text, or a hybrid. Each cell gets nDCG@5 and answer accuracy. The report is the deliverable.

Check: Concept0/11
1. What is late interaction in retrieval?Define
2. What does it mean that a ColQwen index is multi-vector?Define
3. How does ColPali-style retrieval compute the MaxSim score of a page for a query?Define
4. What is an evidence region?Define
5. What is the OCR fallback in this design?Define
6. What is the role of the VLM answerer?Define
7. Why does MaxSim keep fine-grained matches that a single pooled page vector would lose?Explain
8. MaxSim sums the per-token maxima instead of keeping one best patch for the whole query. What does the sum add?Explain
9. Why does the index have to be a multi-vector store such as Vespa or Qdrant multi-vector?Explain
10. Why is the evaluation reported as a matrix of content type by retrieval approach rather than one overall score?Explain
11. Your corpus is mostly handwritten lab notes. Which evaluation design fits this capstone's approach?Design

Architecture

PDFs -> page renderer (PyMuPDF, 180 DPI)
           |
           v
  ColQwen2.5-v0.2 embed (multi-vector per page, ~2048 patches)
           |
           +------> DocPruner 50% compression
           |
           v
   multi-vector index (Vespa or Qdrant multi-vector)
           |
query ----+----> retrieve top-k pages (MaxSim)
           |
           v
  VLM answerer: Qwen3-VL-30B | Gemini 2.5 Pro | InternVL3
    inputs: query + top-k page images + optional OCR text
           |
           v
  answer with cited page numbers + evidence regions
           |
           v
  Streamlit / Next.js viewer: highlighted boxes on source page
Check: Architecture0/1
1. Auditors must be able to check where every answer number came from. Which design supports that?Design

Stack

  • Page rendering: PyMuPDF (fitz) at 180 DPI, portrait-normalized
  • Late-interaction model: ColQwen2.5-v0.2 or ColQwen3-omni (vidore team on Hugging Face)
  • Index: Vespa with multi-vector field, or Qdrant multi-vector, or AstraDB with MaxSim
  • Pruning: DocPruner 2026 policy (keep high-variance patches, 50% compression at < 0.5% accuracy loss)
  • OCR fallback (equations / dense tables): dots.ocr or Nougat
  • VLM answerer: Qwen3-VL-30B self-hosted or Gemini 2.5 Pro hosted; InternVL3 as fallback
  • Evaluation: ViDoRe v3 benchmark, M3DocVQA for multi-page reasoning
  • Viewer UI: Next.js 15 with canvas overlay for evidence regions
Check: Stack0/5
1. What does DocPruner do?Define
2. Why does the evaluation run M3DocVQA in addition to a page-retrieval benchmark like ViDoRe v3?Explain
3. You prune 90% of the patches and ViDoRe nDCG@5 drops below the OCR baseline. What does this show?Analyze
4. Your budget rules out Qwen3-VL-30B, and you consider a smaller VLM. How should you decide?Design
5. An index must fit in half the memory the full multi-vector version needs. Which plan fits the lesson's ideas?Design

Build It

  1. Ingest. Walk a corpus of 10k PDF pages across 10-Ks, scientific papers, and scanned documents. Render each page to a 1536x2048 PNG and persist {doc_id, page_num, image_path}.

  2. Embed. Run ColQwen2.5-v0.2 on each page image. The output is ~2048 patch embeddings of dim 128. Apply DocPruner to keep the highest-signal half, then write to a Vespa multi-vector field or Qdrant multi-vector.

  3. Query. Embed each incoming query with the query tower, which gives token-level embeddings. Run MaxSim against the index: for every query token, take the max dot-product over the page's patch embeddings, then sum. Return the top-k pages.

  4. Synthesize. Call Qwen3-VL-30B with the query and the top-5 page images. Prompt: "Answer using only the supplied pages. Cite each claim by (doc_id, page) and name the region (figure, table, paragraph)."

  5. Evidence regions. Post-process the answer to extract cited regions. If the VLM emits bounding boxes (Qwen3-VL does), render them as overlays in the viewer.

  6. OCR fallback. For pages identified as equation-dense (a heuristic on image variance), run Nougat or dots.ocr and pass the OCR text as an extra channel alongside the image.

  7. Eval. Run ViDoRe v3 (retrieval nDCG@5) and M3DocVQA (multi-page QA accuracy). Run the OCR-then-text pipeline on the same corpus with the same synthesizer, and produce a content-type × approach matrix.

  8. UI. Start with a Streamlit prototype, then build the Next.js 15 production viewer with page-by-page evidence-region overlays.

Check: Build It0/14
1. Why is an OCR text channel still spliced in for some pages?Explain
2. How does the viewer get the bounding boxes it overlays on the source page?Explain
3. Why does the synthesis prompt demand answers only from the supplied pages, each claim cited as (doc_id, page)?Explain
4. In main.py, doc_prune is called on a page with 7 patch vectors and keep_fraction=0.5. How many patches remain?Apply
5. In main.py, a query has two tokens. The dot products with a page's two patches are q1:[0.2, 0.7] and q2:[0.5, -0.1]. What does max_sim_score return?Apply
6. In main.py, a query has 4 tokens and a page has 5 patches. How many dot products does max_sim_score compute for that page?Apply
7. In the TypeScript viewer, the 10-K page is 1224 by 1584 and the canvas is half that size. A bbox has x=142. At what x does the overlay draw it?Apply
8. A client sends GET /document/10k.acme to the viewer server. What comes back?Apply
9. A client requests /document/10k-acme-2025 with the header Accept: application/json. What does the server return?Apply
10. In the TypeScript viewer, what does escapeHtml return for the input: a<b & cApply
11. In main.py, doc_prune is called on a page that has exactly 1 patch vector and keep_fraction=0.5. How many patches remain?Apply
12. Retrieval nDCG@5 looks strong on your eval, but answers about formulas keep coming out wrong. What is the most targeted fix?Analyze
13. You compare vision-first retrieval with an OCR-then-text baseline. What must stay identical for the comparison to be fair?Analyze
14. In the viewer, overlay boxes show up at twice their correct position on the half-size canvas. What was most likely skipped?Analyze

Use It

$ doc-qa ask "what was the 2024 operating margin change for segment EMEA?"
[retrieve]   top-5 pages in 320ms (ColQwen2.5, MaxSim, Vespa)
[synth]      qwen3-vl-30b, 1.4s, cited (form-10k-2024, p. 88) + (..., p. 92)
answer:
  EMEA operating margin moved from 18.2% to 16.8%, a 140bp decline.
  cited: 10-K-2024.pdf p.88 (Table 4, Segment Operating Margin)
         10-K-2024.pdf p.92 (MD&A, Operating Performance)
[viewer]     open with highlighted bounding boxes overlaid on p.88 Table 4
Check: Use It0/3
1. The doc-qa answer cites both p.88 (Table 4) and p.92 (MD&A). Why cite two pages?Explain
2. The doc-qa run reports retrieval in 320ms and synthesis in 1.4s. Against the under-2s serving goal, what follows?Apply
3. The doc-qa output says EMEA operating margin moved from 18.2% to 16.8%. What decline in basis points does that match?Apply

Ship It

outputs/skill-doc-qa.md describes the deliverable: a vision-first document QA system tuned to a specific corpus and evaluated against an OCR-then-text baseline on ViDoRe v3.

WeightCriterionHow it is measured
25ViDoRe v3 / M3DocVQA accuracyBenchmark numbers vs OCR-text baseline and published leaderboard
20Evidence-region groundingFraction of cited regions that actually contain the answer span
20Storage and latency engineeringDocPruner compression ratio, index p95, answer p95
20Multi-page reasoningAccuracy on a hand-labeled 100-question multi-page set
15Source-inspection UXViewer clarity, overlay fidelity, side-by-side comparison tools
100
Check: Ship It0/4
1. Why does the rubric score evidence-region grounding separately from answer accuracy?Explain
2. Why should the OCR-then-text baseline be built before the vision pipeline?Explain
3. Your answers cite the correct pages, but the highlighted boxes often miss the answer text. Which rubric measure captures this?Analyze
4. Single-page questions pass, but questions needing facts from two pages fail. Which rubric line is most likely low?Analyze
Author's Note

I'd build the OCR-then-text baseline before the vision pipeline, not after. The whole claim of this capstone is a gap between the two, and the Ship It rubric scores you against that baseline, so a ColQwen system with nothing to compare to has no result to report.

My bet is that evidence-region grounding is the number to watch, more than nDCG@5. The rubric defines it as the fraction of cited regions that actually contain the answer span. An answer that is right but points at the wrong box is a citation nobody can check, and the viewer exists so that a person can check.

Exercises

  1. Measure ColQwen2.5-v0.2 against ColQwen3-omni on the same corpus. Which pages does one get right and the other miss? Add a "content class" tag to the index to route by type.

  2. Prune embeddings aggressively (75%, 90%) and find the compression cliff, the point where ViDoRe nDCG@5 drops below the OCR baseline.

  3. Build a hybrid. Run OCR-then-text and ColQwen in parallel, fuse with RRF, and rerank with a cross-encoder. Does the hybrid beat either alone? Where does it help most?

  4. Swap Qwen3-VL-30B for a smaller VLM (Qwen2.5-VL-7B) and measure the accuracy-per-dollar curve.

  5. Add handwritten-note support. Render the handwriting corpus, embed it with ColQwen, and measure retrieval. Compare against a handwriting OCR pipeline.

Key Terms

TermWhat people sayWhat it actually means
Late interaction"ColPali-style retrieval"Query tokens score against page patches independently; MaxSim aggregates
Multi-vector"Per-patch embedding"Each document has many vectors, not one pooled vector
MaxSim"Late-interaction scoring"For every query token, take max similarity over document vectors; sum
DocPruner"Patch compression"2026 pruning that keeps 50% of patches with negligible accuracy loss
ViDoRe v3"Document-retrieval benchmark"The 2026 standard for measuring visual-document retrieval
Evidence region"Cited bounding box"A bbox on the source page that localizes the answer span
OCR fallback"Equation channel"Text pipeline used alongside vision for equation- or table-heavy pages

Further Reading

Code

Ships With