TAI BUI
Back to curriculum
BuildPython

Document and Diagram Understanding

Documents are not photos. A PDF, scientific paper, invoice, or handwritten form carries layout, tables, diagrams, footnotes, headers, and semantic structure that plain image understanding misses. The pre-VLM stack was a pipeline of Tesseract OCR, LayoutLMv3, and table-extraction heuristics. The VLM wave replaced it with OCR-free models (Donut in 2022, Nougat in 2023, DocLLM in 2023) that emit structured markup directly. By 2026 the frontier is to feed the page image to Claude Opus 4.7 at 2576px native and get structured markup back without extra machinery. This lesson walks the three-era arc of document AI.

TIME: ~180 minutesPREREQUISITES: Phase 12 · 05 (LLaVA), Phase 5 (NLP)

Learning Objectives

  • Explain the three eras of document AI: OCR pipeline, OCR-free, VLM-native.
  • Describe LayoutLMv3's three input streams: text, layout (bbox), image patches, with unified masking.
  • Compare Donut (OCR-free, image → markup), Nougat (scientific paper → LaTeX), DocLLM (layout-aware generative), PaliGemma 2 (VLM-native).
  • Pick a document model for a new task (invoices, scientific papers, handwritten forms, Chinese receipts).

The Problem

"Understand this PDF" sounds simple and isn't. The information sits in:

  • Text content (90% of the signal).
  • Layout (headers, footnotes, sidebars, two-column format).
  • Tables (rows, columns, merged cells).
  • Figures and diagrams.
  • Handwritten annotations.
  • Fonts and typography (title vs body).

Raw OCR dumps the text and drops the rest. A system that cares about invoices needs to know that "Total: $1,245" came from the bottom-right, not from a footnote.

Warm-up0/3
1. In a LLaVA-style VLM, how does an image reach the language model?Warm-up
A photo frame and a text prompt asking what is in the photo both point into a dashed box with a question mark, which feeds a box labelled LLM that produces a reply.
Typed schematic.
2. A text-only NLP model receives a document as one flat dump of OCR text. What information is lost?Warm-up
A page with the large words Spring Sale at the top, Everything half price in the middle and small words Offer ends Friday at the bottom, and an arrow to a box holding the same words as one line of plain text.
Typed example page.
3. A LLaVA-style encoder reads fixed 336px tiles. What is the risk when a dense A4 page is squeezed into a few tiles?Warm-up
A page filled with many thin lines of small text, 2500 by 3500 pixels at 300 DPI, with an arrow to a grid of four square tiles each labelled 336 by 336.
Typed schematic. The 2500 x 3500 figure is the lesson's main.py.
Check: The Problem0/1
1. Why does a system that processes invoices need layout information and not only the text?Explain
Seven text chips in reading order: INVOICE, Lamp, $55.00, Desk, $90.00, Total, $145.00, with an arrow down to a box for an invoice system whose task is to read the total.
Typed example. The invoice is made up.

The Concept

Era 1: OCR pipeline (pre-2021)

The classic stack:

  1. PDF → image per page.
  2. Tesseract (or commercial OCR) extracts text with per-word bounding boxes.
  3. Layout analyzer identifies blocks (header, table, paragraph).
  4. Table structure recognizer parses tables.
  5. Domain rules + regex extract fields.

This works for clean printed text. It breaks on handwriting, skewed scans, complex tables, and non-English scripts, and each failure mode tends to need its own custom exception path.

TrOCR (2021)

TrOCR (Li et al., arXiv:2109.10282) replaced Tesseract's classic CNN-CTC with a transformer encoder-decoder trained on synthetic + real text images. It was a clean win on handwritten and multilingual text. The stack was still a pipeline (detector, then TrOCR, then layout), but the OCR step got much better.

Era 2: OCR-free (2022-2023)

The first OCR-free models skipped detection entirely and mapped image pixels straight to structured output.

Donut (Kim et al., arXiv:2111.15664):

  • Encoder-decoder transformer, encoder is Swin-B.
  • Output is JSON for form understanding, markdown for summarization, or any task-specific schema.
  • No OCR, no layout, no detection.

Nougat (Blecher et al., arXiv:2308.13418):

  • Trained specifically on scientific papers.
  • Output is LaTeX / markdown.
  • Handles equations, multi-column layout, figures.
  • A common choice for parsing arXiv.

These are specialists, not generalists. Donut on a scientific paper fails, and so does Nougat on an invoice.

LayoutLMv3 (2022)

LayoutLMv3 (Huang et al., arXiv:2204.08387) takes a different track. It keeps OCR and adds layout understanding:

  • Three input streams: OCR text tokens, per-token 2D bounding boxes, image patches.
  • Masked training objective across all three modalities (masked text, masked patches, masked layout).
  • Downstream: classification, entity extraction, table QA.

This is the peak of OCR-based document understanding. It is strong on forms and invoices, needs OCR upstream, and had the best pre-VLM accuracy on standardized document benchmarks.

DocLLM (2023)

DocLLM (Wang et al., arXiv:2401.00908) is LayoutLM's generative sibling. It generates free-form answers conditioned on layout tokens, which suits QA on documents, but it still depends on OCR input.

Era 3: VLM-native (2024+)

By 2024 VLMs were good enough to replace the pipeline. Feed the full page image at high resolution to a VLM, ask the question, get an answer.

  • LLaVA-NeXT 336-tile AnyRes works for small documents.
  • Qwen2.5-VL dynamic-resolution handles 2048+ pixels natively.
  • Claude Opus 4.7 supports 2576px documents.
  • PaliGemma 2 (April 2025) trains specifically for documents + handwriting.

The gap between VLM-native and OCR-pipeline closed fast. By 2026, VLM-native wins on:

  • Scene text (hand-written + printed, mixed scripts).
  • Complex tables with merged cells.
  • Math equations embedded in text.
  • Figures with text annotations.

OCR pipelines still win on:

  • Pure-scan workloads at massive scale where per-page latency matters.
  • Pipeline reliability (deterministic failures vs VLM hallucinations).
  • Regulated environments that need auditable OCR output.

The Claude 4.7 / GPT-5 frontier

At 2576-pixel native input, frontier VLMs reach near-human accuracy on document understanding. The benchmark numbers from early 2026:

  • DocVQA: Claude 4.7 ~95.1, PaliGemma 2 ~88.4, Nougat ~77.3, pipelined LayoutLMv3 ~83.
  • ChartQA: Claude 4.7 ~92.2, GPT-4V ~78.
  • VisualMRC: Claude 4.7 ~94.

The closed-model gap comes mostly from resolution and base-LLM scale. Open models at 7B are a few points behind and catching up.

Math equations and LaTeX output

Scientific papers need exact LaTeX for equations, and Nougat was trained on exactly that. VLMs trained with LaTeX targets (Qwen2.5-VL-Math, Nougat derivatives) produce usable LaTeX. Without explicit LaTeX training, VLMs give readable but imprecise transcriptions.

For scientific-paper pipelines in 2026, chain Nougat on the PDF, then a VLM on the tricky pages.

Handwriting

Handwriting is still the hardest sub-task. Mixed printed + handwritten content (doctors' notes, filled forms) is where OCR pipelines still beat VLMs on cost. Handwritten-only VLMs are improving (Claude 4.7, PaliGemma 2).

2026 recipe

For a new document-AI project:

  • Pure-printed invoices at scale: LayoutLMv3 + rules, cost-efficient.
  • Mixed documents (scientific + handwritten + forms): VLM-native (PaliGemma 2 or Qwen2.5-VL).
  • Full arXiv ingestion: Nougat for math, VLM for figures.
  • Regulatory: OCR pipeline + VLM validator for cross-check.
Check: The Concept0/22
1. What is an OCR pipeline?Define
A scanned line reading Hello world and an arrow to a small table with one row per word showing text, x, y, width, height and a confidence value.
Typed illustration. The numbers are made up.
2. What makes a document model layout-aware?Define
A short form with Name, Date and Amount lines, an arrow to a box labelled LayoutLMv3, and an arrow to a list of the fields found.
Typed example. The form values are made up.
3. What does OCR-free mean for a document model?Define
A page image with an arrow into a large box labelled Donut, encoder-decoder transformer with a Swin-B encoder, with an arrow to a dashed box labelled output.
Typed schematic from the lesson's Donut notes.
4. What is a VLM-native approach to documents?Define
A mock chat window: a page image attachment named page.png and a user message asking what the total due on the page is, with an empty reply bubble below.
Typed mock-up, not a real chat.
5. What is DocVQA?Define
A small page announcing a staff meeting on Thursday in Room 4B, with a question card asking on which day the meeting is and a reference answer card reading Thursday.
Typed illustration of a question-answer item. It is not an item from any real benchmark.
6. Why do models like Donut and Nougat emit markup output such as JSON, markdown or LaTeX instead of free-form text?Define
A receipt page with two arrows to output cards: one a plain sentence giving the shop, the amount and the payment method, the other a JSON object with shop, paid and method fields.
Typed example outputs for one made-up receipt.
7. What are LayoutLMv3's three input streams?Define
A page with the word INVOICE and grey text lines, an arrow labelled input into a box labelled LayoutLMv3, and arrows out to three downstream tasks: classification, entity extraction and table QA.
Typed schematic. The downstream tasks are the ones named in the lesson.
8. What did TrOCR (2021) replace in the classic OCR pipeline?Define
A vertical stack of five classic OCR pipeline steps on the left, and on the right a card describing TrOCR from 2021 as a transformer encoder-decoder trained on synthetic and real text images.
Typed from the lesson's Era 1 list and TrOCR section.
9. Why does Nougat fail on an invoice?Explain
A made-up invoice with two item lines and a price column, an arrow into a box labelled Nougat, and an arrow to a dashed box labelled result.
Typed schematic. The invoice is made up.
10. How does DocLLM relate to LayoutLMv3?Explain
Two cards: LayoutLMv3 from 2022 with its downstream tasks, and DocLLM from 2023 with document QA, joined by a dashed box holding a question mark.
Typed from the lesson's LayoutLMv3 and DocLLM sections.
11. Where do OCR pipelines still beat VLM-native approaches?Explain
A pile of pages with two arrows, one to a card for the OCR pipeline (Tesseract, TrOCR, LayoutLMv3) and one to a card for VLM-native (Qwen2.5-VL, PaliGemma 2, Claude 4.7).
Model names from main.py's eras table.
12. Why is a stage-wise OCR pipeline described as fragile?Explain
Five boxes in a row for the Era 1 stack: PDF to image, OCR plus word boxes, layout analyzer, table recognizer, rules plus regex, with page in on the left and fields out on the right.
Typed from the lesson's Era 1 list.
13. What does an OCR-free model such as Donut remove compared with a LayoutLMv3 setup?Explain
Two cards side by side: LayoutLMv3 from 2022, strong on forms and invoices, and Donut, an encoder-decoder transformer with a Swin-B encoder.
Typed from the lesson.
14. Why does the 2026 recipe for scientific papers run Nougat on the PDF first and then a VLM on the tricky pages?Explain
A two-column paper PDF, then a Nougat box that runs first on the PDF, then a VLM box that runs on tricky pages, then a parsed paper.
Typed schematic from the stem.
15. Frontier closed VLMs lead open 7B models on document benchmarks. What accounts for most of that gap?Explain
Two horizontal bars: Claude 4.7 at about 95.1 on DocVQA, and a dashed shorter bar for open 7B models marked a few points behind, with the gap between them bracketed.
Claude score from the lesson. The open-model bar is a sketch with no exact value.
16. You must ingest 2M arXiv papers that contain equations and figures. Which stack fits?Apply
A paper page with two text columns, a boxed equation and a boxed figure, next to a card reading 2M arXiv papers to ingest.
Typed sketch of one page. The equation is only an example.
17. A teammate cites Claude 4.7's DocVQA lead and proposes it for 10M invoices a day. What does the evidence suggest?Analyze
Two cards, Claude 4.7 with DocVQA about 95.1 and a proposal card reading Use Claude 4.7 for the invoice work, above a banner reading Workload: 10M invoices a day.
DocVQA scores as quoted in the lesson.
18. A LayoutLMv3 pipeline returns nothing for a page that was a skewed phone photo. What is the likeliest cause?Analyze
A tilted photo of a page with lines of text, an arrow into a LayoutLMv3 pipeline box, and an arrow to a dashed red box reading nothing.
Typed schematic.
19. A VLM-native system returns a confident invoice total that appears nowhere on the page. Which tradeoff does this show?Analyze
An invoice listing Chair 85.00, Desk 210.00 and Total 295.00, an arrow to a VLM box, and an output card reading Total due 305.00 stated confidently.
Typed example with made-up numbers.
20. A scientific-paper pipeline uses a VLM never trained on LaTeX targets. Its equations read well but do not compile. What is the likely cause?Analyze
An equation on a page goes into a VLM, whose output is an equation text that reads well; below it a banner says the LaTeX compile step does not compile.
Typed schematic. The equation is only an example.
21. A bank needs auditable processing of scanned forms and also wants an AI sanity check. What design do you pick?Design
A stack of scanned forms beside two need cards: processing that can be audited, and an AI sanity check as well.
Typed sketch of the stem's two needs.
22. You must read receipts that mix Chinese and Latin text at moderate volume, with no audit mandate. Which approach fits best?Design
A receipt sketch where Chinese characters are drawn as square glyph placeholders next to Latin words and amounts, with cards stating moderate volume and no audit mandate.
Typed sketch. Square glyphs stand in for Chinese characters. Amounts are made up.

Use It

code/main.py is stdlib-only and does three things:

  • A toy layout-aware tokenizer: given (text, bbox) pairs, it produces the LayoutLMv3-style input.
  • A Donut-style task schema generator: JSON templates for invoices and forms.
  • A comparison of token budgets per page across OCR-pipeline, Donut, Nougat, and VLM-native.
Check: Use It0/6
1. In the toy invoice from mock_page(), which token has the bbox (400, 440, 550, 470)?Apply
Axes with the origin at the top-left corner and x to the right and y downward, and a small orange rectangle with corners (400, 440) and (550, 470) containing a question mark.
Coordinates from the stem. Scale is drawn proportionally.
2. layoutlm_input is called with patch_grid=(8, 8). How many entries does patch_ids hold?Apply
Code for layoutlm_input with the text_ids expression elided and two lines hidden, then a call with patch_grid set to (8, 8) and a prompt asking for the length of patch_ids.
Real source from main.py with two lines hidden and the text_ids expression elided. The result is left as a question mark.
3. What does donut_schema("receipt") return?Apply
Code for donut_schema with a schemas dict that has invoice and form entries whose bodies are collapsed and a hidden return line, then a call with the argument receipt and a question mark for the result.
Real structure from main.py with bodies and the return line hidden.
4. Using the per-page numbers in main.py's token-budget table, roughly how many input tokens per day does OCR pipeline plus LayoutLMv3 use at 10M pages a day?Apply
A six-row table of stacks from main.py with the tokens-per-page column hidden behind question marks, and a note that the workload is 10,000,000 pages a day.
Stack names from main.py. The token counts are masked in this picture.
5. Which top-level keys does donut_schema("form") define?Apply
A made-up form with Name, Date and City lines and an arrow to a box reading donut_schema("form") that returns a dict, followed by a dashed box with a question mark for its top-level keys.
Typed example form.
6. You run main.py twice and the text_ids differ. Why?Analyze
A terminal showing two runs of python main.py: the text_ids line differs between the runs while the bbox_stream line is identical.
Captured from two real runs of main.py. The other output lines are left out.

Ship It

This lesson produces outputs/skill-document-ai-stack-picker.md. Given a document-AI project (domain, scale, quality, regulatory), it picks between OCR pipeline, OCR-free specialist, and VLM-native.

Check: Ship It0/1
1. You are picking a stack for a new document project and want to avoid choosing on benchmark scores alone. What should you settle first?Design
Four horizontal bars of DocVQA scores: Claude 4.7 about 95.1, PaliGemma 2 about 88.4, pipelined LayoutLMv3 about 83, Nougat about 77.3.
Scores as quoted in the lesson text.
Author's Note

I'd run main.py and read its output before the prose. The mock invoice has eight tokens, and each one carries a bbox next to its text. That pairing is the whole idea behind LayoutLMv3. "Total" and "$1,245.00" sit at y=400 and y=440, well below the line-item price at y=240, so a model that sees coordinates can tell a grand total from a unit price without guessing from the text alone.

I'd also treat the toy as a shape demo and nothing more. The text_ids come from Python's hash() of each string, so they are not a real vocabulary, and string hashing is salted per process by default, so the ids can change between runs. patch_ids is just range(256) for a 16x16 grid with no pixels behind it. The token-budget table is a list of hardcoded rows, not something the script measures. I'd read it for orders of magnitude (512 for the OCR pipeline, 12000 for Claude 4.7 at 2576px) and not as a benchmark.

If I were picking a stack for a real project, I'd settle scale and audit requirements first and look at accuracy second. At 10M invoices a day the token counts in that table decide more than a few DocVQA points do, and a regulated workload wants the deterministic OCR output whatever the VLM scores.

Exercises

  1. Your project is 10M invoices per day. Which stack minimizes cost-per-page without losing accuracy?

  2. Why does LayoutLMv3 outperform pure-CLIP-VLMs on form QA but underperform at scene-text? What does the bbox stream give up?

  3. Nougat generates LaTeX. Propose a test case where VLM-native output beats Nougat on LaTeX fidelity, and a case where Nougat wins.

  4. Read PaliGemma 2 paper (Google, 2024). What was the key training-data addition that lifted document accuracy vs PaliGemma 1?

  5. Design a regulatory-safe hybrid: OCR pipeline as primary, VLM as secondary cross-check. How do you resolve disagreement?

Key Terms

TermWhat people sayWhat it actually means
OCR pipeline"Tesseract-style"Stage-wise stack: detect -> OCR -> layout -> rules; deterministic, fragile
OCR-free"Donut-style"Image-to-output transformer that skips explicit OCR; single model
Layout-aware"LayoutLM"Input includes per-token bbox coordinates; unified masking across modalities
VLM-native"Frontier VLM"Feed page image directly to Claude/GPT/Qwen VLM at high resolution; no pipeline
DocVQA"Doc benchmark"Document VQA standard; most-cited score
Markup output"LaTeX / MD"Structured output format instead of free-form text; enables downstream automation

Further Reading

Code

Ships With