The leaderboard cannot tell you what broke
Most document AI evaluations begin with an input page and end with a task score. That makes procurement easy: one system gets 84, another gets 87, and the larger number wins. It also removes the diagnostic information an engineering team needs. The final answer combines at least three separate events: the system read the marks on the page, preserved their structure and relationships, and reasoned from the right evidence. A failure in any one produces the same zero.
The reverse is possible too. A model can miss a label, infer the intended field from context, and still return the expected answer. The benchmark records a success even though the intermediate representation is unsafe for retrieval, audit or a second question. A single score therefore hides both false confidence and wasted remediation: teams tune prompts to fix an OCR defect, replace OCR to fix a reasoning defect, or celebrate an answer that cannot be traced back to the page.
The DISCO benchmark makes the useful move: it evaluates parsing and question answering separately across handwriting, multilingual text, forms, infographics, charts and multi-page documents. Its findings do not produce one universal winner. OCR pipelines are stronger in some long-document and handwriting settings; direct visual question answering is stronger on some single-page, visually structured tasks. The architecture should follow the document, and the benchmark should reveal why.
Define two contracts before choosing metrics
Parsing and reasoning are not merely two columns in a results sheet. They are two contracts with different inputs, outputs and failure evidence.
| Layer | Contract | Failure evidence |
|---|---|---|
| Parsing fidelity | Turn the source page into a faithful representation of text, structure, geometry and reading order without adding meaning. | Character substitutions, missing blocks, broken cells, lost coordinates, incorrect order and unsupported text. |
| Reasoning quality | Answer a defined question from the supplied representation or image, cite the evidence used and abstain when the evidence is insufficient. | Wrong answer with correct evidence, correct answer with wrong evidence, unsupported inference, missed relationship and failed abstention. |
This boundary matters even when a VLM performs both operations in one call. The benchmark can still ask for a transcription or structured representation first, score it, then ask questions against either the original image or the captured representation. The point is not to force a two-stage production architecture. It is to preserve enough intermediate evidence to locate the defect.
Build the test set around document characteristics
A representative set is not a random folder of easy pages. It is a declared matrix of the characteristics that change system behaviour. Start with production intake, include rejected and low-quality documents, and stratify before sampling.
- Document form — native PDF, scan, mobile photograph, fax, handwriting and mixed digital-plus-image pages.
- Structure — plain paragraphs, multi-column text, nested tables, forms, charts, footnotes and repeated headers.
- Length — single page, short packet and long multi-page document where evidence may cross page boundaries.
- Language and script — the actual language distribution, including diacritics and non-Latin scripts rather than an English-only convenience set.
- Operational difficulty — blur, rotation, low contrast, stamps, handwriting over print, cropped edges and duplicated pages.
- Consequence — fields and questions whose error changes money, eligibility, compliance or a customer commitment.
Publish the count in every cell. A mean over 500 pages is not reproducible if 420 are clean single-page forms and the other 80 carry every failure mode. Report results for each stratum and keep a fixed regression subset that never enters prompt development.
Use metrics that correspond to the artifact
No metric describes all parsing errors. Use a small family and state what each one misses.
| Artifact | Primary measure | Why it is not enough alone |
|---|---|---|
| Running text and handwriting | Character error rate and word error rate | They do not preserve layout or tell you whether a value moved to the wrong label. |
| Fields and identifiers | Exact match plus normalized match | Normalization can hide material punctuation, leading zeros or date ambiguity. |
| Tables | Cell content, row-column structure and header association | A flat text match can score well after destroying the table relationships. |
| Layout | Block detection, class, reading order and geometry overlap | Correct boxes do not prove the text inside them is correct. |
| Question answering | Exact or normalized answer plus evidence support | Answer match alone rewards lucky guesses and unsupported correction. |
| Abstention | Precision and recall of accepted versus rejected answers | A high answer score can coexist with a dangerous willingness to answer without evidence. |
DISCO uses complementary parsing and QA measures, including CER, WER, ANLS and answer matching, because their disagreements are informative. A semantically similar answer with poor string fidelity may be acceptable for search and unacceptable for an account number. The benchmark must encode that difference instead of averaging it away.
Score the evidence, not only the answer
Every reasoning case should contain four things: the question, the expected answer, the exact supporting evidence and the rule for insufficiency. Evidence can be a page and bounding region, a table cell, or a set of spans across pages. Then classify each response into a diagnostic outcome.
| Answer | Evidence | Diagnosis |
|---|---|---|
| Correct | Correct and complete | Supported success. |
| Correct | Wrong, missing or incomplete | Lucky or unsupported success; do not count as production-safe. |
| Wrong | Correct | Reasoning or output-format failure. |
| Wrong | Wrong because the representation is wrong | Parsing or retrieval failure propagated downstream. |
| Abstained | Evidence genuinely insufficient | Correct refusal. |
| Answered | Evidence insufficient | Unsupported inference. |
This table converts a leaderboard into an engineering queue. Parsing failures go to capture, OCR or layout work. Correct parsing with failed reasoning goes to question design, retrieval, model selection or validation. Unsupported successes go to the same high-priority queue as wrong answers because they cannot survive an audit.
Three errors a single score conceals
1. The right amount from the wrong row
An invoice contains the same amount in a subtotal, tax summary and remittance block. The model returns the expected total, but its cited region points to the subtotal. Exact match says pass. Evidence scoring says the system has learned a coincidence that will fail as soon as those values diverge.
2. Perfect text in the wrong reading order
A two-column policy page is transcribed with every word correct and the columns interleaved. Character error rate can remain deceptively strong while a clause becomes a different sentence. The correct parsing test includes reading-order edges, and the reasoning test asks a question whose evidence crosses the affected boundary.
3. A wrong transcription repaired by prior knowledge
A VLM reads a blurred dosage incorrectly, recognizes the medication name, and emits the common dosage instead. The final answer may match the label. Parsing fidelity exposes the substitution; evidence scoring exposes that the answer is not grounded in what the page actually supports.
Keep architectures and prompts controlled
A fair comparison changes one variable at a time. Run the same cases through at least three explicit paths where they are relevant: OCR followed by text reasoning, VLM parsing followed by text reasoning, and direct visual question answering. Pin provider, model version, prompt, decoding settings, image resolution and page handling. Record latency, cost and failure evidence beside quality.
Do not silently give one model a task-aware prompt and another a generic instruction. DISCO reports mixed effects from task-aware prompting: it can help one document type and degrade another. Prompt variants are experimental conditions, not tuning trivia. Version them and report each condition separately.
Multi-question workloads also change the economics. A parsed representation can be reused across many questions, while direct visual QA may process the image again for each one. Quality per call is not quality per completed document. Report cost and latency at the workload shape you actually operate.
A result sheet that can drive a decision
The final report should not begin with an overall score. Begin with the document strata and the failure decomposition.
- Parsing panel — text, fields, tables, layout and reading order, each split by document characteristic.
- Reasoning panel — answer correctness, evidence support and abstention quality on both gold parsing and system parsing.
- Error propagation panel — the loss between reasoning over gold representation and reasoning over each system's representation.
- Operational panel — latency distribution, cost per completed document, retry rate and human-review rate.
- Change panel — model, prompt and dataset versions with deltas against the fixed regression set.
Reasoning over gold parsing is the control that makes attribution possible. If the model fails there, the parser is not the bottleneck. If it succeeds on gold and fails on the system representation, the loss belongs upstream. If direct visual QA succeeds where both two-stage paths fail, the intermediate representation may be discarding spatial evidence.
The reproducibility checklist
- Freeze a manifest — immutable case IDs, source hashes, licenses, strata and train-development-test boundaries.
- Version the ground truth — text, blocks, tables, answers and evidence annotations with reviewer agreement recorded.
- Publish the pipeline contract — input conversion, page resolution, prompts, schemas, timeouts, retries and post-processing.
- Pin every moving part — model and API versions, OCR engine, libraries, container image and configuration.
- Save raw outputs — never keep only the aggregate; retain intermediate representations, answers, evidence and errors.
- Run repeated trials where output can vary — report dispersion, not only the best run.
- Separate development from evaluation — prompt iteration never sees the held-out regression set.
- Report by stratum — include sample counts and confidence intervals; do not let the largest easy group dominate the mean.
- Verify links and artifacts — every cited source, dataset reference and internal path must resolve at publication time.
- Define the release gate — state which parsing, reasoning, evidence and abstention regressions block deployment.
The open DISCO repository is useful because it exposes datasets, pipeline phases, prompts, metrics and results structure rather than publishing only a chart. The principle matters more than adopting one suite unchanged: a benchmark is reproducible when another team can reconstruct the run and diagnose the same failures.
What the benchmark should change
An evaluation has done its job when it changes routing and release decisions. Handwriting may remain on specialized OCR. A visually dense single-page form may go direct to a VLM. Long documents may use explicit text grounding and retrieval. A high-consequence field may require both an exact parsing check and evidence-backed reasoning before automation can proceed. One architecture for every page is easier to draw and harder to defend.
At Cogneris, that diagnostic boundary is part of the operating model: extracted values retain provenance, validation names the check that failed, and document paths can escalate by evidence rather than by a model's self-reported confidence. The practical next step is not another leaderboard. It is a small, versioned evaluation set made from your own intake, with parsing and reasoning measured separately and every failure preserved well enough that somebody can fix it.