OCR accuracy: measure the output you use.

What does OCR accuracy measure?

Start with the output the next system consumes. A transcription test asks whether the recognized characters match a reference. A document-extraction test asks whether the right value was assigned to the right field. A workflow test asks whether the document can proceed without a person correcting it. A system can perform well on one and still disappoint on another.

For example, a statement may contain thousands of correctly recognized characters and one incorrect closing balance. The transcription result can look strong while the record is unusable for reconciliation. Report the metric and denominator beside every result, rather than describing all of these outcomes as one accuracy percentage.

Choose metrics before choosing a model

• Text recognition — character error rate counts substitutions, insertions and deletions against reference characters. Word error rate applies the comparison to words. State normalization rules and how empty references are handled.
• Field extraction — compare each required value with reviewed ground truth after agreed normalization of dates, currencies and whitespace. Keep missing fields, unsupported fields and incorrect values distinguishable.
• Tables and layout — check row and column association, merged cells and reading order separately from the characters inside cells. Correct text in the wrong row is still a downstream error.
• Operational acceptance — record documents that pass your acceptance checks, correction effort, failed requests, latency and cost. Define the configured review policy before calculating a rate.

The Cogneris extraction benchmark guide describes how to structure a comparison around your fields and acceptance criteria. A proposed evaluation method is not a measured Cogneris performance result.

Build a representative evaluation set

Separate native PDFs, clean scans, phone photos, handwriting and difficult layouts. Include the languages, document types and low-quality inputs your service actually receives. Keep a held-out set so examples used to tune prompts or routing do not also supply the final score.

Have a person check the reference values and document ambiguous cases. For each run, retain the model identifier, preprocessing settings, schema, prompt version, dataset version and scoring rules. Record errors and refusals instead of removing them from the denominator. A score without the input mix and the failed cases is hard to reproduce.

Why input quality still matters

Skew, noise, uneven backgrounds and unsuitable segmentation can damage recognition before a field is extracted. The Tesseract quality guide documents these failure modes and possible preprocessing changes. Compare the original and processed images on your own test set; an image transformation is useful only if the measured output improves.

Preserve the source file and the relationship between any crop or rotated page and its original coordinates. That lets a reviewer inspect the same evidence the pipeline processed, especially when a digit, date or row assignment is uncertain.

Compare OCR and multimodal pipelines fairly

Run each candidate on the same files and score the same output. If one pipeline returns plain text and another returns structured fields, add an explicit extraction stage before comparing field correctness. Include that stage's cost, errors and latency in the result.

OmniDocBench provides a public framework for document-parsing evaluation with separate text, table and layout considerations. Pin the dataset and evaluator version when reporting its results. A public benchmark can inform a shortlist, but its document distribution does not establish performance on your own workload.

Confidence is a signal to evaluate

A confidence value is not proof that a field is correct. Check whether the scores separate correct and incorrect values on held-out documents. Inspect important fields individually and count confident errors as well as uncertain correct values. Set acceptance thresholds from the consequences of a wrong value and the correction work your team can support.

Cogneris extraction responses can include per-field confidence and source locations when the value was located on the page. Location coverage depends on the output; it is not guaranteed for every field. A manual review phase must be explicitly configured for the tenant's workflow. See the extraction response contract and module availability for the current boundaries.

Measure the pipeline before replacing it

Compare candidates by document class, not only by the average. A useful report includes required-field correctness, rejected or failed inputs, manual correction time, complete workflow cost and latency. Keep the existing route available until the replacement meets the acceptance criteria you set before the experiment.

When a model, prompt, schema or preprocessing step changes, rerun the held-out set and inspect the changed errors. Keep results tied to the version actually evaluated. The decision to deploy belongs to the workflow owner; a higher score on one metric does not automatically settle that decision.

Start with a scoped extraction evaluation

Use the OCR API guide for scans and images, the document extraction API for structured fields, and contact Cogneris to agree an evaluation on representative documents. Define the fields, evidence coverage and review requirements before interpreting any percentage as a usable result.