Accuracy per field is not completion.

Every document AI evaluation eventually produces one number that goes on a slide. Field-level accuracy is usually that number, because it is the easiest to compute and the most flattering: average the per-field match rate across a test set and you land in the high nineties almost immediately. It is also the number least connected to whether the work got done.

A field is not a unit of work. An invoice with 23 extracted fields at 98% accuracy has, on average, half a wrong field — and one wrong payment amount stops the document. The operation does not care that 22 fields were right. It cares whether the document left the queue without a person touching it, and whether the record it left behind would hold up if someone asked, 6 months later, why that payment went out.

That is what a completion rate measures, and it is why the metric deserves a definition rather than a quotation. Completion rate — often called straight-through processing rate — is the share of in-scope documents that finished with a valid, evidenced result and no human intervention. Every clause in that sentence is load-bearing, and most published numbers quietly drop two or three of them.

Say what “complete” means before computing a rate

Completion is a predicate on a document, not a score. A document either satisfies it or it does not, and the predicate has to be written down before anything is divided, because everything interesting happens in the definition rather than the arithmetic.

A workable predicate has four conjuncts:

• Required fields present — every field the downstream system treats as mandatory has a value, per document type. Optional fields do not participate.
• Validations passed — each value satisfies its own rules and the cross-field rules: totals reconcile, dates order correctly, identifiers resolve against the system of record.
• Evidence attached — every extracted value carries a page, a region and the source text, and that citation resolves. A value with no traceable origin is a guess that happened to be right.
• No human touch — nobody opened, edited, approved or overrode the document inside the measurement window.

Notice what this excludes. A document where the model was confident and correct but produced no citation is not complete here, and it should not be, because the record is what makes the result defensible later. Notice also what it does not require: it does not require the extraction to be perfect against a ground truth the operation does not have.

Completion asks whether the pipeline finished the job with evidence. Correctness asks whether the answer was right, and it is measured separately, against sampled review. Conflating the two is the most common way a completion number becomes untrustworthy — a pipeline can complete 95% of documents and be wrong on a tenth of them. Publish both numbers or neither.

The denominator is the whole argument

The numerator is easy once the predicate exists. The denominator is where a completion rate is usually won by construction.

The distance between the first row and the last is commonly 10 to 20 points, and it is entirely a definitional choice. A completion rate quoted over “documents processed” has excluded the population that made the problem hard.

The defensible construction starts from every file received in the period, then states each exclusion with its own count:

• Duplicates — removed by a written identity rule, not by discretion.
• Out of scope — types the system was never asked to handle, itemized by type.
• Test and replay traffic — tagged at ingestion, never inferred afterwards.
• Still in flight — arrived inside the window but not yet terminal; out of both numerator and denominator, and reported as a count so the reader can see how large that tail is.

Everything else stays in. Ingestion failures stay in and count as incomplete, because a file the system could not open is a file the operation still has to deal with.

The formula, written so someone else can recompute it

complete(d) :=
      required_fields_present(d)
  AND validations_passed(d)
  AND evidence_resolves(d)
  AND human_touches(d) = 0

D  := files received in [t0, t1)
E  := duplicates + out_of_scope + test_traffic + in_flight_at_t1
D* := D \ E                        # the stated denominator

completion_rate = |{ d in D* : complete(d) }| / |D*|

Two properties matter more than the division. First, complete(d) is computable from the audit record alone: no human judgement is invoked at measurement time, which is what lets an auditor who has the logs and not the team reproduce the figure. Second, E is enumerated rather than implied, so the same period can be recomputed under a different exclusion set and the difference attributed to the change.

A figure that cannot be recomputed from a stored record is an estimate. Treat it as one.

Segment by document type, or the rate will lie

A single blended completion rate is a weighted average, and weighted averages move when the mix moves even when nothing about the system changed.

Move 10 points of volume from invoices to correspondence — a seasonal shift, a new counterparty, one large customer changing how it sends things — and the blend falls to 82% with no change to the pipeline at all. A team watching only the blend spends a quarter hunting a regression that does not exist. A team watching segments sees four flat lines and one changed weight.

Segment floors, not a blended target, are therefore what belongs in a goal or a contract. Report the blend for context; hold the segments accountable. Three cuts earn their keep beyond document type:

• Source channel — email attachment, portal upload and API submission carry different input quality, and a channel regression hides comfortably inside a stable blend.
• Counterparty or tenant — one large sender changing a template is the most common cause of a sudden drop, and it is invisible without this cut.
• Vintage — documents received this week and documents that entered the exception loop 3 weeks ago answer different questions; averaging them answers neither.

Completion and human review are one system

The “no human touch” clause creates an incentive that has to be managed in the open: any pipeline can raise its completion rate by lowering the bar for auto-approval. The rate goes up and the errors move downstream, where the metric no longer sees them.

Two counterweights keep the number honest. The first is a correction window. A document that completed automatically and was corrected 2 days later by someone downstream was not complete; it was released early. Measure completion twice — once at terminal state, once after a fixed window, with corrections subtracted. For finance and claims, 14 days is a reasonable default. The second figure is durable completion, and the gap between the two is the current price of the auto-approval threshold.

The second counterweight is a sampled accuracy audit on the completed population. Draw a random sample of documents that finished without review, have a person score them against the source, and publish that error rate beside the completion rate. A completion rate published without one is an unverified claim about work nobody checked.

This is also why the exception loop deserves attention on its own terms. We have argued that straight-through is the wrong number to lead with, because the residue is where the cost concentrates and where improvement stalls. That argument and this one are the same argument: lead with the exception loop, and define the completion rate strictly enough that it cannot be used to hide the residue.

Price the residue and the number becomes money

Completion is an operational metric until it is joined to cost, at which point it becomes a commercial one. The join is short. Let c_auto be the marginal cost of a completed document — compute, storage, per-page fees — and c_exc the fully loaded cost of an exception, including the reviewer, the waiting, and the second person who gets pulled in.

cost_per_document = r * c_auto + (1 - r) * c_exc

Because c_exc is typically 20 to 100 times c_auto, the second term dominates at any realistic rate. At 90% completion with an exception costing 40 times an automated document, exceptions are about 82% of total processing cost. Moving completion from 90% to 93% cuts cost per document by roughly a quarter — and that sentence, not the accuracy figure, is what justifies the engineering.

It also settles which segment to work on. In the table above, freeform correspondence is 6% of volume and roughly a third of the exception load. The blended rate cannot see that. The segmented rate joined to cost points at it directly.

What can actually go into an SLA

Not all of this is committable, and pretending otherwise is how these clauses end in disputes. Committable:

• The definition — predicate, exclusion set and segmentation, fixed for the term and changed only by amendment. The most valuable clause in the section, and the one most often missing.
• Segment floors — a minimum completion rate per document type, measured monthly on the stated denominator.
• The measurement artifact — a monthly record the customer can recompute: counts per segment, exclusions itemized, the sampled accuracy audit and its sample size.
• A remedy — service credits tied to a missed floor, with a stated cure period.

Not committable, and worth saying so plainly:

• A blended rate — the customer controls the mix, so the supplier cannot underwrite it.
• A floor on document types not yet seen — the schema and the failure modes are unknown until they arrive.
• Completion on badly produced inputs — scan quality belongs to whoever controls the scanner, and that clause belongs there too.

Our accuracy SLA page makes the same argument for correctness, for the same reason: a number is a commitment only when the other party can recompute it.

Four ways the number quietly rots

• Mix shift read as regression — the blend moves, every segment holds, and a team spends weeks chasing a phantom.
• Denominator drift — an exclusion is added for a defensible reason, nobody restates the baseline, and a 3-point improvement turns out to be definitional.
• Threshold tuning against the metric — auto-approval loosens, completion rises, the correction window absorbs the difference, and nobody reads the second number.
• Evidence decay — citations stop resolving after a storage migration or a re-paginated source, and the predicate keeps passing on a check that no longer runs. Test that evidence resolves, not that the field is populated.

Six lines instead of one

None of this makes completion the only number worth having. It makes it a number worth trusting. The version worth publishing states its predicate, enumerates its exclusions, breaks out by segment, reports again after a correction window, carries a sampled accuracy figure beside it, and converts to a cost per document.

That is six lines where most decks carry one. The six lines are the reason the one line means anything.