Hand-tuning stops scaling before the pipeline does
An intelligent document processing pipeline is a stack of decisions, not a model. Where to split a packet. Which capture path to use on a scanned page. What the prompt says. How the schema constrains the answer. Which threshold sends a field to a human. Each of those has several defensible settings, and the settings interact: a better splitter changes what the prompt sees, which changes which schema constraint fires, which changes the review rate.
Teams tune this by hand, one knob per incident. That works for a while. It stops working at the point where the number of document types exceeds the number of people willing to babysit them, and it produces a configuration nobody can explain — a set of values that were each rational when they were set and are collectively undocumented.
The obvious answer is to let an optimizer search the space. The obvious failure mode is to hand that optimizer production and call the result autonomous. What follows is the version that stays inspectable: what gets tuned, what the loop is optimizing for, what it is forbidden from touching, and how a bad configuration gets reverted.
Name the search space before automating it
An optimizer that can change anything is indistinguishable from an outage. Declare the parameters it owns, their legal ranges, and the parameters it may propose but not apply.
| Stage | Typical parameters | What a bad setting costs |
|---|---|---|
| Splitting | Boundary detection sensitivity, page-window size, classification confidence for a new sub-document | A merged packet sends two documents to one schema; an over-split packet loses evidence that crossed the boundary. |
| Capture | Capture path per page class, de-skew and de-noise, resolution floor, retry on low-confidence pages | Cheap capture on a hard page corrupts every downstream step and looks like a reasoning failure. |
| Prompting | Instruction wording, few-shot examples, context assembly, whether the raw page image is attached | Longer prompts raise cost and latency; shorter ones quietly drop edge-case handling. |
| Schema | Field types, enumerations, required-versus-optional, normalization and validation rules | An over-strict schema rejects valid documents; a loose one accepts values that break the system of record. |
| Routing | Which model handles which page class, escalation on disagreement, batch composition | Frontier models everywhere is affordable in a demo and not in production. |
| Thresholds | Field-level confidence floors, abstention policy, review-queue entry rules | This is the dial that trades a visible review cost against an invisible error rate. |
Write this down as configuration, not code. A parameter the optimizer can only change by editing a source file is a parameter that cannot be rolled back in one action, and rollback is the feature that makes the rest of this safe.
The objective function is where the honesty lives
Accuracy alone is not an objective. An optimizer told to maximize accuracy will send every page to the most expensive model, attach every image, retry on every low-confidence field, and route almost everything to human review. It will succeed, and the result will be unusable.
A workable objective has four terms and states its exchange rates explicitly:
- Weighted accuracy — per-field correctness against ground truth, weighted by consequence. A wrong bank account and a wrong document title are not the same error and must not carry the same weight.
- Cost per document — inference, capture, retries and storage, measured on the real document mix rather than a clean sample.
- Human review load — the fraction of documents and fields that reach a person, priced at what review actually costs.
- Latency — usually a constraint rather than a term: a hard ceiling the optimizer may not cross, not a quantity it trades away.
The weights are a business decision that engineering should refuse to invent. Ask what a single missed error costs in the worst realistic case, and what an hour of review costs. Those two numbers determine the shape of the whole search. If nobody will state them, the optimizer will imply them anyway — and nobody will know what it chose.
One term deserves separate treatment: abstention. A system that answers everything scores well on answer accuracy and is dangerous. Score abstention as its own outcome, so that a correct refusal counts as a success and a confident wrong answer costs more than a flagged uncertain one. Our argument for why that boundary matters is in the piece on anti-hallucination architecture.
The loop, stated plainly
Five stages, each with an artifact you can read afterwards.
- Measure — run the current configuration against a fixed evaluation set and record per-field results, cost, latency and review rate. This is the baseline every candidate is compared against.
- Attribute — group the failures by stage. A field that is wrong because the splitter cut the page is a splitting defect, not a prompt defect. Without attribution the optimizer tunes whatever is easiest to tune.
- Propose — generate candidate configurations aimed at the dominant failure group, within declared ranges, changing a bounded number of parameters at a time.
- Evaluate offline — score every candidate on the development set, then re-score survivors on a holdout the proposer has never seen.
- Gate and promote — apply the promotion rules below, or discard the candidate and record why.
Attribution is the step teams skip, and it is the step that makes the loop an engineering process rather than a random search. It requires the pipeline to keep intermediate evidence — the split boundaries, the captured text with coordinates, the prompt actually sent, the raw response before validation. If those are not retained, the optimizer cannot tell a capture failure from a reasoning failure, and neither can you. That is the difference between tuning a system and shaking a black box.
Ground truth is the constraint, not the loop
Every claim above depends on a labelled set that is trustworthy enough to be optimized against. This is where these programmes usually fail, and it fails quietly: the optimizer converges beautifully onto the annotator's mistakes.
- Split three ways — a development set the optimizer sees, a holdout used only for promotion decisions, and a locked regression set that never enters tuning at all.
- Stratify before sampling — by document form, structure, language, quality and consequence, and publish the count in each cell. A mean over an unbalanced set is not a measurement.
- Measure the labels themselves — double-annotate a sample and report the disagreement rate. If annotators disagree on 4% of a field, an optimizer chasing a 2% gain on that field is fitting noise.
- Refresh on a schedule — documents drift. A set assembled once and frozen forever will certify a pipeline against a world that no longer sends those documents.
Treat that corpus as an owned asset with a retention and portability story, not a by-product of a review queue. Building the evaluation harness that consumes it is a related problem, covered in the benchmark piece.
Guardrails: the list of things the loop may not do
Guardrails are not warnings in a runbook. They are conditions checked in code, before promotion, that fail closed.
| Guardrail | Rule |
|---|---|
| No regression on protected fields | High-consequence fields may not lose accuracy, whatever the aggregate score does. A gain averaged across 40 fields can hide a loss on the one that moves money. |
| No silent threshold drops | Changes that improve throughput by lowering a confidence floor require explicit approval, because they convert a visible review cost into an invisible error rate. |
| Bounded change size | A limited number of parameters per promotion. Large simultaneous changes cannot be attributed when they fail. |
| Holdout separation | The proposer never sees holdout results. Once it does, the holdout is a development set and the promotion signal is gone. |
| Statistical floor | Improvements below the noise level of the evaluation set are not improvements. Require an effect large enough to survive re-running the same evaluation. |
| Frozen surfaces | Schema fields consumed by a downstream system of record, retention rules and anything with a compliance owner sit outside the search space entirely. |
The last one is worth stating twice. An optimizer that improves an extraction score by dropping a field somebody's reconciliation job depends on has not optimized anything. It has moved the failure to a team that has no idea the search space existed.
Promotion and rollback
Offline wins are hypotheses. Production is the test, and the test needs an exit.
- Version the whole configuration — one immutable, addressable object per candidate, not a set of edits. Rollback means pointing at the previous version, and it has to be a single action.
- Shadow before serving — run the candidate alongside the incumbent on live traffic without acting on its output. Disagreements between the two are the cheapest signal available and cost nothing but compute.
- Promote on a slice — a bounded share of traffic, with the document mix pinned so the comparison is not confounded by a quiet Tuesday.
- Define reversion in advance — the metric, the threshold and the observation window, written before the promotion. A rollback criterion invented during an incident is a negotiation.
- Keep the evidence — which configuration processed which document, retained for as long as the extraction is auditable. A result nobody can attribute to a configuration cannot be defended later.
That last point is the one that turns a tuning loop into something a regulated buyer can accept: not that the pipeline changed, but that every change is dated, attributable and reversible. The same reasoning applies to non-deterministic output generally, which we cover in the audit-trail piece.
Where autonomy ends
"Autonomous" describes the search, not the authority. The loop can propose continuously and evaluate without supervision. It should not decide certain classes of question, because those questions are not technical.
| Decision | Owner | Why |
|---|---|---|
| Parameter search within declared ranges | The loop | Reversible, measurable, bounded. |
| Objective weights and cost of error | Business owner | An exchange rate between money and risk is not an engineering constant. |
| Confidence thresholds on high-consequence fields | Human approval | Trades a visible cost for an invisible one. |
| Schema changes with downstream consumers | Change control | The blast radius sits outside the pipeline. |
| Anything touching retention or residency | Compliance | Not a performance question at all. |
| Promotion after a failed rollback | Human | Two failures in a row mean the model of the system is wrong. |
A loop that respects this table will improve more slowly than one that does not, and it will still be running in a year.
How to tell it is working
Report the loop itself, not only the pipeline it tunes. Four numbers are usually enough: the share of proposals that survive offline evaluation, the share of promotions that survive their observation window, the time from a detected failure group to a promoted fix, and the movement in each objective term separately — never as a single blended score, which conceals exactly the trade the loop just made.
If promotions rarely survive, the offline set does not represent production. If almost everything survives, the gates are too loose or the improvements are noise. If the aggregate score rises while review load also rises, the loop has bought accuracy with someone else's time and has not been asked to account for it.
The goal is not a pipeline that tunes itself. It is a pipeline whose tuning is legible: a declared search space, a stated objective, guardrails that fail closed, an evaluation set that is honest about its own error, and a promotion record that can be read a year later by someone who was not there. Everything else is a black box with better manners.