Classify before you extract.

A document arrives before the pipeline knows what it is. That sounds obvious, but many extraction systems skip the decision and send every file to one large prompt, one general schema or a sequence of extractors. The system pays for work it did not need, waits for models that could never answer the right question, and then discovers too late that an invoice was treated as a bank statement.

Document classification is the routing layer that decides which extraction contract runs. It is not a cosmetic label attached after OCR. The class determines the schema, validation rules, model, page budget, retention policy and exception queue. A wrong class can produce plausible fields under the wrong contract, which is more dangerous than a clean failure.

The useful design question is therefore not whether a classifier can name a document. It is whether the classification decision makes the whole pipeline cheaper, faster and safer than extraction without it.

Classification chooses a contract

Consider a packet containing an invoice, a bank statement, an identity document and a free-form letter. Their bytes may all be PDFs, but the downstream contracts have almost nothing in common.

A universal schema either becomes an enormous bag of optional fields or forces every extractor to run. The first design makes missingness ambiguous: a blank value can mean absent, not applicable or missed. The second design makes cost scale with the number of possible classes rather than the number of documents.

Classification restores a narrow interface. Each class points to one versioned schema, and each schema states which fields are required, which validations run and where exceptions go. That is why our classification documentation belongs before extraction in the flow, not after it as reporting metadata.

Measure the routing decision, not only the label

Top-1 accuracy is necessary but incomplete. A classifier can score well on a balanced benchmark while routing the production long tail badly. It can also add enough latency and cost that the simpler pipeline remains better.

Run the comparison on the same frozen arrival set. Preserve the original class distribution, including unsupported and ambiguous documents, and execute two paths:

• Baseline — send every document through the general extraction path used today.
• Classified route — classify once, apply the confidence policy, then invoke only the selected extractor or review queue.

For both paths, record end-to-end latency, model calls, pages processed, compute or provider cost, schema-valid completion and human touches. Classification wins only if the system-level outcome improves.

routing_value =
    avoided_extraction_cost
  + avoided_exception_cost
  - classification_cost
  - added_latency_cost

valid_route(d) :=
    predicted_class(d) selects the intended schema
  AND schema_validation_passes(d)
  AND evidence_resolves(d)

Report the distribution, not only the mean. Median latency hides the low-confidence path, and average cost hides a small class that triggers an expensive model. At minimum publish p50 and p95 latency, cost per document by class, confusion pairs, review rate and schema-valid completion.

Confidence needs a policy, not a threshold copied from a notebook

A score has no operational meaning until it maps to an action. The same 0.82 confidence can be acceptable for routing a newsletter and unacceptable for routing an identity document into an automated KYC decision.

A production policy has three bands:

• Auto-route — the predicted class clears the class-specific threshold and no competing class is close enough to create ambiguity.
• Disambiguate — ask for a second signal: page-level voting, sender context, filename only as weak evidence, or a smaller targeted model that compares the top two classes.
• Review or unknown — preserve the file, predicted candidates and evidence, then route it to a named queue. Never force the nearest known class.

Thresholds should be class-specific because error cost is class-specific. Confusing an invoice with a receipt may be recoverable. Confusing a passport with a generic image can bypass mandatory checks. Choose each threshold against the cost of that confusion, then remeasure when the input mix changes.

The margin between the first and second class often carries more information than the top score alone. A 0.79 invoice score against 0.12 receipt is a different decision from 0.79 against 0.77. Store both candidates and the policy version so the route can be reconstructed later.

Unknown is a first-class document type

Closed-set evaluation assumes every document belongs to one of the classes the model knows. Production intake does not. Customers upload screenshots, blank pages, bundled packets, new templates and files from processes nobody mentioned during implementation.

If the classifier must choose a known class, the pipeline turns novelty into false certainty. The extractor then emits fields that fit the schema syntactically but have no semantic relationship to the document. Validation catches some of this; plausible dates, amounts and names let the rest through.

An explicit unknown outcome changes the failure mode from silent corruption to visible work. Track its volume, cluster it, sample it and promote stable clusters into new versioned classes only when there is an extraction contract ready for them. A class name without an owned schema and validation path is inventory, not capability.

Packets need classification at more than one level

A 60-page upload can be one document, a packet of many documents or one document with appendices. File-level classification alone cannot tell which. Page-level classification alone can split a table or contract at every visual change.

Use a hierarchy:

1. Packet detection — decide whether the file contains multiple logical documents.
2. Boundary detection — find candidate starts and continuations using layout, page numbers and semantic continuity.
3. Document classification — classify the reconstructed units, not isolated pages.
4. Cross-document reconciliation — keep the packet identity so extracted records can still be compared.

This is especially important in KYC document extraction, where an identity document, proof of address and supporting ownership evidence may arrive together but must follow different schemas and checks. Splitting them loses the case context; flattening them loses the document contracts.

Latency is a budget, not a reason to skip classification

Classification adds a hop, but a hop is not automatically slower end to end. A 120 ms classifier that avoids two multi-second extraction calls is a latency optimization. A large multimodal classifier in front of a single cheap extractor may be the opposite.

Budget the path explicitly:

Cache only against a content identity that survives renames and retries. Make the decision idempotent, include the classifier and policy versions, and invalidate the route when either changes. Otherwise a retry can silently select a different schema from the same bytes.

The audit record is part of the API

A routing response should explain enough for another system to act and for a reviewer to reconstruct the decision. A minimal record includes:

{
  "document_id": "...",
  "class": "invoice",
  "confidence": 0.94,
  "alternatives": [{"class": "receipt", "confidence": 0.04}],
  "action": "auto_route",
  "schema": "invoice@3",
  "classifier": "intake-classifier@7",
  "policy": "routing-policy@12",
  "evidence": [{"page": 1, "region": [0.08, 0.12, 0.41, 0.24]}]
}

Do not log raw document text merely to explain the route. Store bounded evidence references under the same access and retention controls as the source. Classification metadata can itself reveal sensitive categories, so tenant isolation and deletion rules still apply.

A rollout that can prove it helped

Start in shadow mode. Classify every arrival, record the proposed route, but keep the current extractor in control. Compare routes against the downstream result and human corrections. This produces a confusion matrix on the real mix without changing customer outcomes.

Then enable auto-routing one class at a time. Pick a class with a narrow schema, sufficient volume and a reversible downstream action. Hold out a control slice so latency, cost and completion remain comparable. Expand only when the class-specific threshold holds across at least one full business cycle.

The release gate should answer five questions:

• Route quality — are false routes below the risk budget for each class?
• Unknown handling — do novel and ambiguous inputs fail visibly?
• Economics — did avoided calls exceed classification and review cost?
• Latency — did p95 end-to-end time improve or stay inside the SLA?
• Traceability — can every route be reconstructed from versions and evidence?

Classify first, then make the promise narrower

Document classification does not make extraction universally intelligent. It makes each extraction promise narrower and testable. The invoice path can promise invoice fields and invoice validations. The identity path can promise identity checks. The unknown path can promise that uncertainty will not be disguised as a known class.

That is the operational value of the layer: one early decision removes unnecessary work, selects the right contract and gives uncertainty somewhere honest to go. Measure it on the whole pipeline, and classification stops being a benchmark label. It becomes infrastructure.