Someone has to maintain the model. Decide who.

The question is asked as accuracy and settled by maintenance

The standard evaluation takes a week. Someone assembles a folder of documents, runs it through two or three candidates, and produces a table of per-field accuracy. The vendor's ready-made invoice model scores one number, a custom-trained pilot scores another, the difference is discussed, and the decision is made. Nothing about that process is unreasonable. It is just measuring the wrong horizon: it describes one afternoon against one sample of documents that was, in most cases, assembled by the person who wanted the project approved.

What determines whether the choice was right shows up in month seven. A supplier moves their totals block. A new region joins with documents nobody sampled. A field becomes mandatory because of a rule change. The vendor improves the model behind the endpoint and the same page now returns a slightly different value, with no deployment on your side and no line in your change log. Every one of those is a maintenance event, and the prebuilt-or-custom decision is mostly a decision about who handles them, on what cadence, at whose cost, and with what evidence afterwards.

So the useful reframing is not which option is more accurate. It is which failure am I equipped to fix. A team with a labelling pipeline, a regression set and someone whose job includes the model can fix a fitted model that has drifted. A team without those cannot, and buying a custom model gives them an asset that depreciates quietly from the day it ships.

Four things get called "the model"

Half of the confusion in these conversations is vocabulary. At least four distinct artifacts are sold under the same word, and they have different cost structures, different owners and different failure modes.

Conflating the second and third is why "build us a custom model" quotes vary by an order of magnitude between vendors — one is quoting a labelling programme, the other is quoting a week of configuration. Conflating the first and fourth is why buyers hear "no training required" and conclude "no configuration required", then discover that the schema, the thresholds and the validation rules are the work and always were. The template-versus-zero-shot dial is a related but separate question: that one is about how a page is read, this one is about who owns the thing doing the reading.

The eight dimensions, and which way each one points

A decision matrix is only useful if each row can actually be answered with something you know or can measure this week. These eight can.

Nobody scores eight out of eight on one side. The matrix earns its keep precisely because it produces a split answer — prebuilt for most of the schema, custom for two fields, zero-shot for the long tail of senders nobody will ever template. A split answer is not indecision. It is the shape the problem actually has.

Variability decides; volume only pays

Volume is the number everyone leads with, and it is the wrong first question. Volume determines whether a fitted model can be afforded — amortising a labelling and retraining programme across fifty thousand documents a month is a different arithmetic from five hundred. It says nothing about whether the fitted model will hold.

Take two intakes of the same size. The first is twelve suppliers who have sent the same layout for four years; a custom model trained once will still be right next spring, and the retraining cadence is close to annual. The second is four thousand senders where a meaningful share of each quarter's volume comes from layouts that did not exist the quarter before. Same volume, and in the second case the fitted model is systematically out of date on exactly the part of the intake that costs the most to handle, because the pages it has never seen are also the pages most likely to end up in review.

The single most useful number here is the share of last month's volume that came from layouts unseen ninety days ago. It is a query against data you already have, it takes an afternoon, and almost nobody runs it before deciding. High and rising says the generalising path is structural rather than a stopgap. Low and flat says a fitted model has something durable to be fitted to.

The answer is per field, not per document type

The matrix above reads as though the unit of decision is a document class. In practice it is a field, and treating the class as the unit is what produces the most common disappointment in this whole category: a prebuilt model that performs beautifully on the fields every document of that class has, and indifferently on the three that your business actually runs on.

That is not a defect. A model trained on a public distribution of invoices knows what an invoice number, a total and a tax line look like across millions of examples. It has no reason to know that your operation depends on a cost-centre code that your suppliers type into a free-text note in whatever format they feel like. Averaged accuracy across the schema hides this completely — twenty-two fields at 97% and two fields at 40% reports as a strong result and behaves like a broken one.

Two consequences follow. First, rank the fields by consequence before anything else: which values, if wrong, cause a payment, a policy decision, a filing or a customer commitment to be wrong. Those get their own accuracy bar and possibly their own extraction path, while the cosmetic fields do not deserve a training programme. Second, expect the production shape to be mixed — a standard model for the standard schema, a narrow fitted or rule-based extractor for the handful of fields that are yours, and a defined exception path for whatever neither clears.

Training time is not the cost. Labelling is.

Vendors quote training in hours, because hours are short and impressive. It is the least interesting number in the project. The cost line is the labelled set, and it has five parts that all have to be paid: sourcing documents that represent the intake rather than the archive, writing an annotation guideline precise enough that two people labelling the same page agree, getting the labelling done, holding out an evaluation set that never gets trained on, and then keeping the whole thing alive as corrections come back from human review.

That last part is the one that gets skipped, and it is what separates a dataset from a snapshot. Reviewers correct extracted values every day. Those corrections are the highest-quality labels your operation will ever produce — they are real documents, real fields, and someone was paid to be right about them. If they flow only into a database column and never back into the training set, you are running a labelling operation whose output you throw away, and paying separately for labelling later.

The labelled set is also where lock-in accumulates most quietly. Annotations produced by your operation, on your documents, sitting in a vendor's console in a format nobody has ever exported. Three questions belong in the contract before the first label exists: who owns the annotations, in what documented format they can be exported — including geometry and version history, not just field values — and whether review corrections land in an exportable store or only in someone's weights. Weights rarely travel between platforms. Labels can, if you asked in time.

The maintenance obligation exists on both sides of the line

The custom side of this is well understood, if often underfunded. A fitted model drifts as the intake moves. It needs a retraining cadence, a regression set that catches a regression before a customer does, and a person whose job description contains it. A model with no assigned owner degrades on a schedule that nobody is watching, and the first signal is usually an exception queue getting longer for reasons nobody can name.

The prebuilt side is the one that surprises people. A managed model is updated by the vendor, and an update means the same document can return a different value with nothing changing on your side. Sometimes that is an improvement. Either way it is a change to your output, and if your reconciliation, your thresholds or your downstream automation were tuned against the previous behaviour, it is a change to your results too. What that requires is unglamorous and non-negotiable: version pinning where the vendor offers it, change notice where they do not, a regression set of your own documents that runs against every version, and the model version recorded against every processed document so a dispute six months later can be traced to a specific behaviour rather than to a general impression that things got worse in the spring. That record belongs with the rest of the per-field audit trail, and the pinning decision belongs in the control plane under change control, because it changes output for identical input.

Which produces the sentence worth taking away from this whole piece: you need the regression set either way. Prebuilt does not remove the obligation to know when your extraction quality moved; it only removes your ability to do anything about it directly. The prebuilt-or-custom decision settles one thing — whether you also need the training loop.

Governance asks different questions of each

The two options sit differently under every framework currently landing on document processing, and the difference is not one of severity but of who holds the evidence.

Train your own model and you are much closer to the party that built the system. That means the record is yours to produce: where the training documents came from and on what lawful basis, what the annotation guideline said, how the evaluation set was constructed and held out, what the measured performance was per field and per document class, which version is in production and what changed in the last one. This is more work, and it is work you can actually do, because you are holding all the inputs. It is the same technical file discipline that has to exist for any system carrying consequential decisions.

Use a supplied model and the obligation to evidence performance in your own context stays exactly where it was, while the evidence about the model itself belongs to someone else. The diligence therefore moves into procurement: what the vendor documents about training data and evaluation, whether they commit to notifying you before a model change, how long your documents are retained and where, whether your documents are used to improve their models by default, and which sub-processors touch the page on the way through. Several of those are frequently answered by a default setting rather than by the contract, which is a poor place for the answer to live. A documented procurement baseline is what stops that from being rediscovered deal by deal.

Portability is decided before the first label

The exit test is a single question, and it is worth asking out loud early, when the answer is still cheap to change: if you had to move to another platform within ninety days, what would you carry, and could you reproduce today's accuracy with it?

Note the asymmetry that falls out of the table. The custom path is the one that generates a genuinely valuable, genuinely portable asset — the labelled set — and it is also the path where that asset is most likely to be created inside somebody else's product without an export clause. Whether custom-trained means ownership or lock-in is settled by a contract term, not by the technology.

The hybrid that ships

Teams that get this right rarely pick a side. They sequence it, and the sequence is close to the same in every intake we have seen.

• Start on the path with no training cost — a prebuilt model or schema-driven zero-shot extraction gives you a measured failure set in two weeks rather than a project plan in two quarters, and the failure set is a far better specification than anything written in advance.
• Promote failures, not document types — only the fields and layouts that fail repeatedly, expensively or dangerously earn a fitted model; everything else stays where it is, which keeps the labelling programme small enough to actually maintain.
• Keep the checks common to both paths — arithmetic validation, cross-field consistency, master-data lookups and acceptance thresholds belong to the pipeline rather than to whichever engine ran, because shared checks are the only thing that makes two paths comparable.
• Let sender history do the routing — what happened the last two hundred times this layout arrived is the strongest cheap signal you own, and it is the same signal that routes effort per document.
• Let stable layouts graduate — a format that has arrived unchanged for months can move to a cheaper deterministic path, with the general model kept as the fallback for the day it changes.
• Keep one exception path — whichever engine failed, the queue, the explanation and the resolution loop should look identical to the person working it.

Four ways this goes wrong

1. The evaluation set was chosen by the person who wants approval

Sample folders are curated, usually unconsciously. The pages that were rejected at capture, the photograph taken in a warehouse, the fax that somebody scanned twice — those are exactly the documents that decide the economics, and they are exactly the ones that do not make it into a demo folder. The fix is procedural rather than technical: draw the evaluation set as a random sample of a real month of intake, including everything that failed, label it once, and never let a vendor see it before the run. Anything else measures the sampling, not the model. It is also the cheapest defence against a pilot that does not survive production.

2. Training on the archive instead of the intake

The most available labelled documents are historical, and history was captured by different scanners, from a different set of counterparties, under a different set of formats. A model fitted to that distribution scores well on a held-out slice of the same archive and underperforms in production from the first day, for a reason that is genuinely hard to see from inside the metrics. If the archive is what you have, at minimum measure how far it has drifted from last month's intake before treating it as training data — and weight the recent material heavily.

3. Low-code extractors that nobody owns

The console is the seductive option: an analyst builds an extractor in an afternoon, it works, and it goes to production without a regression set, without a review, and without a record of why the threshold is set where it is. Eighteen months later the analyst has moved teams, forty extractors exist, six of them contradict each other, and nobody will touch any of them. Configuration is code — it changes the output for the same input — and it needs the same version history, the same owner and the same change control that code gets, whether or not it is stored in a repository.

4. Optimising the model instead of the finished result

Two points of field accuracy that do not shorten the exception queue are spend, not improvement. The metric that decides whether a custom model was worth building is the cost of a finished, accepted document — extraction, checks, review time and rework included — measured against what the previous path cost. Field accuracy is an input to that number and is frequently mistaken for it, which is how projects end with a better model and an unchanged operation. That is the same arithmetic as cost per successful task, applied to a build-or-buy decision.

The trade-off, stated honestly

Three things that cut against the enthusiasm on both sides.

Prebuilt is right more often than the custom-model business admits. For common commercial documents with an enormous public distribution, a few thousand labels of yours will rarely beat a model trained on orders of magnitude more examples of the same artifact. When that is the situation, the money spent on a labelling programme buys more accuracy if it is spent on capture quality, on validation rules and on the exception loop instead. Fixing a blurred photograph at the moment of capture beats every model in the comparison.

Custom is genuinely necessary more often than the general-model narrative admits. Where the document is a domain artifact — a broker's policy schedule, a lab result in a house format, a customs instruction with conventions that exist inside one industry — no general distribution contains it, and no amount of prompt work substitutes for having seen the thing a few thousand times. In those cases the labelled set is the differentiator and treating it as one is correct.

And a share of custom-model projects are solving a problem that something cheaper solves. A missing field is sometimes a schema problem. A recurring error is sometimes a validation rule that was never written. A low accuracy score is sometimes an intake with thirty percent unusable images. Model work is more interesting than intake work, which is a bias rather than an argument, and it is worth naming before a quarter goes into the wrong one.

We sell a platform built around schema-driven zero-shot extraction, so the section above should be read with that interest in view. The honest version of our own position is that we think the fitted model is the right answer less often than it is sold, and that when it is right, the thing that made it right was the labelled set rather than the training run.

Where to start

A short sequence, ordered by information gained per hour spent.

• Measure the new-layout share — what percentage of last month's volume arrived in layouts you had not seen ninety days earlier; this one number moves the decision more than any accuracy comparison and takes an afternoon.
• Rank fields by consequence, not by frequency — the values that can cause a payment, a policy decision or a filing to be wrong get their own bar; the rest do not deserve a training programme.
• Build the evaluation set before you talk to anyone — a random sample of real intake including the rejects, labelled once by you, held back from every vendor.
• Price the whole obligation, not the training run — labelling, re-labelling, the owner's time, the retraining cadence and the regression runs, against the per-page cost of the managed path over the same period.
• Run the no-training path for two weeks first — the failure set it produces is a better specification for a custom model than anything you could write in advance, and sometimes it removes the need for one.
• Settle the three ownership questions before the first label — who owns the annotations, in what format they export, and whether review corrections come back out.
• Pin the version and record it per document — whichever path you take, the model or configuration identifier belongs in the record next to what was extracted and what was checked.
• Set a review date — input mix, model capability and prices all move; this is a standing decision with a next occurrence, not a completed project.

Closing thought

The prebuilt-versus-custom question feels like a technology choice and behaves like an operating decision. Both options work. Both degrade. The one that survives contact with a real intake is the one whose maintenance has a name attached to it — someone who owns the labels, or someone who owns the vendor relationship and the regression set that keeps it honest. Teams that skip that step do not get a worse model on day one; they get an unowned one, and find out in the quarter when the documents change.

At Cogneris we start from the schema rather than from a fitted artifact: tenant-configurable classification and zero-shot extraction, so a new document type is a schema and a set of rules rather than a labelling project, with per-field confidence and provenance on every value, validation that names what failed and where on the page, and tamper-evident audit metadata covering what was extracted, what was checked and what was released. Results and their audit records are yours, retrievable through the API, because a document platform you cannot leave is a document platform you cannot evaluate. If you want to know which fields in your intake genuinely justify a fitted model and which ones never will, see the pricing page or talk to our team and bring one real month of documents — including the ones your last evaluation left out.