Test the pipeline without borrowing customer documents.

An evaluation starts with a reasonable request: give us a sandbox so we can try the extraction without touching anything real. Somebody sets up an account, somebody else is asked for "some test documents", and within a day a folder of last quarter's supplier invoices is sitting in a shared drive, renamed sample_01.pdf through sample_40.pdf. Nobody decided that customer data would be used for testing. It simply became the fastest way to get a meaningful result.

That request hides two separate questions. The first is where the test calls run. The second is which documents are allowed to go into them. Most of the effort goes into the first question, and most of the risk sits in the second.

A sandbox answers the wrong question first

A sandbox, in the usual API sense, is a separate host or key that accepts requests and returns plausible responses without real consequences. For a payment API that is exactly what you want: the dangerous part is the money, and a sandbox removes it. For document AI the dangerous part is not the call. It is the document you send, and a separate host does nothing about that.

It also cannot answer the question most evaluations exist to answer. If the sandbox runs a different model, a cached response or a reduced pipeline, then it tests whether your code can parse a response shape. It says nothing about whether the extraction is good enough on your documents. A team can pass every sandbox test and learn on the first production day that stamps, skewed scans and handwritten corrections behave differently.

Cogneris does not have a separate sandbox host. The environments guide states it directly: trial traffic runs against the same regional host and the same xtkt_live_ key shape as any other tenant, on a signup credit. There is no environment migration after the trial. The benefit is that what you measure during the evaluation is what you get afterwards. The consequence is that every document you send during the trial is sent to production, in the region you chose at sign-up. The test-data decision cannot be deferred to "when we go live", because you are already live.

A test of a document pipeline tests three things

"Testing the pipeline" usually means three different activities that need different inputs. Mixing them is how real documents end up in places where a placeholder file would have worked just as well.

• Plumbing — authentication, request construction, async job handling, webhook verification, retries and error handling. Any valid file exercises this. The content is irrelevant.
• Extraction quality — whether the fields come back correct, complete and calibrated on the document population you will actually process. Only representative documents exercise this.
• Operating behavior — volume, limits, cost per page and latency under a realistic mix of file sizes and page counts. This needs realistic shapes, but not necessarily real content.

Once the three are separated, the data question becomes smaller. Plumbing tests never need a customer document. Operating tests rarely do. Only quality evaluation truly depends on documents that resemble production, and that is the one place where the decision deserves a named owner.

Keep plumbing tests off the metered path

Because trial and production share a host, a continuous integration job that uploads a PDF on every build is a production request on every build. It spends credit, it adds audit entries, and it creates a habit of treating the live API as a test fixture. The better pattern is the one most teams already use for any third-party dependency: record a small number of real responses once, store them as fixtures in the repository, and replay them in the build.

The recorded responses should come from documents that contain nothing sensitive. A generated invoice with invented supplier names and round numbers produces a response with exactly the same structure as a real one. That is all a parser test needs. When the response contract changes, you record again, deliberately, and the diff in the fixture shows what changed.

Some checks do need the live service. Connectivity and region can be verified against the health endpoint, GET /v1/health, which the environments guide documents as returning status, region and version without authentication. Webhook signature verification is best tested with a real delivery once and with stored deliveries afterwards; the callback contract covers what those tests should assert. Key handling belongs to the credential design, not to the test corpus, and is covered in the article on key scope and rotation.

What a synthetic document can prove

Synthetic documents are generated on purpose for testing: rendered invoices, forms filled with invented values, statements assembled from a template. They are the right default for a large part of the work, because they carry no personal or commercial data and can be committed, shared and regenerated freely.

They are good at testing the schema side of the pipeline. You control every value, so you can create the edge cases that are rare in real traffic: a total with three decimal places, a date in an unusual format, a missing optional field, a line-item table that runs to a second page, a currency symbol after the amount. Each one checks a normalization or validation rule you wrote, with an expected value you know exactly.

They are also good at load and cost rehearsal. Page count, file size and file type drive most operating behavior, and a generator can produce the realistic distribution of those shapes without anything real inside them.

This is a different use of the word from generative fraud. Synthetic documents made by an attacker to pass as genuine are a detection problem, and they are discussed in the article on AI-generated document fraud. Here the synthetic document is declared, labeled and kept inside the test suite.

What a synthetic document cannot prove

A generated document is clean in the ways real documents are not. It has the fonts the generator chose, text that sits exactly on the baseline and a layout that never drifts. Real traffic contains the things a generator does not think of: a fax header, a coffee stain across the total, a stamp over the signature block, a page scanned upside down, a supplier who changed their template in March and did not tell anyone.

That creates three blind spots that no amount of synthetic volume closes:

• Capture noise — skew, blur, compression artifacts, low contrast and handwriting affect reading far more than content does, and they are hard to simulate honestly.
• Layout population — the real distribution of templates, including the long tail of one-off layouts, is a property of your suppliers or customers, not of any generator.
• Absence patterns — which fields are legitimately missing, and how often, is learned from real documents. A generator fills what it was told to fill.

A quality score measured only on synthetic documents is therefore an upper bound. It tells you the pipeline can work. It does not tell you how often it will.

When the synthetic corpus stops being representative

The useful question is not whether to use synthetic data, but when it has stopped standing in for the real thing. That can be measured instead of argued, using a small approved real sample as the reference.

Run the same schema, the same validation rules and the same decision thresholds over both sets, then compare three numbers field by field:

• Accuracy gap — the difference in field-level accuracy between the synthetic set and the real sample. A large gap on a field means the synthetic version of that field is easier than reality.
• Abstention gap — the difference in how often values come back empty, uncertain or rejected by validation. Real documents usually abstain more, and that rate drives review volume.
• Layout coverage — the share of layout families in the real sample that have at least one synthetic counterpart. Families with no counterpart are untested by definition.

The tolerance is a team decision, and it should be written down before anyone looks at the results. For illustration only: a team might accept synthetic results for a field while the accuracy gap stays within two percentage points and the layout family is covered. Once a field falls outside that tolerance, its quality number must come from real documents, and the synthetic cases stay in the suite only as regression tests for the rules they were built to check.

Re-run the comparison whenever the real population changes: a new supplier group, a new country, a new scanner fleet at intake. Representativeness is not a property a corpus keeps. It decays as production moves.

Production data in a test is a compliance decision

The real sample has to come from somewhere, and that is where evaluations most often go wrong. Documents collected for accounts payable, onboarding or claims were collected for that purpose. Using them to evaluate a new processor is a new use, sent to a new party, and it deserves the same scrutiny as any other change in processing.

The questions are not exotic, but they need answers from people who can give them:

• Purpose — is evaluation of a processor compatible with the purpose for which the documents were collected, or does it need its own justification?
• Minimization — does the evaluation need forty full documents, or would ten with the fields that matter be enough?
• Agreement — is the vendor covered by a data processing agreement before the first upload, not after the pilot succeeds?
• Retention — how long will the evaluation documents and results be kept, and who deletes them when the evaluation ends?
• Approval — who signed off, and where is that decision recorded?

On the vendor side, two of those questions have concrete settings. Retention controls are configurable today, according to the environments guide, so the retention period for evaluation documents can be set before the first upload rather than inherited. The region is fixed when the tenant is created; the residency article explains what that decision does and does not settle. Neither setting answers whether the documents should have been sent at all. That answer belongs to the data owner.

Redaction is a third option with its own blind spots

Between synthetic and real sits the redacted document: a real file with personal or confidential values removed. It keeps the layout, the capture noise and the template population, which are exactly the things synthetic documents lose. For many evaluations it is the best compromise.

It has two traps. The first is that redaction removes the values you are trying to extract. A redacted tax identifier cannot be checked for accuracy, so the fields that matter most for compliance are the ones a redacted set cannot evaluate. The second is that redaction is often cosmetic. A black rectangle drawn over a PDF may leave the text layer underneath intact, and an extraction pipeline will read it perfectly. Redaction that is meant to protect data has to remove the content from the file, and someone should verify that on a sample before the set is used.

Make one corpus concrete

A workable evaluation corpus has tiers, each with its own storage and rules:

• Tier 0, fixtures — synthetic documents and recorded responses, committed to the repository and used on every build. No approval needed.
• Tier 1, redacted — real layouts with verified redaction, stored outside the repository, used for layout and noise coverage.
• Tier 2, approved real sample — the smallest set of real documents that makes the quality measurement meaningful, with an approval record, a retention date and a named owner.

For every document in tiers 1 and 2, keep a manifest entry: where it came from, which layout family it represents, who approved its use, the expected values for the fields under test, and the date it must be deleted. The manifest turns an informal folder of samples into something a reviewer can audit. It is also how you prove, at the end of the evaluation, that the tier 2 documents are gone.

Ownership of the expected values is a separate question. If the labels become a reusable evaluation set, the ground truth article covers who owns them and how they leave with you. This article is about which documents were allowed into the test in the first place.

Before the first real document goes in

Five checks turn the evaluation from an improvised upload into a decision:

• Separate the tests — plumbing and parser tests run on fixtures and never send a document on every build.
• Start synthetic — schema rules, edge values and load shapes are exercised with generated documents first.
• Measure the gap — accuracy, abstention and layout coverage are compared against a small approved real sample, with the tolerance written down in advance.
• Record the approval — purpose, minimization, agreement, retention and the approver are documented before tier 2 exists.
• Plan the deletion — retention is set on the tenant, the manifest carries deletion dates, and someone checks them when the evaluation ends.

There is no sandbox that makes these decisions unnecessary. A separate host would only make it easier to forget that they were made. Testing against the real service with documents you are allowed to send gives you a result you can trust and a record you can defend.