Self-hosted document AI moves the boundary.

Residency is one requirement, not the architecture

A request for self-hosted document AI usually arrives as one sentence: the documents cannot leave our environment. That sentence is important, but it is incomplete. Which environment — a tenant VPC, a private cloud, a corporate data centre or an isolated network? May model artifacts cross the boundary? May telemetry leave? Who patches the runtime at 02:00? What happens when a new model improves table extraction but changes the output?

Those questions are the architecture. The GPU location is only one line in it. A managed service can satisfy a regional-residency requirement while failing a control requirement. An on-prem OCR API can keep every byte inside the perimeter while failing availability, update discipline or evidence retention. Self-hosting removes a vendor-operated data path and adds an infrastructure product your team now owns.

This is also why self-hosting and air-gapping are not synonyms. A private service may still pull model artifacts, images and patches from external registries. A genuinely isolated deployment needs a controlled artifact route, signed bundles and an offline upgrade procedure. We cover that narrower case in the air-gapped deployment guide; here the subject is the broader operating-model decision.

Four deployment shapes, four ownership boundaries

These labels are not standards. Vendors use “private”, “dedicated” and “self-hosted” differently, so replace every label in a proposal with a diagram and a responsibility table. Mistral's current Document AI material, for example, explicitly advertises on-premises and private-cloud deployment. Docling demonstrates another end of the spectrum: local document conversion, an API server that can run on one machine, and a Redis-backed worker model for horizontal scale. The commercial and open-source routes can both be valid; they transfer different work to the buyer.

The responsibility matrix decides whether self-hosting is real

Ask each candidate architecture to fill every cell below. “Customer managed” is not an answer until a named team accepts it.

The last row is easy to miss. Keeping the service alive is not enough. A driver, OCR engine or model update can leave the endpoint healthy while changing table structure, coordinates or confidence distributions. Infrastructure monitoring cannot detect a semantic regression. A protected evaluation set and a benchmark that separates parsing from reasoning are part of operating the service.

Draw the threat model around every plane

A network boundary does not describe the whole system. Trace at least five planes, and treat every crossing as a claim that needs evidence.

• Document plane — original files, rendered pages, temporary images, extracted text and structured output. Record storage locations, encryption keys, deletion behaviour and retry queues.
• Model plane — weights, OCR language packs, container images and configuration. Decide who signs them, how they are scanned and how the runtime proves which artifact processed a document.
• Control plane — deployment APIs, admin consoles, schedulers and remote support. A local data plane with a vendor-accessible control plane is not fully self-contained.
• Telemetry plane — logs, traces, crash dumps and quality samples. Redacting the document while exporting its extracted fields can still disclose the document.
• Human plane — reviewers, support engineers and administrators. Private infrastructure does not replace tenant isolation, least privilege or a reviewed access path.

For each plane, write who can read, write and export; where the evidence is retained; and which action is possible during an incident. Then test the answers. A policy diagram without a packet trace, access log or restore exercise is intent, not proof. The same standard should apply to the output itself: an audit trail must identify the exact configuration and model behind one extraction.

Model updates become a release train

Managed services hide much of the release machinery. A self-hosted service exposes it. The buyer needs a way to receive artifacts, verify provenance, test compatibility, promote a candidate and return to the previous version. Skipping any step turns “version control” into a filename convention.

1. Import — receive immutable model, runtime and dependency artifacts through an approved route; verify signatures and create a software bill of materials.
2. Qualify — run security checks, compatibility tests and the locked document evaluation set. Measure output drift, not only whether the container starts.
3. Shadow — replay representative traffic without acting on the new result. Store disagreements by document class and field consequence.
4. Promote — pin the full artifact set, not just the model name, and record the change against the documents processed after it.
5. Rollback — restore the previous set in one bounded action and replay jobs whose outcome is no longer trusted.

Open-source tooling makes this concrete. Docling's offline guidance requires model artifacts to be staged locally, and its server can point to a pre-populated artifact path. That solves remote dependency at runtime; it does not solve artifact governance for you. Someone still owns acquisition, scanning, approval, distribution and retirement.

Observability must measure documents, not just machines

CPU, GPU memory and queue depth tell you whether the service is busy. They do not tell you whether it is right. Add document-level signals that expose the work without leaking its content:

• Input profile — pages, file type, language, scan quality and document class, using bounded labels rather than raw customer text.
• Stage timing — render, OCR, layout, extraction, validation and human review as separate spans.
• Quality movement — abstention rate, validation failures, coordinate coverage and measured accuracy on a sampled labelled stream.
• Version attribution — runtime image, model bundle, OCR engine, prompt and schema version on every result.
• Queue integrity — accepted, started, completed, failed, retried and expired jobs must reconcile. Throughput without a denominator hides loss.

Docling's server documentation exposes the operational shape clearly: a local engine for one host, or a Redis queue with separate workers for horizontal scale. It also notes that an API can accept jobs while no worker processes them. That is the kind of failure a health endpoint misses and a queue reconciliation catches.

Total cost is the operating system around the model

Do not compare an API price with a GPU invoice. Compare complete systems over the same workload and service level.

Express the result per processed page and per completed document, at expected, low and peak volume. The low-volume case exposes idle capacity. The peak case exposes burst handling. The completed-document view captures review and retry work that a per-page infrastructure number hides. If the result depends on 90% accelerator utilization from day one, it is not a forecast; it is a sales target.

A decision rule that survives procurement

Start with hard constraints, then compare economics. If policy requires physical isolation, a managed regional service is not a candidate. If the organization cannot staff a release train or a 24-hour incident path, an on-prem deployment is not a candidate either. Control requirements eliminate options in both directions.

Two current reference points make the range visible. Mistral Document AI advertises secure on-premises and private-cloud deployment alongside its document processing capabilities. Docling's deployment documentation shows how a local conversion server can grow from one host into queued workers, while its offline guidance explains how pre-staged model artifacts support isolated operation. These are implementation facts, not a recommendation for either product.

The correct choice is the smallest ownership boundary that satisfies the hard requirements. Sometimes that is managed SaaS with contractual controls. Sometimes it is a private endpoint. Sometimes it is a platform inside the buyer's account or data centre. The mature decision is not “cloud or on-prem”. It is a named boundary, an accepted responsibility matrix, a tested release train, a threat model that covers every plane and a cost model that includes the people keeping it true.