A connector is not a migration.

Pointing a document pipeline at a source demos well. Aim it at an SFTP folder or a document library, watch a few hundred files come back as structured JSON, and the work looks finished.

It is not finished, because a one-time import ends and a connected source does not. Files keep landing. Old files get corrected. Someone changes who may read a folder. The schema improves and last quarter's results are suddenly the wrong shape. Then finance asks what it costs to run the whole archive again.

Continuous document processing is four systems wearing one name: a live queue, a backfill, a reprocessing policy, and a permission model that travels with the document. How a PDF, a spreadsheet or a scan is accepted in the first place is a different contract, settled before any of this — we wrote that one up separately in the ingestion contract. This article is about what happens on day 2 and every day after.

The first sync and the steady state are different systems

A backfill is bounded. You can count the documents, estimate the cost, schedule it for a weekend and get an approval before it runs. A live queue is unbounded, latency-sensitive, and has to survive the source being unreachable for 40 minutes without losing its place.

Running both through one undifferentiated queue is the most common way an integration disappoints in its second week: a two-year archive lands in front of today's invoices, and documents that mattered this morning clear this evening.

Give them separate queues, separate rate limits and separate budgets. They only look alike because both end in an extraction call.

Backfill is a plan, not a button

Treat the backfill as a first-class operation with its own design, because the questions it raises have no default answers:

• Scope — how far back, which folders, which document classes, and what the cut-off is.
• Order — newest first is usually right, because recent documents are the ones people ask about.
• Checkpointing — item 40,000 of 200,000 fails, and the run must resume from there, not from zero.
• Throttle — the source system has its own rate limits, and saturating them is an outage you caused.
• Termination — an explicit definition of done, with a count that reconciles against the source.

The last one is worth insisting on. A backfill that ends because the loop ran out of pages is not a backfill that ended because everything was processed. Reconcile the count, publish the difference, and name the documents that never completed.

Reprocessing is routine, not an incident

Documents get processed again for ordinary reasons: the schema gained a field, a validation rule changed, a model improved, a customer disputes a figure, or a bug is found in a transform that ran for three weeks. None of those are emergencies, and none should require a bespoke script.

Make reprocessing addressable — by document, by class, by date range, by the schema version that produced the original result:

POST /reprocess
{
  "selector": {
    "documentClass": "invoice",
    "receivedBetween": ["2026-07-01", "2026-07-31"],
    "producedBySchema": "2026-06-02"
  },
  "targetSchema": "2026-08-14",
  "reason": "tax split into base and withheld",
  "writeMode": "new_revision"
}

That writeMode matters more than the rest of the payload. Reprocessing that overwrites in place destroys the audit trail: the approved value from July is replaced by a value nobody approved, and the record of which schema produced which number is gone. Append a revision, keep the pointer to the current one, and let consumers ask for either.

Reconciling a fresh result against an older one is the same problem as reconciling any two extraction outputs — it needs stable identities and attached evidence, which is why the shape of the result governs whether a reprocess is auditable at all.

Permissions travel with the document, or the pipeline leaks

The source system knows who may read a file. The pipeline usually reads it with a service account that may read everything. Unless that difference is handled deliberately, the extraction output becomes a copy of the document with the access control removed.

It fails in three recognisable shapes:

• Flattening — every result is readable by anyone who can query the destination, because the connector's own credentials became the effective ACL.
• Staleness — access was captured at ingest and never refreshed, so someone removed from a folder in March still resolves as a reader in September.
• Derived leakage — the document is protected and the extraction JSON, the page thumbnails and the search index built from it are not.

Carrying access alongside the document means recording it as data, versioning it, and naming what inherits it:

{
  "documentId": "doc_8412",
  "source": {
    "system": "document-library",
    "itemId": "01H9…",
    "sourceVersion": "etag:9f21c4"
  },
  "access": {
    "capturedAt": "2026-09-03T04:10:22Z",
    "aclVersion": "acl-3391",
    "readers": ["group:ap-clerks", "group:controllers"],
    "inherits": [
      "extraction:doc_8412:rev3",
      "thumbnail:doc_8412:p1",
      "index:doc_8412"
    ]
  }
}

Two properties make this real rather than decorative. First, aclVersion gives you something to compare, so a connector can detect that permissions moved without diffing member lists. Second, inherits enumerates the derived artifacts explicitly — a thumbnail that nobody listed is a thumbnail nobody will revoke.

Then check at read time, not only at write time. Permissions captured hours ago describe the past; the question a consumer asks is whether this person may read this result now. Use the stored ACL as a fast path and re-resolve against the source when its version has moved.

Deletion and drift at the source

Files do not only arrive. They get renamed, moved between folders, superseded by a corrected version, and deleted. A connector that only listens for creation events builds a destination that slowly stops describing the source.

Deletion is the case with consequences. When a document is removed at the source, the extraction output, the evidence crops and the index entries derived from it outlive it — which turns a routine deletion into a retention problem and, when the document held personal data, a privacy one. Propagate the deletion to everything in that inherits list, and record a tombstone rather than a silent gap, so a later audit can tell "deleted upstream on this date" from "never processed".

Renames and moves deserve the opposite treatment: they must not create a second document. Identity comes from the source system's stable item ID, never from the path or the filename, or every reorganisation of a shared folder reprocesses the whole tree at full price.

Cost belongs in the design, not the invoice

Per-page pricing makes the arithmetic of continuous processing unusually legible, which is an argument for doing it in advance. A 200,000-page backfill, a monthly full reprocess and a retry storm are three very different numbers, and only the first is usually anticipated.

The expensive default is reprocessing everything whenever anything changes. Most schema changes touch a few classes; a selector that names them turns a five-figure run into a three-figure one. Estimate before executing, publish the estimate next to the request, and require an approval above a threshold you choose deliberately.

What the connector has to remember

A connector without durable state is a connector that reprocesses the world every time it restarts — at full cost, producing duplicates. The state is small, and each field earns its place:

• Cursor — the source's own change token, so polling resumes rather than rescans.
• Per-item status — pending, processing, done, failed, skipped, with the reason attached.
• Source version — the etag or revision that was read, so a changed file is distinguishable from a re-seen one.
• Schema version — what produced the current result, which is what makes a reprocess selector possible later.
• Attempt count — with a ceiling, so a poison document stops rather than billing forever.

Those fields compose into the only idempotency key worth having: source system, item ID, source version and schema version together. It answers the question a connector asks thousands of times a day — have we already done this exact work? — and a key missing any one of the four answers it wrongly.

The minimum useful contract

None of this requires a distributed systems research project. It requires deciding seven things before the first sync rather than after the first surprise:

1. Live intake and backfill run on separate queues with separate budgets.
2. The backfill has a cut-off, a cursor, a throttle and a reconciled definition of done.
3. Reprocessing is addressable by class, date range and schema version.
4. Reprocessing appends a revision and never overwrites an approved result.
5. Access is captured as versioned data and every derived artifact is named as inheriting it.
6. Deletion at the source propagates to derived artifacts and leaves a tombstone.
7. Identity is the source item ID plus its version, never the path.

Delivery to the destination sits at the far end of this and has a contract of its own — signing, retries, replay and idempotent receipt, which we covered in the callback contract. The patterns for the systems in between are catalogued in the integration docs.

The demo ends when the first files come back. The system starts when the second batch arrives, when someone changes a permission, and when a schema change makes last quarter worth reading again. A connector that only handles the first of those is a migration that has not admitted it yet.