The extraction finished, the callback fired, and the receiver returned 200. Six hours later the finance team found the same invoice posted four times. Nothing had failed. The endpoint had answered every retry with a success it had not earned, parsed a payload it had never verified, and treated four deliveries of one event as four events.

A webhook is not a notification. It is an inbound API that you operate and we depend on. The moment you register a URL, you have taken on a contract with a signature scheme, a retry schedule, a replay window and an idempotency key — whether or not you implemented any of them.

This is the receiver side of the webhooks API: eleven operations across 8 paths, all Bearer-authenticated. The endpoint management is the easy half. The half that decides whether your data stays correct is what your handler does in the first 50 milliseconds.

Verify before you parse

Every delivery carries three headers, and the first one is the only reason to trust the other two:

Cogneris-Webhook-Signature:  t=1747654200,v1=9c8f…7e2a
Cogneris-Webhook-Id:         3f1a5c62-1f4e-4a1e-9d2b-6c0f7a8b1234
Cogneris-Webhook-Attempt-Id: 8b2d0e91-7c33-4f5a-bb10-2e9d4a6c7788

The signature is HMAC-SHA256 over the timestamp and the raw body. Four details decide whether a verification function is correct or merely looks correct:

  • Decode the secret first — the signingSecret is base64url. Decode it to raw bytes and key the HMAC on those bytes, not on the string.
  • Sign the timestamp and the body — the signed value is t, a literal dot, then the raw request body exactly as received.
  • Compare in constant time — the digest is lowercase hex in the v1 field, and a byte-by-byte comparison leaks its own answer.
  • Bound the age — reject when now - t exceeds your tolerance. 300 seconds is a reasonable default.

In Node.js, that is the whole function:

import crypto from "node:crypto";

function verify(rawBody, header, signingSecret, toleranceSec = 300) {
  const parts = Object.fromEntries(
    header.split(",").map(p => p.trim().split("="))
  );
  const t = Number(parts.t);
  if (Math.abs(Date.now() / 1000 - t) > toleranceSec) return false;

  const key = Buffer.from(signingSecret, "base64url");
  const expected = crypto.createHmac("sha256", key)
    .update(Buffer.concat([Buffer.from(`${t}.`), rawBody]))
    .digest();
  const supplied = Buffer.from(parts.v1, "hex");

  return supplied.length === expected.length
    && crypto.timingSafeEqual(supplied, expected);
}

And in Python, the same four rules, the same order:

import base64, hashlib, hmac, time

def verify(raw_body: bytes, header: str, signing_secret: str,
           tolerance_sec: int = 300) -> bool:
    parts = dict(p.strip().split("=", 1) for p in header.split(","))
    t = int(parts["t"])
    if abs(time.time() - t) > tolerance_sec:
        return False

    padding = "=" * (-len(signing_secret) % 4)
    key = base64.urlsafe_b64decode(signing_secret + padding)
    expected = hmac.new(key, f"{t}.".encode() + raw_body,
                        hashlib.sha256).digest()
    supplied = bytes.fromhex(parts["v1"])

    return hmac.compare_digest(supplied, expected)

Neither of these is a library call. Both are Node.js and Python samples you own, because a verification routine you cannot read is a verification routine you cannot audit.

The raw body is the part that breaks

The most common verification bug is not in the crypto. It is that rawBody was never raw.

Body-parsing middleware runs before your handler, reads the bytes, produces an object and discards the original. Re-serialising that object gives you JSON that is semantically identical and byte-wise different — a reordered key, a dropped space, a re-escaped character — and the HMAC covers bytes, not meaning. The signature fails, and the failure looks like a key problem.

So the receiver has to be built to keep the bytes. In Express that means a raw body parser mounted on the webhook route specifically; in Flask it means reading request.get_data() before anything touches the JSON. Verify against the bytes, then parse. The order is not stylistic.

The inverse failure is quieter: a handler that parses first and verifies second has already deserialised attacker-controlled input. It may never reach the signature check, because it crashed in the parser.

Retry is ours, replay is yours

These two words describe different systems, and conflating them is how outages turn into data loss.

Retry is automatic and bounded. When a delivery fails, we retry four times on a fixed schedule — 1 minute, 5 minutes, 30 minutes, then 2 hours after the preceding attempt. Retryable responses are 408, 425, 429 and any 5xx. Every other status is treated as final, which means a 400 from a receiver that mis-parsed the payload will never be retried. The schedule spans a little under 3 hours in total; an outage longer than that exhausts it.

Replay is manual and unbounded in time. Every attempt is recorded, and any delivery can be re-sent on request:

POST /api/v1/webhooks/deliveries/{id}/replay

No request body. Handle 409, which is what you get when the delivery's current state conflicts with the request — replay is not valid from every state.

The operational consequence is the part worth internalising. If your receiver was down for a morning, the answer is not to ask for a resend and not to wait for a retry that already stopped. It is to list the deliveries in that window and replay them:

GET /api/v1/webhooks/deliveries?EndpointId={id}&State=failed
    &CreatedFrom=2026-09-02T06:00:00Z&CreatedTo=2026-09-02T11:00:00Z

Filters are EndpointId, State, EventType, CreatedFrom and CreatedTo, with cursor pagination: pass PageSize, then send the returned nextCursor back as Cursor until it comes back null. Recovery is a query you can run, not a support ticket you file.

The delivery id is the idempotency key

A retry repeats Cogneris-Webhook-Id. A replay repeats it too. That is the design: the value identifies the delivery, not the attempt, and Cogneris-Webhook-Attempt-Id is what changes between tries.

So the receiver needs exactly one rule — record the delivery id inside the same transaction that records the effect. Not before, or a crash mid-handler makes the event permanently invisible. Not after, or two concurrent retries both pass the check.

INSERT INTO processed_webhooks (delivery_id) VALUES ($1)
ON CONFLICT (delivery_id) DO NOTHING;
-- 0 rows affected => already handled, return 200 and stop

Deliveries are at-least-once. Any receiver without that unique constraint is not idempotent, it is merely lucky about timing — and the four-times-posted invoice is what running out of luck looks like.

Return 2xx fast, work later

The response to a webhook should be an acknowledgement, not a result. Acknowledge first and process asynchronously: write the payload to a queue or a durable table, return, and do the work outside the request.

A handler that calls three internal services before responding has made our request timeout into your dependency graph. When one of those services is slow, the delivery is recorded as a failure, retried on the schedule above, and your handler runs a second time — often while the first run is still going. Slowness becomes duplication.

The delivery record makes this visible rather than theoretical. Each attempt carries attemptNumber, state, queuedAt, latencyMs, httpStatusCode, errorCode and safeErrorDetail. A receiver whose latencyMs is climbing toward the timeout is a receiver that is about to start duplicating work, and the number is there before the incident is.

Subscribe to versions, not to shapes

Event types carry an explicit version:

com.cogneris.document.processed.v1
com.cogneris.signature.completed.v1
com.cogneris.ticket.created.v1
com.cogneris.workflow.completed.v1
com.cogneris.webhook.test.v1

A breaking change to a payload ships as a new event type rather than a silent change to an existing one. A subscriber never receives a shape it did not opt into, which means an integration written against .v1 keeps receiving .v1 until someone changes the events array on the endpoint.

It also means the migration path is explicit and dual-running: subscribe to both versions, handle both, confirm the new one in your own logs, then drop the old. There is no flag day, because there is no in-place mutation to be caught by.

Rotate the secret without dropping deliveries

The signingSecret is returned once, on create and on rotate. There is no operation that reads it back, so a secret that was logged instead of stored is a secret that has to be rotated.

POST /api/v1/webhooks/endpoints/{id}/rotate-secret
POST /api/v1/webhooks/endpoints/{id}/test
POST /api/v1/webhooks/endpoints/{id}/disable

Rotation is a live operation on a live endpoint, so the receiver should accept either the old or the new secret during the changeover, then drop the old one once the test delivery verifies. The /test operation exists for exactly this: it queues a com.cogneris.webhook.test.v1 delivery that travels the whole path — signature, headers, retry semantics — without waiting for a real document.

A receiver that has never verified a test delivery has never verified anything. The first real event is a bad place to discover the secret was stored with its base64url padding intact.

What the errors are telling you

Webhook operations return Problem Details per RFC 9457, and the status codes separate cleanly into problems of authorisation, addressing and state:

  • 401 and 403 — the Bearer token is missing, expired, or lacks access to the tenant. The API-key scheme is not accepted on these routes at all.
  • 404 — no endpoint or delivery with that UUID in this tenant. Cross-tenant addressing does not return a permission error, it returns an absence.
  • 409 — the operation conflicts with current state, most often a replay of a delivery that is not replayable.
  • 429 — rate limited. Back off; this is quota, not a defect.

That distinction matters when you automate recovery. A 409 loop means your replay logic is not reading delivery state before acting. A 404 loop usually means an endpoint id from one environment is being used against another — see environments for why those ids do not travel.

The contract, in one paragraph

Verify the signature over the raw bytes before parsing. Bound the timestamp. Treat Cogneris-Webhook-Id as the idempotency key and commit it with the effect. Return 2xx quickly and do the work asynchronously. Let retries handle transient failures, and use replay — not a resend request — for anything longer than the schedule. Subscribe to versioned event types and migrate by dual-running.

None of it is exotic, and all of it is the difference between an integration that survives its first outage and one that discovers, six hours late and four invoices deep, that 200 was never the same thing as handled. The integrations documentation covers where these callbacks land downstream; the webhooks reference is the operation-by-operation contract itself.