Zid webhook health: recover broken endpoints without losing orders

Zid Webhook Health Tracking is a documented circuit-breaker-style control for webhook subscriptions. Zid marks an endpoint Healthy, Degraded or Broken from its recent delivery failures; when a hook becomes Broken, Zid suspends new dispatches and discards pending undelivered events. For teams connecting Zid to an ERP, warehouse, payment workflow or customer-support system, recovery therefore requires more than changing a URL: replace the receiver safely, restore delivery, and reconcile the affected business records from an authoritative API.

This guide uses Zid's official documentation as verified on 22 October 2026. State thresholds, retry timing, scopes and recovery behavior are documented platform facts. The durable-inbox design, alert thresholds, reconciliation workflow and rollout controls below are engineering recommendations that must be tested against your application and current Zid contract.

What Zid's health states actually mean

The official Webhook Health Tracking guide describes three states evaluated over a sliding one-hour window:

A successful delivery resets the failure count and restores Healthy. Zid also states that HTTP 429 responses do not count toward the health failure total. That exception prevents an intentionally throttling receiver from being marked unhealthy merely because it asked the platform to slow down, but it is not permission to run indefinitely at saturation. A receiver returning 429 can still delay business state, create operational backlog and consume retry capacity.

Zid documents three retry intervals after a failed delivery: one minute, five minutes and fifteen minutes. A failure is counted after the retry sequence is exhausted. This means a Degraded transition is not a single transient error; it is evidence that multiple deliveries have already failed through the platform's retry policy. Alerting only after Broken waits too long.

The Health Summary endpoint returns counts for Healthy, Degraded and Broken webhooks and requires the third_webhook_read scope. The Broken Webhooks endpoint lists broken subscriptions and uses the same read scope. The documented recovery operation requires third_webhook_write.

Why Broken creates a data gap, not just downtime

When a subscription becomes Broken, Zid's documentation says new events are no longer dispatched, pending undelivered events are discarded, and the hook remains Broken until explicit recovery. The old record keeps a broken_at timestamp for audit. Recovering it does not turn the same receiver back on: the workflow soft-deletes the old subscription and creates a fresh Healthy hook with the same event subscription but a new target URL.

The new URL requirement prevents an operator from pointing recovery at the same unverified failure path and calling the incident resolved. It also encourages a versioned receiver such as /webhooks/zid/v2 or a blue-green hostname. But the platform cannot reconstruct business events that were discarded while the original subscription was Broken. Your integration must find the affected time window and compare its local state with Zid's authoritative resources.

For an order integration, define the reconciliation interval from the earliest reliable warning, not merely from the moment an engineer noticed the incident. A conservative starting boundary is the minimum of the local last-success timestamp and Zid's broken_at, with an overlap large enough to cover clock skew, queue delay and delayed observation. The overlap is safe only when writes are idempotent.

This is why webhook recovery and order reconciliation are separate operations:

1. receiver recovery restores future delivery;
2. reconciliation repairs past gaps;
3. deduplication prevents the overlap from producing duplicate effects;
4. an audit record proves which subscriptions and business entities were repaired.

A related guide on webhook synchronization that survives missed events explains the general durable-inbox pattern. This article focuses on Zid's explicit health and recovery contract rather than repeating basic webhook setup.

Design the receiver for fast, durable acknowledgement

The request path should do only the work required to authenticate the sender, preserve the original event and return a correct HTTP response. Do not perform ERP writes, shipment creation, customer messaging or inventory recomputation before acknowledgement.

A practical ingestion sequence is:

1. receive the request and retain the exact bytes needed by your authentication contract;
2. validate the request using the current Zid webhook authentication requirements;
3. derive a stable delivery identity from a documented event identifier when available;
4. insert the raw event, selected headers, subscription identity and received time into a durable inbox in one transaction;
5. return a success response only after that transaction commits;
6. let an asynchronous worker parse, validate and apply the business transition.

If the platform payload lacks a unique delivery identifier for a particular event type, build a bounded deduplication key from stable business fields such as store, event type, entity identifier and source version or timestamp. Do not hash the entire JSON blindly: harmless field ordering or a newly added optional property can turn the same business event into a different hash.

A minimal inbox can contain:

CREATE TABLE zid_webhook_inbox (
  delivery_key text PRIMARY KEY,
  store_id text NOT NULL,
  event_type text NOT NULL,
  entity_id text,
  received_at timestamptz NOT NULL DEFAULT now(),
  payload jsonb NOT NULL,
  status text NOT NULL DEFAULT 'pending',
  attempts integer NOT NULL DEFAULT 0,
  processed_at timestamptz,
  last_error text
);

The primary key is an idempotency gate, not a promise that the handler is correct. The worker must also make downstream writes idempotent. A “send refund” command needs its own business idempotency key and state transition; otherwise replaying an inbox row can double a financial side effect.

Keep the request handler isolated from slow dependencies. If the inbox database is unavailable, returning a success code loses the event. Return success only when durable storage succeeds. If downstream work fails, retry it from the inbox under a bounded policy and move poison messages to a visible quarantine state rather than blocking every later event.

Monitor health without creating another failure mode

Use the documented health endpoints as a control-plane signal. A scheduled monitor can read the summary at a modest interval, record state transitions and alert only on meaningful changes. Zid documents a general API limit of 60 requests per minute per application per store in its rate-limiting guide, so a five-minute health check is an engineering example—not a platform requirement—and should be adjusted for fleet size and the rest of your API budget.

Recommended signals include:

• Healthy, Degraded and Broken counts by application and store;
• the first time each subscription entered its current state;
• consecutive observations of Degraded;
• durable-inbox acceptance rate and acknowledgement latency;
• worker queue age, processing failures and quarantine volume;
• the newest source timestamp successfully applied per entity type;
• the time since the last successful authoritative reconciliation.

Do not page on a single failed delivery if the platform is already retrying it. Do page when Degraded persists across observations, when inbox persistence fails, or when business-state lag breaches the product's service objective. Broken should trigger an incident immediately because the documentation explicitly says dispatch is suspended and pending deliveries are discarded.

Zid also documents a batched email containing a CSV of unreviewed Broken webhooks. Treat that message as a secondary operational signal, not the primary detector. Email timing and human attention are not a safe recovery control for order or payment flows.

Recover with a versioned replacement endpoint

Prepare a replacement receiver before invoking recovery. The new URL should reach the same durable ingestion boundary through a versioned route or a separately deployable target. Verify TLS, DNS resolution, authentication configuration, request-size limits, timeouts, database connectivity, queue permissions and observability from the production network.

A controlled recovery workflow is:

1. fetch the health summary and broken list using a credential with third_webhook_read;
2. open an incident record containing hook identity, event type, old URL, broken_at and last local success;
3. deploy a new target URL without deleting the old route;
4. run a synthetic probe that tests the receiver's own health without pretending to be a Zid delivery;
5. verify that production secrets and store mappings are present;
6. submit the recovery batch with third_webhook_write;
7. inspect the result of every item because Zid documents partial batch failures;
8. confirm the new subscription is Healthy and accepting durable writes;
9. reconcile the affected data window;
10. retire the old path only after evidence is complete.

Do not log access tokens, Basic Authentication credentials or full sensitive payloads in the incident record. Store secrets in a managed secret service and log identifiers or hashes that support correlation without exposing customer data.

The orchestrator should model recovery as a state machine rather than a one-click action:

DETECTED
  -> REPLACEMENT_READY
  -> RECOVERY_REQUESTED
  -> DELIVERY_VERIFIED
  -> RECONCILING
  -> VERIFIED
  -> CLOSED

Any failed step -> NEEDS_REVIEW

Persist every transition with an actor, timestamp, input hash and result. If the recovery request times out, do not immediately submit it again. First query current subscriptions and health to distinguish “request failed” from “request succeeded but response was lost.”

For a related cutover pattern, see safe webhook endpoint rotation. The same blue-green principle applies, but Zid's Broken recovery adds an explicit new-URL constraint and an API-recorded health state.

Reconcile missing orders and other entities

Recovery restores the event channel; it does not prove that the ERP, warehouse or analytics database matches Zid. Reconciliation should read authoritative resources for the incident window and compare them with local records using stable identifiers and source versions.

For each entity:

1. load the authoritative representation and its current status;
2. find the local record and its last applied source version or timestamp;
3. insert a missing entity or apply a newer valid transition;
4. leave newer local evidence untouched until reviewed;
5. enqueue downstream effects only when their idempotency key has not completed;
6. write a reconciliation result such as matched, repaired, conflicted or failed.

Do not replay arbitrary historical events merely to make the queue look complete. A stale “order created” event applied after a current cancellation can move the local model backward. Prefer current-state reconciliation and guard transitions by source version, event time and allowed state-machine edges.

Financial and fulfillment effects need separate controls. A repaired order row does not automatically authorize a refund, shipment or inventory movement. Reconstruct the intended command, confirm its business preconditions and use an immutable idempotency key. The Zid customer-wallet safety guide shows the same distinction between state evidence and irreversible commands.

Finish reconciliation with aggregate checks: counts and totals by status, currency, fulfillment location and time bucket. Row-by-row equality can miss an entire page omitted by a pagination bug; aggregate comparisons make systemic gaps visible.

Best practices and anti-patterns

Use least-privilege credentials: monitoring needs third_webhook_read, while recovery needs third_webhook_write. Separate those credentials or runtime roles so a read-only monitor cannot mutate subscriptions.

Treat Degraded as a warning window. Investigate receiver errors, certificate expiration, authentication changes, database saturation and deployment regressions while delivery still continues. Validate the replacement target before recovery, record per-item batch results, and reconcile from an authoritative source after every Broken incident.

Avoid acknowledging before durable storage, doing slow ERP work in the request thread, treating 429 as harmless, recovering to an untested URL, treating a Healthy new hook as proof that old data is complete, replaying stale events without transition guards, retrying an uncertain recovery without reading current state, storing secrets in logs, or polling so aggressively that monitoring competes with production API work.

The platform's health states protect delivery infrastructure; they do not guarantee end-to-end business correctness. Your system owns durable acceptance, idempotent effects, gap detection and evidence that the repair succeeded.

Production checklist

Before launch, document the subscribed event types, authentication method, target URL ownership and on-call owner. Test invalid signatures, duplicate deliveries, out-of-order events, inbox outages, worker failure, HTTP timeout, a sustained Degraded state, a Broken recovery with one failed batch item, and reconciliation across the exact incident window.

Set dashboards for platform state and application state. Keep a versioned replacement route ready. Restrict write scope. Store raw evidence with a retention policy appropriate to customer data. Verify every downstream command independently. Run scheduled reconciliation even when webhooks appear Healthy, because no delivery system alone proves that two databases agree.

The desired outcome is not merely a green webhook status. It is a closed recovery loop: Zid detects delivery failure, your monitor raises the right signal, a verified replacement endpoint restores future events, reconciliation repairs the discarded interval, and audit evidence shows that orders, payments and fulfillment state are complete without duplicate effects.

For help designing reliable commerce integrations, see the software and integration services.

Prepared by: Noor Yasser