How-to guide

Should you build webhook retries yourself? Test the tail first

The first version of webhook delivery is small: read an event, POST it, retry if it fails. If that is all your flow needs, write it — this page says when it is. What decides the question is what grows around that loop once it runs in production, and who operates it afterwards. Below is that list, and five of its failures you can run against Queuey's sandbox in under a minute, with no signup.

What grows around the first loop

  • durable storage, queues and workers, with retry schedules, backoff and Retry-After
  • per-destination concurrency and per-customer rate limits, so one slow receiver does not stall the rest
  • signing and secret rotation, idempotency, per-key ordering
  • failure classification: a 503 waits, a 429 slows down, a 401/403 needs a person, a 422 will never succeed unchanged — a generic retry scheduler treats them all the same
  • holding and probing an unhealthy destination instead of a retry storm into it
  • a dead-letter queue, and replay that is safe to run on a filtered set
  • a delivery log, alerts, and a way for support to answer "was it delivered, and why not?" without an engineer reading logs

None of these is hard alone. The cost is keeping all of them correct together, on call, as the product changes.

Five receiver failures, side by side

A plain retry loop — retry anything that is not a 2xx, a fixed number of times — against what Queuey decided when this sequence ran. The last column is what the sandbox reports for each attempt.

The receiver answersA plain retry loopQueueyDecision per attempt
503 twice, then 200Retries and gets there. A plain loop handles this one.Retries with backoff and delivers on the third attempt.retry_transient_target_unavailable → retry_transient_target_unavailable → success
429 with Retry-After: 5, then 200Retries on its own schedule, usually too early, and spends the receiver's rate limit doing it.Waits the five seconds the receiver asked for, then delivers.retry_after_target_hint → success
422 on every attemptSends the same payload until the attempts run out. It will never be accepted unchanged.No retry. Dead-lettered on the first answer, as a payload problem.no_retry_bad_payload
401 on every attemptKeeps retrying against a credential that no longer works.No retry. Dead-lettered on the first answer, as an access problem a person has to fix.no_retry_authentication_failed
503 for 30 seconds, then 200Depends on the budget: gives up before the receiver is back, or retries without knowing it is down.Keeps retrying on a paced schedule and delivers once the receiver answers. The events are delivered in order, so nothing overtakes it.retry_transient_target_unavailable → … → success

Run it yourself

The sandbox runs the real delivery engine against an internal receiver; it never calls your endpoints or anyone else's. It needs curl and jq, and the report is complete about 45 seconds after step 2.

the tail, in the sandbox
JAR=$(mktemp)
# 1. An isolated run: no signup, an internal receiver, deleted after ~2 hours
RUN=$(curl -s -c "$JAR" -b "$JAR" -X POST https://api.queuey.ai/trial/runs \
  -H "Content-Type: application/json" -d '{}' | jq -r .publicId)

# 2. Five events, each meeting a different receiver failure
curl -s -c "$JAR" -b "$JAR" -X POST https://api.queuey.ai/trial/runs/$RUN/fire-sequence \
  -H "Content-Type: application/json" -d '{"steps":[
  {"label":"503-then-recovers","outcome":1,"failCode":503,"failCount":2},
  {"label":"429-with-retry-after","outcome":1,"failCode":429,"failCount":1,"rateLimitRetryAfterSec":5},
  {"label":"422-bad-payload","outcome":2,"failCode":422},
  {"label":"401-access-revoked","outcome":2,"failCode":401},
  {"label":"down-for-30s","outcome":5,"failCode":503,"failForSeconds":30}
]}' | jq .stepsAccepted   # 5

# 3. About 45 seconds later: each event's final status (2 delivered, 6 dead-lettered)
#    and the decision behind every attempt, in the order the steps were sent
curl -s -b "$JAR" https://api.queuey.ai/trial/runs/$RUN \
  | jq -c '.reports[] | [.finalStatus, (.timeline[] | "\(.responseCode // "timeout") \(.decisionReason)")]'

Each line of the output is one event: its final status — 2 delivered, 6 dead-lettered — and the answer and decision behind every attempt. GET /trial/runs/$RUN/events/{eventId} shows the full record for one of them.

What the sandbox does not show
Fan-out, signing, per-customer ordering and a slow receiver sharing a queue with healthy ones need more than one receiver, and the sandbox has one. The fanout, signing and ordering guides cover those.

When building it is the right call

Building is still the right call when delivery is the product's own differentiator, the protocol or guarantees are unusual, regulation rules out a managed service, or the team already runs mature shared delivery infrastructure.

So is keeping the small loop, when there are few events, one receiver you control, and a provider whose own retries are enough.

What Queuey costs. Free €0/month (5,000 tokens / month), Team €49/month (100k tokens / month included), Growth €249/month (1M tokens / month included), excl. VAT. A token is one started 16 KB of payload; retries are free. Compare it with the engineering and on-call time the list above takes, not with the cost of the first version.

Related