Webhooks are deceptively simple. An HTTP POST to a URL when something happens. What could go wrong?
As it turns out, quite a lot. At scale, webhook delivery becomes one of the hardest problems in distributed systems. Let's break down the most common failure modes.
1. Endpoint downtime
Your consumer's server goes down. Your webhook fires into the void. Without retry logic, that event is lost forever. Most implementations retry 3 times with fixed intervals — but that's not enough.
The fix: Exponential backoff with jitter, combined with a dead-letter queue for events that exhaust all retries. Queuey handles this automatically, with configurable retry policies per endpoint.
2. Timeout mismatches
You send a webhook. The consumer takes 45 seconds to process it. Your system times out at 30 seconds, marks it as failed, and retries — causing duplicate processing.
The fix: Separate acknowledgment from processing. The consumer should return 200 immediately and process asynchronously. Queuey enforces reasonable timeout windows and tracks delivery receipts independently from processing.
3. Ordering guarantees
Events arrive out of order. A user update arrives before the user creation event. Your consumer's state becomes inconsistent.
The fix: Include sequence numbers or timestamps in your event payload. Better yet, use an event transport layer like Queuey that maintains ordering guarantees within a partition.
4. Payload size limits
A bulk operation generates a massive webhook payload. The consumer's reverse proxy rejects it. The event is lost.
The fix: Use event references instead of full payloads. Send a lightweight notification with an event ID, and let the consumer fetch the full payload when ready.
Building reliable delivery
Reliable webhook delivery isn't just about retries. It's about observability, dead-letter queues, replay capabilities, and per-consumer configuration. That's what Queuey provides out of the box.