A sender transmits an event.
The receiver processes it successfully.
The acknowledgement never makes it back.
Nothing particularly exotic happened. The network disappeared at exactly the wrong moment.
Now the two systems disagree about reality.
The receiver knows the event was accepted.
The sender does not.
This is a lost acknowledgement, and it is one of the simplest ways to understand why reliable delivery inevitably has to deal with duplicates.
The sender cannot infer success
Imagine an event with ID evt-1042.
The sender makes a request:
POST /events/evt-1042
The receiver stores it and returns 200 OK.
Before that response reaches the sender, the connection dies.
The sender sees a timeout.
What should it do?
If it assumes the request succeeded, it risks losing the event in the case where the receiver never actually got it.
If it assumes the request failed, it has to send it again.
Reliable systems normally choose the second option.
That means the receiver may see evt-1042 twice.
Retry is not duplication if identity survives
The second request should not represent a second business event.
It is another delivery attempt for the same event.
That difference needs to exist somewhere in the protocol.
A stable event identifier is the usual answer.
The sender retries evt-1042, not a newly created evt-2088.
The receiving side can then recognise that the event is already known and avoid applying the business effect twice.
Without stable identity, retrying safely becomes much harder.
The sender is attempting to improve reliability, but every retry looks indistinguishable from new work.
HTTP status codes do not remove uncertainty
There is a tendency to think of delivery in terms of responses:
200 means success. 500 means failure. Timeout means retry.
The timeout case demonstrates the limitation.
A timeout tells the sender that it did not receive a response in time.
It does not tell the sender whether the receiver processed the request.
The uncertainty is fundamental.
The same applies outside HTTP. Distributed systems cannot always know the remote side's state after communication fails.
A reliable architecture needs behaviour that remains safe despite that uncertainty.
This is why at-least-once is practical
At-least-once delivery accepts that duplicate delivery may happen.
The sender's priority is to avoid losing the event. If the sender cannot prove that responsibility transferred, it tries again.
The cost is that downstream systems need a way to tolerate or remove duplicates.
That trade-off is often much easier to operate than pretending exactly one physical delivery can always be guaranteed across independently failing systems.
The key is to distinguish logical event identity from delivery attempt identity.
One logical event can have several delivery attempts.
Custody becomes uncertain during the gap
Lost ACKs are also a good example of event custody.
Before the first attempt, the sender owns the event.
After the receiver stores it, the receiver has effectively accepted responsibility.
But the sender does not know that because the acknowledgement was lost.
For a period, both systems may behave as though they own responsibility.
That is safer than neither system owning it.
The duplicate can be reconciled later. A lost event cannot.
This is one reason durable delivery tends to favour overlap over gaps when ownership is uncertain.
For a deeper treatment of the responsibility model, see Who Owns the Event? Understanding Durable Event Custody.
The problem gets worse after restart
Suppose the sender times out and plans to retry in thirty seconds.
Five seconds later it crashes.
After restart, will it remember that evt-1042 is still unresolved?
If delivery state lives only in memory, perhaps not.
Now reliability depends on whether the sender persisted both the event and enough information to know that delivery was incomplete.
The receiver may already have the event.
The sender needs to be willing to resend it anyway.
This is where persistent identity and persistent delivery state work together.
Duplicates need to be safe downstream
Deduplication at the first receiver is useful, but the principle often needs to extend further.
Suppose the receiver accepts the duplicate but then publishes a new downstream event each time. The original duplicate has simply moved one step further into the system.
For important workflows, idempotency needs to exist at the point where repeated processing would create an unwanted effect.
Examples include:
- charging a card
- creating an invoice
- opening a physical lock
- incrementing a counter
- provisioning a subscription
For other operations, duplicates may be harmless.
Again, the requirement depends on the meaning of the event.
The goal is not to prevent every duplicate
Trying to design a system where duplicate delivery can never occur often creates a lot of complexity for very little benefit.
A more practical objective is:
- important events are not silently lost
- retries keep the same logical identity
- receivers can recognise repeated delivery
- business effects are idempotent where needed
- delivery attempts are observable
Once those properties are in place, a lost ACK stops being a mysterious edge case.
It becomes a normal delivery condition the architecture already knows how to handle.