Distributed event systems become easier to reason about when responsibility is explicit.
An application creates an event and sends it somewhere else. That system may send it further. Eventually, a destination is expected to act on it.
At every handoff there is a basic question:
Who is responsible for making sure this event does not disappear?
We use the term event custody for that responsibility.
It is not a new protocol or delivery guarantee. It is a way to describe where responsibility currently sits.
A successful request may not be enough
An application publishes an event to a local service.
The service immediately returns 200 OK.
The application deletes its local copy.
A few milliseconds later, the service crashes.
If the event only existed in memory, it is gone.
The HTTP request succeeded, but the handoff was not durable.
For a system where the application is allowed to forget the event after success, the receiving side needs to have accepted enough responsibility to justify that.
In many architectures that means the event is durably persisted before success is returned.
The contract changes from:
I received your request.
to:
I have taken responsibility for this event.
Custody moves in stages
Consider:
Application → Edge → Cloud → Destination
Before publishing, the application owns the event.
After the Edge layer has durably persisted it:
Application owns event → Edge accepts → Edge owns event
After the cloud durably accepts it:
Edge owns event → Cloud accepts → Cloud owns event
And once the final destination accepts it according to the delivery contract:
Cloud owns event → Destination accepts → delivery complete
The components may be different in another architecture. The useful part is that each handoff has a clear meaning.
Ownership gaps are dangerous
The worst case is when both sides think responsibility moved when it did not.
The application receives success and forgets the event.
The next system did not durably store it.
A crash occurs.
Nobody has the event anymore.
That is a custody gap.
The opposite situation is less dangerous: both sides temporarily act as though they still own the event.
That can lead to a resend and possibly a duplicate, but stable event identity can reconcile that later.
For important data, temporary overlap is usually safer than a gap.
Lost acknowledgements expose the model
A receiver durably stores an event and sends an acknowledgement.
The acknowledgement disappears.
The receiver believes custody transferred.
The sender cannot know that, so it keeps responsibility and retries.
Now both sides temporarily behave as owners.
That is exactly what an at-least-once delivery model should allow.
The duplicate is not the fundamental problem. The system just needs to recognise that the retry represents the same logical event.
For the full failure sequence, see Lost ACKs: The Failure Nobody Thinks About.
Success needs a precise definition
Consider a method called:
PublishAsync(event)
A successful return could mean:
- accepted into process memory
- placed in an in-memory queue
- written to durable local storage
- accepted by a cloud service
- processed by the final destination
Every one of those APIs could reasonably call itself successful.
But they are very different contracts.
The producer needs to know which one it is getting.
If success means the producer may safely forget the event, then the next layer has to provide a guarantee strong enough to support that decision.
This is particularly important at the edge, where a network connection may disappear immediately after the call.
Durable local acceptance simplifies the producer
Suppose the producer can hand an event to a local component that persists it before returning success.
The application no longer needs its own copy after that point.
It does not need to know whether the internet is currently available. It does not need to schedule cloud retries. It does not need to reconstruct a delivery backlog after restart.
Those responsibilities have moved with custody.
Queuey Edge uses that contract: when PublishAsync returns successfully, the event has been durably accepted locally.
That does not guarantee the final destination has already received it. It means the application no longer owns the delivery problem.
The distinction is important.
Custody does not imply exactly-once delivery
A custody model does not eliminate uncertainty between systems.
If Edge sends an event to the cloud and the acknowledgement is lost, it may need to send the event again.
If the event retains a stable identity, that retry can still represent the same logical event.
The objective is not to guarantee that only one network request will ever exist.
It is to make sure the event remains safe while delivery attempts are repeated.
This is often a much more practical guarantee.
Not every event needs durable custody
Some data is naturally replaceable.
A device might publish its current CPU temperature every ten seconds. If one update disappears, the next one may be enough.
Persisting every reading and guaranteeing recovery could add cost without adding meaningful value.
Other events represent facts that happened once:
- a fault was detected
- a production run completed
- a payment was requested
- a safety threshold was crossed
- a device changed configuration
For those events, knowing who owns the event is much more important.
A useful architecture can support different levels of reliability rather than treating every byte the same.
Use custody as an architecture test
Take one important event in your system and follow it from creation to final processing.
At each boundary ask:
Who owns it before the call?
What does success actually mean?
Has the next component persisted enough state to survive a crash?
What happens if the acknowledgement is lost?
Can the same logical event be sent again safely?
If any handoff produces an unclear answer, that is probably where the reliability discussion should start.
Event custody does not solve distributed systems.
It does make the gaps much easier to see.