The Message That Broke FIFO

*The queue preserved its guarantees. The application still ended up in the wrong state.*

An account is granted access. A minute later, that access is revoked. Both events enter the same FIFO queue, in the correct order.

The first event contains a field the receiving API doesn't recognize. The API rejects it, and the event ends up in the dead-letter queue (DLQ). The second event is accepted, and the account is disabled.

Two hours later, someone fixes the compatibility issue and replays the failed event. The receiving application accepts it, applies the old AccessGranted operation, and enables the account again.

No messages were lost. The DLQ preserved the failed event, and replay did what it was supposed to do. The delivery system behaved as configured.

Unfortunately, that wasn't enough.

The account ended up in the wrong state.

This is one of the more interesting problems with ordered event delivery. We spend considerable effort making sure messages aren't lost, retries work, and events are delivered in order. But those mechanisms don't necessarily guarantee that the receiving application ends up with the correct result.

And it raises a question we don't always ask when designing reliable delivery systems: Should an event that failed two hours ago still be allowed to change the application today?

FIFO didn't fail

First-in, first-out sounds simple enough. Messages enter a queue in one order and leave in the same order. But the order in which messages are accepted, the order in which they're delivered, and the order in which their effects are successfully applied are not necessarily the same thing.

Here's what happened to our account:

The original order was 100, 101. The successful processing order became 101, 100.

The queue hasn't necessarily violated its FIFO guarantee. Event 100 received a terminal delivery outcome when it was moved to the DLQ, allowing event 101 to proceed. When 100 was eventually replayed, the application had already processed the newer event.

Microsoft makes an important distinction in its Azure Service Bus documentation. Sequence numbers establish queuing and extraction order, but processing order requires additional coordination. Service Bus provides sessions for that purpose.

There's another catch. Even sequential HTTP delivery doesn't guarantee sequential processing if the receiver acknowledges events before completing the work. The delivery system may preserve FIFO while the application finishes operations in a different order.

So even when ordered delivery is implemented correctly, we eventually face a difficult decision: What should happen when one message cannot be processed, but everything behind it is ready?

The DLQ changes the rules

A DLQ solves a familiar operational problem. An invalid event shouldn't necessarily block thousands of valid events waiting behind it. We preserve the message, investigate the failure, fix the problem, and replay it when we're ready.

The alternative is to stop processing until someone intervenes. That might be appropriate for certain workflows, but it's hardly an attractive default for every integration.

The catch is that moving an event to the DLQ can change what we can guarantee about its original position.

AWS explicitly warns against using DLQs with FIFO queues when the exact order of operations must be preserved.

Azure Service Bus documents what happens when a session message exceeds its maximum delivery count. The message moves to the DLQ, and subsequent messages can proceed. If the failed message is later resubmitted, it gets a new enqueue time and sequence number.

AWS describes comparable consequences during SQS DLQ redrive, where redriven messages can be interleaved with new messages arriving from producers.

Different systems implement replay differently. In Queuey, for example, replay reactivates the original event record rather than creating a new one. The event retains its identity and original ordering position relative to events still waiting in its lane.

That's useful, especially when several failed events need to be replayed. But it doesn't undo deliveries that have already completed. If 101 has been processed, replaying 100 cannot put that operation back in front of 101 in the receiving application's history.

None of these behaviors necessarily represents a bug. They're consequences of allowing processing to continue after an earlier event has failed.

There's also an important distinction between a technical decision and a business decision. A permanently rejected event can be moved to the DLQ, allowing delivery to continue. But whether subsequent events should actually be processed depends on what the rejected event represents.

The DLQ preserved the event. It didn't guarantee that the application's original business order could be restored.

Lanes help. But choose the boundaries carefully.

Much of the operational impact can be reduced through architecture. We don't necessarily need strict global ordering, and we certainly don't need every event to follow the same failure policy.

Imagine an application processing events for 10,000 customer accounts through one FIFO stream. A single problematic event could hold up processing for every account, even though most of those accounts have nothing to do with each other.

A more practical approach is often to partition events by the resource they affect.

Events for Account A belong to one ordered lane, while accounts B and C have their own. An event-local failure in A doesn't have to prevent B and C from progressing.

This is the principle behind message groups in Amazon SQS, sessions in Azure Service Bus, and per-key lanes in systems like Queuey.

In Queuey, Partition = ByKey and Ordering = Fifo provide one active delivery per key. The next event in the lane waits until the previous one has received a delivery outcome, such as a successful HTTP response, DLQ or skip.

Different lanes can make progress concurrently. They aren't completely isolated, though. Some failures, including an ambiguous timeout under the default Hold policy for a non-idempotent destination, can pause the entire queue.

That distinction matters. A permanent rejection and an uncertain delivery outcome require different handling. With a timeout, the receiving system might have completed the operation without returning a response. Simply retrying it could result in a duplicate operation.

Lanes are useful because they limit unnecessary dependencies between events. They don't remove dependencies that actually exist.

The same goes for separating event types.

A temperature reading usually doesn't need the same delivery policy as a payment instruction. Independent telemetry can often continue after an earlier measurement fails. A sequence of dependent commands may need to stop until the failed operation has been resolved.

Using separate queues for independent workloads allows different retry, ordering and DLQ policies without forcing everything through the same rules.

There is one trap, though.

AccessGranted and AccessRevoked are different event types, but they affect the same account state. Sending them through separate, independently ordered queues could make our original problem worse. Meanwhile, two events of the same type might concern different customers and have no dependency whatsoever.

The important thing isn't the event type. It's the relationship between the operations.

Get that right, and lanes and delivery policies can remove a lot of unnecessary blocking. But when an old event eventually returns from the DLQ, something still needs to decide whether it should be applied.

What do we do when the old event comes back?

Let's return to event 100. It has been sitting in the DLQ for two hours while newer events have been processed.

We could replay it immediately, but that assumes the original operation is still relevant.

Sometimes it is. Sometimes it isn't.

An old temperature measurement might still be useful as historical data. An account status change may have been superseded. A missing inventory adjustment could be essential for calculating the correct balance, even if later adjustments have already been processed.

For events representing complete, replaceable state, a revision number is often a straightforward solution.

Suppose our account events contain revisions assigned by the authoritative source when the changes were committed:

The receiving application stores the latest applied revision for each account. When revision 100 eventually arrives, it checks whether a newer revision has already been applied.

A conditional database update can enforce this:

UPDATE account_access
SET enabled = TRUE,
    revision = 100
WHERE account_id = 42
  AND revision < 100;

If revision 101 is already stored, the update changes nothing. The old grant cannot overwrite the newer revocation.

The comparison and update happen atomically. This example assumes the account record already exists and that the event represents a complete replacement of the relevant state.

Notice where the revision comes from. It is assigned by the authoritative business system, not by the delivery platform. Queue sequence numbers and delivery timestamps are useful operational information, but they don't necessarily reflect the original order of business changes.

The application also needs to handle the result correctly.

If it has safely established that revision 100 is obsolete, it can acknowledge the event without changing the state. There's no reason to return an HTTP error and send a harmless stale event into another retry cycle.

Of course, zero updated rows doesn't automatically prove that an event is stale. The account might not exist, for example. Those situations must be distinguished.

And revision checks aren't a universal fix.

Consider an event called InventoryReducedByFive. If that operation never succeeded, rejecting it merely because a newer revision exists could leave the inventory balance incorrect.

For dependent operations, the receiving system may need to detect missing revisions, hold subsequent processing, or reconstruct the correct state from an authoritative record.

This is where the difference between ordering and freshness becomes important.

Preserve ordering where operations depend on one another. Preserve freshness where newer state supersedes older state.

If an operation depends on the previous one, a missing event may require the affected sequence to stop until the problem is resolved. If an event describes replaceable state, a newer authoritative revision can often take precedence over an older one.

Lanes help isolate these decisions so unrelated workloads don't have to wait. But it's the receiving application that ultimately needs to protect its own state, whether through revisions, dependency checks or reconciliation.

Replay isn't always recovery

Imagine someone opens the DLQ on Monday morning and replays everything that failed over the weekend. The original technical problems have been fixed, and the events are now accepted.

That doesn't necessarily mean the application's state is correct.

Replaying a failed event into a running system is different from rebuilding state from an ordered history. In the first case, newer changes and external side effects may already have occurred. In the second, we deliberately reconstruct state from a known starting point using the authoritative event sequence.

Sometimes replay is enough. Other times, the event has been superseded or requires reconciliation before it can be safely handled.

Idempotency doesn't necessarily solve this either.

Event 100 might never have been successfully processed before. Its identifier is unique, and the consumer has no record of applying it. An idempotency check sees no duplicate.

The problem isn't that we're processing the same event twice. We're processing an old event for the first time, after the application has already moved beyond it.

Test the failure, not just FIFO

Most ordering tests are fairly straightforward. Send a sequence of events and verify that they arrive in order.

It's a useful test, but not a particularly demanding one.

I'd be more interested in what happens when event 100 fails permanently while 101 and 102 are waiting in the same lane.

Does processing stop, or does 100 move to the DLQ while the others continue? What happens when 100 is replayed? Does the application accept it, ignore it as stale, or identify a missing dependency?

Then I'd run the same test across independent lanes. A failure for Account A shouldn't unnecessarily block Account B when the chosen failure policy permits independent progress.

I'd also distinguish a permanent rejection from a timeout. In the latter case, the receiver might have applied the operation without returning a response. Retrying blindly can introduce an entirely different problem.

Finally, I'd inspect the actual business state, not just the delivery logs.

There's little comfort in seeing three green delivery statuses if the account is now enabled when it should be disabled.

These tests tell us much more than proving that a queue can deliver thousands of successful messages in numerical order.

FIFO is a starting point

FIFO queues, dead-letter queues, retries and replay are all useful mechanisms. Lanes and different delivery policies make them considerably more practical, especially when applications process many independent customers, devices or workflows.

A sensible architecture can isolate failures, preserve ordering where it matters, and allow unrelated operations to continue. But lanes primarily solve an isolation problem. They don't automatically prevent an old operation from changing newer business state.

For some workflows, the right approach is to stop the affected sequence until the failed operation has been resolved. For others, continued progress is more important, provided the application can recognize stale updates and handle missing dependencies.

The important thing is to make that choice before the first serious failure, rather than discovering the consequences when someone replays a DLQ on a Friday afternoon.

Because FIFO can tell you which message came first.

It cannot tell you whether that message still belongs in the present.

References and further reading

• AWS — Using dead-letter queues in Amazon SQS — FIFO ordering and the implications of DLQs.

• AWS — FIFO queue delivery logic — Message groups and ordered processing.

• AWS — Dead-letter queue redrive — What happens when failed messages are reintroduced.

• Microsoft — Azure Service Bus message sessions — Ordered processing, sessions and resubmission.

• Microsoft — Event Sourcing pattern — Event history, projections and state reconstruction.

• Queuey — Ordered delivery and per-key lanes — Queue configuration and ordered lanes.

*The examples illustrate documented messaging behaviors and their architectural consequences. They are not presented as measurements from a specific production deployment.*