A device can produce perfectly valid telemetry for hours while nobody outside the site can see it.
The obvious response is to buffer the data locally and send it later.
The interesting problems start after that sentence.
How long can the device stay offline? What survives a restart? How do you know what was already delivered? What happens when six hours of telemetry suddenly has to move through a connection that is only barely back?
Offline telemetry is not mainly a networking problem. It is a state-management problem.
During the outage
Assume a device produces one event every second.
The internet connection disappears at 10:00.
The application can continue to produce events because its local job does not depend on the cloud. Those events now need to exist somewhere until delivery becomes possible again.
If they only exist in RAM, the backlog is vulnerable to any process or machine restart.
If they are written to durable local storage, the system gains a much stronger guarantee, but it also gains a growing dataset that needs to be managed.
After one minute there are 60 pending events.
After six hours there are 21,600.
Across a fleet of thousands of devices, a network incident can create a substantial amount of data waiting to move.
The fact that every individual event is small does not make the backlog irrelevant.
The event time must travel with the event
One of the easiest mistakes to make is recording only when telemetry reaches the cloud.
Suppose a machine reports excessive vibration at 10:14 while the site is offline. Connectivity returns at 13:30 and the event is delivered at 13:31.
The cloud now has at least two relevant times:
- when the event occurred
- when the event was received
Those should not be confused.
If the downstream system treats 13:31 as the time of the measurement, the data has survived but its meaning has changed.
The longer the outage, the more obvious the distortion becomes.
This is one reason telemetry events should carry their occurrence time from the point where they are created, rather than relying only on ingestion timestamps later in the pipeline.
We go deeper into that in Your IoT Data Is Lying to You.
Then the network returns
Recovery is not simply the opposite of being offline.
The application may have a large backlog while new events continue to arrive.
If it immediately sends everything at maximum speed, several things can happen. The newly restored connection may become saturated. The cloud ingestion service may rate-limit the device. A destination further downstream may not be able to absorb the burst.
A good recovery mechanism needs to drain old events while continuing to accept new ones.
Depending on the use case, ordering may matter as well. If machine.stopped happened before machine.started, replaying them in the opposite order can leave the receiving system with the wrong state.
There is no universal ordering requirement for IoT. High-volume telemetry may tolerate substantial reordering. State transitions often do not.
The important part is that the behaviour is defined rather than accidental.
Recovery has to survive another failure
Imagine the device has accumulated 30,000 events.
Connectivity returns and 8,000 are delivered.
The device then restarts.
What happens to the remaining 22,000?
A durable backlog needs enough delivery state to resume correctly. Otherwise the system may restart from the beginning and resend everything, or worse, assume the backlog is complete and lose the remainder.
This is where stable event identity becomes useful.
A resend of an existing logical event should still be recognisable as the same event. That makes retries and recovery much safer even when the sender is uncertain about which acknowledgements made it back.
Storage has a limit
“Store locally until the network returns” can sound unlimited.
It never is.
Disk space is finite, and the amount available to one application should normally be bounded even when the physical disk is much larger.
What happens when the local spool reaches that limit is an architectural decision.
Silently dropping the oldest event may be correct for replaceable sensor readings.
Refusing new events may be correct when every event matters and the producer needs to know that durability can no longer be guaranteed.
Different products will make different choices. The important part is that the behaviour is explicit and observable.
An offline mechanism that works until storage fills and then quietly changes semantics is difficult to trust.
Not all telemetry deserves the same treatment
There is a useful distinction between state and events.
If a sensor reports current humidity every five seconds, the value from thirty minutes ago may have little operational value once a newer reading exists.
If a device reports that a safety alarm was triggered, that is a historical fact. A newer event does not replace it.
Many IoT systems contain both types.
Applying maximum durability to everything can create unnecessary storage and traffic. Treating everything as replaceable can lose important history.
Reliability starts by deciding which data actually matters.
Store-and-forward should be boring
When it works well, offline telemetry is not especially dramatic.
The device keeps producing data. Events are persisted. Connectivity returns. The backlog drains. The cloud sees the original occurrence times. Restarts do not change the result.
That is the desired property.
The operational complexity is still there, but it no longer has to leak into the application that produced the telemetry.
For a broader discussion of that separation, see Your IoT Application Shouldn't Depend on the Cloud Being Online.
And if your telemetry arrives safely but still tells the wrong story, the next article is Your IoT Data Is Lying to You.