How Do You Keep Plant Data During a Network Outage?
To store and forward industrial data during a network outage, buffer records near the equipment and upload them when the connection returns. Specify durable local storage, original timestamps and a stable identity for each record. Then agree how much outage time the buffer must cover, how uploads resume and what happens when storage fills.
The aim is a history you can account for: which records were captured, which reached the destination and where any gaps remain.
Where should the buffer sit?
Put the buffer on the equipment side of the connection you expect to lose. For a failed connection between a plant and a remote reporting system, that could mean a local data collector.
Draw the whole path before choosing a location: equipment, collector, plant network and destination. Mark the failures each buffer must cover.
A collector cannot preserve readings it never receives. If the equipment-to-collector link fails, you need storage closer to the source or an explicit gap in the history. Treat a stopped collector and a failed remote connection as separate test cases.
For records that must survive a restart, require a durable write before the collector treats capture as complete. Include power interruption and restart recovery in acceptance testing.
How much buffer capacity do you need?
Start with the longest outage the plant intends to cover. Include time to notice the problem, repair it and restore access. Base the calculation on stored record size, including metadata and storage overhead.
Illustrative worked example — these are planning assumptions, not measured results:
| Planning input | Illustrative value |
|---|---|
| Record rate | 200 records per second |
| Average stored size | 300 bytes per record |
| Outage to cover | 12 hours |
| Calculated record volume | 2,592,000,000 bytes, about 2.6 GB |
The calculation is 200 × 300 × 43,200 seconds. This is a starting point, not the required disk size. Reserve space for logs, storage maintenance and record-size variation. Check the estimate against representative data, including busy periods.
Specify capacity as both usable bytes and time at an agreed record rate. “Twelve hours at the agreed peak load” gives operations a clearer boundary than a disk size alone.
Also size the recovery path. Once connected, the collector must handle new records while draining the backlog. Measure net drain capacity during testing and agree an acceptable recovery time.
Which timestamps should each record keep?
Keep the original event timestamp through storage, retries and upload. Do not replace it with the time the destination receives the record.
Define these fields separately:
| Field | Meaning |
|---|---|
| Event time | When the source says the event happened |
| Capture time | When the collector received the record |
| Receipt time | When the destination accepted the record |
| Time quality | Whether the event time is trusted, estimated or unavailable |
If the source has no timestamp, use capture time as the documented fallback and mark it accordingly.
Agree the time basis, clock synchronisation method and behaviour after a clock correction. Retain a source sequence number where available. Do not assume timestamps alone establish an unambiguous order.
For production reports, specify whether a delayed event belongs to its original shift and whether historical totals are recalculated. Our guide to what an OEE number can hide covers the wider need to define the inputs behind a reported figure.
How should duplicate detection work?
Design for this case: the destination saves a record, but the collector never receives its acknowledgement. The collector should retry without adding the event twice.
Assign each record a stable identity before its first upload. Require the destination to recognise that identity on every retry. An illustrative identity could combine the source, a persistent sequence number and an identifier that distinguishes separate source sessions.
Define what happens when equipment restarts or sequence numbers reset. Keep the identity unchanged when a record is retried.
Avoid treating equal values as duplicates. Two consecutive temperature readings may legitimately match. Duplicate detection should identify the same captured record.
Keep duplicate-detection history long enough to cover the agreed retry and recovery window. If the destination cannot recognise repeated records, resolve that limitation before relying on automatic retries for production totals.
In what order should records upload?
Specify ordering per source or process stream. Decide whether any downstream calculation needs a wider order across several sources.
Oldest-first upload is a useful starting policy for rebuilding history. If operators also need current visibility, consider separate live and backlog paths with defined capacity for each. Mark live views clearly while historical recovery remains incomplete.
Write down the answers to four questions:
- Must each source's sequence remain intact?
- Can the destination accept late records?
- Which reports must be recalculated after backfill?
- What happens when a record is rejected?
For a rejected record, choose a documented response: halt the affected stream or isolate it for review and continue. Preserve its identity and rejection reason. Avoid an endless retry loop with no operator visibility.
Only remove a local record after an acknowledgement whose meaning is explicit. For durable delivery, require confirmation that the destination has committed it, rather than merely received a request.
What happens when local storage fills?
Choose the policy before commissioning. A full buffer needs an explicit loss or collection rule.
| Policy | Decision to document |
|---|---|
| Overwrite oldest records | Which history may be discarded, and how the lost range is recorded |
| Reject new records | How new gaps are counted and made visible |
| Reduce collection | Which signals may be sampled less often, and how changed collection is marked |
| Reserve space by priority | Which record classes receive reserved capacity |
There is no universal choice. Make it with the people responsible for production records and reporting.
Show buffer occupancy, remaining capacity, oldest unsent record and dropped-record count. Set warning levels that leave time to respond. Test how the plant receives those warnings while the remote connection is unavailable.
Do not let a reporting buffer's storage policy silently become a machine-stop rule. Any safety-related response needs separate design with a qualified assessor.
What should you test before relying on it?
Use a known sequence of records so missing, repeated and reordered entries are detectable. Test the complete path through destination storage and reporting.
- Disconnect the remote link while collection continues.
- Restart the collector with a backlog present.
- Interrupt power during a local write.
- Restore a slow or intermittent connection.
- Interrupt an acknowledgement after destination acceptance.
- Fill the buffer and observe the agreed policy.
- Reject a record and check recovery behaviour.
- Compare identities, timestamps, gaps and report totals after recovery.
Record expected results before testing. Put capacity, acknowledgement, ordering and full-storage rules in the functional design specification, so acceptance does not depend on an informal demonstration.
Which common mistakes leave gaps?
Common mistakes include sizing for normal traffic only, deleting records when an upload starts, assigning a new identity on retry and using upload time as event time.
Another is declaring recovery complete when the connection returns. Define completion as an accounted-for backlog, resolved or reported exceptions, and checked destination totals.
Store and forward also has a boundary: it addresses delayed delivery of captured records. Treat sensor failure, failed collection, destroyed local storage and safety control as separate requirements. Do not promise uninterrupted history unless the design and tests support that promise.
Related reading: aligning timestamps across plant data systems.
Frequently asked questions
Can we guarantee that no data is lost?
Define the guarantee against specific failures and a capacity limit. Require evidence for the agreed outage duration, restart behaviour and recovery process. Beyond those limits, require visible gaps and loss counts.
Should we upload the backlog as one large batch?
Agree batch size with the destination and test interruptions. Require safe retries and confirmation of which records were accepted before deleting anything locally.
Can dashboards show current readings during recovery?
Make that an explicit requirement. If current readings arrive ahead of history, show that backfill is still running and identify reports whose totals remain provisional.
Planning a plant data connection? Explore our automation services and define the outage, recovery and storage rules before commissioning.