Data
Lessons from building a streaming pipeline for sensor data
Real-time data from meters and sensors looks simple until the sensors misbehave. Design notes for utilities and public works.
We built a streaming pipeline for sensor telemetry as a reference architecture: devices publish readings, a broker buffers them, a processor validates and aggregates, storage keeps history, and dashboards and alerts consume the result. The architecture is standard. The lessons were in the details.
Sensors lie
Readings arrive late, out of order, duplicated, and occasionally impossible. A water meter reporting negative flow, a temperature sensor stuck at one value for a week. The pipeline must validate every reading, flag anomalies, and keep the raw value alongside the cleaned one so that nobody has to guess later what happened.
Time is harder than it looks
Device clocks drift. Some devices batch readings and send them hours later. Every reading needs both the device's timestamp and the arrival time, and every aggregation needs a rule for late data.
Resolution costs money
Storing every reading at full resolution forever is expensive and rarely useful. Recent data at full resolution, older data aggregated, and a retention policy agreed with the people who use the data.
Alerts need hysteresis
A threshold alert on a noisy signal fires constantly. Alerts should require a condition to persist and should clear only after a margin. Operators stop reading alerts that cry wolf.
Devices are an attack surface
Every device has credentials and firmware. Unique credentials, encrypted transport, a segmented network, and an update path are requirements from day one, because retrofitting them across hundreds of field devices is painful.
Open formats
Device vendors prefer their own platforms. Insist on standard protocols and the ability to export data, or the pipeline is hostage to one vendor's roadmap.
For agencies
When buying sensors and platforms, specify data ownership, export, protocols, and security features, and separate the device purchase from the platform where you can. The pipeline is where the value is, and it should belong to you.