Designing an IoT Telemetry Pipeline
A hundred thousand sensors reporting once a second don't care whether your database is ready for them.
Overview
An IoT telemetry pipeline faces a write pattern unlike any of the previous architectures in this journey: a huge number of small, timestamped, append-only writes arriving continuously and often in bursts, from devices that don't coordinate with each other or with the backend at all.
Why It Exists
A fleet of sensors doesn't pause its writes to wait for the backend to catch up, and doesn't retry gracefully if a database rejects a burst of writes it wasn't provisioned for. This chapter exists to walk through the two structural decisions that make IoT ingestion survivable: buffering incoming writes through a stream so the storage layer never sees the raw burst directly, and storing the resulting data in a Time-Series DatabaseA database (TimescaleDB, ClickHouse, InfluxDB) purpose-built for timestamped, append-only data like metrics and logs.Learn more built specifically for this exact write pattern. Neither decision is optional at this write volume — skip either one, and the same architecture that handles 100,000 readings a second collapses under its own burst traffic.
Real World Example
A fleet of 100,000 temperature sensors each reports a reading once per second — a sustained 100,000 writes per second under normal conditions, with brief spikes when many devices reconnect after a network blip simultaneously. Writing directly to a database at that volume, with those bursts, would require the database to be provisioned for worst-case spikes at all times. Instead, an ingestion stream (like Kafka) absorbs the burst, buffering it until the time-series database can catch up at its own sustainable write rate — the sensors and the database are decoupled from each other's exact timing.
The Ingestion-to-Storage Pipeline
Ingestion — Buffering Bursty Writes With a Stream
Incoming readings are written first to a durable, high-throughput stream, which absorbs bursts and decouples the rate devices send data from the rate the database can sustainably ingest it.
Time-Series Database for Storage
Readings are consumed from the stream and written into a time-series database (see Vector & Time-Series Databases), which is built specifically for timestamped, append-heavy, rarely-updated data like sensor readings.
Downsampling and Retention Policies
Raw, full-resolution readings are useful for recent debugging, but a year of second-by-second data at full precision is rarely needed — downsampling (e.g. to 1-minute averages) after a defined age keeps storage bounded while preserving the trend.
Tiering Old Telemetry to Cold Storage
Downsampled historical data that's kept for long-term compliance or trend analysis, but rarely queried, is a natural fit for the storage tiering strategy covered earlier in this journey — moved to a colder, cheaper tier via a lifecycle policy.
Diagram
Readings flow through a buffer before reaching storage, then age out to cheaper tiers
100,000 sensors
bursty, uncoordinated writes
Ingestion stream
absorbs bursts, smooths the write rate
Time-series database
full resolution, recent data
Downsampled + tiered
older data, lower resolution, cheaper storage
Common Mistakes
Writing directly from devices to the database with no buffering layer in between
Why: The database then has to be provisioned for worst-case burst load at all times, even though sustained average load is much lower — an expensive and fragile way to handle bursty traffic.
Fix: Buffer incoming writes through a durable stream, letting the database consume at its own sustainable rate.
Storing all telemetry at full resolution indefinitely
Why: Second-by-second data from months or years ago is rarely queried at that resolution, and keeping it all at full precision forever means storage cost grows without bound, indefinitely.
Fix: Downsample data older than a defined age to a coarser resolution, keeping only what's actually useful for long-term trend analysis.
Using a general-purpose relational database for time-series data at this volume
Why: General-purpose databases aren't optimized for the append-heavy, time-ordered, rarely-updated access pattern of sensor telemetry, and tend to perform and compress far worse than a purpose-built time-series database at this scale.
Fix: Use a database designed specifically for time-series workloads once ingestion volume reaches a meaningful scale.
Interview Questions
Why does an IoT telemetry pipeline typically put a stream between the devices and the database, instead of writing directly?
Devices send data in bursts that don't coordinate with the database's actual capacity — a stream absorbs those bursts and lets the database consume data at a steady, sustainable rate instead of needing to be provisioned for worst-case burst load at all times.
Why would you downsample old telemetry data instead of just deleting it or keeping it all at full resolution?
Full-resolution old data is rarely queried at that precision, but the overall trend it represents can still be valuable for long-term analysis or compliance. Downsampling — reducing to a coarser time resolution — keeps that trend information at a much smaller storage footprint than the full-resolution data, striking a better balance than either deleting it entirely or keeping every single raw reading forever.
A fleet of IoT devices experiences a network outage and reconnects simultaneously, sending several minutes of buffered readings all at once. How does the architecture in this chapter handle that specific scenario?
The ingestion stream is exactly the layer designed to absorb this: it can accept a large burst of buffered readings quickly, without requiring the time-series database itself to instantly handle that spike, because the stream holds the data durably until the database's consumers can process it at their normal sustainable rate. Without the stream in between, that reconnection burst would hit the database directly, risking write rejections or serious latency degradation across the whole system at exactly the moment a lot of data needs to be captured.
Production Best Practices
Do
✓Buffer incoming device writes through a durable, high-throughput stream.
✓Downsample data past a defined age instead of keeping everything at full resolution forever.
✓Use a purpose-built time-series database once ingestion volume reaches meaningful scale.
Don't
✗Don't write directly from devices to the database with no buffering layer.
✗Don't keep all historical telemetry at full resolution indefinitely.
✗Don't default to a general-purpose relational database for high-volume time-series ingestion.
Comparison
| Handles Bursts? | Optimized For | Cost at Scale | |
|---|---|---|---|
| Direct writes to DB | No — requires worst-case provisioning | General queries | High — always provisioned for peak |
| Stream + time-series DB | Yes — stream absorbs bursts | Append-heavy, time-ordered data | Lower — provisioned for average, tiered over time |