Design a real-time IoT monitoring platform that ingests temperature, humidity, and pressure readings from millions of sensors worldwide. Support real-time dashboards with 5-minute rolling averages, threshold-based alerting, anomaly detection, and querying up to 3 months of historical data with automatic downsampling for older ranges.
What to design
What are the core functional requirements for the IoT monitoring platform?
What are the non-functional requirements? Consider ingestion throughput, query latency, data retention, and reliability.
Design the core APIs — device registration, data ingestion, real-time queries, historical queries, and alert management.
How would you design the data ingestion pipeline to handle 1 million events per second reliably?
How would you store time-series data with different retention tiers — raw, 1-minute, 1-hour, and daily aggregates?
How would you compute real-time 5-minute rolling averages using stream processing?
How would you implement threshold-based alerting and anomaly detection on the streaming data?
How would you serve the real-time dashboard and handle historical range queries efficiently?
How would you handle late-arriving data from sensors with intermittent connectivity?
How would you monitor sensor health and detect offline or malfunctioning devices?