Simor

Data Infrastructure for Production AI

Practical writing on AI data engineering, feature stores, and the infrastructure choices that determine whether AI systems work in production.

Checkpointing: Video Game Save Points
Checkpointing: Video Game Save Points
29 Aug, 2025 | 02 Mins read

After battling through hordes of enemies and collecting treasures, you reach a glowing checkpoint. If you fail now, you restart from the save, not the beginning. That's checkpointing: periodically sav

Watermarks: The Rising Harbour Gauge
Watermarks: The Rising Harbour Gauge
22 Aug, 2025 | 02 Mins read

The harbormaster watches a gauge showing tide level. Ships can only depart when the tide rises above their draft mark. Some arrive on time, others are delayed by storms, a few drift in days late. Whe

CDC: The Gossip Column
CDC: The Gossip Column
15 Aug, 2025 | 03 Mins read

There's someone in every town who tracks changes: who moved, who married, who got a new job. They don't track static facts (John lives on Oak Street). They track changes (John moved from Oak to Elm).

Metadata-Driven ELT: Designing Declarative Pipelines
Metadata-Driven ELT: Designing Declarative Pipelines
15 Aug, 2025 | 03 Mins read

A data engineer at an e-commerce company stared at a mess of SQL scripts, Python notebooks, and configuration files. What started as a simple ETL job had mutated into a hydra of interdependent transfo

Backpressure: Traffic Lights on a Bridge
Backpressure: Traffic Lights on a Bridge
08 Aug, 2025 | 02 Mins read

A narrow bridge holds 50 cars safely. When car 51 tries to enter, the light turns red. Cars queue on the approach road, then the streets leading to it, then the highways beyond. The bridge is protect

DataOps Automation with Dagster, Prefect 2 & Airflow 2
DataOps Automation with Dagster, Prefect 2 & Airflow 2
08 Aug, 2025 | 04 Mins read

A fintech company's data platform ground to a halt when a schema change cascaded through dozens of pipelines. Their homegrown orchestration system—a maze of cron jobs and bash scripts—offered no visib

Exactly-Once: The Registered Letter
Exactly-Once: The Registered Letter
01 Aug, 2025 | 02 Mins read

You're sending a $10,000 check. Regular mail might get lost. Send two copies, recipient might cash both. What you need: tracked, signed for, proof of delivery. Your check arrives exactly once. Not zer

Time-Series Forecasting Pipelines: From TSDB to Model Monitoring
Time-Series Forecasting Pipelines: From TSDB to Model Monitoring
01 Aug, 2025 | 04 Mins read

An energy company's AI predicted electricity demand would peak at 6 PM, as typical. The first game of the World Cup had millions turning on TVs at 4 PM, creating an unprecedented spike their models co

Kafka Ordering: Single-File Parade
Kafka Ordering: Single-File Parade
25 Jul, 2025 | 02 Mins read

A parade where everyone maintains exact position. The drummer at position 10 stays at position 10. The flag bearer at position 50 remains at position 50. Even if they take breaks, when they reassemble