Simor

Data Infrastructure for Production AI

Practical writing on AI data engineering, feature stores, and the infrastructure choices that determine whether AI systems work in production.

Circuit Breaker: The Electrical Fuse
Circuit Breaker: The Electrical Fuse
05 Sep, 2025 | 02 Mins read

Your home's electrical panel has circuit breakers. Plug in too many appliances, the breaker trips, cutting power to prevent fires. You can't use those outlets until you flip it back on. Annoying, but

Data Contracts: Aligning Producers & Consumers
Data Contracts: Aligning Producers & Consumers
05 Sep, 2025 | 03 Mins read

An engineering team renamed a critical field from 'user_signup_date' to 'account_created_at' without warning. It cascaded through dozens of data pipelines, breaking executive dashboards, halting marke

Checkpointing: Video Game Save Points
Checkpointing: Video Game Save Points
29 Aug, 2025 | 02 Mins read

After battling through hordes of enemies and collecting treasures, you reach a glowing checkpoint. If you fail now, you restart from the save, not the beginning. That's checkpointing: periodically sav

Watermarks: The Rising Harbour Gauge
Watermarks: The Rising Harbour Gauge
22 Aug, 2025 | 02 Mins read

The harbormaster watches a gauge showing tide level. Ships can only depart when the tide rises above their draft mark. Some arrive on time, others are delayed by storms, a few drift in days late. Whe

CDC: The Gossip Column
CDC: The Gossip Column
15 Aug, 2025 | 03 Mins read

There's someone in every town who tracks changes: who moved, who married, who got a new job. They don't track static facts (John lives on Oak Street). They track changes (John moved from Oak to Elm).

Metadata-Driven ELT: Designing Declarative Pipelines
Metadata-Driven ELT: Designing Declarative Pipelines
15 Aug, 2025 | 03 Mins read

A data engineer at an e-commerce company stared at a mess of SQL scripts, Python notebooks, and configuration files. What started as a simple ETL job had mutated into a hydra of interdependent transfo

Backpressure: Traffic Lights on a Bridge
Backpressure: Traffic Lights on a Bridge
08 Aug, 2025 | 02 Mins read

A narrow bridge holds 50 cars safely. When car 51 tries to enter, the light turns red. Cars queue on the approach road, then the streets leading to it, then the highways beyond. The bridge is protect

DataOps Automation with Dagster, Prefect 2 & Airflow 2
DataOps Automation with Dagster, Prefect 2 & Airflow 2
08 Aug, 2025 | 04 Mins read

A fintech company's data platform ground to a halt when a schema change cascaded through dozens of pipelines. Their homegrown orchestration system—a maze of cron jobs and bash scripts—offered no visib

Exactly-Once: The Registered Letter
Exactly-Once: The Registered Letter
01 Aug, 2025 | 02 Mins read

You're sending a $10,000 check. Regular mail might get lost. Send two copies, recipient might cash both. What you need: tracked, signed for, proof of delivery. Your check arrives exactly once. Not zer