Legacy mainframe to cloud-native: the data migration they said was impossible

Legacy mainframe to cloud-native: the data migration they said was impossible

Simor Consulting | 21 Jul, 2026 | 06 Mins read

An insurance company running on an IBM mainframe had accumulated forty years of policy data in VSAM files and DB2 tables. The mainframe processed 600,000 transactions per day across policy administration, claims processing, and billing. The system worked. The system was also forty years old, written in COBOL, maintained by a team of eight developers whose average age was sixty-one. The youngest COBOL developer on the team was fifty-four. The company’s board had approved a cloud migration five years earlier. Three outside firms had assessed the migration and concluded it was too risky to attempt while the mainframe was processing live transactions.

The risk assessment was correct. A big-bang migration — copy all data, rewrite all applications, switch over a weekend — was not feasible. The mainframe contained 1.2 billion policy records, 340 million claims records, and business logic encoded in 4.7 million lines of COBOL that no living person fully understood. A migration that stopped the mainframe for even one day would delay claims payments for 2.3 million active policyholders, trigger regulatory scrutiny, and cost approximately $4 million in operational disruption.

The question was not whether to migrate. The question was how to migrate without stopping the mainframe.

The strangler fig approach

We designed a migration using the strangler fig pattern — named after the tropical vine that grows around a tree, gradually replacing it without killing it. The cloud-native system grew around the mainframe, absorbing functionality one domain at a time while the mainframe continued to process live transactions. Over time, the cloud-native system handled an increasing share of the workload. The mainframe handled a decreasing share. The transition was measured in years, not months.

This diagram requires JavaScript.

Enable JavaScript in your browser to use this feature.

The routing layer was the hinge. Every transaction — a policy change, a claim submission, a billing adjustment — entered the routing layer. The router determined whether the transaction’s domain had been migrated to the cloud-native system or was still on the mainframe. Migrated transactions were routed to cloud-native services. Legacy transactions were routed to the mainframe. The policyholder experienced no difference regardless of which system processed their transaction.

The data sync service kept the two databases consistent. When a transaction was processed on the mainframe, the sync service replicated the relevant records to the cloud database. When a transaction was processed in the cloud, the sync service replicated the results back to the mainframe. This dual-write pattern was the highest-risk component in the architecture, because inconsistencies between the two databases could cause incorrect policy calculations or missed claims payments.

The sync problem

The data sync service was the source of the project’s three most significant incidents. The first incident occurred during month four, when a sync lag of forty-five seconds caused a billing adjustment made on the mainframe to be processed twice — once by the mainframe and once by the cloud billing service, which received the synced record and treated it as a new transaction. The duplicate billing affected 1,200 policyholders. The team resolved it by adding idempotency keys to every synced record, ensuring that a record processed by one system could not be re-processed when it synced to the other.

The second incident occurred during month nine, when a schema change on the mainframe added a field to the policy record without updating the sync service. The sync service dropped the new field during replication, causing cloud-native services to produce incorrect premium calculations for policies that had been updated on the mainframe after the schema change. The team resolved it by adding schema validation to the sync service, which detected and alerted on schema mismatches before they caused data loss.

The third incident occurred during month fourteen, when a network partition between the mainframe and the cloud environment caused a sync backlog that grew to six hours. During the backlog window, the two databases were inconsistent. A claims adjuster on the cloud system approved a claim based on policy data that had been updated on the mainframe four hours earlier. The update had changed the policy’s coverage limits. The approved claim exceeded the updated limits. The team resolved it by implementing a consistency check that blocked transactions in the cloud system when the sync backlog exceeded a fifteen-minute threshold.

These incidents taught the team that the sync service was not an infrastructure component. It was a critical business system that required the same monitoring, testing, and change management as the mainframe itself.

The migration sequence

We migrated by business domain, not by technical layer. The first domain was billing, because it had the fewest integrations with other domains and the most straightforward data model. Billing was migrated in month six. The second domain was policy administration, because it was the largest and most complex, and the team wanted billing’s operational experience before tackling it. Policy administration was migrated in month eighteen. Claims processing was last, because it had the most regulatory constraints and the most complex business logic. Claims was migrated in month thirty.

Each domain migration followed the same protocol. The cloud-native service was built and tested against production data replicated from the mainframe. The service ran in shadow mode for at least sixty days, processing the same transactions as the mainframe and comparing outputs. When the shadow comparison showed consistent agreement for sixty consecutive days, the routing layer was updated to send live traffic to the cloud service. The mainframe continued to receive the same transactions for an additional thirty days as a fallback. After thirty days of live operation without incident, the mainframe’s corresponding modules were decommissioned.

Total migration time was thirty months. The board had allocated three years. The project came in six months early, primarily because the billing migration went smoothly and gave the team confidence to accelerate the policy administration timeline.

What we gave up

The mainframe processed transactions in batch cycles — four batch windows per day. The cloud-native system processed transactions in real time. This was an improvement, not a trade-off. However, some downstream systems had been built to consume the mainframe’s batch output. These systems had to be re-engineered to consume real-time event streams. The re-engineering effort added four months to the claims migration timeline.

The second trade-off was operational knowledge. The COBOL developers understood the mainframe’s behavior in ways that no one understood the cloud-native system. Forty years of edge cases, regulatory patches, and business rule exceptions were encoded in the COBOL code. The migration team spent roughly twenty percent of their time reverse-engineering COBOL logic to ensure that the cloud-native implementation preserved every edge case. Some edge cases were discovered only after the cloud system went live and produced a different result than the mainframe for a rare transaction type.

The third trade-off was cost during the parallel-run period. Running both systems simultaneously was expensive — the mainframe’s operating costs continued, and the cloud infrastructure added new costs. Total infrastructure cost during the thirty-month migration was approximately double the steady-state cost of either system alone. The team accepted this because the alternative — a risky big-bang migration — could have cost far more in a single incident.

Results

After the migration was complete, annual infrastructure costs decreased by thirty-five percent compared to the mainframe. Transaction processing latency decreased from batch-cycle averages of six hours to real-time processing under 500ms. The development team grew from eight COBOL developers to fourteen engineers working on cloud-native services, with a median age of thirty-four. The company’s ability to respond to regulatory changes improved from an average of four months to three weeks, because cloud-native services could be updated and deployed in days rather than mainframe release cycles measured in quarters.

The decision heuristic

If your legacy system is too critical to stop and too old to maintain, do not attempt a big-bang migration. Build the replacement around the legacy system using the strangler fig pattern. Accept that the parallel-run period will be long, expensive, and operationally complex. The cost of running both systems is the insurance premium against a catastrophic migration failure. Migrate one domain at a time, validate each domain with a shadow-mode period measured in months, and do not decommission the legacy system’s corresponding module until the replacement has been in production without incident for at least thirty days. Patience is the strategy.

Shipping a production AI system?

Find the control gaps before they turn into incidents. Take the AI Production Scorecard for a fast baseline across the seven layers, or book an architecture review and we will turn it into a hardening plan.

Similar Articles

The Modern Data Stack for AI Readiness: Architecture and Implementation
The Modern Data Stack for AI Readiness: Architecture and Implementation
28 Jan, 2025 | 03 Mins read

Existing data infrastructure often cannot support ML workflows. The modern data stack offers a foundation, but it requires adaptation to become AI-ready. This article covers building a data architectu

How a retailer reduced inference latency 90% with feature store caching
How a retailer reduced inference latency 90% with feature store caching
21 Apr, 2026 | 04 Mins read

A mid-market e-commerce retailer with roughly $200M in annual revenue had invested eighteen months building a product recommendation engine. The models were accurate. Offline evaluation showed meaning

dbt vs SQLMesh: which transformation tool wins in 2026?
dbt vs SQLMesh: which transformation tool wins in 2026?
23 Apr, 2026 | 06 Mins read

Every analytics team eventually faces the same choice: how do you transform raw data into something analysts can actually use? For years, dbt was the only serious answer. SQLMesh arrived with a differ

The data pipeline that cost $50K/month — and the audit that found why
The data pipeline that cost $50K/month — and the audit that found why
22 Apr, 2026 | 04 Mins read

A financial services firm running analytics on trade settlement data came to us with a specific complaint: their cloud data platform cost had tripled in eighteen months, and nobody could explain why.

Data Lakehouse Security Best Practices
Data Lakehouse Security Best Practices
22 Feb, 2024 | 02 Mins read

Data lakehouses combine lake flexibility with warehouse performance but introduce security challenges from their hybrid nature. Securing these environments requires layered approaches covering authent

Migrating from batch to streaming: a 6-month journey
Migrating from batch to streaming: a 6-month journey
28 Apr, 2026 | 05 Mins read

A logistics company processing two million shipments per day ran their entire operational reporting stack on nightly batch ETL. Every morning at 6 AM, operations managers reviewed dashboards built on

When RAG failed: a knowledge retrieval project post-mortem
When RAG failed: a knowledge retrieval project post-mortem
29 Apr, 2026 | 05 Mins read

A legal technology company had invested six months building a retrieval-augmented generation system to help contract attorneys find relevant precedent clauses across a corpus of 180,000 executed agree

From 3-hour dashboards to 3-minute insights: a BI modernization story
From 3-hour dashboards to 3-minute insights: a BI modernization story
05 May, 2026 | 05 Mins read

A manufacturing company with facilities in twelve countries ran its operational reporting on a traditional BI stack: a data warehouse, an ETL pipeline, and a dashboard tool that had been deployed six

Orchestration face-off: Airflow vs Prefect vs Dagster
Orchestration face-off: Airflow vs Prefect vs Dagster
07 May, 2026 | 06 Mins read

The orchestration market has a clear incumbent and two serious challengers. Apache Airflow has been the default choice since 2015. Prefect and Dagster both emerged to address Airflow's pain points, bu

The vector database that couldn't scale — and what we did instead
The vector database that couldn't scale — and what we did instead
12 May, 2026 | 05 Mins read

A media company with a library of twelve million articles, transcripts, and research documents had built a semantic search system on a managed vector database. The system was designed to let journalis

Building an AI operating system for a 10,000-person company
Building an AI operating system for a 10,000-person company
19 May, 2026 | 05 Mins read

A diversified industrial company with 10,000 employees across manufacturing, logistics, and field services had accumulated forty-seven separate AI projects over three years. Each business unit had bui

Real-time streaming: Kafka vs Redpanda vs Pulsar
Real-time streaming: Kafka vs Redpanda vs Pulsar
21 May, 2026 | 05 Mins read

Kafka has dominated event streaming for a decade. It processes trillions of messages daily across thousands of companies. Its dominance created an ecosystem so large that "streaming" became synonymous

How we killed our ETL pipeline (and productivity went up)
How we killed our ETL pipeline (and productivity went up)
26 May, 2026 | 05 Mins read

A B2B SaaS company running a customer success platform had a data pipeline that consumed sixty percent of the data engineering team's time. Not feature work. Not analytics. Pipeline maintenance. The p

A compliance-first AI rollout in financial services
A compliance-first AI rollout in financial services
03 Jun, 2026 | 05 Mins read

A regional bank with $12 billion in assets wanted to use machine learning to improve its commercial loan underwriting process. The existing process was manual, relying on credit analysts who spent fou

Semantic Layer Implementation: Challenges and Solutions
Semantic Layer Implementation: Challenges and Solutions
20 Mar, 2024 | 02 Mins read

A semantic layer provides business-friendly abstraction over technical data structures, enabling self-service analytics and consistent metric interpretation. Implementing one involves technical challe

The $2M model that never made it to production
The $2M model that never made it to production
09 Jun, 2026 | 05 Mins read

A retail chain with 400 stores spent two years and $2.1 million building an inventory optimization model. The model was technically excellent. It reduced predicted stockouts by thirty-two percent and

Data cataloging tools: Atlan, Alation, DataHub, Amundsen
Data cataloging tools: Atlan, Alation, DataHub, Amundsen
11 Jun, 2026 | 05 Mins read

A data catalog solves a trust problem. When an analyst cannot find the right table, does not know what a column means, or cannot tell whether data is fresh, they either guess or ask someone. Both outc

Data mesh in practice: year 2 retrospective
Data mesh in practice: year 2 retrospective
16 Jun, 2026 | 05 Mins read

An insurance company with $400 million in premium volume adopted data mesh two years ago. The central data team had become a bottleneck. Every business unit — claims, underwriting, actuarial, and dist

When your AI vendor goes bankrupt — surviving platform lock-in
When your AI vendor goes bankrupt — surviving platform lock-in
23 Jun, 2026 | 05 Mins read

A healthcare analytics company received notice on a Tuesday afternoon that their primary AI infrastructure vendor was filing for Chapter 7 bankruptcy. The platform hosted their patient risk stratifica

Real-time fraud detection: from proof-of-concept to production in 90 days
Real-time fraud detection: from proof-of-concept to production in 90 days
30 Jun, 2026 | 05 Mins read

A payment processor handling twelve million transactions per day had a fraud detection system that was accurate but slow. The system reviewed transactions in batch, four times per day. A fraudulent tr

Consolidating 47 data sources into one knowledge layer
Consolidating 47 data sources into one knowledge layer
01 Jul, 2026 | 05 Mins read

A global professional services firm with 8,000 consultants maintained institutional knowledge across forty-seven separate systems. Project proposals lived in a document management system. Client engag

The GDPR audit that reshaped our entire ML pipeline
The GDPR audit that reshaped our entire ML pipeline
07 Jul, 2026 | 05 Mins read

A European fintech with twelve million customers received a GDPR audit notice from their national data protection authority. The audit focused on the company's machine learning pipeline, which powered

How a healthcare org deployed LLMs without violating HIPAA
How a healthcare org deployed LLMs without violating HIPAA
14 Jul, 2026 | 05 Mins read

A hospital system with twelve facilities and 14,000 clinical staff wanted to use large language models to assist with clinical documentation. Physicians spent an average of two hours per day on docume

Data quality platforms: Great Expectations vs Soda vs Monte Carlo
Data quality platforms: Great Expectations vs Soda vs Monte Carlo
15 Jul, 2026 | 06 Mins read

Data quality failures are expensive and silent. A broken pipeline does not crash — it produces wrong data that flows into dashboards, models, and decisions. The error is discovered weeks later when a

The modern data stack is dead — here's what replaced it
The modern data stack is dead — here's what replaced it
23 Jul, 2026 | 05 Mins read

The modern data stack was a marketing category that outlived its usefulness. Between 2019 and 2023, it described a specific architecture: Fivetran or Airbyte for ingestion, dbt for transformation, Sno

Building trust in AI recommendations — the change management story
Building trust in AI recommendations — the change management story
28 Jul, 2026 | 06 Mins read

A consumer goods company built an AI system that recommended reorder quantities for 12,000 SKUs across 340 distribution points. The system optimized for a multi-objective function that balanced invent

Schema registry showdown: Confluent vs Apicurio vs AWS Glue
Schema registry showdown: Confluent vs Apicurio vs AWS Glue
30 Jul, 2026 | 05 Mins read

When producers and consumers share a Kafka topic without agreeing on the data format, things break in production. A producer adds a field. A consumer expects the old schema. The deserialization fails,

Serverless Data Pipelines: Architecture Patterns
Serverless Data Pipelines: Architecture Patterns
05 Jun, 2024 | 08 Mins read

# Serverless Data Pipelines: Architecture Patterns Serverless computing eliminates server management and provides automatic scaling with pay-per-use billing. These benefits matter for data pipelines

Event-Driven Data Architecture
Event-Driven Data Architecture
15 Sep, 2024 | 02 Mins read

Event-driven architectures treat changes in state as events that trigger immediate actions and data flows. Rather than processing data in batches or through scheduled jobs, components react to changes

From Data Silos to Data Mesh: The Evolution of Enterprise Data Architecture
From Data Silos to Data Mesh: The Evolution of Enterprise Data Architecture
15 Feb, 2025 | 03 Mins read

Traditional centralized data architectures worked for BI but struggle with AI workloads. Centralized teams become bottlenecks as data volumes grow. Domain experts who understand the data are separated

Case Study: End-to-End RAG Platform for Customer Support
Case Study: End-to-End RAG Platform for Customer Support
05 Dec, 2025 | 05 Mins read

A SaaS company with 200 support agents and 10,000+ knowledge base articles had an 18-hour average response time and 23% first-contact resolution. Their largest enterprise client threatened to cancel a

Case Study: Building a Production AI Knowledge Layer for Financial Services
Case Study: Building a Production AI Knowledge Layer for Financial Services
01 Mar, 2026 | 10 Mins read

A regional bank's investment research team spent 60% of their time gathering information and 40% doing analysis. Analysts had to search through regulatory filings, internal research memos, market data

Feature Stores for AI: The Missing MLOps Component Reaching Maturity
Feature Stores for AI: The Missing MLOps Component Reaching Maturity
12 Mar, 2026 | 11 Mins read

A recommendation system team built their tenth model. Each model required feature engineering. Each feature engineering project started by copying code from the previous project, then modifying it for

Case Study: Multi-Agent System for Supply Chain Optimization
Case Study: Multi-Agent System for Supply Chain Optimization
13 Jun, 2026 | 12 Mins read

A mid-size automotive parts manufacturer with operations spanning 15 countries and relationships with over 200 suppliers faced a supply chain coordination problem that was consuming too much of their

The AI Data Pipeline: Special Considerations for Unstructured and Structured Data
The AI Data Pipeline: Special Considerations for Unstructured and Structured Data
11 May, 2026 | 13 Mins read

Data pipelines for AI are not the same as data pipelines for traditional software systems. The outputs are different. The failure modes are different. The tolerance for data quality issues is differen