A payment processor handling twelve million transactions per day had a fraud detection system that was accurate but slow. The system reviewed transactions in batch, four times per day. A fraudulent transaction detected at the next batch window had already been settled, the funds had moved, and the recovery process had begun. The average time between a fraudulent transaction and its detection was six hours. In six hours, a compromised account could process dozens of fraudulent transactions.
The fraud team had a proof-of-concept model that could detect fraud in real time — within the 200ms authorization window that every card transaction must pass through. The model was accurate. The model was fast. The model had been a proof-of-concept for fourteen months. It had not shipped because every stakeholder had a feature they considered essential for production.
The compliance team wanted explainability. The operations team wanted a human review queue for borderline cases. The data science team wanted continuous retraining. The infrastructure team wanted multi-region failover. The product team wanted a merchant-facing dashboard. Each of these features was reasonable. Together, they constituted a twelve-month roadmap that was preventing the most important feature — real-time detection — from shipping.
The scoping decision
We proposed a ninety-day launch with a single success criterion: detect and block fraudulent transactions within the authorization window, with a false positive rate below 0.5 percent. Everything else was deferred.
This was politically difficult. The compliance team argued that explainability was a regulatory requirement. The operations team argued that the human review queue was necessary for edge cases. Both were right in the abstract. Neither was right about the timeline. The cost of waiting twelve months for a complete system was approximately $14 million in fraud losses that a stripped-down system could have prevented. The cost of shipping a minimal system and adding features incrementally was a temporary reduction in operational convenience.
The CTO made the call: ship the minimum, measure the impact, add features based on measured need rather than anticipated need.
This diagram requires JavaScript.
Enable JavaScript in your browser to use this feature.
What went into the 90-day build
Days 1 through 15: infrastructure. The fraud detection service was deployed as a sidecar to the existing authorization gateway. The sidecar pattern meant that the new service intercepted the authorization request, scored it, and returned a block or approve decision without modifying the authorization gateway’s code. This eliminated the highest-risk integration point — modifying a system that processed twelve million transactions per day.
The sidecar communicated with a feature store that held pre-computed transaction features. The feature store was populated by a streaming pipeline that consumed the transaction event stream. Feature computation latency was under 100ms, which fit within the 200ms authorization window when combined with the model inference time of 30ms.
Days 16 through 45: model hardening. The proof-of-concept model was retrained on the most recent six months of labeled fraud data. The training pipeline was automated but not continuous — retraining happened weekly, not in real time. The team accepted weekly retraining because the fraud patterns in payment processing evolved on a weekly, not hourly, cadence. Daily retraining would have added infrastructure complexity without meaningful accuracy improvement.
The false positive rate was tuned to 0.4 percent on the validation set, below the 0.5 percent target. The team used a cost-sensitive loss function that weighted false negatives ten times higher than false positives, reflecting the business reality that missing a fraudulent transaction was ten times more expensive than flagging a legitimate one for review.
Days 46 through 75: shadow mode. The fraud detection service ran in shadow mode for thirty days. It scored every transaction and logged its decision, but the authorization gateway ignored the scores and used the existing batch system. Shadow mode served two purposes: it validated that the service could handle production traffic volumes without latency spikes, and it provided a comparison dataset for measuring the real-time model’s accuracy against the batch system’s accuracy.
The comparison showed that the real-time model detected fraud 4.2 hours earlier than the batch system on average. It also detected seventeen percent more fraudulent transactions, because some fraud patterns that were invisible in batch — rapid sequences of small transactions — were detectable in real time with streaming features.
Days 76 through 90: staged rollout. The service was enabled for one percent of transactions in week one, ten percent in week two, fifty percent in week three, and one hundred percent in week four. At each stage, the fraud team monitored false positive rates, latency, and system stability. No rollback was required.
What we deferred
Explainability was deferred to phase two. The model was a gradient boosted tree, which provided feature importance scores but not the regulatory-grade explanations that the compliance team required. The compliance team accepted a three-month deferral after the fraud team demonstrated that the blocked transactions were logged with feature-level detail that could support manual explanation if needed.
The human review queue was simplified. Instead of a sophisticated triage system, blocked transactions were logged to a database table that the operations team reviewed on the next business day. This was not ideal for the operations team, but it was functional. A sophisticated triage dashboard was deferred to phase two.
Multi-region failover was deferred. The service ran in a single region. If the region went down, the authorization gateway would fall back to the batch system. This degraded to the pre-project state — six-hour detection — but did not create a new failure mode. Multi-region deployment was deferred to phase two.
Continuous retraining was replaced with weekly retraining. The team built the automation for daily retraining but ran it weekly to simplify the monitoring and validation workload during the initial production period.
Results
In the first ninety days of production operation, the real-time fraud detection system blocked $3.8 million in fraudulent transactions that the batch system would have missed or detected too late. The false positive rate held at 0.38 percent, below the 0.5 percent target. Authorization latency increased by an average of 40ms, well within the 200ms window.
The phase two features were delivered over the following six months based on measured need. Explainability was the first priority, because two blocked merchants requested explanations and the manual process was time-consuming. The human review queue was second, because the operations team’s backlog of manual reviews grew beyond their daily capacity. Continuous retraining was last, because weekly retraining was producing adequate accuracy.
The decision heuristic
When you have a working proof-of-concept and a twelve-month feature wishlist, the right question is not “what do we need for production.” The right question is “what is the minimum that is better than what we have today.” Every feature that is not strictly necessary for the core capability adds time to the timeline and risk to the deployment. Ship the core capability. Measure the impact. Add features in response to measured problems, not anticipated problems. The features you think you need before launch are rarely the features you actually need after launch.
