When the model was right but nobody believed it

When the model was right but nobody believed it

Simor Consulting | 04 Aug, 2026 | 05 Mins read

An agriculture technology company built a crop yield prediction model that combined satellite imagery, soil sensor data, weather forecasts, and historical yield records. The model predicted per-field yields with an accuracy that exceeded the agronomists who had been advising farmers for decades. In validation against three years of historical data, the model’s yield predictions were within eight percent of actual yields, compared to the agronomists’ average error of fourteen percent.

The agronomists did not trust it.

The model was deployed to support agronomists in making fertilizer application recommendations. An accurate yield prediction allowed the agronomist to calculate the right fertilizer rate — too little and the crop underperformed, too much and the farmer overpaid while creating environmental runoff. The model’s recommendation was displayed alongside the agronomist’s own recommendation. The agronomist chose which to follow.

For two growing seasons, the agronomists overrode the model’s recommendation seventy-eight percent of the time. They used their own judgment, which was less accurate, because they did not trust a system that could not explain its reasoning in terms they recognized.

The expertise mismatch

Agronomists make recommendations based on observable crop indicators: leaf color, plant height, root development, pest pressure, and soil moisture measured by hand. These indicators are concrete, visible, and verifiable. An agronomist can walk a field, observe that the corn is showing nitrogen deficiency in the lower leaves, and recommend a sidedress application. The reasoning chain is short, observable, and grounded in decades of field experience.

The model’s reasoning chain was different. The model correlated satellite-derived vegetation indices with historical yield outcomes, adjusted for weather patterns and soil composition. The model could not walk a field. The model could not see nitrogen deficiency in lower leaves. The model saw spectral reflectance patterns that correlated with yield outcomes at a population level.

Both approaches were valid. They operated on different evidence and at different scales. The agronomist’s evidence was local and observable. The model’s evidence was statistical and population-level. When the two disagreed — which happened frequently — the agronomist defaulted to their own evidence because they could verify it with their own senses.

The trust problem was not about accuracy. The agronomists acknowledged that the model was more accurate in validation. The trust problem was about evidence type. The agronomists trusted evidence they could observe. The model’s evidence was invisible to them.

The failed approach: accuracy arguments

The data science team’s first instinct was to present the accuracy comparison. They showed the agronomists the validation results: the model’s eight percent error versus the agronomists’ fourteen percent error. They showed that in 217 out of 300 validation fields, the model’s prediction was closer to the actual yield than the agronomist’s prediction.

The accuracy argument backfired. The agronomists interpreted it as an attack on their expertise. Some pointed out that the model had access to data they did not — satellite imagery processed by algorithms they could not inspect — and that the comparison was therefore unfair. Others pointed out that the validation was retrospective and that the model’s performance in novel conditions — a new pest, an unusual weather pattern, a soil anomaly — was unknown.

Both objections were legitimate. The accuracy comparison was retrospective, and the model’s behavior in novel conditions was uncertain. But the deeper issue was psychological. Telling experts that a model is more accurate than they are does not build trust. It builds resentment. The experts disengage or actively resist, not because they are irrational, but because the framing positions the model as a replacement rather than a tool.

The approach: augment, not replace

We redesigned the interaction model so that the model’s output was presented as additional evidence for the agronomist’s decision, not as a recommendation to accept or override.

This diagram requires JavaScript.

Enable JavaScript in your browser to use this feature.

The combined evidence view showed the agronomist four things: the model’s yield estimate with its confidence interval, the satellite-derived vegetation index map for the field, the agronomist’s own field observations from their most recent visit, and the soil test results. The agronomist weighed all four sources of evidence and made a recommendation.

The critical design decision was that the model’s output was positioned as one evidence source among several, not as a recommendation. The agronomist was not asked “do you agree with the model’s recommendation.” The agronomist was asked “given the model’s yield estimate, your field observations, and the soil tests, what fertilizer rate do you recommend.” This framing preserved the agronomist’s agency and expertise while making the model’s evidence available for consideration.

Over time, agronomists started using the model’s evidence to sharpen their own recommendations. An agronomist who observed good crop conditions but saw a declining vegetation index trend in the satellite data would investigate whether there was a localized stress factor they had missed during their field visit. The model’s population-level evidence complemented the agronomist’s local-level evidence. Neither was sufficient alone.

The calibration partnership

We also built a feedback mechanism that allowed agronomists to flag when the model’s yield estimate seemed inconsistent with their field observations. These flags were not overrides — they were calibration signals. When an agronomist flagged a prediction, the data science team investigated whether the model was missing a feature that the agronomist’s observation implied.

In the first season, agronomists flagged forty-three predictions. Investigation revealed that the model was systematically overestimating yields for fields with compacted soils, because the soil sensor data did not capture compaction — it captured moisture and nutrient levels, which were adequate even in compacted soil. The model saw healthy soil chemistry. The agronomist saw poor root development. The agronomist was right, and the model was missing a feature.

The data science team added a compaction indicator derived from historical tillage records and soil type classifications. The model’s accuracy for compacted fields improved from a twelve percent error to a seven percent error. More importantly, the agronomists who had flagged the compagation issue saw their feedback directly improve the model. This was the turning point for trust.

What we gave up

The evidence-based interaction model was slower than a direct recommendation. An agronomist reviewing the combined evidence view spent an average of six minutes per field, compared to two minutes for a direct recommendation. At 200 fields per week, this was an additional thirteen hours of agronomist time.

The team accepted this because the agronomists were making better recommendations. Fertilizer application accuracy improved in the second growing season, with the combined approach producing yield outcomes within six percent of optimal, compared to eight percent for the model alone and fourteen percent for the agronomist alone. The combination was more accurate than either source of evidence independently.

The second trade-off was model development velocity. The calibration partnership meant that the data science team spent approximately twenty percent of their time investigating agronomist flags and integrating feedback. This slowed feature development but produced a model that the agronomists trusted, which was a precondition for adoption.

Results

In the third growing season — the first after the interaction model redesign — agronomists used the model’s evidence in eighty-three percent of their recommendations. Override rate dropped from seventy-eight percent to nineteen percent. The overrides that remained were concentrated in fields with known data gaps — sensor failures, satellite image cloud cover, or historical records older than five years.

Fertilizer cost savings reached twenty-two percent across the customer base, exceeding the eighteen percent that the model alone could have achieved if it had been trusted from the start. The additional four percent came from the agronomists’ local knowledge correcting the model’s population-level blind spots.

The decision heuristic

When domain experts reject a more accurate model, the problem is not their irrationality. The problem is that the model’s evidence is invisible to them. Do not present the model’s conclusion and ask them to agree or disagree. Present the model’s evidence alongside their own evidence and let them synthesize. Position the model as an additional data source, not as a recommendation engine. And when their expert observation reveals a gap in the model’s features, fix the gap. The fastest path to trust is demonstrating that the expert’s knowledge makes the model better, not that the model makes the expert obsolete.

Shipping a production AI system?

Find the control gaps before they turn into incidents. Take the AI Production Scorecard for a fast baseline across the seven layers, or book an architecture review and we will turn it into a hardening plan.

Similar Articles

Anatomy of an AI Incident: Post-Mortem of a Model Provider Outage
Anatomy of an AI Incident: Post-Mortem of a Model Provider Outage
19 Jun, 2026 | 09 Mins read

On a Tuesday at 2:14 PM, a major model provider began returning elevated error rates for a specific model endpoint. By 2:31 PM, a customer support platform that depended on that endpoint was producing

Agent Guardrails: Containing What an Agent Can Do in Production
Agent Guardrails: Containing What an Agent Can Do in Production
25 Jun, 2026 | 09 Mins read

Input guardrails check whether a user prompt is safe. Output guardrails check whether a model response is appropriate. Agent guardrails check whether the actions an agent takes are within bounds. Thes

How a retailer reduced inference latency 90% with feature store caching
How a retailer reduced inference latency 90% with feature store caching
21 Apr, 2026 | 04 Mins read

A mid-market e-commerce retailer with roughly $200M in annual revenue had invested eighteen months building a product recommendation engine. The models were accurate. Offline evaluation showed meaning

The data pipeline that cost $50K/month — and the audit that found why
The data pipeline that cost $50K/month — and the audit that found why
22 Apr, 2026 | 04 Mins read

A financial services firm running analytics on trade settlement data came to us with a specific complaint: their cloud data platform cost had tripled in eighteen months, and nobody could explain why.

EU AI Act enforcement begins: what data teams must do now
EU AI Act enforcement begins: what data teams must do now
25 Apr, 2026 | 04 Mins read

The first enforcement window of the EU AI Act opened in February 2026, and the grace periods that protected early movers are expiring on a rolling schedule through 2027. This is no longer a policy dis

Migrating from batch to streaming: a 6-month journey
Migrating from batch to streaming: a 6-month journey
28 Apr, 2026 | 05 Mins read

A logistics company processing two million shipments per day ran their entire operational reporting stack on nightly batch ETL. Every morning at 6 AM, operations managers reviewed dashboards built on

When RAG failed: a knowledge retrieval project post-mortem
When RAG failed: a knowledge retrieval project post-mortem
29 Apr, 2026 | 05 Mins read

A legal technology company had invested six months building a retrieval-augmented generation system to help contract attorneys find relevant precedent clauses across a corpus of 180,000 executed agree

From 3-hour dashboards to 3-minute insights: a BI modernization story
From 3-hour dashboards to 3-minute insights: a BI modernization story
05 May, 2026 | 05 Mins read

A manufacturing company with facilities in twelve countries ran its operational reporting on a traditional BI stack: a data warehouse, an ETL pipeline, and a dashboard tool that had been deployed six

The vector database that couldn't scale — and what we did instead
The vector database that couldn't scale — and what we did instead
12 May, 2026 | 05 Mins read

A media company with a library of twelve million articles, transcripts, and research documents had built a semantic search system on a managed vector database. The system was designed to let journalis

Building an AI operating system for a 10,000-person company
Building an AI operating system for a 10,000-person company
19 May, 2026 | 05 Mins read

A diversified industrial company with 10,000 employees across manufacturing, logistics, and field services had accumulated forty-seven separate AI projects over three years. Each business unit had bui

How we killed our ETL pipeline (and productivity went up)
How we killed our ETL pipeline (and productivity went up)
26 May, 2026 | 05 Mins read

A B2B SaaS company running a customer success platform had a data pipeline that consumed sixty percent of the data engineering team's time. Not feature work. Not analytics. Pipeline maintenance. The p

A compliance-first AI rollout in financial services
A compliance-first AI rollout in financial services
03 Jun, 2026 | 05 Mins read

A regional bank with $12 billion in assets wanted to use machine learning to improve its commercial loan underwriting process. The existing process was manual, relying on credit analysts who spent fou

Regulators are coming for your training data — are you ready?
Regulators are coming for your training data — are you ready?
06 Jun, 2026 | 03 Mins read

The regulatory focus on AI is narrowing from the models themselves to the data that trains them. The EU AI Act requires documentation of training data provenance and composition. The US Copyright Offi

How to audit your AI pipeline for bias -- step by step
How to audit your AI pipeline for bias -- step by step
07 Jun, 2026 | 06 Mins read

Bias in AI systems is not a theoretical risk. It is a measurable property that can be detected, quantified, and mitigated at every stage of the pipeline. The teams that treat bias as an audit problem

The $2M model that never made it to production
The $2M model that never made it to production
09 Jun, 2026 | 05 Mins read

A retail chain with 400 stores spent two years and $2.1 million building an inventory optimization model. The model was technically excellent. It reduced predicted stockouts by thirty-two percent and

Data mesh in practice: year 2 retrospective
Data mesh in practice: year 2 retrospective
16 Jun, 2026 | 05 Mins read

An insurance company with $400 million in premium volume adopted data mesh two years ago. The central data team had become a bottleneck. Every business unit — claims, underwriting, actuarial, and dist

Designing guardrails: a practical architecture guide
Designing guardrails: a practical architecture guide
21 Jun, 2026 | 06 Mins read

The guardrail problem in AI is a tension between two failure modes. Too few guardrails and the system produces harmful, inaccurate, or brand-damaging outputs. Too many guardrails and the system refuse

When your AI vendor goes bankrupt — surviving platform lock-in
When your AI vendor goes bankrupt — surviving platform lock-in
23 Jun, 2026 | 05 Mins read

A healthcare analytics company received notice on a Tuesday afternoon that their primary AI infrastructure vendor was filing for Chapter 7 bankruptcy. The platform hosted their patient risk stratifica

Sovereign AI: why countries are building their own models
Sovereign AI: why countries are building their own models
27 Jun, 2026 | 03 Mins read

France released a fully open-source large language model trained on curated French-language data. India announced a multilingual model covering 22 scheduled languages. The UAE expanded its Falcon mode

Real-time fraud detection: from proof-of-concept to production in 90 days
Real-time fraud detection: from proof-of-concept to production in 90 days
30 Jun, 2026 | 05 Mins read

A payment processor handling twelve million transactions per day had a fraud detection system that was accurate but slow. The system reviewed transactions in batch, four times per day. A fraudulent tr

Consolidating 47 data sources into one knowledge layer
Consolidating 47 data sources into one knowledge layer
01 Jul, 2026 | 05 Mins read

A global professional services firm with 8,000 consultants maintained institutional knowledge across forty-seven separate systems. Project proposals lived in a document management system. Client engag

The GDPR audit that reshaped our entire ML pipeline
The GDPR audit that reshaped our entire ML pipeline
07 Jul, 2026 | 05 Mins read

A European fintech with twelve million customers received a GDPR audit notice from their national data protection authority. The audit focused on the company's machine learning pipeline, which powered

How to write an AI incident response plan
How to write an AI incident response plan
12 Jul, 2026 | 07 Mins read

AI systems fail differently than traditional software. A traditional software bug produces incorrect output deterministically -- the same input always produces the same wrong output, and a fix elimina

How a healthcare org deployed LLMs without violating HIPAA
How a healthcare org deployed LLMs without violating HIPAA
14 Jul, 2026 | 05 Mins read

A hospital system with twelve facilities and 14,000 clinical staff wanted to use large language models to assist with clinical documentation. Physicians spent an average of two hours per day on docume

Legacy mainframe to cloud-native: the data migration they said was impossible
Legacy mainframe to cloud-native: the data migration they said was impossible
21 Jul, 2026 | 06 Mins read

An insurance company running on an IBM mainframe had accumulated forty years of policy data in VSAM files and DB2 tables. The mainframe processed 600,000 transactions per day across policy administrat

The procurement checklist for AI vendors
The procurement checklist for AI vendors
26 Jul, 2026 | 07 Mins read

AI vendor procurement is where organizations make binding commitments that are expensive to unwind. A three-year contract with a model provider locks you into their pricing, their rate limits, their m

Building trust in AI recommendations — the change management story
Building trust in AI recommendations — the change management story
28 Jul, 2026 | 06 Mins read

A consumer goods company built an AI system that recommended reorder quantities for 12,000 SKUs across 340 distribution points. The system optimized for a multi-objective function that balanced invent

AI safety regulation roundup: US, EU, UK, and Asia compared
AI safety regulation roundup: US, EU, UK, and Asia compared
01 Aug, 2026 | 04 Mins read

The regulatory landscape for AI safety has fractured along jurisdictional lines. The EU has taken a prescriptive, risk-based approach. The US has taken a sector-specific, agency-led approach. The UK h

Metadata Management for AI Governance
Metadata Management for AI Governance
24 May, 2024 | 03 Mins read

# Metadata Management for AI Governance AI systems in production require metadata management to support compliance, auditing, and model oversight. Without systematic tracking of model lineage, traini

Case Study: End-to-End RAG Platform for Customer Support
Case Study: End-to-End RAG Platform for Customer Support
05 Dec, 2025 | 05 Mins read

A SaaS company with 200 support agents and 10,000+ knowledge base articles had an 18-hour average response time and 23% first-contact resolution. Their largest enterprise client threatened to cancel a

The Governance Layer: Managing AI Risk, Compliance, and Audit
The Governance Layer: Managing AI Risk, Compliance, and Audit
07 Feb, 2026 | 13 Mins read

A healthcare system deployed an AI triage assistant. It worked well in testing. In production, it started routing patients with chest pain to low-priority queues. The error was subtle and infrequent.

Case Study: Building a Production AI Knowledge Layer for Financial Services
Case Study: Building a Production AI Knowledge Layer for Financial Services
01 Mar, 2026 | 10 Mins read

A regional bank's investment research team spent 60% of their time gathering information and 40% doing analysis. Analysts had to search through regulatory filings, internal research memos, market data

Case Study: Multi-Agent System for Supply Chain Optimization
Case Study: Multi-Agent System for Supply Chain Optimization
13 Jun, 2026 | 12 Mins read

A mid-size automotive parts manufacturer with operations spanning 15 countries and relationships with over 200 suppliers faced a supply chain coordination problem that was consuming too much of their

Responsible AI by Design: Integrating Ethics into AI Architecture
Responsible AI by Design: Integrating Ethics into AI Architecture
02 Jun, 2026 | 09 Mins read

Responsible AI is not a checklist you complete before deployment. It is a set of architectural decisions that you make throughout the design process, each of which involves trade-offs that are real an