A consumer goods company built an AI system that recommended reorder quantities for 12,000 SKUs across 340 distribution points. The system optimized for a multi-objective function that balanced inventory carrying costs, stockout risk, and supplier lead time variability. In backtesting, the system would have reduced inventory costs by eighteen percent while maintaining the same service levels.

The supply chain planners did not use it.

The system had been deployed for seven months. Adoption rate was twelve percent — meaning eighty-eight percent of planner decisions were made without consulting the system’s recommendation. Planners who did consult the recommendation overrode it sixty-four percent of the time. The system was technically available. It was organizationally invisible.

The VP of supply chain asked us to determine why. The answer was not what the data science team expected.

The trust gap was not about accuracy

The data science team assumed the adoption problem was an accuracy problem. They ran a fresh backtest and confirmed that the system’s recommendations were within five percent of optimal for eighty-nine percent of SKUs. The system was accurate. Accuracy was not the issue.

The issue was explainability — not in the technical sense of model interpretability, but in the organizational sense of decision reasoning. When the system recommended reducing reorder quantity for a SKU by thirty percent, the planner needed to understand why. Was it because demand was declining? Because the supplier’s lead time had improved? Because a promotion was ending? The system provided a number. The planners needed a reason.

Without a reason, the planner could not defend the decision to their manager. Supply chain decisions had downstream consequences — stockouts triggered customer complaints, excess inventory triggered carrying cost reviews. Every decision was implicitly auditable. A planner who followed a system recommendation and caused a stockout could not explain their reasoning. A planner who used their own judgment and caused a stockout could at least describe their thought process. The system recommendation, without explanation, was a liability.

This is the trust gap that kills AI adoption in operational settings. The system does not need to be more accurate. It needs to be more defensible.

What the team tried

The data science team added feature importance charts to the recommendation interface. Each recommendation was accompanied by a bar chart showing which input features contributed most to the reorder quantity suggestion. The charts were technically correct. They showed that “last 30 days demand” and “supplier lead time” were the top features for a given recommendation.

The planners did not find this helpful. Feature importance tells you what the model used. It does not tell you what happened in the business that made the recommendation sensible. A planner seeing that “last 30 days demand” was the top feature still did not know whether demand dropped because of seasonality, a competitor promotion, or a distribution channel change. The feature importance chart answered a machine learning question. The planner needed a business question answered.

The second attempt was a confidence score. Each recommendation was tagged with a confidence level — high, medium, or low — based on the model’s prediction interval. Planners were told to trust high-confidence recommendations and override low-confidence ones. This created a perverse incentive: planners treated “high confidence” as permission to follow the recommendation without understanding it, and “low confidence” as permission to ignore it. The confidence score became a proxy for thinking rather than a tool for thinking.

The approach: decision narratives

We redesigned the recommendation interface to present each recommendation as a decision narrative — a short, structured explanation that described what changed, why the recommendation changed, and what the planner should verify before accepting.

The narrative generator did not use an LLM. It used a rule-based template system that mapped model features to business language. The templates were written by experienced supply chain planners, not data scientists. This was a critical design choice. The planners who would use the system designed the explanations that would make the system trustworthy.

A typical narrative read: “Reorder quantity reduced by 30% because trailing 30-day demand for this SKU dropped by 25%, primarily in the Southeast region. Supplier lead time is stable. Before accepting, verify that the demand drop reflects a permanent shift and not a temporary distribution disruption.”

The narrative answered three questions. What changed: the reorder quantity recommendation moved. Why it changed: demand dropped, concentrated in a specific region. What to verify: whether the demand drop was permanent or temporary.

The verification step was the most important component. It gave the planner a specific action to take before accepting the recommendation. Instead of trusting the system blindly or ignoring it, the planner checked a specific business condition. If the verification confirmed the system’s reasoning, the planner accepted the recommendation with confidence. If the verification contradicted the system’s reasoning, the planner overrode the recommendation and the override reason was fed back into the system.

The override feedback loop

Every override was logged with the planner’s reason. The reasons were categorized: demand anomaly, supplier issue, promotion not reflected, seasonal adjustment needed, and system error. The data science team reviewed override categories weekly. If a specific category dominated overrides for a specific SKU or region, it indicated a feature the model was missing.

Over the first three months, the override feedback led to four model improvements. The “promotion not reflected” category revealed that the model did not account for retailer-specific promotions, which were tracked in a system the model had not been connected to. The “seasonal adjustment needed” category revealed that the model’s seasonal decomposition was too coarse for certain product categories. Each improvement was validated by tracking whether the override rate for the affected category decreased after the model was updated.

This created a virtuous cycle. Planners overrode recommendations for specific, categorizable reasons. The data science team used the override reasons to improve the model. The improved model produced recommendations that required fewer overrides. The planners’ trust in the system increased because they could see that their overrides led to improvements rather than disappearing into a log file.

What we gave up

The narrative generation system added latency to the recommendation interface. Each recommendation required a feature analysis pass and a template selection step before the narrative was displayed. Total interface load time increased from 1.2 seconds to 2.8 seconds. The planners accepted the additional latency because the narrative made each recommendation actionable.

The template system required ongoing maintenance. As the model evolved and new features were added, new narrative templates had to be written. The planning team designated one senior planner to spend ten percent of their time maintaining narrative templates. This was a recurring cost that did not exist in the original recommendation-only interface.

The structured override process added a step to every decision. Instead of simply following or ignoring a recommendation, the planner had to categorize their override reason. This added approximately thirty seconds per override. For planners who overrode twenty recommendations per day, this was ten additional minutes of work.

Results

After six months, adoption rate increased from twelve percent to seventy-one percent. Override rate decreased from sixty-four percent to twenty-eight percent. The remaining overrides were concentrated in specific SKU categories where the model’s features were demonstrably incomplete, giving the data science team a clear prioritization for model improvement.

Inventory carrying costs decreased by fourteen percent, slightly below the eighteen percent that the backtesting had predicted. The four-percentage-point gap was attributable to the SKUs where planners still overrode the system’s recommendations. As model improvements addressed the override reasons, the gap was closing.

The decision heuristic

If your AI system is accurate but unused, the problem is not the model. The problem is that the system provides conclusions without reasoning, and your users need reasoning to make defensible decisions. Do not optimize for accuracy. Optimize for explainability — not the technical kind that satisfies a model auditor, but the organizational kind that satisfies a decision-maker’s manager. Present recommendations as decision narratives that answer what changed, why it changed, and what to verify. Make the override process a feedback mechanism rather than a failure mode. Trust is not a property of the model. It is a property of the interface between the model and the human.