Your iPhone prompts you: iOS 18.4 is available. It includes improvements to battery performance, new photo editing tools, and a fix for crashes in third-party apps. You can install it now or wait. If something goes wrong, you can restore from a backup.
You have just received a software release notification. The update is versioned, described, and reversible. These properties are so ordinary in software that you barely notice them. They are not yet ordinary in AI.
The Versioning Problem
When you call an AI model, which version are you using? With most cloud model APIs, the answer is: whatever the provider decided to give you. You might get a newer model with different behavior and no notice. Your application might start producing different outputs without any code change on your end.
This is a problem. Software teams spend significant effort avoiding unintended behavior changes. A function that returned a sorted list last week should not start returning an unsorted list this week without a code change and a code review. The same expectation should apply to model behavior, but it does not, because model versions are not exposed the way software versions are.
The providers have incentives that do not fully align with yours. They want to roll out improvements that benefit all customers. They may not want to support old model versions indefinitely. But when a new version behaves differently on your inputs, it is your users who encounter the change, not theirs.
What Good Versioning Looks Like
A versioned model is identified, described, and reversible. Identification means a version number or hash that uniquely names what you are running. Description means a changelog that tells you what changed and why. Reversible means you can fall back to a previous version if the new one causes problems.
Some providers now offer model versioning with these properties. You can pin to a specific version, read the changelog, and roll back if needed. This is the software release model applied to AI. The cost is the overhead of managing versions: deciding when to upgrade, testing new versions, tracking which version is deployed where.
The overhead is real but manageable. For applications where output consistency matters, it is worth the cost. For exploratory applications where the latest model capabilities are more important than consistency, pinning may be unnecessary constraint.
The Upgrade Discipline
Without a version upgrade discipline, you get two failure modes. You never upgrade, and your model falls further behind as the provider improves the base model. Or you upgrade indiscriminately, and unexpected behavior changes reach production before you notice.
A middle path works better: upgrade on a schedule, test the new version against your evaluation suite, and promote through environments before reaching production. This is the release candidate process that software teams use, and there is no reason AI model deployment should be different.
The schedule should match your stability needs. Applications with high reliability requirements might upgrade quarterly. Applications that want the latest capabilities might upgrade monthly. The key is that the upgrade is a deliberate decision, not a passive drift.
Testing new versions requires an evaluation suite that can detect regressions. If you have a set of inputs with known good outputs, you can run the new version against them and compare. If the known-good outputs change in ways you did not expect, you have found a regression. If you do not have an evaluation suite, building one is worth the investment once your AI usage reaches production scale.
The Floating Version Trap
Many APIs default to a “latest” or “auto-update” version that changes over time. This is convenient for providers because they can improve the model without coordinating with you. It is dangerous for you because you cannot predict when behavior will change.
If you are using a floating version in production, you are essentially accepting provider-side changes without review. That might be acceptable if the provider’s incentives fully align with yours. In practice, they often do not. A model change that improves performance on benchmark tests might degrade performance on your specific inputs. You would want to know that before your users do.
Pinning to a version and testing upgrades deliberately is the equivalent of not automatically accepting every software update on your phone. The phone prompts you. You decide whether to install. The same discipline applies to AI models.
Versioning and Prompt Sensitivity
Some prompts are more sensitive to model version changes than others. A prompt that relies on specific formatting or specific reasoning patterns may break when the model version changes even if the model’s overall capability is similar.
This is another reason to maintain an evaluation suite. If you change model versions and your evaluation results change, you need to investigate whether the prompt needs adjustment, whether the new model version is simply worse on your specific task, or whether something else is happening.
Prompts that are brittle, relying on specific model behaviors that may not be guaranteed, are a risk. Over time, as you observe which prompts are stable across versions and which are sensitive, you can refactor the sensitive ones to be more robust.
Real-World Scenario: The Quiet Regression
A team deploys a customer service AI. The evaluation suite tests a set of 200 representative customer queries and checks that the answers meet quality standards. When the team originally built the system, it passed all 200 tests.
Six months later, the provider has updated the underlying model twice. The team, not having pinned versions, is now on the latest. They run the evaluation suite again. One hundred eighty tests pass. Twenty fail. The model has quietly become worse at certain query types.
Without the evaluation suite and the habit of running it, the team would not have known. Customer satisfaction scores might be declining for reasons nobody could explain. With the suite, the team identifies which query types regressed and can decide whether to pin to an older version, adjust prompts, or accept the regression.
Real-World Scenario: The Capability Upgrade
A different team pins to a specific model version. When the provider releases a new version with significantly better reasoning, the team evaluates it against their suite. The new version scores higher on their tests, especially for complex multi-step questions.
The team still does not upgrade immediately. They spend two weeks retesting edge cases and checking that the improvement is consistent across their specific use cases. Only then do they roll the new version into staging, where a broader set of users tests it for another week.
The deliberate upgrade process takes a month. The team ends up on the new version and benefits from its improved capability. They also avoided the risk of deploying a new version that might have worked well on benchmarks but poorly on their specific inputs.
The Prompt Sensitivity Spectrum
Not all prompts are equally sensitive to model version changes. Understanding where your prompts fall on this spectrum helps you prioritize testing effort.
Prompts that rely on exact token matching or specific formatting are highly sensitive. If your prompt says “Format the output as a JSON object with the field ‘name’ first, then ‘value’,” the output may change if the model’s tokenization or JSON generation behavior changes.
Prompts that rely on reasoning capabilities are moderately sensitive. If your prompt asks the model to “explain the trade-offs,” the specific trade-offs may change with model version, but the general capability to identify trade-offs may persist.
Prompts that rely on factual knowledge are sensitive in ways that are hard to predict. A model that knew the capital of France in one version may still know it in the next. But a model that did not know the capital of a small country might learn it in a training update, changing outputs unexpectedly.
Versioning for Multi-Tenant Systems
When your AI system serves multiple customers or products, versioning gets more complex. Different customers may need different model versions at the same time. One customer may be mid-evaluation of a new version while another wants to stay on a stable old version.
This requires a versioning architecture that supports simultaneous deployment of multiple versions. The routing layer directs each request to the appropriate version based on customer configuration, feature flag, or other criteria.
This is more operationally complex than running a single version everywhere. The benefit is that you can offer different stability guarantees to different customers. Enterprise customers who pay more may get access to stable versions. Smaller customers may accept the latest version risk.
The Version Drift Problem
When you pin to a version and do not upgrade for a long time, the pinned version becomes increasingly distant from the current version. The provider has moved on. Security patches may not apply to old versions. Performance improvements are missed.
Drift has costs. An old pinned version may have known security vulnerabilities that the provider will not patch because the version is too old. An old pinned version may be running on infrastructure that the provider is deprecating.
Managing drift requires periodic upgrade even for pinned versions. You do not have to be on the latest, but you cannot fall too far behind. Set a maximum drift threshold: if the pinned version is more than N releases behind current, it is time to evaluate an upgrade.
The Rollback Decision
At some point, a new model version will cause problems in production. The question is whether to rollback or fix forward. Rollback means returning to the old version while you investigate. Fix forward means adjusting prompts or application logic to work with the new version.
The decision depends on the severity of the problem, the expected time to fix, and the cost of staying on the old version. If the new version is causing incorrect outputs that affect customer interactions, rollback may be the only acceptable choice while you investigate.
Fix forward makes sense when the problem is manageable and the new version provides benefits you want. If the new version is significantly cheaper or more capable, you may prefer to invest in making your application work with it rather than giving up those benefits.
The evaluation suite is essential for this decision. If the evaluation suite can characterize the regression precisely, you know exactly what you are dealing with. If the regression is vague, the decision is harder.
Versioning and Prompt Engineering
Your prompts are often tuned to a specific model version. A prompt that works well on version 3.5 may produce worse outputs on version 4.0. The upgrade process has to account for this.
This means version upgrades include prompt re-evaluation. When you evaluate a new model version against your evaluation suite, you may find that the same prompts produce different quality. Some may improve. Some may degrade. You need to know which before you can decide whether the upgrade is worth it.
This argues for keeping your evaluation suite focused on outcomes rather than specific outputs. If your evaluation suite checks whether the model produces correct answers, the prompts can change as needed to achieve correctness. If your evaluation suite checks whether the model uses specific phrasing, you are locked into a prompting style that may not transfer.
The Semantic Versioning Question
Model providers do not always use semantic versioning. A change from model 3.5 to model 4.0 does not tell you whether the change is a bug fix or a breaking change. A model called “gpt-4” might have internal version updates that change behavior without any version number change.
This is a real problem for teams that pin to version names rather than specific model hashes. If the provider updates the internal version of “gpt-4” without changing the name, your pinned deployment may silently change behavior.
Some providers now offer dated versions or specific model hashes that represent a specific trained checkpoint. These are more reliable for pinning than version names that do not specify a training checkpoint.
When evaluating providers, ask about their versioning policy. A provider that offers specific checkpoint pinning is more suitable for production systems that need version stability than a provider that updates models in place without version changes.
The Cost of Falling Behind
There is a cost to being too conservative with upgrades. Model capabilities improve over time. If you pin to an old version indefinitely, you do not benefit from improvements in reasoning, factual accuracy, or efficiency.
The cost of falling behind is opportunity cost. Your competitors are using newer, better, cheaper models. Your pinned old version is costing more and producing worse outputs.
Balancing this cost against the stability benefit of pinning requires a deliberate strategy. Some organizations upgrade on a fixed schedule, say quarterly, regardless of whether they have seen problems. They accept the testing overhead in exchange for staying current.
Decision Rules
Pin model versions in production when:
- Output consistency matters for your application
- You have an evaluation suite that can detect regressions
- Your users or downstream systems are sensitive to unexpected behavior changes
- Your provider offers versioned model access with specific checkpoint pinning
- You have the engineering capacity to manage upgrades deliberately
- You serve multiple customers who may need different version stability guarantees
- Your prompts are sensitive to model version and require retuning to maintain quality
Accept floating versions when:
- You are in active development and want the latest improvements
- Your application tolerates behavior variation
- You have strong downstream testing that will catch unexpected changes
- You have no evaluation suite and cannot build one in reasonable time
- The cost of version management exceeds the value of consistency
- You are running experiments that require the latest model capabilities
- Your provider uses semantic versioning and updates models rarely
Upgrade on a schedule when:
- You want to stay current but not expose yourself to unannounced changes
- Your evaluation suite can catch regressions before they reach production
- Your stability needs allow for periodic testing cycles
- You have engineering capacity dedicated to version management
Treat model upgrades like software releases: versioned, described, tested, and reversible. The discipline is not about avoiding upgrades; it is about making them deliberate.