← All articles
Production AI5 min read

Your AI provider can change the model under you. Plan for it

Model providers retire and update versions on their schedule. Pin versions, run regression evals, and keep a tested fallback so changes never surprise you.

Your AI provider can change the model under you. Plan for it

You pinned the model version, so you assumed you were safe. Then the provider announced the version's retirement date, or quietly updated the "stable" alias you were actually calling, and your system started behaving differently without a single line of your code changing. This is not a hypothetical risk. It is a maintenance reality of building on someone else's model: the ground under your product moves on a schedule you do not control.

The changes are rarely dramatic, which is what makes them dangerous. A model update shifts tone, tightens refusal thresholds, or formats tool calls slightly differently. None of that shows up as downtime. It shows up as a support ticket about answers that used to be crisp, or a parser that starts failing on one response in twenty. Teams that only monitor uptime miss all of it. The system is "up" and quietly worse.

The first defence is boring and non-negotiable: pin exact versions in production, never build on an unversioned alias, and treat the provider's changelog like a dependency update. Deprecation deadlines go in the calendar the day they are announced, with an owner and a migration slot — not in a Slack thread everyone forgets. If your system cannot tell you exactly which model version served each request last month, you cannot diagnose a behaviour change, and you cannot prove to a customer that nothing changed on your side.

The second defence is a regression eval set that runs whenever the model changes, not whenever someone remembers. Before any cutover, run the new version in shadow against the golden set: same prompts, same expected behaviours, diffed outputs. Set a threshold in advance for what counts as a regression, and refuse the cutover if it fails. The teams that survive provider changes are not the ones with the best prompts. They are the ones who can measure a new model's behaviour before their customers do.

The third defence is architectural: keep the provider behind one adapter, not scattered through your codebase. Prompts should avoid provider-specific tricks that break on the next model. A fallback model should be configured and tested, not just documented — a fallback you have never actually run is a hope, not a plan. For systems where a behaviour change is expensive, a second provider for the critical path is worth the integration cost. You need an exit for the one path that would hurt most.

None of this means providers are unreliable. It means they are vendors with roadmaps, and your product's stability cannot depend on their roadmap standing still. When a provider retires a model, your response should be a scheduled migration with passing evals — not a scramble.

We build AI systems that survive provider changes — pinned versions, regression evals, and fallback paths, so a model update is a planned event instead of a production incident.

Field note: A client learned this the expensive way when their provider retired a model version on sixty days' notice. The migration looked trivial until the golden set caught it: the new version formatted a structured output differently in about one case in fifteen, just enough to break a downstream parser intermittently. Because the eval set ran in shadow before cutover, the team fixed the parser and the prompt before a single customer saw the new model. The migration took a week of planned work instead of a weekend of firefighting. The lesson stuck: the eval set is not quality theatre. It is the thing that turns a provider's roadmap change into your routine Tuesday.

Related service AI products

Building something worth shipping?

We take on a small number of AI product engagements. Tell us what you are building — we reply quickly.