After go-live an AI system needs four kinds of attention. Model retirements: Anthropic and OpenAI publish deprecation schedules and retire models with notice, so every system needs someone who reads them, tests the replacement on the fifty samples, and switches before the deadline. Drift: accuracy falls as documents, inputs and customer behaviour change, so a weekly sample against the threshold continues indefinitely and the exceptions rate is watched. Cost creep: usage grows and prices change, so the per-unit cost and the ceilings are reviewed monthly against the invoice. And people: the person who worked the exceptions leaves, the runbook goes stale, a connector is granted more than it needs. A support retainer should cover those four, with a monthly report against the threshold and the cost, and should cost a small fraction of the build per month; it should not be the only place the knowledge lives, because the code, the accounts, the prompts and the runbook are yours.
- Model retirements
- Published schedules; test and switch before the date
- Drift
- Weekly sample against the threshold, forever
- Cost
- Per-unit and ceilings reviewed monthly
Model retirements
Providers retire models. Anthropic and OpenAI both publish deprecation pages with notice periods, and a system built on a model that is retired without a switch stops working or silently falls back to something untested. The routine is simple: someone owns reading the deprecation pages; when a model the system uses is scheduled for retirement, the candidate replacement is run against the fifty samples and the threshold; prompts are adjusted if needed; the switch is made in a controlled window before the deadline; the runbook records the change. A system on the hyperscaler routes has the same job with the cloud provider’s calendar. This is the single most common cause of an AI system breaking a year after go-live, and the easiest to prevent.
Drift
The documents the knowledge assistant reads change and nobody removes the old ones. The invoices start arriving in a new supplier’s format. Customers start asking about a product that did not exist at go-live. The model behaves slightly differently after a provider update. Each moves accuracy, usually downwards and slowly, which is why the weekly sample of real outputs scored against the threshold continues as long as the system runs, and why the exceptions rate is watched: a rising rate is drift showing itself. When the sample falls below the threshold, the cause is found (documents, inputs, model, prompts) and fixed, and the fix is recorded.
RelatedHow to check AI output before you rely on it A knowledge assistant on your own documents
Cost creep
Usage grows because the system works, and a per-unit cost that was pence becomes a noticeable line. Providers change prices, usually downwards for a given capability but not always for the model you are on. A new feature added in month four doubles the tokens per request. So the monthly routine compares the invoice with the per-unit cost stated at go-live, checks the ceilings still fire, and asks whether a smaller model now passes the threshold for part of the work. Our cost-governance case study is what that looks like when it is done properly.
RelatedWhat AI features cost to run Case study: AI cost governance
People and access
The person who worked the exceptions queue leaves and their replacement was never shown the runbook. A connector granted for one purpose is still live for another. A credential set up by a developer who has since left is still in use. The quarterly routine reviews who has access to what, rotates credentials on leavers, checks the runbook against reality, and confirms the owner of each process still owns it. This is also what a client questionnaire and an ISO 42001 assessor will ask about.
What a support retainer should cover
The four routines above, on a cadence, with something you receive each time.
| Cadence | What is done | What you receive |
|---|---|---|
| Weekly | Sample scored against the threshold; exceptions rate checked; ceilings and errors reviewed | A line in the monthly report |
| Monthly | Invoice against per-unit cost; deprecation pages read; small prompt or document fixes | A one-page report: accuracy, cost, incidents, changes |
| Quarterly | Access and credential review; runbook check; model options retested for cost and quality | An updated runbook and a recommendation |
| On notice | Model retirement handled: test, switch, record | Confirmation before the deadline |
RelatedYour data when a supplier builds your software AI vendor due diligence checklist
If the supplier disappears
The test of a well-built system is that another supplier can pick it up from the runbook in a day. That means the code in your repository, the accounts and credentials in your name and your vault, the prompts and thresholds documented, the deprecation routine written down, and the fifty samples kept. If any of those sits only with the supplier, ask for it now, while the relationship is good.
RelatedHow long an AI implementation takes AI implementation
Common questions
Does an AI system need maintenance after launch?
Yes: model retirements handled before their deadlines, a weekly accuracy sample against the threshold, a monthly cost review against the per-unit figure, and a quarterly access and runbook review.
What happens when the AI model we use is retired?
Providers publish deprecation schedules with notice. The replacement is tested on your samples against the threshold, prompts adjusted, and the switch made before the deadline. Unhandled, the system breaks or silently degrades.
What is model drift?
Accuracy falling over time as documents, inputs, customer behaviour or the model itself change. It is caught by the continuing weekly sample and a rising exceptions rate.
What should an AI support retainer include?
The weekly sample, the monthly cost and deprecation check with a one-page report, the quarterly access and runbook review, and model retirements handled on notice. It should be cancellable and a small fraction of the build per month.
How do we avoid depending on the supplier?
Code in your repository, accounts and credentials in your name and vault, prompts and thresholds documented, the deprecation routine written down, and the samples kept. Another supplier should be able to take over from the runbook in a day.
Why did our AI system stop working a year after launch?
Most often a retired model nobody switched, followed by a document set nobody tidied and a credential nobody rotated. All three are routine to prevent.