Retiring a Credit Model: The Stage Nobody Documents | HL Hunt
Retiring a Credit Model: The Stage Nobody Documents
Model documentation covers development, validation, approval and monitoring. It almost never covers switch-off, which is the one stage every model reaches. A model stops deciding on the day it is turned off, and the decisions it made keep generating obligations for years: disputes, examinations, litigation, back-book forecasting. The standard that matters is not retention but reproducibility, and keeping the code does not achieve it, because a score is code plus feature definitions plus version plus cutoff plus overlays.
What you'll learn
The step nobody plans for
Model lifecycle documentation covers development, validation, approval, and monitoring. It almost never covers what happens when the model stops being used, which is the one stage every model reaches.
The result is a predictable set of problems that arrive months after anyone has stopped thinking about the model:
- A decision has to be explained and the model that made it is no longer running.
- A dispute requires reproducing a score and the feature definitions have moved on.
- An examination asks for the approved version and three versions exist with no record of which was live when.
- A pricing question requires the overlays in force at a date, per our overlay analysis, and the overlay register was kept in a spreadsheet by someone who has left.
- The replacement model inherits a feature that nobody can now define, because its definition lived in the retired model's code.
Every one of those is a documentation failure rather than a modelling failure, and all of them are cheap to prevent at the moment of retirement and expensive to fix afterward.
What the retention window actually is
A model stops making decisions on the day it is switched off. The decisions it made keep generating obligations for years.
| What persists | How long |
|---|---|
| Explaining a decline, per our adverse action analysis | Through the applicant's dispute and complaint window |
| Accounts booked under it | The life of the account, which for a term loan is years |
| Fair lending review of a period | Examination cycles reaching back years |
| Litigation about a decision | The applicable limitation period, per our limitations analysis |
| Loss forecasting on the back book | Until the vintages mature, per our loss forecasting analysis |
Reproducibility is the standard, and it is higher than retention. Keeping the code is not sufficient, because a score is a function of code, feature definitions, input data at a point in time, the version in force, the cutoff applied, and any overlay. Per our audit trail analysis, missing any one of those makes the decision unexplainable even with the model in hand.
What has to be preserved
- The scoring code, at the exact version that was live, with its dependencies pinned.
- Feature definitions as code, not as prose. A feature described in a document and computed in a script will diverge.
- The development sample, or a defensible statement of why it cannot be retained and what stands in its place.
- The validation report and the approval record, per our validation analysis, including conditions attached to the approval.
- Every cutoff and policy rule applied on top, with effective dates, because the model's output was not the decision.
- The overlay register with dates on and off, which per our overlay analysis is the part most often held informally.
- Monitoring history, which is the evidence of whether the model was performing while it was live.
- The proxy and disparity testing performed, per our proxy testing analysis, since a question about a period will be a question about the testing done in it.
- A reproducibility test that was actually run, which is the only item that proves the rest are sufficient.
Item nine is the one that distinguishes an archive from a working archive. Before decommissioning, take a sample of historical decisions, regenerate the scores from the archived artifacts alone, and confirm they match. Per our accuracy analysis, an archive that has never been tested is an assumption, and the time to discover it is incomplete is while the people who built it are still available.
The reproducibility test has a deadline
The window in which a model can be made reproducible is the window in which the people who built it still work there and the systems that fed it still run. Both close, and neither closes on a date anyone announces. A decommissioning checklist completed six months after the model was switched off is a checklist completed from memory.
Running the switch-over
Replacing a model is not an event, and treating it as one is where most of the avoidable damage occurs.
- Run both in parallel first, with the new model scoring and not deciding, per our testing analysis.
- Compare decision distributions, not just scores. Two models with similar discrimination can approve substantially different populations, per our acceptance analysis.
- Identify who changes category, and check the change is not concentrated in a way that raises a fair treatment question.
- Reconcile the cutoff. A new model's score distribution differs, so an unchanged cutoff is a changed policy, per our threshold analysis.
- Migrate in a staged way where volume allows, so the effect is observable before it is universal.
- Keep the old model runnable during the overlap, not merely archived, because the first weeks are when you will want to compare.
- Record the switch-over date and time precisely, since every later question about a decision begins with which model made it.
- Update the reason code mapping, per our adverse action analysis, because reasons generated from a new model are not the same reasons.
Item four is the most common error in a model replacement. A cutoff is a point on a distribution, and a new model redraws the distribution. Carrying the old number across looks like continuity and is a silent change in approval rate, direction unknown until someone measures it.
Item seven sounds trivial and is the foundation of everything afterward. An operation that cannot say precisely which model scored a given application cannot answer any question about that application, and the record has to be at the decision level rather than at the deployment level, because deployments overlap.
Decisions that stay explainable after the model is gone
HL Hunt AI Underwriting stamps every decision with the model version, cutoff, and overlays in force at the moment it was made, archives feature definitions as executable code alongside the scoring logic, and runs a reproducibility test against archived artifacts before any model is retired.
When to retire a model
The decision to replace is usually made on performance, and performance is the weakest of the available triggers.
| Trigger | Strength |
|---|---|
| Input no longer available or redefined | Decisive. A model missing an input is not a degraded model, it is a different one |
| Population has shifted materially | Strong, per our monitoring analysis |
| Discrimination has decayed | Moderate; recalibration may suffice |
| Overlays now carry the decision | Strong and frequently missed. A model needing large adjustments to work is being replaced informally already |
| A better model exists | Weakest on its own; the cost of change is real |
| Nobody who built it remains | Not a performance trigger and a real risk trigger |
The fourth row is the one worth adding to a monitoring pack. An overlay register that keeps growing is evidence that the model no longer fits the conditions, and per our overlay analysis overlays go on faster than they come off. Tracking the share of the decision attributable to adjustments rather than to the model gives you a retirement trigger that arrives earlier than discrimination decay does.
The last row deserves to be treated as a governance matter rather than an inconvenience. A model nobody can explain is a model that cannot be defended, and the practical response is either to document it properly while anyone remains who can, or to plan its replacement on that basis alone.
Retire a model without losing the decisions
Get started with HL Hunt AI Underwriting for decision-level version stamping, overlay registers held as data rather than spreadsheets, parallel running with decision-distribution comparison, and cutoff reconciliation built into every model change.
Frequently asked questions
Because decisions keep generating obligations for years after switch-off: dispute windows, account lives, examination cycles, litigation periods, and back-book forecasting.
No. A score is code plus feature definitions plus input data plus version plus cutoff plus overlays. Missing one makes the decision unexplainable.
A reproducibility test: regenerate a sample of historical scores from the archive alone and confirm they match, while the people who built it are still available.
Carrying the old cutoff across. A new model redraws the distribution, so the same number is a silent policy change in an unknown direction.
Key takeaways
- Switch-off is the one lifecycle stage every model reaches and the one almost never documented.
- Decisions keep generating obligations for years after a model stops running, through disputes, examinations, litigation and back-book forecasting.
- Reproducibility is a higher standard than retention: code alone cannot regenerate a score.
- Preserve the live code version, feature definitions as executable code, cutoffs, the overlay register with dates, validation and approval records, and monitoring history.
- Run a reproducibility test from the archive alone before switch-off, while the people who built the model are still available.
- Compare a replacement on decision distribution rather than score, and identify who changes category.
- Carrying the old cutoff across a new model looks like continuity and is a silent policy change.
- A growing overlay register is an earlier and better retirement trigger than discrimination decay.
This guide is educational and does not constitute legal, compliance, or model risk advice. Record retention periods, adverse action and dispute windows, examination look-back practices, limitation periods, model risk management expectations, and documentation requirements vary by institution type, by product, and by jurisdiction and change over time. The retention illustrations here are indicative only. Consult your model risk, compliance, legal, and records functions before retiring any model or setting a retention schedule.