Reproducing a Decision Three Years Later

Someone asks why an applicant was declined in March of a year that's now two years gone. To answer it you need what the system saw at that moment — the exact input values, the model version, the attribute definitions, the rules that fired, and anything a person did. Most operations retain the outcome, frequently retain the score, and retain almost none of the rest — which means the decision can be reported and cannot be explained. The data has since refreshed, the model has been replaced, and the values that produced the score no longer exist anywhere.

When you'll be asked

Occasion	Reaches back	Asks for
Consumer dispute	Months to years	Why this decision
Fair lending review	Years	Every decision in a period
Examination	Years	Samples, with full reconstruction
Litigation	Years	Everything about a specific case
Model validation	The development period	Per our validation analysis
Your own analysis	Whatever exists	Why performance shifted

The second row is the demanding one and the one that catches operations out. A fair lending review examines decisions in aggregate, which requires the inputs for a population rather than for a case — so retention that works for individual disputes fails here, because storing a narrative per file doesn't produce an analyzable dataset.

And the last row is the one operations forget is a use case at all. The most common internal question — why did approvals shift last quarter — needs the same data, which means the trail pays for itself outside compliance.

Why the score isn't enough

The core mechanical problem, and it isn't obvious until you try.

A score is a summary of inputs that no longer exist in the form the model saw them. Per our attribute analysis:

• Credit data refreshes continuously, so pulling the file today returns a different picture.
• Attribute definitions change, sometimes silently, so the same raw data yields different values now.
• Model versions get replaced, so today's model would produce a different score from the same inputs.
• Policy rules change, so the same score might now produce a different outcome.

Four independent reasons why re-running the decision today doesn't reproduce it, and each is invisible in the stored outcome.

The consequence: an operation that stored only the outcome can say what happened and cannot say why. "The model declined it" is not an explanation, and per our explainability analysis, a reason code stored without the values it was derived from can't be verified either — which means even the reasons on the adverse action notice per our notices guide become unverifiable.

What to capture

At decision time, because none of it can be recovered afterwards.

1. Every input value as used — the attribute values the model consumed, not a reference to a file that will change.
2. The raw source data where practical, or a reference with enough detail to identify the exact pull.
3. Missing indicators and substituted values. Per our attribute analysis, whether a value was observed or supplied is essential and is the thing most often lost — a reason code derived from a substituted value describes something the applicant never had.
4. The model identifier and version.
5. The attribute definition version.
6. The score or output.
7. Every rule evaluated and its result — not just the ones that fired.
8. The policy version in force.
9. The outcome, including amount, price, and terms.
10. The reasons generated, and what was sent.
11. Any manual action, with who and why.
12. Timestamps throughout.

Item three is the one to prioritize if you can only fix one thing. The distinction between an observed value and a substituted one determines whether a decision is defensible, and it's routinely discarded because the model consumed a single number either way.

Item seven matters for a reason that isn't obvious: recording only the rules that fired makes it impossible to establish that a rule didn't fire, which is exactly what a fair lending analysis needs to know.

Versioning

The discipline that makes reconstruction possible, and the gap in most otherwise-rigorous programs.

A decision is reproducible only if every component that shaped it is versioned and retained. That's:

• The model, including the artifact itself, not just a version number.
• The attribute definitions.
• The rule set.
• The policy parameters — cutoffs, limits, pricing.
• Any lookup tables or reference data.

Retaining a version number without the versioned artifact is not retention. If the model file was overwritten, knowing it was version 4.2 tells you nothing you can act on — and this is a common finding, because model artifacts are large and version control on them is frequently weaker than on code.

Per our attribute analysis, the attribute layer is where governance is thinnest: an operation with rigorous model versioning and no attribute versioning has pinned the visible component and left the layer underneath free to move. Since attribute changes arrive through a vendor relationship rather than through model risk, they change every model's behaviour without triggering any review.

Rules and overrides

Two sources of decisions that the model didn't make.

Policy rules frequently determine the outcome, per our policy analysis — a knockout rule can decline an applicant with a strong score, and if only the score is retained the record suggests a model decision that never happened.

What that requires:

• Record which rules were evaluated and each result.
• Record which rule was determinative, where one was.
• Version the rule set, since rules change more often than models.
• Record the parameter values, since a rule with a threshold needs the threshold.

Manual overrides are the second gap and the more damaging one. Per our override analysis, overrides carry genuine information — near-cutoff approvals frequently outperform predictions — but only if the reasoning was captured.

What to record on any manual action:

• Who, with authority level.
• What changed, from and to.
• Why, in structured categories rather than free text — free text can't be analyzed across a population, which is the whole value of override data.
• What information they had that the system didn't.
• Approval, where required.

An override without a recorded reason is indefensible and uninformative at once — you can't explain it to an examiner and you can't learn from it, which are the two things it was potentially good for.

Third-party components

Where reconstruction most often fails, because the component isn't yours.

Per our vendor analysis, using a purchased model or an external decision service doesn't transfer the obligation to explain the decision — but the ability to reconstruct depends on the vendor.

What to establish contractually, before deployment:

• That you receive and can retain the inputs sent and outputs returned.
• That version information accompanies every call.
• That the vendor retains their side for at least your retention period.
• That they'll support reconstruction during examinations and litigation.
• That definition changes are notified in advance, per our attribute analysis.
• What happens if the relationship ends — a terminated vendor is under no obligation to help you explain a decision from three years ago, which makes this the clause most worth having and most often absent.

The last point is the practical trap. Operations negotiate retention obligations for the term and not beyond it, then discover that the decisions made during the relationship still need explaining after it ended.

How long

Longer than operational convenience suggests, and it's a question for counsel rather than for storage planning.

What drives the period:

• Regulatory record requirements, which vary by product and jurisdiction.
• Limitations periods for potential claims.
• Examination cycles, which reach back to the last one.
• Model development needs — per our validation analysis, you need decisions old enough to have outcomes, which for some products is years.
• Any litigation hold, which suspends everything the moment a matter arises, per our records guide.

The failure mode is a default deletion schedule set for storage reasons. Data deleted on an ordinary cycle is unavailable regardless of why it's needed, and the deletion is usually automated and unnoticed — which per our records guide is the same problem in a different setting.

And retention isn't the same as retrievability. Records in a format nobody can read, or in a system that's been decommissioned, are retained and useless — which the test below is designed to catch.

Testing the trail

The control that reveals whether any of this works, and it takes an afternoon.

Select real decisions from more than a year ago and reconstruct them end to end.

1. Pick a sample — approvals, declines, and overrides.
2. Retrieve everything you'd need.
3. Re-run the decision with the retained inputs and the retained versions.
4. Compare to the recorded outcome. It should match exactly.
5. Attempt to explain each decision as you would to an examiner.
6. Document what was missing.

Operations running this for the first time reliably find gaps, most commonly: the input values weren't retained, the model artifact was overwritten, attribute versions weren't recorded, override reasons are unusable free text, or the data is in a decommissioned system.

Then run it annually, and after any material change — a model deployment, a vendor change, a platform migration. Migrations are where trails break most often, because retention is rarely on the migration checklist and the loss isn't discovered until something is asked.

The framing worth carrying: a retention design is a set of assumptions until it's exercised. The cost of the test is a few hours; the cost of discovering the gap during an examination is a different order entirely, and by then it's not fixable.

Frequently asked questions

---

This guide is educational and does not constitute legal or compliance advice. Record retention requirements, examination expectations, and documentation standards for credit decisions vary by product, institution type, and jurisdiction, and are described here only in general terms. Consult qualified counsel and your compliance and model risk functions when setting a retention schedule or designing a decision record.