Model Validation: What It Is Supposed to Catch

A validation report that reproduces the developer's performance figures and confirms they're correct has verified the arithmetic and tested nothing that matters. The failures that damage lenders aren't calculation errors — they're models predicting the wrong outcome accurately, fitted on a population that doesn't match the applicants, relying on a variable that won't hold, working well in aggregate and badly in the segment carrying most of the volume, or deployed in a form that differs from the one documented. None of those show up in a performance metric, and all of them are what validation exists to find.

Start with the outcome definition

The first question, and the one most validations reach last if at all.

What is the model predicting, and is it what the business needs predicted?

A model trained on ninety-day delinquency predicts ninety-day delinquency. If the business cares about charge-off, or about profitability, or about accounts that go delinquent and cure, the model is answering a related but different question — and the gap shows up as calibration error that nobody attributes to the definition.

What to examine:

• The definition itself, precisely — what event, at what point, measured how.
• Whether it matches the decision. An approval decision about profitability isn't well served by a pure delinquency model.
• The performance window — was it long enough for outcomes to mature? Per our vendor analysis, an immature window understates losses systematically.
• Treatment of cures and partial recoveries.
• Whether it distinguishes credit loss from fraud loss, per our attribution analysis — a model trained on a blended outcome learns a blended and unstable relationship.

An outcome definition mismatch is the most consequential finding a validation can make, because it invalidates every performance figure downstream. It's also the cheapest thing to check, and it's checked last because performance metrics are what everyone expects a validation to contain.

The development sample

The second question: who was in the data, and how do they differ from who will be scored?

What to test:

• Composition versus the applicant population — score distribution, channel mix, product mix, geography, tenure.
• Whether it's approved accounts only. Almost always yes, and per our reject inference analysis that means the model has never observed the population below the cutoff — so its behaviour there is extrapolation, and a validation should say so rather than reporting performance as though it applies.
• The period covered, and whether conditions during it were representative.
• Volume of matured defaults, not accounts — the count that determines whether the fit is stable.
• Overrides and exceptions in the sample, per our override analysis.
• Missing data patterns, per our missing data analysis — the imputation approach encodes an assumption, and validation is where that assumption should be surfaced.

For a purchased model, this is the section that usually can't be completed, and that's itself the finding. A vendor's validation was conducted on the vendor's development population, which isn't yours — so the validation that matters tests the model on your data, and where the vendor won't disclose enough for that, the validation should say the model cannot be validated.

Variable review

Every variable, examined by someone capable of noticing a problem.

For each one:

Question	What it catches
Is the direction plausible?	Leakage and data errors
Is it available at decision time?	Variables derived from post-decision information
Will it remain stable?	Reliance on a source or definition that will change
Is it acceptable to use?	Fairness and proxy concerns
How is it constructed?	Transformation and imputation assumptions
What's its missing rate?	Variables doing less work than they appear to

The first row catches the most damaging error. A variable predicting in an implausible direction is usually leakage — something in the data that correlates with the outcome because it was recorded after or because of it. Models with leakage perform beautifully in development and fail immediately in production, and the only reliable detection is someone looking at the direction and asking whether it makes sense.

The fourth row is where validation meets fair lending. Per our governance framework, variables correlating with protected characteristics require justification — and a variable nobody has examined cannot be assessed. That's the position our vendor analysis identifies as untenable when a provider won't disclose the variable list.

Segment performance

The test that most often changes a conclusion.

Aggregate performance is a weighted average that can conceal segments where the model doesn't work. A model with good overall discrimination can be near-random on a population representing a fifth of volume, and the aggregate won't show it.

Segments to test separately:

• Thin file versus established, per our segmentation analysis — the most common place performance diverges.
• By channel, since channel shifts level substantially.
• By product, where one model covers several.
• By score band, particularly near the cutoff where decisions concentrate.
• By vintage, to detect degradation.
• By protected characteristic, for the outcome testing in our governance framework.
• Near the boundaries of the input ranges, where extrapolation begins.

The near-cutoff test deserves emphasis: that's where the model's output actually changes decisions, so performance there matters more than performance in the tails where everyone is approved or declined regardless.

And by-protected-characteristic testing has to be done per segment as well as in aggregate, since a disparity confined to one segment can be invisible in a combined analysis.

The deployed model

The check most frequently skipped, and it finds real errors.

Does the model running in production match the one documented? Differences are common and silent:

• Variable definitions constructed differently in production than in development.
• Data feeds supplying values in a different format, scale, or timing.
• Missing value handling implemented differently from the development approach.
• Rounding and precision differences.
• Rules layered on top of the score that alter the effective decision.
• Version drift, where the deployed version isn't the approved one.

The test is direct: score a set of records through both the documented specification and the production system, and compare. Any difference is a finding.

This connects to our explainability analysis: reason codes are generated in production, so an implementation difference in preprocessing produces reason codes describing values the applicant never reported. Validating the implementation is what catches that, and it's an afternoon's work.

What independence means

Structural, not titular.

A validator is independent when they have no stake in the outcome and the authority to conclude that a model should not be used. Which means:

• They don't report to the person who built it.
• Their findings can't be removed without record. An override should be documented and approved, not negotiated away in a draft.
• They aren't measured on the model's approval.
• They can access everything — data, code, documentation.
• Their conclusion goes to the approver, not through the developer.

Full organizational separation is impractical for small lenders — the person who could validate a model is frequently the person who built it. The workable substitutes: an external reviewer for periodic validation, a documented arrangement protecting the validator's conclusions, board or committee visibility of unresolved findings, and at minimum a rule that findings are recorded as raised rather than as resolved.

Per our institutional cost analysis, validation is a fixed cost that falls hardest on smaller institutions — which is a reason to scope it proportionately, not a reason to skip it.

When to validate

A schedule alone will validate on time and miss what actually went wrong.

Scheduled: before deployment, and periodically thereafter at an interval matching the model's materiality.

Event-driven, and these matter more:

• A new population — a channel, product, or geography the model wasn't built on.
• A data source change, including a vendor changing a definition.
• A product change affecting terms or the customer mix.
• Monitoring showing drift, per our monitoring guide.
• A material economic shift, where the development period no longer resembles current conditions.
• Any implementation change, however minor it appears.

The failures that cause most damage follow a change rather than accumulating slowly, which is why the trigger list matters more than the calendar. A model validated annually and changed in March is unvalidated for nine months in its current form.

Writing findings that get acted on

A validation that doesn't conclude anything can't be acted on, and most don't.

What a finding needs:

1. A conclusion, not an observation. "The development sample under-represents the current applicant population in channel mix, which is likely to degrade performance on direct applications" beats "channel mix differs."
2. A severity, so the approver can prioritize.
3. An implication — what happens if it isn't addressed.
4. A recommendation, including whether the model should be used meanwhile.
5. An owner and a date.

And the overall conclusion should be explicit: approved for use, approved with conditions and limitations, or not approved. A report ending without a recommendation has transferred the decision back to whoever commissioned it, which defeats the purpose.

The limitations section is the most useful part of a good validation and the most frequently omitted. Stating clearly where the model shouldn't be relied on — which populations, which ranges, which decisions — is more valuable than any performance metric, because it's what prevents the model being used somewhere it was never tested.

Frequently asked questions

---

This guide is educational and does not constitute legal or compliance advice. Model risk management expectations, validation scope, independence requirements, and documentation standards vary by institution type, size, and regulator, and continue to develop. Consult qualified counsel and your model risk function about the requirements applicable to your institution.