Explainability Is Four Different Requirements, Not One

"Is the model explainable?" is asked as though it had a single answer, and it doesn't. A lender has to explain an individual decision to the applicant, the model's construction to an examiner, the relationships it learned to its own risk function, and any disparities to fair lending analysis — four audiences, four questions, four mechanisms. They're routinely collapsed into one debate about model complexity, which produces the wrong conclusion in both directions: teams reject useful models for a reason that doesn't apply, and teams deploy complex models believing an attribution method has satisfied a requirement it doesn't address.

The four requirements

Audience	Question	Satisfied by
The applicant	Why was I declined?	Accurate specific reasons for this decision
The examiner	How was this built and controlled?	Development and validation documentation
Your risk function	What did it learn, and does it make sense?	Understanding of the relationships
Fair lending analysis	Does it produce disparities, and is there a better alternative?	Outcome testing and variable review

Three observations that follow immediately.

Only the first is per-decision. The other three concern the model as a whole and are answered once per version, not per applicant. That's the confusion that drives most of the debate: a requirement to explain individual outcomes gets treated as a requirement for the model itself to be simple.

The third is the hardest and the least discussed. Attribution methods answer the first well. They don't tell you what relationship the model learned, which is what a risk function needs in order to judge whether it's sensible and will hold.

The fourth doesn't require explaining the model at all in the usual sense — it requires measuring outcomes and reviewing variables, which is empirical work rather than interpretive work.

Explaining a decision to an applicant

The obligation, per our notices guide, is to provide accurate specific principal reasons for the adverse action.

What that does and doesn't require:

• Doesn't require a simple model, disclosure of the model, or an explanation of how scoring works generally.
• Does require reasons that genuinely describe what drove this outcome for this person, in terms they can act on.

So the question isn't model complexity — it's whether you have a reliable method of identifying the factors that mattered for an individual decision. A complex model with sound per-decision attribution can satisfy this. A simple model with reason codes assigned from a generic list can fail it.

The test worth applying to any implementation: would the applicant, acting on these reasons, actually improve their position? A notice citing something the applicant can neither verify nor influence has met a formality and provided nothing — and providing something is the point of the requirement.

What makes a reason code wrong

Two failures that produce notices that look compliant and describe something other than what happened.

Reasons derived from imputed values. Per our missing data analysis, models substitute values for blank fields — and if a reason code traces to one of those, the notice describes a fact the applicant never reported and that isn't in their file. The applicant is told they were declined because of a number your system invented.

This is the accuracy problem that survives every review, because nobody traces a reason code back through the preprocessing layer. The check is mechanical: for a sample of notices, confirm each cited factor came from an observed value rather than a substituted one. An afternoon's work, and most operations have never done it.

Reasons drawn from a generic list. Where reason codes are mapped by rule rather than derived from the decision, they're plausible rather than actual. A declined applicant with high utilization and thin file gets "high utilization" because the mapping says so, when the model may have been driven by something else entirely.

Two further cases worth checking:

• Transformed variables. If the model uses a ratio or an interaction, the reason has to be expressed in terms of something the applicant recognizes — which requires mapping back through the transformation, and the mapping is where accuracy is lost.
• Policy declines dressed as model declines. An applicant declined by a rule rather than by a score should be told the rule's reason. Attributing it to the score is inaccurate.

Understanding versus attribution

The distinction that matters most and gets least attention.

Attribution answers: for this prediction, which inputs contributed and how much? Post-hoc methods do this, and they do it usefully.

Understanding answers: what relationship has the model learned between this input and the outcome, across the population?

These are different, and only the second lets you judge whether the model is sensible:

• Is the direction right? A variable predicting in an implausible direction is a signal of leakage or a data problem — and you can only see it if you know the direction.
• Is the relationship stable? One driven by a narrow slice of the training data will break, and the monitoring in our monitoring guide won't catch it until it has.
• Does it hold outside the observed range? Extrapolation behaviour is where the boundary errors in our reject inference analysis originate.
• Is the variable acceptable on grounds beyond prediction?

Attribution answers none of these. A method telling you that a variable drove this decision doesn't tell you whether the relationship it encodes is one you'd endorse.

What produces understanding: partial dependence and similar summaries, monotonicity constraints where a direction is known, checking that relationships match domain expectation, and testing behaviour at the edges of the input range. None of this requires a simple model — it requires deliberately examining a complex one, which is work that attribution tooling can make feel unnecessary.

The failure to name: an operation with excellent attribution and no understanding can explain every decision and cannot tell you whether the model is sound. That's a comfortable position and a fragile one.

The fair lending requirement

Largely empirical rather than interpretive, per our governance framework:

1. Measure outcomes by group, which requires no explanation of the model at all.
2. Where disparities appear, identify the contributing variables — this is where understanding matters.
3. Assess whether those variables are justified by a legitimate business need.
4. Search for less discriminatory alternatives — a model or specification performing comparably with smaller disparity.
5. Document the search, since the search itself is part of what's expected.

Step four is where interpretability earns its keep operationally. Searching for a less discriminatory alternative requires being able to modify the model in a targeted way — dropping or constraining a specific variable and observing both effects. That's straightforward with an interpretable specification and harder with one where the contribution is distributed across interactions nobody can isolate.

So the fair lending case for interpretability isn't about explaining decisions. It's about being able to change the model deliberately, which is a different and more practical argument than the one usually made — and it applies whether or not any disparity has yet been found.

Note also that the variable review in step three requires someone to look at the variable list. A model with variables nobody has examined cannot be assessed, which is the position our vendor analysis identifies as untenable when a vendor declines to disclose them.

Measuring the actual tradeoff

The performance cost of interpretability is asserted constantly and measured rarely.

What the assumption gets wrong:

• On structured credit data with well-constructed variables, the gap is frequently modest. Much of what a complex model finds in raw data is already captured by variables a credit team would build deliberately.
• The gap varies enormously by population and data richness, so a general claim about it is not usable.
• Complexity's cost isn't only performance. Explaining, validating, monitoring, and modifying a complex model all cost more — and per our vendor analysis, the obligations don't reduce.
• Some gains are fragile, resting on relationships that don't survive a population shift.

What to do instead of assuming: fit both on your own data and compare, on your own outcome definition, by segment. Then price the operational difference and compare that to the performance difference expressed in dollars.

The comparison worth making explicitly: if a complex model gains a small amount of discrimination and costs meaningfully more to explain, validate, and modify, the interpretable model may be the better business decision on economics alone — before any compliance argument. That's a legitimate conclusion and it's rarely reached because the tradeoff is debated in principle rather than measured.

Testing your explanations

An explanation method needs validating like anything else, and almost nobody does it.

What to test:

• Stability. Do near-identical applicants receive near-identical reasons? An explanation method giving different reasons for materially the same input isn't explaining anything, and this is the single most useful test available.
• Faithfulness. If you change the cited factor, does the decision change as the explanation implies?
• Coverage. Do a small number of reasons dominate all notices? That usually indicates a mapping problem rather than a genuine concentration.
• Actionability. Can the applicant do anything about the cited factor?
• Traceability. Does each reason correspond to an observed value?
• Consistency with understanding. Do the attributed factors match what you know about the model's learned relationships? A mismatch means one of the two is wrong.

The stability test deserves the emphasis. It's cheap — score pairs of very similar applicants and compare the reasons — and it catches the most common implementation failure, which is an attribution method producing unstable output that nobody has looked at because it always returns something plausible.

What to build

1. Per-decision attribution that traces to observed applicant values, for the adverse action requirement.
2. A reason-code mapping derived from the decision rather than from a rule.
3. Preprocessing traceability, so any reason can be tied back through imputation and transformation to a real fact.
4. Model-level understanding — documented relationships, directions, and edge behaviour, produced at development.
5. Monotonicity constraints where the direction is known, which improves both understanding and stability at little cost.
6. Full variable documentation, reviewed by people who can spot a problem.
7. Development documentation written at development, per our cold start analysis — documentation assembled during an examination is visibly assembled during an examination.
8. Outcome testing infrastructure for fair lending.
9. The ability to modify the model in a targeted way, for alternative searches.
10. Explanation validation, run and documented like any other validation.

Items three and ten are the ones almost universally missing, and both are cheap. Preprocessing traceability and explanation stability testing together would catch most of the accuracy failures described here, and neither requires changing the model.

Frequently asked questions

---

This guide is educational and does not constitute legal or compliance advice. Adverse action, model risk management, and fair lending requirements vary by institution type and regulator and continue to develop, particularly regarding the use of complex models in credit decisions. Consult qualified counsel and your model risk function.