Rules and Models: Which One Is Really Deciding | HL Hunt
Rules and Models: Which One Is Really Deciding
Validation, monitoring, documentation and fair lending testing are all directed at the model. Meanwhile the policy rules in front of it decline most of the applicants. On the volumes below, policy rules make 76% of declines and the model makes 24%, and the governance is distributed in almost exactly the opposite proportion. Rules are not the problem; ungoverned rules are, and the thing deciding three quarters of outcomes deserves at least the scrutiny given to the thing deciding one quarter.
What you'll learn
Who is actually making the decisions
Model risk management, validation, monitoring, documentation and fair lending testing are directed at the model. Meanwhile the policy rules sitting in front of it decline most of the applicants.
Take 9,400 monthly applications with a knockout rule layer before scoring:
- Declined by a policy rule before being scored: 31%, which is 2,914
- Remaining and scored: 6,486
- Declined by score: 14% of those, which is 908
- Total declines: 3,822
That asymmetry is the subject of this article. Per our governance analysis, the model is validated, monitored, documented and tested because it is identifiable as a model. The rule layer is configuration, so it is changed by a ticket and reviewed by nobody with the same rigour.
Per our ambiguity analysis, this is not an argument that rules are wrong. Several of them are legal requirements and most of the rest are defensible. It is an argument that the thing deciding three quarters of outcomes should be governed at least as carefully as the thing deciding one quarter.
What belongs in a rule and what belongs in a model
| Question | Where it belongs | Why |
|---|---|---|
| Is this lending permitted at all? | Rule | Age, jurisdiction, product eligibility. Binary and legal |
| Is this person who they say they are? | Rule | Identity is a gate, not a probability |
| Is there an active bankruptcy or deceased flag? | Rule | Per our prioritization analysis, a legal restriction expressed as a weight can be outvoted |
| How likely is this person to repay? | Model | Continuous, estimable, and the model is better at it than a threshold |
| Is one adverse item disqualifying? | Model | A rule here discards information the model would weigh |
| Is the debt-to-income ratio acceptable? | Contested | Usually a rule and arguably better as a model input |
| Does this look like fraud? | Both, separately | Per our stacking analysis, a different question needing a different model |
The fifth row is where most of the avoidable damage sits. A knockout on a single adverse item discards everything else about the applicant, and the model exists precisely to weigh that item against the rest. Per our threshold analysis, converting a continuous variable into a binary gate throws away the information in its magnitude and its age.
The sixth row is the live argument in most credit shops. A debt-to-income ceiling is a rule, and per our counteroffer analysis it produces the specific distortion that extending a term passes the test while making the loan worse. As a model input it would be weighed against everything else; as a ceiling it decides alone.
How the rule layer grows
- A loss happens and a rule is added. Specific, justified at the time, and permanent.
- Nobody owns the aggregate. Per our stipulation analysis, each rule is individually defensible and the total is nobody's responsibility.
- The rationale is lost. Per our institutional memory analysis, a rule nobody can justify survives review because nobody can say it is unnecessary.
- Rules are cheap to add. A configuration change rather than a model version, so no validation is triggered.
- Declines are invisible. Per our selection analysis, nobody observes what a declined applicant would have done, so a rule that declines good applicants generates no evidence against itself.
Point five is the asymmetry that makes the layer grow without limit. A rule that is too tight produces losses nobody sees; a rule that is too loose produces defaults everybody sees. Per our ambiguity analysis, when the cost of one error is attributable and the cost of the other is not, the set ratchets in one direction.
A falsifiable claim
A substantial share of rule-layer knockouts will decline applicants whose model score would have placed them comfortably above the cutoff. This is testable without lending anything: score the rule-declined population anyway and look at the distribution. If rule-declined applicants cluster at the bottom of the score range, the rules and the model agree and the layer is redundant but harmless. If a meaningful share scores well above the cutoff, the rules are declining people the model would have approved, and each such rule is a decision to override your own best estimate with a threshold nobody has validated.
Governing the rule layer
- Maintain an inventory. Every rule, its owner, its date, its rationale, and the last time it was reviewed. Most lenders cannot produce this.
- Score the rule-declined population anyway. Costs nothing, requires no lending, and produces the test above.
- Report decline volume by rule. Per our stipulation analysis, the rules doing the most work are frequently not the ones anyone would name.
- Give every rule an expiry date. Per our overlay analysis, a rule that must be renewed is reviewed, and a rule that persists by default is not.
- Test removals the way you test models. Per our testing analysis, waive a rule for a randomized share and compare outcomes.
- Run fair lending testing on the rule layer, per our proxy analysis, because a rule can be a proxy exactly as a model input can, and it is less likely to have been examined.
- Require accurate decline reasons from rules, per our adverse action analysis, which is harder than it sounds when a rule fires before scoring.
- Version the rule set and stamp decisions with the version, per our decommissioning analysis, because reconstructing a decision requires knowing which rules were live.
Item two is the highest-return item and the easiest to run. Scoring applicants you already declined requires no risk, no approval and no new data, and it directly answers whether the rule layer is agreeing with your model or overriding it.
Item four is the structural fix. An expiry date converts the default from retention to review, which is the only mechanism that counteracts the ratchet described above. Per our fixed cost analysis, requirement sets only shrink when something forces the question.
Order matters, and it is rarely deliberate
A rule that fires before scoring produces a decline with no score attached, which means no counteroffer can be computed, no reason code can be derived from the model, and the applicant cannot be reconsidered without rerunning the whole case. Rules that exist for legal or identity reasons belong first. Rules that are credit judgments belong after scoring, where the score is available to inform them and to populate the decline reason. In most systems the order reflects the order the rules were written.
Govern what is actually deciding
HL Hunt AI Underwriting holds the rule layer as versioned policy with owners, rationales and expiry dates, scores rule-declined applicants so the overlap with the model is visible, reports decline volume per rule, and runs disparity testing on rules as well as on model inputs.
What to report
| Metric | Why |
|---|---|
| Share of declines by rule against by score | The 76 against 24 figure, for your own book |
| Decline volume per individual rule | Identifies which single rule to examine first |
| Score distribution of rule-declined applicants | Whether the rules agree with the model or override it |
| Rules with no recorded rationale | The pruning candidates, per our institutional memory analysis |
| Rules not reviewed in the last year | The governance gap, stated as a count |
| Decline rate by rule across groups | Fair lending exposure in the layer least likely to have been tested |
The third row is the one that reframes the conversation internally. A chart showing rule-declined applicants spread across the score range, including well above the cutoff, makes the argument in this article in a single picture and is produced from data the lender already holds.
The last row is the one with the most exposure attached. Per our proxy analysis, a rule is as capable of being a proxy as a model variable, it is easier to construct carelessly, and because it is treated as configuration rather than as a model it is frequently the part of the decision process that has never been tested at all.
Score the ones you declined
Get started with HL Hunt AI Underwriting to run the no-risk test in this article on your own book: score the rule-declined population, see how much of it sits above your cutoff, and find the single rule worth examining first.
Frequently asked questions
In the worked case, 76% of declines come from policy rules and 24% from the model, which receives essentially all the governance attention.
Rules for binary legal questions and identity; the model for likelihood of repayment. A knockout on one adverse item discards everything else.
Because a tight rule produces losses nobody sees while a loose one produces visible defaults. One error is attributable and the other is not.
Score the rule-declined population anyway. If a meaningful share scores above the cutoff, the rules are overriding your own best estimate.
Key takeaways
- In the worked case policy rules make 76% of declines and the model makes 24%, with governance distributed the other way round.
- Rules belong on binary legal and identity questions; likelihood of repayment belongs in the model.
- A knockout on a single adverse item discards everything else about the applicant that the model would have weighed.
- A debt-to-income ceiling as a rule produces the distortion where extending a term passes the test while worsening the loan.
- Rule layers ratchet because a too-tight rule produces losses nobody observes and a too-loose one produces visible defaults.
- Scoring the rule-declined population costs nothing and shows whether the rules agree with your model or override it.
- An expiry date on every rule converts the default from retention to review, which is the only counter to the ratchet.
- Rules can be proxies exactly as model inputs can, and are frequently the part of the process never tested at all.
This guide is educational and does not constitute legal, compliance, or model risk advice. All figures are stylized illustrations using assumed application volumes and decline rates, and do not describe any particular lender. Which eligibility criteria are legally required, model risk management expectations, adverse action reason requirements for rule-based declines, record retention, and fair lending testing expectations for policy rules vary by institution type and jurisdiction. Consult your compliance, legal, and model risk functions before adding, removing, or reordering any decision rule.