Proxy Variables: Testing What a Model Is Really Using | HL Hunt
Proxy Variables: Testing What a Model Is Really Using
Almost every useful credit variable correlates with something, because financial outcomes are not randomly distributed across a population. So correlation alone cannot be the test, and a rule excluding every correlated input excludes nearly all of them. The sharper question is whether a variable’s predictive power lives in the part that tracks a characteristic or in the part that does not, and that is directly testable. Remove the tracking part, refit, and see what survives. The answer is a number rather than an argument, which is what makes it usable as a control.
What you'll learn
What makes a variable a proxy
Every model input is a stand-in for something. A credit score stands in for repayment behaviour, which stands in for capacity and willingness, neither of which is observable. That is not a defect; it is how measurement works.
A problematic proxy is narrower: a variable whose predictive power comes substantially from its correlation with a characteristic the model is not permitted to use, rather than from anything it independently measures.
The distinction matters because the two look identical in a model's output. Both improve fit. Both have a plausible story attached. Per our governance analysis, the difference is not visible in the coefficient and has to be established by a separate test.
The common error is to treat correlation with a protected characteristic as itself disqualifying. Almost every useful credit variable correlates with something, because financial outcomes are not randomly distributed across a population. A rule excluding every correlated variable excludes nearly all of them, which is why the question has to be sharper than that.
The residualization test
The sharper question is this: is the predictive power in the part of the variable that tracks the characteristic, or in the part that does not? That is directly testable.
Suppose a variable correlates 0.62 with a characteristic the model may not use:
- R squared: 0.62 squared = 0.38
- Share of the variable's variance explained by the characteristic: 38%
- Share that is something else: 62%
Now remove the explained part. Regress the variable on the characteristic, keep the residual, and refit the model using the residual in place of the original variable. The residual is by construction uncorrelated with the characteristic, so whatever predictive power survives is coming from somewhere else.
| Result | Interpretation |
|---|---|
| Gini 0.48 falls to 0.47 | The predictive content is not in the proxy channel; the variable measures something real |
| Gini 0.48 falls to 0.39 | The predictive content largely was the proxy channel |
This test answers a question that inspection cannot. Looking at a variable and asking whether it seems like a proxy produces an argument. Residualizing it and refitting produces a number, and per our validation analysis a number is what a model risk function can act on.
What the test does not settle
A variable can pass residualization and still be objectionable. If the residual is uncorrelated with the characteristic but the variable as used still produces materially different outcomes across groups, the disparity is real regardless of the mechanism. Residualization tells you where the predictive power lives; it does not tell you whether the resulting decisions are acceptable. Those are two questions and both need answering.
Where proxies enter
Rarely through a variable anyone would defend if it were named plainly. Almost always through one of these routes:
- Geography. Anything derived from where someone lives carries everything else that is distributed geographically, and aggregation does not remove it.
- Institution identity. Which bank, lender, or provider someone uses carries the composition of that institution's customer base.
- Product history. Per our thin file analysis, which products someone has held reflects which products were marketed and available to them.
- Channel and device. Per our channel analysis, how someone reached you correlates with a great deal about them.
- Interactions and engineered features. Two acceptable variables combined can produce something that tracks a characteristic more closely than either alone.
- Model inputs from vendors. A purchased attribute or score is a composite whose components you may not be able to inspect at all.
- Timing and behaviour. When applications are submitted, how long forms take, how many sessions are used.
Item five is the one that defeats variable-by-variable review. A model examined input by input can pass while an interaction term reconstructs the excluded characteristic from components that are individually fine. This is why the testing has to be done on the model as fitted rather than on the input list.
Item six is the one with the least available remedy. Per our attribute analysis, a vendor attribute is frequently a derived composite, and you are accountable for the decision regardless of whose model produced the input. The practical response is to run the same residualization test on the vendor attribute as on your own variables, which requires no visibility into how it was built.
When a correlated variable is legitimate
The case for keeping correlated variables is strong and it is not a technicality.
- Direct measures of the thing you are predicting. Payment history on existing obligations is the most predictive input in consumer credit and it correlates with everything, because outcomes correlate with everything.
- Removing them harms the people the exclusion was meant to protect. A model stripped of predictive power approves fewer people overall and prices less accurately, per our thin file analysis, and the borrowers who lose most are the ones whose good behaviour was the evidence removed.
- The alternative is not neutrality, it is a worse model. Per our rationing analysis, less information means more rationing, and rationing is not distributed evenly either.
- Some correlation is causal in the right direction. A variable can correlate with a characteristic because both are downstream of the same real circumstance, which makes it informative rather than substitutive.
The second bullet is the argument that should be made more often. The instinct that removing a correlated variable protects people assumes the counterfactual is the same decision without it. The actual counterfactual is a less accurate decision, and less accuracy falls hardest on applicants whose files are thin, per our verification analysis, because they have the least evidence to spare.
Running the testing properly
- Test the model as fitted, not the variable list. Interactions and engineered features are where proxies hide.
- Run residualization on every material input, including vendor attributes you cannot see inside.
- Record the performance loss for each. That table is the document that shows the work was done and is what a reviewer will ask for first.
- Search for a less correlated alternative before removing a variable outright. Per our overlay analysis, the narrowest change that addresses the problem is usually the right one.
- Measure outcome disparity separately from mechanism, because a variable can pass the mechanism test and still produce unacceptable outcomes.
- Re-test after every refit. Per our monitoring analysis, a variable that was not a proxy at fitting can become one as populations shift, and nothing announces it.
- Document why each variable was kept, not only which were removed, because the kept ones are what a decision is made on.
- Keep the test reproducible. Per our audit trail analysis, an analysis that cannot be re-run is an assertion rather than a control.
Item six is the one most programmes lack entirely. Proxy testing is typically treated as a one-time step at model build. The relationship between a variable and a characteristic is a property of the population, not of the variable, so it drifts, and a model validated once is a model validated against a population that no longer exists.
Item four is where the most value is created. Finding a variable that captures the same signal with less correlation preserves the model's accuracy and reduces the exposure, which is a strictly better outcome than either keeping the original or removing it and accepting the loss.
Proxy testing as a standing control
HL Hunt AI Underwriting runs residualization tests across every material input including purchased attributes, records the performance loss for each, re-runs the full suite on every refit rather than at build only, and reports mechanism and outcome disparity as separate results.
What the report should contain
| Element | Why a reviewer asks for it |
|---|---|
| Correlation of each input with each characteristic tested | Establishes what was looked at, including what was found to be fine |
| Performance loss under residualization, per variable | The mechanism finding, expressed as a number |
| Alternatives considered and why they were rejected | Shows the narrowest remedy was sought before the broadest |
| Outcome rates by group, before and after any change | The disparity finding, which is separate from mechanism |
| The business justification for each retained variable | Retention is a decision and needs a reason on the record |
| Re-test schedule and last run date | Shows this is a control rather than a one-time exercise |
The first row is more useful than it looks. A report listing only the variables that failed leaves a reviewer unable to tell whether the rest were tested and passed or were never examined. Recording the passes is what converts a set of findings into evidence of a process.
A test you can hand to a reviewer
Get started with HL Hunt AI Underwriting for reproducible residualization runs across every input, a retained-variable justification record rather than a removal list, and separate mechanism and outcome reporting on a schedule instead of at build only.
Frequently asked questions
No. Almost every predictive credit variable correlates with something. The question is whether the predictive power comes through that correlation or from something else.
Regress the variable on the characteristic, keep the residual, and refit. If performance barely moves the variable measures something real; if it collapses, it was working through the proxy.
Through geography, institution identity, product history, channel, vendor attributes, and above all interactions, which is why the model as fitted must be tested rather than the input list.
Often not, because the counterfactual is a less accurate model, and thin-file borrowers have the least evidence to spare. Finding a less correlated alternative is usually better.
Key takeaways
- Correlation with a characteristic does not make a variable a proxy; nearly every predictive credit input correlates with something.
- A problematic proxy is one whose predictive power comes substantially through that correlation rather than from what it independently measures.
- Residualize the variable and refit: a Gini falling 0.48 to 0.47 means the content is elsewhere, 0.48 to 0.39 means it was the proxy channel.
- Residualization identifies the mechanism and does not settle whether the resulting outcome disparity is acceptable; both need testing.
- Interactions and engineered features defeat variable-by-variable review, so test the model as fitted.
- Run the same test on purchased vendor attributes, since accountability for the decision does not transfer with the input.
- Removing a correlated variable usually produces a less accurate model, and the loss falls hardest on thin-file applicants.
- Re-run the suite on every refit, because the correlation is a property of the population and drifts without announcing it.
This guide is educational and does not constitute legal, compliance, or model risk advice. All figures are stylized illustrations and do not describe any particular model or portfolio. Fair lending requirements, permissible testing methodologies, the treatment of protected characteristic data for testing purposes, documentation expectations, and model risk management standards vary by institution type and jurisdiction and change over time. Consult your fair lending, compliance, legal, and model risk functions before designing or acting on any testing programme.