Champion-Challenger: The Test That Cannot Finish | HL Hunt
Champion-Challenger: The Test That Cannot Finish
A challenger model is meant to prove itself in production. On 18,000 applications a year, detecting the improvement it was built for, 7.2% to 6.3% default, needs 12,197 accounts per arm, which is 16 years at a 10% allocation. Both models will be gone first. The design is still worth running, for reasons that have nothing to do with the comparison it is usually described as performing.
What you'll learn
The test cannot finish
A champion-challenger design routes some share of applications to a new model, keeps the rest on the incumbent, and compares the outcomes. It is the standard answer to how a model gets proven in production.
Put the usual numbers in it. 18,000 applications a year, a 46% approval rate, and a hope that the challenger takes default from 7.2% to 6.3%, which is a 12.5% relative improvement and a result worth having.
Detecting a difference that size at conventional standards requires about 12,197 matured accounts in each arm.
| Challenger allocation | Approvals a year | Years to accrue | Plus maturation |
|---|---|---|---|
| 10% | 828 | 14.7 | 16.0 years |
| 50% | 4,140 | 2.9 | 4.2 years |
And it is not only the default rate. Detecting a change in approval rate from 46% to 44% needs 9,712 per arm, which is 5.4 years at a 10% allocation, and approval rate is the fast-resolving metric in this design because it is observed immediately.
So the standard framing of champion-challenger is wrong at this scale. Per our small numbers analysis, the limitation is not the design but the arithmetic underneath it, and the same ratio of base rate to volume governs every attempt to learn from a small portfolio.
What a challenger actually proves
The design is still worth running. The justification is different from the one usually given, and stating it accurately changes how the challenger is set up and what is reported from it.
- It proves the pipeline. A model scoring live traffic for a year has demonstrated that its inputs are available at decision time, in the format it expects, within the latency budget.
- It proves the data exists. Per our point-in-time analysis, an attribute that was present in the development sample may not be retrievable in production, and a challenger is how that is discovered before the model is relied on.
- It exposes failure modes. Per our data source analysis, how a model behaves when a feed is late or missing is learned from production and not from a notebook.
- It establishes the operational runbook. Monitoring, alerting, score distribution reporting and override handling all exist before the model carries volume.
- It detects large regressions quickly. A challenger that is dramatically worse shows up fast, because large effects need small samples.
- It generates a disagreement set, which is the part with real analytical value and is covered below.
Item five is the one that is statistically sound and routinely undersold. The same arithmetic that makes a 0.9-point improvement undetectable makes a 3-point deterioration obvious, so a challenger is a good instrument for catching a model that is badly wrong and a poor one for confirming that a model is modestly right. Per our monitoring analysis, that asymmetry should determine what the challenger is monitored for.
The measurement that does work
Comparing aggregate outcomes needs thousands of accounts. Comparing decisions needs none, because every application produces two decisions immediately.
At a 10% allocation, 1,800 applications a year go to the challenger. If the two models agree on 91% of them, 162 decisions differ, and those 162 are the entire informational content of the experiment.
| Disagreement type | What to do with it |
|---|---|
| Challenger approves, champion declines | The valuable set. Per our reject inference analysis, outcomes are unobserved unless some are funded deliberately |
| Challenger declines, champion approves | Directly testable, because the account was funded and the outcome is observed |
| Same decision, very different score | Per our counteroffer analysis, matters where score drives price or structure rather than approval |
| Disagreement concentrated in one segment | Per our proxy analysis, the first thing to examine for a fair lending question |
| Disagreement driven by one attribute | Usually a definition difference rather than a modelling one |
The second row is the only comparison in this whole design that yields an answer in months rather than years. Where the challenger would have declined an account the champion approved, the outcome is observable, so the challenger's judgement on that slice can be scored directly. A small number of such cases carries more information than a much larger number of agreements.
The fifth row explains most disagreement in practice and is mistaken for insight. Two models that compute utilization or tenure on slightly different definitions will disagree on a stable, predictable set of applications, and that is a data lineage finding rather than a modelling one. Per our attribute analysis, reconciling the attribute definitions usually removes a large share of the disagreement before any modelling question is reached.
A challenger proves the pipeline, not the point estimate
Detecting a 0.9-point default improvement needs 12,197 accounts per arm, which is 16 years at a 10% allocation. HL Hunt AI Underwriting runs challengers for what they can establish, which is data availability, latency, failure behaviour and the disagreement set, and reports them that way.
Four ways the comparison gets broken
Even where the design is sound in principle, the implementation commonly destroys the comparison. These are the failures to check for.
| Failure | Effect |
|---|---|
| Allocation by segment rather than at random | The arms differ in population, so any outcome difference is unattributable |
| The challenger score visible to reviewers | Per our review analysis, manual decisions then blend both models and neither arm is clean |
| Policy rules applied after both models | Per our rules analysis, if rules make most declines, the models are being compared on a minority of decisions |
| Different cut-offs on the two models | You are comparing two policies, not two models, and cannot separate them |
| Allocation changed mid-test | Cohorts from different periods face different economic conditions |
The third row is the largest and least discussed. Per our rules analysis, where policy rules account for the great majority of declines, the model is deciding a minority of the file and a challenger comparison measures a difference on that minority while being reported as a comparison of the whole decision. A challenger that improves scoring on 24% of declines and is credited with the portfolio's performance is being misread.
The fourth row is the most common unforced error. A challenger deployed at the cut-off that produces the same approval rate is a different experiment from one deployed at its own optimal cut-off, and operations frequently do the second while interpreting the first.
The argument for deploying without a completed test
If the statistical comparison cannot finish, one honest conclusion is that the challenger should be deployed on the strength of its development evidence, its operational record and its structural reasoning, with the champion retired. That is a defensible position and better than the common alternative, which is running a challenger indefinitely at 10% while describing it as validation. What makes it defensible is saying so. A model deployed with a documented statement that its production performance has not been and cannot be measured at this volume, alongside an out-of-time development estimate and a monitoring plan for large regressions, is honestly governed. Per our validation analysis, what is not defensible is presenting a 16-year test that has run for 18 months as evidence that the model works.
How to set one up at this scale
- Allocate at random, at the application level, with the assignment recorded and immutable.
- Use the largest allocation you can tolerate. A 50% split answers in 4.2 years where 10% answers in 16, and the cost of a wrong challenger is bounded by monitoring for large regressions.
- Keep the challenger score invisible to reviewers and out of every downstream rule.
- Hold the cut-off policy identical unless comparing policies is the explicit purpose.
- Report the disagreement set every month, with the five breakdowns above, rather than reporting an outcome comparison that cannot yet mean anything.
- Fund a deliberate sample of challenger-approved, champion-declined applications, since that is the only way those outcomes are ever observed, and size it as a controlled cost rather than an experiment.
- Reconcile attribute definitions first. Per our attribute analysis, this removes disagreement that looks like signal.
- Set a decision date in advance, with the evidence that will be available by then, so the challenger does not run forever by default. Per our decommissioning analysis, a challenger with no end date becomes permanent infrastructure.
Item six is the only route to the question everyone actually wants answered. Whether the challenger safely approves applications the champion declines cannot be learned from a comparison, because those accounts are never funded. Per our reject inference analysis, statistical reconstruction of those outcomes is a modelling assumption rather than an observation, and a small funded sample is the one thing that converts it into data.
Item eight is the governance point. A challenger running at 10% with no decision date is not a test, it is a second model in production with no owner and no retirement plan, and per our decommissioning analysis it will accumulate the same dependencies as the champion while nobody is accountable for its performance.
The disagreement set is the experiment
At a 10% allocation, 162 decisions a year differ between champion and challenger, and those are the entire informational content. HL Hunt AI Underwriting reports the disagreement set by type and segment every month, instead of an outcome comparison that cannot yet mean anything.
Frequently asked questions
Not at realistic volumes. A 7.2% to 6.3% default improvement needs 12,197 accounts per arm, which is 16 years at a 10% allocation.
To prove the pipeline, data availability, latency and failure behaviour, and to catch large regressions quickly. Not to confirm a small improvement.
The disagreement set: about 162 differing decisions a year at a 10% allocation. The directly testable slice is challenger declines on champion approvals.
Segment-based allocation, reviewers seeing the challenger score, policy rules deciding most declines, different cut-offs, and changing allocation mid-test.
Key takeaways
- Detecting a 7.2% to 6.3% default improvement needs 12,197 matured accounts per arm.
- That is 16 years at a 10% challenger allocation and 4.2 years at 50% on 18,000 applications a year.
- Even a 46% to 44% approval rate change needs 9,712 per arm, or 5.4 years at 10%.
- What a challenger does establish is data availability, latency, failure behaviour and the runbook.
- It also catches large regressions fast, because large effects need small samples.
- The disagreement set is the real experiment: 162 differing decisions a year at a 10% allocation.
- Where policy rules make most declines, a challenger comparison measures a minority of the decision.
- A challenger with no decision date is not a test but a second production model with no owner.
This guide is educational and does not constitute financial, legal, or compliance advice. All figures are stylized illustrations using assumed application volumes, approval rates, default rates, agreement rates, and allocation shares, and do not describe any particular model or portfolio. Sample size calculations use conventional significance and power conventions and will differ under other assumptions or with correlated outcomes. Model risk management expectations, fair lending requirements, and documentation standards for model deployment are governed by federal and state law and by supervisory guidance that varies by institution type and changes over time; consult qualified counsel and model risk advisers before relying on any approach described here.