Population Stability Index: What a PSI of 0.05 Does Not Tell You
Population Stability Index: What a PSI of 0.05 Does Not Tell You
Two shifts in an applicant population read 0.052 and 0.046 on the population stability index, both comfortably below the 0.10 threshold that convention calls insignificant. One cut the approval rate from 60% to 51% and raised the approved bad rate. The other raised approvals to 69% and lowered it. The index gave them the same number because it measures how far a distribution moved and not which way, and at 400 applications a month pure noise reads about 0.04.
What you'll learn
What the index measures
The population stability index compares the distribution of a variable, usually a score, between a baseline period and a current one. Split the score into bins, take each bin's share in both periods, and sum (current share minus baseline share) times the log of their ratio across bins. A perfectly stable distribution scores zero. The convention in scorecard practice is that below 0.10 means no meaningful shift, 0.10 to 0.25 means a moderate one, and above 0.25 means the population has changed enough to investigate.
Those thresholds are not a statistical test. They are a rule of thumb that spread through credit scoring textbooks and vendor documentation, and they carry no information about sample size, bin count, or what kind of shift occurred. Each of those matters, and each can produce a reading that is confidently wrong.
The index cannot see direction
Take a score split into ten equal deciles in the baseline, with bad rates running from 20% in the lowest decile to 2% in the highest, and a policy that approves the top six deciles. Baseline approval is 60% and the approved book's bad rate is 4.42%.
Now two different things happen.
| Shift | PSI | Approval rate | Approved bad rate |
|---|---|---|---|
| Baseline | 0.000 | 60% | 4.42% |
| A marketing campaign brings in lower-scoring applicants | 0.052 | 51% | 4.71% |
| A partner channel brings in higher-scoring applicants | 0.046 | 69% | 4.24% |
A nine-point change in approval rate is not a minor operational event. Per our denominator analysis, it changes the base of every ratio the lender reports, and per our channel analysis it is exactly the kind of shift a new acquisition channel produces. The index reported it as nothing to see, because the convention says 0.05 is nothing.
The symmetry is by construction. The index is a divergence measure: it penalizes movement in either direction equally. That makes it good at one thing, which is flagging that the population is not the one the model was built on, and bad at the thing people use it for, which is deciding whether the change matters.
The index also cannot see sample size
A lender scoring 400 applications a month against a ten-bin baseline will see a positive index every month even if nothing has changed, because 400 draws from a distribution never reproduce it exactly.
| Monthly volume | Median PSI with no real shift | 95th percentile with no real shift |
|---|---|---|
| 400 | 0.021 | 0.042 |
| 5,000 | 0.002 | 0.003 |
At 400 applications, pure sampling noise regularly produces a reading of 0.04, which is close to the marketing shift above. A useful rule of thumb is that the index expected from noise alone is about (number of bins minus one) divided by the sample size: with ten bins and 400 observations, about 0.022, which matches the table. At 5,000 observations it is 0.002, and a reading of 0.05 is real.
So the same 0.05 means nothing at one lender and something at another. Per our small numbers analysis, a small operation cannot distinguish a real shift from a noisy month on this index, and per our launch analysis a lender entering a new market with low initial volume will see the index jump for reasons that have nothing to do with the population.
Bin count changes the number too
The marketing shift reads 0.049 with five bins, 0.052 with ten, and 0.062 with twenty. The differences are modest here and can be large where the shift is concentrated in a narrow score range that coarse bins blur. A threshold of 0.10 applied to a five-bin index and a twenty-bin index is two different thresholds, and a monitoring program that changed its binning at some point has a discontinuity in its history that will look like a population event.
What the index is good for
None of this means the index should be dropped. It should be used for what it does.
- As an alarm that the population moved, prompting a look at the bins, not as a verdict on whether the move matters.
- On inputs as well as scores. Per our monitoring analysis, a shift in one input variable with a stable score usually means two shifts are cancelling, which is worth knowing.
- With a noise floor set from volume. Compute the expected index from sampling alone and treat readings below twice that as noise.
- Alongside direction. Report the approval rate and the expected bad rate of the approved population beside the index, so the two shifts above are told apart.
- Against a baseline that is itself point-in-time. Per our point-in-time analysis, a baseline rebuilt from today's data is not the development population.
- As a trend, not a threshold. A reading that rises from 0.01 to 0.04 over six months at 5,000 a month is a real drift; a single 0.08 at 400 a month is probably not.
Item four is the one that would have caught both shifts in the table. The index said 0.05 twice. The approval rate said 51% and then 69%. A dashboard showing both would have prompted the right question each time, and per our alert analysis a single threshold on a direction-blind index produces either too many alarms or the wrong ones.
A stability index of 0.05 cut approvals by nine points
The index reports how much a distribution moved and not which way. HL Hunt AI Underwriting monitors score and input distributions with a noise floor set from volume, and reports approval rate and expected approved bad rate beside the index so direction is never lost.
What a shift should trigger
| Finding | Response |
|---|---|
| Population shifted, score still ranks correctly | No model change. Per our policy analysis, the cut-off may need to move to hold approval or loss targets |
| Population shifted because of a channel or campaign | Treat as a business decision already made; monitor that channel's cohort separately |
| Inputs shifted, score did not | Investigate for cancelling changes or a data definition change |
| Score shifted, performance not yet observable | Per our loss forecasting analysis, use early indicators by vintage rather than waiting for maturity |
| Score shifted and ranking degraded | Per our validation analysis, this is a validation event, and per our decommissioning analysis it may be a retirement one |
| Index rose when binning or baseline changed | A measurement artefact; recompute the history on one basis |
The first row is the common case and the one most often mishandled. A population that moved does not mean the model stopped working. It means the model is being asked about different people, and if it still orders them correctly the response is a policy adjustment rather than a rebuild. Per our rules analysis, where policy rules make most declines, the index on the model score may also be measuring the wrong thing entirely.
The defence of the thresholds
The 0.10 and 0.25 conventions persist because they are simple, because every vendor and auditor recognizes them, and because for a large lender at steady volume with consistent binning they work reasonably well as a first filter. A program that reported a different statistic would have to explain it to every examiner. The honest position is to keep the index and the conventions as the shared vocabulary, and to add the three things the index lacks: a noise floor from volume, a direction, and a consistent binning history. That is not a rejection of the convention. It is the convention with its assumptions written down.
A monitoring program that uses it properly
- Fix the bins and the baseline and record both, so the series is comparable over time.
- Compute the noise floor for each month's volume and show it on the chart.
- Report direction: approval rate and expected bad rate of approvals, beside the index.
- Compute the index per channel and per segment, because a stable total can hide two moving parts.
- Track inputs and score together.
- Use the index to decide what to investigate, and early performance by vintage to decide what to change.
- Per our champion-challenger analysis, apply the same monitoring to any challenger, so a shift is not mistaken for a model difference.
Item four is where the index earns its place. A total index of 0.03 with one channel at 0.18 is a channel problem that the total hides, and per our prequalification analysis a channel that sends a different population than the model expected is the usual origin of a widening gap between early decisions and final ones.
Use the index to decide what to look at, not what to change
A stable total can hide a channel at 0.18. HL Hunt AI Underwriting computes stability by channel and segment on fixed bins and a recorded baseline, and pairs every reading with the early performance by vintage that decides whether anything should change.
Frequently asked questions
A divergence measure between a baseline and current score distribution. The 0.10 and 0.25 thresholds are convention, not a statistical test.
It measures how far the distribution moved, not which way. Two opposite nine-point approval swings both read about 0.05, below the threshold.
Roughly (bins minus one) over sample size: about 0.022 at 400 applications, where a noise month can read 0.04, and 0.002 at 5,000.
Investigation, not automatic rebuild. If the score still ranks correctly, adjust policy; if ranking degraded, validate; if binning changed, recompute.
Key takeaways
- The population stability index measures how far a score distribution moved from baseline; the 0.10 and 0.25 thresholds are convention.
- Two opposite shifts read 0.052 and 0.046; one cut approvals from 60% to 51%, the other raised them to 69%.
- The index is direction-blind by construction, so it cannot say whether a shift matters.
- Expected PSI from noise alone is about (bins minus one) over sample size: 0.022 at 400 applications, 0.002 at 5,000.
- At 400 applications a month, a noise month regularly reads 0.04, close to the real shifts above.
- Bin count changes the reading: the same shift read 0.049, 0.052 and 0.062 at five, ten and twenty bins.
- Report approval rate and expected approved bad rate beside the index to restore direction.
- Compute the index per channel; a stable total can hide a channel at 0.18.
This guide is educational and does not constitute financial, legal, or compliance advice. All figures are stylized illustrations using assumed score distributions, decile bad rates, approval policies, and simulated sampling, and do not describe any particular model, lender, or portfolio. Monitoring conventions, threshold choices, and model risk management expectations vary by institution and supervisory guidance and change over time; consult qualified model risk advisers before relying on any approach described here.