Modeling Rare Defaults: Imbalance and Calibration

When 3% of loans default, a model that predicts no defaults is 97% accurate and worthless. The usual fix, training on a 50/50 sample, helps the model find defaulters but leaves its scores meaning something else: a score of 0.20 corresponds to a true default probability of 0.77%. Ranking metrics will not reveal the problem. Every price, limit and loss forecast built on the raw scores will.

Why defaults are hard to model

In most credit portfolios default is rare. If 3% of loans default, a dataset of 10,000 loans contains 300 defaults and 9,700 loans that performed. The model is trying to learn the pattern of the 300 from a sample dominated by the 9,700. Three problems follow, and the common fixes for the first one often create the second.

• Accuracy is meaningless. A model that predicts “no default” for every applicant is 97% accurate and useless.
• Rebalancing distorts probabilities. Oversampling defaults or undersampling good loans to reach a 50/50 sample helps some algorithms find the minority pattern, but the scores that come out no longer mean what they appear to mean.
• Few events limit what the data can support. Per our small numbers analysis, a segment with 60 defaults cannot support a model with dozens of variables, however many good loans it has.

The accuracy paradox

Accuracy counts correct predictions. When one outcome is 97% of the data, predicting that outcome always scores 97%. A model that correctly identifies half the defaulters while wrongly flagging 5% of good loans scores lower on accuracy, about 93.7%, despite being enormously more useful.

The fix is to measure what the decision needs. Ranking quality, measured by the area under the ROC curve or the Gini coefficient, asks whether defaulters score higher than non-defaulters. Calibration asks whether a predicted 4% default rate turns out to be about 4%. The Brier score, the mean squared difference between predicted probability and outcome, combines both. A model that predicts 3% for everyone has a Brier score of about 0.029, which sets the baseline any real model must beat. Per our model validation analysis, a credit model is validated on discrimination and calibration separately, because each can fail while the other looks fine.

What rebalancing does to the scores

Suppose a model is trained on a sample rebalanced to 50% defaults when the true rate is 3%. The model learns that roughly half its training cases default, and its scores sit around that base rate. A score of 0.20 from such a model does not mean a 20% chance of default.

Score from the 50/50 model	Corrected probability at a 3% true rate
0.20	0.77%
0.40	2.02%
0.50	3.00%
0.60	4.43%
0.80	11.01%

The correction is a prior adjustment. Convert the score to odds, multiply by the ratio of the true odds of default to the sample odds, and convert back. For a logistic regression, King and Zeng showed the slope coefficients from such a sample are consistent and only the intercept is wrong; with a 50/50 sample and a 3% true rate, the intercept should fall by ln(0.97 ÷ 0.03), about 3.48. For other algorithms the same odds correction applies to the output, though it is exact only under the assumption that sampling depended on the outcome alone.

The ranking is unchanged; the probabilities are not. The correction is monotonic, so the AUC and Gini are identical before and after. That is exactly why the mistake survives: a validation report focused on ranking will show nothing wrong, while every price, limit and loss forecast built on the raw scores is badly off.

What the error costs

Use of the score	Using the raw 0.20	Using the corrected 0.77%
Expected loss on a $10,000 loan at 60% loss given default	$1,200	$46
Approve if predicted default is below 5%	Only applicants scoring below 0.05: almost no one	Applicants scoring below about 0.63
Portfolio loss forecast	Roughly the sample rate of default	Roughly the true rate

A lender pricing from raw scores would charge for $1,200 of expected loss on a loan that carries $46. It would decline most good applicants and lose them to competitors, or, if a cutoff was tuned by trial and error to compensate, would carry a policy no one can explain. Per our price decomposition analysis and loss forecasting analysis, the expected loss term in price and in reserves needs a calibrated probability, not a rank.

Synthetic oversampling

SMOTE, introduced by Chawla and co-authors, creates synthetic minority cases by interpolating between real defaulters and their nearest neighbors. It can help some algorithms draw a better boundary around a small class. It also has costs that matter particularly in credit.

• It miscalibrates by construction. Recent work in clinical prediction, which shares credit's rare-outcome structure, found that imbalance corrections including SMOTE produced systematically overestimated risks with little or no gain in discrimination.
• Synthetic borrowers may not be plausible. Interpolating between two defaulters can produce combinations of income, tenure and utilization that no real applicant has.
• It interacts badly with validation. Oversampling before splitting data leaks synthetic copies of test cases into training, inflating performance.

For most credit applications, a better default is to train on the natural distribution with a method that handles imbalance well, such as gradient boosting with class weights, then check calibration and recalibrate if needed with Platt scaling or isotonic regression, which Niculescu-Mizil and Caruana showed can repair the probabilities of many learners. Per our monotonic constraints analysis, constraints that encode known risk directions also help a model learn from few events without inventing patterns.

How many defaults are enough

In logistic regression, a widely cited simulation study by Peduzzi and co-authors found estimates became unreliable when there were fewer than about 10 events per candidate variable. With 2,000 loans and a 3% default rate, the 60 defaults support about 6 variables by that rule. King and Zeng also showed that in small samples of rare events, logistic regression tends to underestimate event probabilities, and proposed a finite-sample correction.

• Pool carefully. Per our point-in-time data analysis, adding older vintages increases events, but only if the data reflects what was known at each decision.
• Use the rejected population with care. Per our reject inference analysis, a model trained only on approved loans sees fewer defaults than the applicant population would produce.
• Penalize. Regularization shrinks coefficients toward zero and reduces overfitting when events are scarce.
• Monitor the base rate. Per our population stability analysis, a calibrated model drifts when the true default rate moves, and recalibration is needed even when ranking holds.

The case for rebalancing

• Some algorithms need it. Methods that optimize accuracy or treat errors symmetrically can ignore a 3% class entirely; rebalancing or class weights force them to learn it.
• Ranking may be all that matters. A lender that sets cutoffs from observed outcomes by score band, rather than from predicted probabilities, is less exposed to miscalibration.
• Case-control sampling saves money. When labels or features are expensive to obtain, keeping all defaults and a sample of performing loans is efficient, as King and Zeng emphasize, provided the correction is applied.

None of these is an argument against calibration. They are arguments for rebalancing as a training choice followed by a correction, not as a substitute for one. The testable implication is simple: group loans by predicted default, compare each group's predicted rate with its realized rate, and if a model trained on rebalanced data has not been corrected, the predicted rates will sit far above the realized ones in every band. Per our credit policy analysis, a policy built on those predictions inherits the error.

Frequently asked questions

References

1. King, G., & Zeng, L. (2001). Logistic Regression in Rare Events Data. Political Analysis, 9(2), 137–163.
2. Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16, 321–357.
3. Niculescu-Mizil, A., & Caruana, R. (2005). Predicting Good Probabilities with Supervised Learning. Proceedings of the 22nd International Conference on Machine Learning, 625–632.
4. Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review, 78(1), 1–3.
5. Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996). A Simulation Study of the Number of Events per Variable in Logistic Regression Analysis. Journal of Clinical Epidemiology, 49(12), 1373–1379.
6. van den Goorbergh, R., van Smeden, M., Timmerman, D., & Van Calster, B. (2022). The Harm of Class Imbalance Corrections for Risk Prediction Models. Journal of the American Medical Informatics Association, 29(9), 1525–1534.

---

This guide is educational and does not constitute model risk or regulatory advice. Default rates, scores, loss given default, cutoffs and sample sizes are stylized illustrations and do not describe any particular lender or model. The prior correction is exact under outcome-based sampling assumptions; validate any correction against realized outcomes before relying on it.