What a Small Portfolio Cannot Learn From Itself | HL Hunt
What a Small Portfolio Cannot Learn From Itself
A lender writes 400 loans and sees 24 defaults, which is 6%. That observation is consistent with a true rate anywhere from 3.67% to 8.33%, a span of 2.27 times. Detecting a move from 6% to 5% would take 8,158 accounts per group, or 20.4 years of originations. Which means most of what a small operator believes about its own book is not evidence. It is the thing that filled the space where evidence would be.
What you'll learn
Four hundred loans tells you almost nothing
A lender originates 400 loans a year. Its default rate comes in at 6%, which is 24 defaults. The number looks solid. It is written in a board pack as a fact.
It is not a fact. It is one draw from a distribution. The standard error on a 6% rate at n=400 is 1.19 percentage points, so the 95% interval around that observation runs from 3.67% to 8.33%.
A 4% book and an 8% book are not minor variations of each other. One is a business and one is not. The same single year of experience is consistent with both, which means the observation does not distinguish between them, which means any decision justified by "our loss rate is 6%" is justified by something the data does not establish.
This is not a criticism of the lender's record-keeping. Per our balance accuracy analysis, data quality is a separate problem with separate remedies. The problem here is that the quantity being estimated is rare and the sample is small, and no amount of care in recording the 24 defaults makes 24 a larger number than it is.
Which is why policy changes cannot be evaluated
The practical consequence arrives when someone wants to know whether a change worked. Suppose a tightening is expected to move default from 6% to 5%, which is a 16.7% relative improvement and a large commercial result.
To detect a difference that size at conventional standards, you need about 8,158 matured accounts in each group.
| Effect to detect | Accounts needed per group | Years at 400 a year |
|---|---|---|
| 6% to 5% | 8,158 | 20.4 |
| 6% to 3% | 748 | 1.9 |
The gap between those two rows is the whole structure of the problem. A small operator can learn from a change that halves its loss rate. It cannot learn from a change that improves it by a sixth, not because it is not measuring carefully, but because twenty years of originations would be needed and the policy, the market and the firm will all have changed several times before then.
So small operators are not short of data in general. They are short of data about small effects. Per our launch analysis, the same arithmetic governs entry into a new state, and per our monitoring analysis it governs whether a rare operational defect can be found by sampling. The common factor is a low base rate against a small denominator.
What gets believed instead
The gap does not stay empty. Something fills the space where evidence would be, and it is worth being specific about what.
- Attribution to the most recent change. Losses fall the year after a tightening, so the tightening worked. The tightening may have done nothing and the fall may be the sampling variation that was always going to happen.
- Attribution to the most recent person. Per our institutional memory analysis, a rule that arrived with a particular hire gets credited to them and survives them.
- Rules that accumulate and never retire. Each was added after a loss and none can be shown to be unnecessary, because showing that would require the same 8,158 accounts. Per our rules analysis, this is why decline rules outnumber model declines in most operations.
- Borrowed benchmarks. Per our reference point analysis, a published industry figure becomes the standard because it is the only number anyone has, regardless of whether the portfolios are comparable.
- Overreaction to individual losses. One $14,000 charge-off in a book of 400 is 0.25 percentage points of loss rate and generates a rule, because it is visible and the 3.67-to-8.33 interval is not.
Item five is the mechanism that does the most damage. A single salient loss carries more weight in the room than the distribution it was drawn from, and the resulting rule declines a population that will never be observed again. Per our ambiguity analysis, the rule is also usually written loosely enough that its scope expands over time.
The event most operators never see
The second half of the problem is not about sample size in a year. It is about how many years there are.
Take an event with a 2% annual probability, which is loosely a one-in-fifty-year event: a regional employment collapse, a payment rail failure, a change in a funding market that closes it.
- Over a 10-year business life, the probability of seeing it at least once is 18.3%.
- Over 30 years, it is 45.5%.
So more than four in five ten-year operators never see it, and conclude from direct experience that it does not happen. Per our correlated failure analysis, that conclusion is not irrational given the evidence available to them. It is the correct inference from a sample that cannot contain the event.
Which makes the longest-serving operator the most confident and not the best informed
A firm that has traded for twelve years without encountering a tail event has twelve years of evidence that its practices are adequate, and that evidence is genuinely consistent with its practices being inadequate. The pattern is self-reinforcing: the operators who took the most exposure and were not hit are the ones still present to describe what works, and per our benchmarking analysis their practices become the reference others copy. The firms whose practices were tested and failed are not in the sample.
The same problem outside lending
This is not a credit phenomenon. It is an arithmetic one, and it shows up anywhere a business tries to learn from its own experience.
| Question | Why the firm's own data cannot answer it |
|---|---|
| Did raising prices 4% cost us volume? | Per our pricing analysis, seasonal variation is larger than the effect being measured |
| Is this marketing channel working? | Conversion differences of a few points need thousands of visits, not hundreds |
| Did the new hiring process improve retention? | Per our turnover analysis, three departures a year cannot separate a 22% rate from a 16% one |
| Is our fraud rate going up? | Per our fraud rate analysis, a low base rate makes monthly movement almost all noise |
| Which of these two quotes is better value? | Per our reference point analysis, two observations give almost no information about a distribution |
In every row, the honest answer is that the firm's own experience is not an adequate instrument, and the useful question becomes which other instrument to use.
What to do about it
The answer is not to give up on evidence. It is to stop asking a small sample to do work it cannot do, and to use the four substitutes that exist.
- Test only large effects. If a change is not expected to move the outcome by a third or more, do not plan to measure it. Decide it on reasoning and accept that it stays unverified.
- Use faster-resolving proxies. A 90-day early-payment indicator accrues observations several times faster than a 24-month default, and per our point-in-time analysis it has to be defined as of the decision date to be worth anything.
- Pool with others. Industry data, bureau performance data and consortium files exist precisely because no individual participant has enough of its own. Per our attribute analysis, this is what a bureau is for.
- Reason from structure rather than from outcomes. A rule that is defensible from how the product works does not need an impact estimate; a rule justified only by a result you cannot measure has nothing holding it up.
- Retire rules on a schedule. Since you cannot prove a rule is unnecessary, the alternative is a default expiry, so the stock of rules does not grow monotonically. Per our decommissioning analysis, the same logic applies to models.
- Write down the interval, not the point. A board pack that says "6%, with a range of 3.7% to 8.3% on this year's volume" supports different decisions from one that says "6%".
- Plan for the event you have not seen. Per our variance analysis, capacity for a shock has to be sized from structure, because the experience that would size it is missing by construction.
Item six is the cheapest and has the largest effect on the quality of decisions. Reporting the interval alongside the estimate costs one line and changes what the number can be used to argue. It is also the single thing most likely to stop a rule being added after one bad quarter, because the bad quarter visibly sits inside the range the previous quarters already implied.
Item one is the one people resist, and it is the honest position. A great many operational decisions cannot be evidenced at the scale they are made, and pretending otherwise produces false confidence rather than knowledge. A decision made on structural reasoning and labelled as such is more defensible than the same decision dressed in a measurement that could not have detected the effect it claims to have found.
Frequently asked questions
A 6% rate on 400 loans is consistent with anything from 3.67% to 8.33%, a 2.27-fold span. It cannot separate a 4% book from an 8% book.
Only for large effects. A 6% to 5% move needs 8,158 accounts per group, or 20.4 years; a 6% to 3% move needs 748, or 1.9 years.
Because a 2% annual event appears in only 18.3% of ten-year business lives, so most operators correctly infer from their own data that it does not happen.
Faster-resolving proxies, pooled industry data, structural reasoning, and scheduled rule retirement. And report the interval, not the point.
Key takeaways
- Observing 24 defaults in 400 loans is consistent with a true rate of 3.67% to 8.33%, a 2.27-fold span.
- That single range covers both a viable 4% book and an unviable 8% one.
- Detecting a 6% to 5% improvement needs 8,158 accounts per group, which is 20.4 years at 400 a year.
- Detecting a 6% to 3% improvement needs only 748, so small operators can learn from large effects alone.
- A 2% annual event appears in only 18.3% of ten-year business lives, so most operators never observe one.
- The survivors who took the most exposure and were not hit become the reference others copy.
- A single salient loss carries more weight in the room than the distribution it was drawn from.
- Report the interval rather than the point estimate; it costs one line and changes what the number can argue.
This analysis is educational and does not constitute financial, investment, or statistical advice. All figures are stylized illustrations using assumed origination volumes, default rates, effect sizes, and event probabilities, and do not describe any particular portfolio or business. Sample size and power calculations use conventional significance and power conventions and will differ under other assumptions, correlated outcomes, or clustered data. Consult a qualified adviser before relying on any approach described here.