Point-in-Time Data: Why Models Validate Well and Underperform | HL Hunt
Point-in-Time Data: Why Models Validate Well and Underperform
A credit file field called current address tenure, pulled today for a decision made 18 months ago, reads up to 18 months higher than the underwriter saw. Every value is correct and nothing is missing, which is why data quality checks never catch it. On a stylized rebuild, Gini falls from 0.54 to 0.41 once attributes are reconstructed as of decision date, meaning 24% of measured performance was never there.
What you'll learn
- The field that means two different things
- What it costs in measured performance
- The opposite error, which is also common
- Where the leakage comes from
- How to find it in a model you already have
- The case against perfect point-in-time discipline
- What to record for every model
- Frequently asked questions
The field that means two different things
A credit file holds a field called current address tenure. It is a useful predictor: people who have lived somewhere a long time behave differently from people who moved last month.
Now build a model on a sample of decisions made 18 months ago, pulling that field from today's file. For an applicant who has not moved since, the field now reads 18 months higher than it did on the day the decision was made. The model learns the relationship between tenure and outcome using a tenure value that did not exist at decision time.
This is not a data quality problem. Every value is correct. The file is accurate as of today, the join worked, nothing is missing. The problem is that the model is trained on a world that the production model will never see, because in production the field is always as of now and now is always the decision date.
And tenure is the benign case, because the error has a known direction. Per our attribute analysis, the fields that cause real damage are the ones whose today-value encodes the outcome: a delinquency count that includes the delinquency on the very account being modelled, a utilization figure measured after the loan was drawn, an inquiry count that includes inquiries the applicant made because they were declined.
What it costs in measured performance
The symptom is a model that validates well and underperforms in production, and the gap is measurable if you rebuild the sample properly.
| Build | Gini | What it means |
|---|---|---|
| Attributes pulled as of today | 0.54 | What the development document reports |
| Attributes rebuilt as of each decision date | 0.41 | What the model will actually deliver |
| Difference | 0.13 | 24% of apparent performance was leakage |
The figure matters more than it looks, because the whole decision rests on it. A cut-off set to achieve a target approval rate at a target loss rate was calibrated on 0.54. At 0.41 the same cut-off produces a higher loss rate at the same approval rate, or a lower approval rate at the same loss rate, and per our launch analysis that difference is large enough to turn a portfolio that was priced to work into one that is not.
It also explains a pattern that is usually blamed on something else. A model that degrades immediately on deployment rather than drifting gradually is not experiencing population drift. Per our monitoring analysis, drift is slow and shows up in the input distributions. Leakage shows up on day one, at full size, with the inputs looking exactly as expected.
The opposite error, which is also common
Point-in-time discipline cuts both ways. The first error uses data from after the decision. The second uses data from too long before it.
A bureau attribute refreshed monthly and consumed by a decision engine on day 3 of the cycle is up to 33 days stale. The underwriter believes they are looking at a current balance and are looking at one from last month, which is a real difference on an applicant whose position changed in the interval.
- In training, using a 33-day-stale attribute is correct, because that is what production will have.
- In production, it is a 33-day blind spot, and it is the window in which per our stacking analysis a borrower can take several loans that none of the lenders can see.
So the two errors require different fixes, and conflating them produces a bad one. Training must reproduce the production latency exactly, including the staleness. Production should reduce the latency where it can. A team that fixes staleness in production without rebuilding the model has changed the input distribution the model was fitted on, which is per our validation analysis a change requiring revalidation rather than an improvement requiring none.
A model that validates at 0.54 and runs at 0.41 was never a 0.54 model
Rebuilding attributes as of each decision date is the difference between measured and delivered performance. HL Hunt AI Underwriting builds every development sample from as-of snapshots, so the development document reports the number the model will produce.
Where the leakage comes from
Some sources are structurally unable to give you a historical view, and knowing which is most of the work.
| Source | Can it be reconstructed as of a past date? |
|---|---|
| Archived bureau pull, stored at decision time | Yes. This is the clean case and the reason to archive pulls |
| Bureau archive product | Usually yes, at a cost, and worth it |
| Live bureau re-pull | No. It returns today, and nothing in the response says so |
| Bank transaction data | Partly. Per our open banking analysis, the transactions are dated but the derived attributes were computed later |
| Internal customer master | Rarely. Most are overwritten in place with no history |
| Fraud and identity flags | Almost never. Flags are added retrospectively, which is exactly the problem |
| Verification outcomes | Per our verification analysis, yes if the result was stored, no if only the conclusion was |
The internal customer master is the worst offender
It is the easiest table to join to, it is owned in-house, it is free, and it is almost always maintained as current state rather than as history. An address, employer, income figure, segment code or risk tier read from it is the value as of today, not as of the decision, and because the table is internal nobody treats it as an external dependency requiring an as-of query. Per our institutional memory analysis, the knowledge that a particular column is overwritten in place typically lives with one person and is not written down anywhere the next modelling team will find it.
How to find it in a model you already have
- Rebuild one attribute as of decision date and compare the distributions. If they differ, the field is not point-in-time, and you now know the direction and size.
- Check the sign and magnitude of every coefficient against the hypothesis. A variable that is implausibly strong is the leading indicator of leakage, not of insight.
- Hold out by time rather than at random. A random split shares the leakage with the holdout and hides it; an out-of-time split does not.
- Look for any field that could have been updated by the outcome. Delinquency counts, utilization, inquiry counts, collection flags and account status all can be.
- Compare development performance against the first three months in production. Per our monitoring analysis, an immediate gap of this size is leakage rather than drift.
- Document the as-of date of every input. Per our data source analysis, a model whose inputs have no recorded as-of date cannot be audited for this at all.
- Ask whether the rules have the same problem. Per our rules analysis, a policy rule calibrated on a leaked attribute carries the error without anyone calling it a model.
Item three is the single most effective test and the cheapest. A random holdout from a leaked sample is leaked in the same way, so it confirms the leaked performance. Splitting by time forces the holdout to be a period the development data could not have known about, which is the only split that reproduces the production situation.
Item two is where the finding usually comes from. A coefficient that is too good is the most reliable symptom there is, and per our proxy analysis the reflex of asking why a variable is strong, rather than being pleased that it is, catches both leakage and proxying.
The case against perfect point-in-time discipline
It is not free, and sometimes it is not worth it
Archive bureau products cost money per record. Building as-of snapshots of internal tables requires history that may not exist, which means instrumenting now and waiting a year to have a sample. For a small portfolio, the choice is frequently between a leaked model available this quarter and a clean model available in four quarters, and the leaked model with a known, measured and documented degradation can be the better business decision. The defensible version of that choice requires the measurement. Rebuilding one or two attributes as of date is enough to size the gap, and a model deployed with a documented expectation of 0.41 rather than a claimed 0.54 is calibrated honestly even if it was built dirty. Per our validation analysis, what makes it indefensible is not the leakage. It is reporting the leaked number as the expected one.
There is also a case for using some future-dated information deliberately, and it is narrower than it sounds. A target variable must be measured after the decision, which is the one place where looking forward is required rather than forbidden. The discipline applies to predictors and not to outcomes, and a team that over-applies it ends up defining a target so close to the decision date that it no longer measures the thing being predicted.
What to record for every model
| Record | Why |
|---|---|
| As-of date of every input, per observation | Without it, this cannot be audited at all |
| Whether each source was archived or re-pulled | Re-pulled sources are leaked by construction |
| Production latency per attribute, in days | Training must reproduce it, not remove it |
| Out-of-time holdout performance, separately from random | The only split that detects the problem |
| Which internal tables are current-state only | The list is short and nobody has written it down |
| Expected production performance, stated as a number | Per our adverse action analysis, it is what the decision is defended against |
| First-90-day performance against that expectation | Per our decommissioning analysis, the gap decides whether the model should stay live |
The third row is the one teams get backwards. Production latency is a constraint to reproduce in training, not a defect to engineer out of the development sample. A model trained on same-day data and deployed against 33-day-old data is being asked a different question from the one it was fitted to answer, and it will answer the question it was fitted to.
The date on the data is part of the data
A field pulled today for a decision made 18 months ago has the same name and a different meaning. HL Hunt AI Underwriting records the as-of date of every input, holds out by time rather than at random, and reports what the model will deliver.
Frequently asked questions
Every predictor reconstructed as it stood on the decision date. The same field name can mean a different quantity today than it did then.
On a stylized rebuild, Gini falls 0.54 to 0.41, so 24% of apparent performance was leakage, and the cut-off was calibrated on the wrong number.
Archived pulls and bureau archive products, yes. Live re-pulls, internal customer masters and retrospective fraud flags, generally not.
Yes, if the gap is measured and the expected performance is stated honestly. Reporting the leaked number as the expected one is what is indefensible.
Key takeaways
- A tenure field pulled today for an 18-month-old decision overstates what the underwriter saw by up to 18 months.
- Every value is correct, so data quality checks do not catch it; the problem is the as-of date, not the value.
- On a stylized rebuild, Gini falls from 0.54 to 0.41, meaning 24% of apparent performance was leakage.
- A model that degrades on day one rather than gradually is showing leakage, not population drift.
- The opposite error is staleness: a monthly attribute used on day 3 is up to 33 days old in production.
- Training must reproduce production latency rather than engineer it out of the development sample.
- Internal customer masters are the worst offender because they are free, easy to join to, and overwritten in place.
- An out-of-time holdout is the cheapest and most effective test; a random holdout shares the leakage.
This guide is educational and does not constitute financial, legal, or compliance advice. All figures are stylized illustrations using assumed Gini coefficients, sample ages, refresh cycles, and latency windows, and do not describe any particular model or portfolio. Model risk management expectations, fair lending requirements, adverse action notice obligations, and permissible use of consumer report data are governed by federal and state law and by supervisory guidance that varies by institution type and changes over time; consult qualified counsel and model risk advisers before relying on any approach described here.