Quality Monitoring in Collections: What a Sample Can Find | HL Hunt
Quality Monitoring in Collections: What a Sample Can Find
Quality programmes usually begin with a scorecard. They should begin with a sample size, because a review programme that cannot detect the defect rate it cares about is an expense rather than a control. Reviewing 1.5% of 22,000 monthly contacts is 330 calls, which catches a one-in-two-hundred defect only 81% of the time; being 95% confident takes 598. That arithmetic says random sampling is the wrong primary tool for rare failures, and it points at what to review instead.
What you'll learn
Start with whether the sample can find anything
Quality monitoring usually starts with a scorecard. It should start with a sample size, because a review programme that cannot detect the defect rate it cares about is an expense rather than a control.
An operation handling 22,000 contacts a month and reviewing 1.5% of them is listening to 330 calls. What can 330 calls find?
| True defect rate | Chance a 330-call sample contains at least one |
|---|---|
| 0.5% | 81% |
| 1.0% | 96% |
| 2.0% | Effectively certain |
Turn that around into a requirement. To be 95% confident of catching a defect at least once:
- A 2% defect rate: 148 calls
- A 1% defect rate: 298 calls
- A 0.5% defect rate: 598 calls
That is the number worth confronting before designing a scorecard. Detecting a rare and serious failure requires a large sample, and measuring its rate precisely requires a much larger one. Per our review economics analysis, this is the same arithmetic that governs any sampling-based control, and the implication is the same: random sampling alone is the wrong primary tool for rare events.
Sample where the defects are
If random sampling cannot efficiently find rare failures, the answer is to stop sampling randomly for that purpose. Random sampling measures the general standard; targeted review finds the problems.
- Every complaint, and the three contacts preceding it. Per our complaints analysis, a complaint is a labelled defect and reviewing only the complained-of call misses what led to it.
- Every contact on an account later found to be in dispute or with a wrong balance, per our accuracy analysis.
- Contacts with accounts carrying vulnerability flags, per our vulnerability analysis, which is where the cost of a defect is highest.
- Outliers in call length, both directions. Very short calls and very long ones are where procedure departs from script.
- Contacts immediately after a cease contact or dispute flag, which tests whether the suppression actually worked.
- Every agent's first calls after training or after a change in script or policy.
- A random sample on top of all of that, sized to measure the general standard rather than to catch rare events.
Item five is the single highest-value target and almost nobody monitors it. A system that records a restriction and an operation that honours it are different things, per our cease contact analysis, and the only way to know whether suppression works is to look for contacts that should not exist. That query takes minutes and finds the defects that matter most.
What the scorecard should measure
Most scorecards measure compliance with a script and call the result quality. Those are different, and conflating them produces agents who read well and resolve nothing.
| Category | What belongs in it | How it should be weighted |
|---|---|---|
| Hard requirements | Required disclosures, identity verification, prohibited statements, honouring restrictions | Pass or fail, never scored |
| Accuracy | Correct balance, correct terms, correct consequences stated | Pass or fail |
| Outcome quality | Affordability actually assessed, arrangement realistic, per our affordability analysis | Heaviest scored weight |
| Treatment | Vulnerability recognized, pressure proportionate, questions answered | Heavy scored weight |
| Process | Notes, disposition codes, follow-up set | Light weight |
| Script adherence | Phrasing and sequence | Lightest, or not scored |
The first row has to be binary. A required disclosure that is worth five points out of a hundred is a requirement an agent can trade away for a good score elsewhere, and per our prioritization analysis the same logic applies here as to exclusions in a work list: a legal obligation expressed as a weight is an obligation that can be outvoted.
The bottom row is where most scorecards put most of their weight, because script adherence is easy to assess consistently. That is a reason to measure it, not a reason to weight it heavily, and per our incentive analysis agents optimize whatever the scorecard weights.
The arrangement that scored well and broke
The clearest test of whether a quality programme measures anything real is to compare scores against what happened afterward. If calls scoring highly produce arrangements that fail at the same rate as calls scoring poorly, the scorecard is measuring conformity rather than quality. Per our payment plan analysis, an arrangement agreed at an unaffordable figure is a defect that every script-based scorecard will mark as a success.
Making the scores mean the same thing
A score is only useful if two reviewers would give the same call the same result, and that is not the default.
- Calibrate regularly. All reviewers score the same call independently, then compare. Dispersion is normal at first and is the point of the exercise.
- Measure reviewer agreement as a reported metric, per our monitoring analysis, exactly as you would measure a model's stability.
- Write failure definitions, not criteria. "Affordability assessed" is interpretable; "asked about income and essential expenditure before agreeing a figure" is checkable.
- Separate the reviewer from the agent's manager where volume allows, since a manager scoring their own team has an interest in the result.
- Review the reviewers. A sample of completed reviews re-scored by someone else is the only check on the control itself.
The third point does most of the work. Scorecard disagreement is almost always definitional rather than a matter of judgment, and rewriting a criterion as the specific observable behaviour that satisfies it removes most of the variance without any further process.
Monitoring that looks where the defects are
HL Hunt AI Debt Collection reviews every contact automatically against hard requirements rather than sampling for them, targets human review at complaints, disputes, flagged accounts, and contacts that should not have occurred, and reports reviewer agreement alongside the scores themselves.
What automated review changes
Reviewing every contact rather than a sample removes the sampling problem entirely, and introduces different ones.
- Hard requirements become fully checkable. A required disclosure either appeared or did not, on every call, which is the single largest gain available.
- Prohibited statements become detectable at full coverage, rather than at 1.5%.
- Outcome quality remains hard to assess. Whether an arrangement was realistic is a judgment, and a system that scores it is approximating something it cannot observe.
- The false positive rate becomes the constraint. At 22,000 contacts, a 2% false positive rate on an automated check produces 440 items a month for humans to dismiss, which is more work than the sampling programme it replaced.
- Transcription errors propagate. A check run on a transcript inherits the transcript's mistakes, and the words most likely to be misheard include names, amounts, and dates.
The fourth point is what determines whether automated review helps. Full coverage is only valuable if the flags are actionable, and per our optimal rate analysis a detection system tuned for sensitivity without regard to precision generates a queue nobody works. The right sequence is to automate the binary checks, where precision is high because the question is factual, and to keep human judgment on the outcome questions where it is irreplaceable.
What to report
| Metric | Why |
|---|---|
| Hard requirement failure rate, at full coverage | The only one with a compliance consequence |
| Contacts that should not have occurred | Tests suppression rather than conduct |
| Arrangement survival by call score | Whether the scorecard predicts anything real |
| Reviewer agreement | Whether scores are comparable between reviewers at all |
| Complaint rate against score | The external check on an internal measure |
| Score dispersion by agent on comparable accounts | Separates agent effect from account difficulty |
The third row is the one that validates the whole programme. If arrangements from high-scoring calls survive longer than those from low-scoring calls, the scorecard measures something worth measuring. If they do not, the operation is paying for a measurement that correlates with nothing it cares about, and per our testing analysis that is a finding about the scorecard rather than about the agents.
Measure what the programme is for
Get started with HL Hunt AI Debt Collection for binary checks applied at full coverage, targeted human review driven by complaints and restriction breaches, reviewer calibration tracked as a metric, and arrangement survival reported against call score so the scorecard itself is tested.
Frequently asked questions
To be 95% confident: 148 calls for a 2% defect rate, 298 for 1%, and 598 for 0.5%. A typical 330-call sample catches a 0.5% defect only 81% of the time.
Complaints and the calls before them, disputed accounts, flagged accounts, length outliers, and above all contacts that occurred after a restriction was recorded.
Hard requirements and accuracy pass or fail, never scored. Heaviest scored weight on outcome quality and treatment; script adherence lightest or not at all.
Factual checks become full coverage, which is the big gain. But false positives become the constraint: 2% on 22,000 contacts is 440 items a month to dismiss.
Key takeaways
- Size the sample to the defect rate that matters: 598 calls to be 95% confident of hearing a 0.5% defect, against a typical sample of 330.
- Catching one instance tells you a defect exists; measuring its rate precisely needs a far larger sample again.
- Random sampling measures the general standard and is the wrong tool for rare failures; targeted review finds them.
- Query for contacts that occurred after a restriction was recorded, which is the highest-value target and almost never monitored.
- Make hard requirements pass or fail, because a legal obligation expressed as a weight can be outvoted by a good score elsewhere.
- Weight outcome quality above script adherence, since script adherence is the easiest thing to assess rather than the most important.
- Write criteria as observable behaviours, which removes most reviewer disagreement without any further process.
- Validate the whole programme by comparing arrangement survival against call score; no relationship means the scorecard measures conformity.
This guide is educational and does not constitute legal or compliance advice. All figures are stylized illustrations using assumed contact volumes, review rates, handling times, and defect rates, and do not describe any particular operation. Probability calculations assume independent contacts and a constant defect rate, which is a simplification. Call recording and monitoring consent requirements, required disclosures, prohibited practices, contact restrictions, and record retention obligations vary by jurisdiction and by product. Consult your compliance and legal functions before designing or changing a monitoring programme.