The Manual Review Queue: What a Second Look Actually Costs

Referral to manual review is treated as a safety valve rather than a line of business, so its cost is known and its output is not. A queue processing 960 cases a month at $8.87 per case is costing $40.30 per decision it actually changes, and the ratio between those two figures is the confirmation rate that almost nobody reports. There is also a cost that appears in no budget: the delay strips the strongest referrals out before the decision lands, which makes the queue look less valuable than it is.

What the queue costs

Referral to manual review is usually treated as a safety valve rather than as a line of business, which means its cost is known and its output is not.

Take 8,000 monthly applications with a 12% referral rate, 14 minutes of analyst time per case, at a loaded $38 an hour:

• Referrals: 960 a month
• Analyst hours: 224, or about 1.4 full-time equivalents
• Direct cost: $8,512 a month
• Cost per case reviewed: $8.87

$8.87 per case is cheap, and that is the figure most operations cite. It is also the wrong figure, because it prices the activity rather than the output.

The number that actually matters

A review only produces value when it changes the decision. If the reviewer confirms what the model already concluded, the applicant experienced a delay and the institution paid for a second opinion it did not act on.

Suppose 78% of reviews end in the same decision the automated policy would have reached:

• Decisions changed: 960 × 22% = 211 a month
• Cost per changed decision: $8,512 ÷ 211 = $40.30

$40.30 is still frequently worth paying. If a review approves an applicant the model would have declined and that account contributes $180, the economics are overwhelming. The point is not that manual review is expensive. It is that the confirmation rate is the single most important number in the queue and almost nobody reports it.

Per our override analysis, the same data that produces a confirmation rate also tells you which referral rules are working, because a rule whose referrals are confirmed 95% of the time is a rule that should be automated in whichever direction it keeps confirming.

The referral rule is the real design decision

What arrives in the queue is determined by the referral criteria, and those criteria are usually accumulated rather than designed.

Referral reason	What it is really saying	Should it be a referral?
Score near the cutoff	The model is uncertain	Yes, per our threshold analysis, this is exactly where a second look has content
Missing or unverifiable data	The model lacks an input	Usually no; this is a verification task, not a judgment task
Inconsistency between sources	Something needs resolving	Yes, but the reviewer needs authority to resolve it
Policy exception requested	The rule does not fit	Yes, and it should be logged as an exception, per our exception analysis
Amount above a threshold	The stakes are high	Only if the review adds something the model cannot
Fraud flag	A different question entirely	Yes, but to a different queue with different skills
Nobody remembers why	Accumulated rule debt	No, and measuring the confirmation rate finds these

The second row is where most wasted capacity sits. Sending a case to an analyst because a document is missing puts a skilled judgment resource on a clerical task. Per our verification analysis, the correct response to a missing input is to request it, automatically, and to re-decide when it arrives.

The last row is the one measurement finds immediately. Referral rules accumulate through incidents: something went wrong once, a rule was added, the circumstance never recurred, and the rule keeps referring cases that are confirmed every time. A confirmation rate by referral reason exposes these in a single report.

The cost the applicant pays

The queue has a second cost that appears in no operating budget, which is the delay it imposes.

Per our latency analysis, a delay is a price charged to the applicant, and in lending it is also a conversion problem:

• Applicants abandon. A decision that takes two days competes against a decision that takes two minutes, and per our acceptance analysis the ones who abandon are not a random sample. Applicants with better alternatives leave first.
• The queue selects adversely. Because the best applicants in the queue have the most options, a slow review systematically loses the ones you most wanted, and the observed performance of reviewed-and-approved accounts is worse than the review's decisions deserve.
• Batch processing concentrates it. A queue worked once a day gives every applicant the full cycle time regardless of when they applied.

The second bullet is the finding worth sitting with. Manual review appears to add less value than it does, because the delay it imposes strips out the strongest applicants before the decision lands. An operation measuring the review's quality on booked outcomes is measuring it on a sample the review's own latency degraded.

Whether the reviews are any good

An unexamined review queue is a second decision system with no validation, which is a strange thing to run alongside a model that is validated continuously.

1. Measure agreement between reviewers. Give the same cases to two analysts periodically. Low agreement means the outcome depends on who picked up the file, per our exception analysis.
2. Measure realized performance of overturned decisions. When a reviewer approves something the model declined, how do those accounts perform? This is the only direct test of whether the review adds information.
3. Hold out a sample. Auto-decide a small random share of referrals so there is a comparison group, per our testing analysis. Without it, the review is evaluated only against itself.
4. Track by reviewer. Per our monitoring analysis, wide dispersion in approval rates across analysts on comparable cases is the same finding as an unstable model and deserves the same response.
5. Check the decline reasons. Per our adverse action analysis, a manually declined application needs accurate specific reasons, and a reviewer's judgment is harder to convert into a compliant reason than a model's output.
6. Report outcomes by group. Discretionary decisions are where uneven treatment accumulates least visibly, per our governance analysis.

Item two is the decisive test and it is rarely run. If overturned declines perform as well as auto-approvals, the reviewers are finding real information the model misses and the queue is creating value. If they perform materially worse, the queue is buying volume at a loss, and the reviewers are being rewarded for approvals rather than for accuracy.

Item one is the cheapest and most uncomfortable. Reviewer agreement is usually lower than anyone expects, and discovering that is the beginning of fixing it rather than a reason not to look.

Running it well

1. Report the confirmation rate by referral reason, monthly. Everything else follows from this one table.
2. Automate any rule confirmed above roughly 90% in the direction it confirms, and delete rules nobody can justify.
3. Route clerical work away from judgment work. Missing documents go to automated requests, not to analysts.
4. Separate fraud from credit into different queues with different skills and different metrics.
5. Give reviewers authority to resolve, not only to decide. A reviewer who can call, request, and counter-offer, per our reconsideration analysis, resolves cases that a reviewer who can only approve or decline must decline.
6. Work the queue continuously, not in batches, and prioritize within it by applicant quality rather than by arrival order, since the best applicants are the ones abandoning.
7. Capture the reason for every overturn in a code. This is the training data for the next model version and it costs one dropdown.
8. Set a latency target and report against it, because an unbounded queue is an unbounded cost borne by the applicant.

Item five produces the largest gain in most operations. A large share of referrals are cases where a missing piece of information would settle the question, and a reviewer empowered only to approve or decline resolves those by declining. Per our rationing analysis, that is a decline caused by the design of the review rather than by the application.

Frequently asked questions

---

This guide is educational and does not constitute legal, compliance, or model risk advice. All figures are stylized illustrations using assumed application volumes, referral rates, handling times, labour costs, confirmation rates, and contribution figures, and do not describe any particular lender. Adverse action requirements, record-keeping obligations, model risk management expectations, and fair lending expectations for discretionary decisions vary by institution type and jurisdiction. Consult your compliance, legal, and model risk functions before changing referral criteria or review authority.