Answer first: A fraud false-positive rate is meaningful only after the firm defines the decision and the verified outcome. Measure legitimate customers or payments interrupted, time to resolution, review effort, abandoned activity, complaints and cohort differences; then compare that customer and operating cost with the fraud or intelligence benefit delivered by the control.

Key takeaways

  • Define what decision is being evaluated and when ground truth is mature.
  • Use legitimate customers, payments or value as a denominator alongside alerts.
  • Measure interruption duration and resolution effort, not only alert closure.
  • Feed reviewed outcomes back without turning uncertain labels into training truth.

Decision framework

Decision area Evidence to require Risk it controls
Alert Rule/model produced review work Queue volume
Intervention Payment/account/customer was actually constrained Customer friction
Verified fraud Mature evidence supports positive outcome Control benefit
Verified legitimate Mature evidence supports false positive Control cost
Unresolved Kept separate until evidence matures Label contamination

A fraud false positive occurs when a control flags legitimate activity as suspicious. Its cost can include manual review, delayed payments, customer support and lost business. A fintech should measure these effects alongside fraud losses before describing tighter controls as an improvement.

This is not an argument for weaker prevention. It is an argument for knowing what a control actually does. A rule that blocks almost everything may produce low fraud losses while making the product unusable.

Define the outcome you can verify

Separate a flagged event from a confirmed false positive. Some cases remain uncertain.

Record the evidence used to classify them and the period allowed for fraud reports to emerge.

FCA financial-crime guidance concerns effective risk assessment and controls. Consumer Duty materials separately address evidence of customer outcomes where the Duty applies.

12 Neither objective is served by a dashboard that counts interventions without examining their consequences.

Choose a unit: application, payment, account or customer. Ten alerts on one payment are not ten independent suspicious transactions. Duplicate alerts can inflate both the apparent workload and the apparent success of a tool.

Add a legitimate-activity denominator

Report the proportion of legitimate transactions challenged, delayed and ultimately rejected.

Show time to resolution and whether the customer completed the intended action.

Do not use only complaints to find the damage. Some customers leave quietly, switch providers or make the payment elsewhere. Link the event to later activity where appropriate and lawful, while recognising that correlation does not establish the reason for departure.

An illustrative rule flags 2,000 of 100,000 payments. Review confirms 100 fraudulent payments, 1,600 legitimate payments and 300 unresolved cases. Calling all 2,000 prevented fraud would be wrong. Calling all 1,900 non-confirmed cases false positives would also overstate what is known.

Calculate review and interruption costs

Measure staff time per case, including escalation and repeat contact. Add the cost of mistaken restrictions, reversals and remediation. Estimate lost business conservatively and show the assumptions.

A hypothetical £5 manual review applied to 1,600 legitimate payments costs £8,000 before customer effects. If a proposed tool reduces that work but introduces another type of loss, compare both changes on the same period and population.

Avoid treating every abandoned customer as the same value. A new applicant, an established payroll user and a high-volume merchant have different patterns and consequences.

Segment analysis should reveal material differences without becoming an exercise in excluding inconvenient customers.

Review thresholds with mature evidence

Fraud labels arrive late. A rule can appear accurate immediately after deployment and deteriorate as later reports arrive. Use a consistent observation window and explain immature cohorts.

External datasets require similar care. The PSR dashboard defines its reporting population and period. 3 Its figures cannot automatically be compared with an internal alert rate or a model vendor's prevented-loss estimate.

Back-test threshold changes and consider controlled deployment where feasible. Monitor both fraud outcomes and legitimate activity. Keep an emergency route to stop a change that creates unexpected harm.

Give customers a route to resolve the problem

Explain what they can do without revealing sensitive control logic. Offer a proportionate review channel and keep support informed about the restriction's status.

If the product repeatedly asks for evidence already supplied, investigate whether systems share the relevant record. More document requests can look like diligence while actually reflecting a broken handover.

Review cases involving urgent payments and customers with additional needs. A standard waiting period can have very different effects depending on the activity being interrupted.

Distinguish a review queue from an effective intervention

A flagged payment may wait hours before anyone acts. The control's effect depends on what happens during that wait, whether the payment is held and what the customer is told.

Measure queue age by value and customer consequence, not only case count. A few time-sensitive business payments can require more urgent attention than a larger low-impact population.

In a hypothetical system, a new rule doubles alerts without increasing reviewer capacity.

Confirmed fraud falls initially, but legitimate customers experience growing delays. The analysis should test whether the improvement remains worthwhile after mature losses and interruption costs are included.

Review feedback loops in the training data

If analysts repeatedly label alerts using incomplete evidence, the model or rule review can inherit those mistakes. Preserve the basis of labels and revisit a sample after outcomes mature.

Watch for cases cleared because the customer abandoned the process. Abandonment is not proof of fraud, just as successful completion is not proof that every risk was absent.

Separate changes to the model from changes to operations. Faster human review can improve outcomes without any change in predictive accuracy. Better customer information can reduce unnecessary alerts without weakening the risk standard.

Those distinctions matter when evaluating vendors and allocating budget. The company should pay for the improvement it can actually observe, whether that comes from better detection, clearer evidence or a more capable response team.

The decision for the risk committee

Present prevented losses, actual losses, uncertain cases, legitimate activity disrupted and review costs together. Explain which estimates are observed and which depend on assumptions.

A threshold decision needs two populations in view: harmful activity prevented and legitimate activity interrupted. Mature outcomes and reviewed cases make that comparison possible.

Alert volume alone cannot establish that the product improved.

A practical 90-day implementation plan

  1. Select one control and document decision, population, intervention and outcome labels.
  2. Build an outcome window long enough for fraud reports, reversals and complaints to mature.
  3. Compare benefit and harm by customer, payment, value and vulnerable cohort.
  4. Change one threshold or feature at a time with fraud-loss and customer-outcome guardrails.
  5. Sample unresolved cases separately; uncertainty should remain visible until the evidence matures instead of being forced into a convenient success or failure label.

Frequently asked questions

What is the best false-positive denominator?

Use more than one: alerts help operations, while legitimate customers, payments and value show customer reach and commercial impact.

Can an alert closed with no action be called a false positive?

Not necessarily. It may be unproductive, but a true/false label requires a defined event and sufficiently mature evidence.

Should fewer alerts always be the goal?

No. The goal is proportionate, effective risk reduction. Lower volume can reflect better targeting or missed risk; review outcomes and intelligence value.

Reporting by Shoreditch Talk