How to run a retrospective fraud backtest on your own book

Write the decision question first

A retrospective backtest is not a search for the highest accuracy number. It asks a narrower operational question: if this score had existed when each application was decided, which later confirmed losses would have appeared inside the institution's available review capacity?

Write the target decision before collecting data. A lender may want to rank applications for manual review. A bank may want a step-up path at account opening. Those actions use different thresholds and carry different error costs. Combining them after scoring makes the result hard to interpret.

The test should also name what it cannot prove. Historical outcomes are incomplete, fraud labels can arrive late and past policy affected which applicants entered the book. A backtest estimates value on the observed book. It does not establish a universal detection rate.

Fix the cohort and observation window

Start with a closed cohort, usually twelve months of applications for one product and decision path. Preserve approved, declined and referred applications when lawful and available. A dataset containing only booked accounts cannot answer how the score would have ranked rejected applications.

Define the outcome window separately from the application window. Each application needs enough later time for the selected outcome to become observable. Recent applications with immature outcomes should be excluded or marked pending rather than treated as clean.

Record policy version, product, channel and decision date. Segment differences matter. A digital personal-loan application and an assisted branch opening should not be mixed into one headline metric unless the production policy also treats them as one population.

Build the smallest useful application table

One row should represent one original application. The stable key can be an institution-generated token. Include the decision timestamp, product and channel, original policy action, later outcome label and the application-time signals permitted for the test.

Keep raw identifiers inside the institution. Name, mobile, email, address, device and beneficiary values can be converted to stable tokens before transfer. The scorer needs consistent equality relationships across rows, not readable personal data.

Missing values must stay missing. Do not replace an absent field with information collected during servicing. The absence itself may be informative, and later enrichment would introduce evidence the production decision did not have.

• Application record

  Stable application token, source timestamp, channel, product and original action.

• Tokenised identity evidence

  Stable values for permitted identity, contact, address, device and beneficiary comparisons.

• Decision-time observations

  Credential age, bureau state, verification outcomes and other signals as they existed then.

• Outcome record

  Named label, confirmation source, observation date and any pending status.

Use keyed tokens, not plain hashes

A plain hash of a mobile number or other low-entropy identifier can be tested against a guessed list. Use institution-controlled keyed tokenisation so the same permitted value maps consistently inside the test while outsiders cannot reproduce the mapping without the secret.

The NIST HMAC standard defines a keyed-hash construction. The institution should choose and govern its implementation, key rotation and access controls. The vendor should receive tokens only after the agreed transformation runs inside the institution's environment.

Keep the key out of the backtest workspace and transfer process. Document which fields were tokenised, which normalisation rules ran first and whether the same key scope was used across the full cohort. Inconsistent tokenisation destroys graph relationships without producing an obvious error.

Prevent time travel in feature reconstruction

Every feature needs an as-of time. If an address became linked to a confirmed ring after the application date, that later label cannot be fed into the original score. The backtest may use the link only if the underlying relationship was observable then and the production system could have computed it.

Split model development and evaluation by time. Randomly mixing older and newer applications can place related ring members on both sides of the split. The evaluation then benefits from relationships that would not have been known during a live first encounter.

A reproducible feature manifest should list the source table, event-time rule, transformation version and missing-value treatment for every input. If a feature cannot be reconstructed under those rules, remove it from the historical comparison or label the limitation.

Define outcomes before looking at scores

Use outcomes the institution can defend: a confirmed fraud disposition, a documented mule closure, a first-payment-default label under an agreed policy or another governed event. Keep operational proxies separate from confirmed fraud so the report does not silently change definitions.

Record who or what confirmed the outcome and when. Disputed, reversed and pending cases need their own states. Treating every unresolved account as clean inflates the apparent negative population and can make precision look better than it is.

The RBI Annual Report 2024-25 reports ₹36,014 crore as the amount involved in banking-sector frauds reported during the year for FY 2024-25. The annual report gives the sector context, but it is not a label source for an institution's row-level test. The backtest must use the institution's own governed outcomes.

Measure the queue the team can work

Accuracy is usually a weak headline for a rare-event ranking problem. A model can classify the large clean majority correctly while placing few confirmed cases near the top. Precision and recall describe different parts of the operating trade-off. The Google Machine Learning classification guide provides standard definitions for both metrics.

Rank every eligible application by score and select the top 5% if that reflects the review team's actual capacity. Precision at that cutoff is the share of reviewed applications with the agreed outcome. Recall at that cutoff is the share of all observed outcome cases contained in the queue.

Report the full precision curve across feasible review volumes. Also show results by product, channel and decision month where sample size permits. Do not choose the best segment after seeing results and present it as the whole book.

Add confidence intervals or resampling ranges when the outcome count is small. A large movement caused by a few cases should be visible to the reader. Publish the denominator beside every rate so an evaluator can distinguish a stable comparison from a sparse segment.

Return evidence, not one performance number

The institution should receive a scored row for every eligible application, ranked reason codes, the model and feature versions, cohort definitions and an exclusions file. Aggregate charts are useful, but they cannot support a case review by themselves.

Review samples from the top of the ranking and from material false-positive groups. Check whether a repeated device reflects a genuine branch workflow, whether an address is common for a legitimate reason and whether a thin-file segment is being over-routed. These reviews inform policy thresholds and data corrections.

Our opinion is that a backtest without a row-level return file is a sales demonstration, not a risk evaluation. The buyer must be able to reproduce the cohort, inspect the reasons and challenge the result.

Close the test with a deletion record

Agree the processing purpose, access list, retention period and deletion event before data moves. A mutual NDA is useful, but it does not replace a documented data-processing basis or the institution's vendor-risk process.

The RBI KYC Master Direction governs customer due diligence and records for regulated entities. The institution should map the backtest workflow to its own obligations, internal policy and approved environment before sharing any tokenised dataset.

At completion, return the output, delete working copies under the agreed schedule and issue a record of deletion. Retain only the artefacts the contract and governance approval permit. That ending is part of the test design, not administrative cleanup.

Record who approved the cohort, who received the output and which environment processed it. If a later production proposal cites the backtest, the approval pack should point back to the exact dataset version and evaluation code rather than a copied presentation chart.