The rule that went quiet.
A rule that fires nothing needs a reason as much as a rule that fires everything.
You work in the risk team at Northbank Pay, a fictional PSP. Tomas notices that R-114, a rule for high-value card-absent payments where the issuer country differs from the merchant’s, has produced no alerts for three weeks. The dashboard is green. Nobody raised it because the queue was quieter. Decide what to do this afternoon. All people, figures and internal procedures in this exercise are synthetic.
Fictional scenario · Synthetic data · No real customer records- days without an alert
- 21
- errors logged
- 0
- alerts in backfill
- 338
What would
you look at?
Open at least two files before making a decision. You can read every file.
0 of 3 files opened
A useful definition01 / Rule dashboardAlert history+
R-114 produced between 11 and 22 alerts a day for six months, 16 on average. The last alert was at 23:50 on 13 October. The rule’s dashboard shows green because it measures errors and latency, not output. Over the six months, analysts resolved 71% of R-114 alerts as ‘no action’, and 9% of alerted transactions later became disputes, against 1.1% for unalerted card-absent transactions. Queue volume fell by a sixth and nobody asked why.
An abrupt stop deserves a data and control investigation. The historical association with disputes does not establish fraud precision or the causal value of the rule. A no-action disposition is not necessarily a false positive, and the comparison cohort may have a different risk mix.
02 / Data team release noteChange record+
On 13 October the data team released schema version 7 of the transaction export. The column ‘card_country’ was renamed ‘issuer_country’. R-114’s condition compares card_country with merchant_country. The rule engine treats a missing column as null, a comparison with null as false, and logs nothing, because null comparisons are a normal case. The release note went to a mailing list. No document records which rules depend on which columns.
The reproduced missing-field behaviour explains this rule’s silence in the synthetic environment. A one-line fix does not establish that every dependent rule is healthy. Test other consumers of the changed schema and distinguish intended engine behaviour from a safe integration design.
03 / Analyst rerunBackfill+
Running R-114 with the corrected column over the 21 days yields 338 transactions that would have alerted. 29 of them have already become disputes, 8.6%, with the dispute window still open for all 338. 212 are settled and paid out to merchants; 126 are inside Northbank’s merchant payout delay; the exercise runbook permits individual review before payout, not an automatic bulk hold. Card settlement and merchant payout are distinct stages. The transactions fall across 40 merchants, none dominant. The fix is a one-line mapping change that takes ten minutes.
Backfill identifies missed review opportunities, not preventable losses. Disputes can have several causes and can arrive later. Preserve the paid-out versus pending-payout distinction without pretending that a rule match determines the correct action for every transaction.
Make the call.
Leave your reasoning.
Choose what you would do under this scenario’s assumptions. Your notes stay in this browser. Do not enter real customer information.
The trade-off
Still unknown
Which other rules depend on changed fields, how the review cohort’s outcomes mature, and whether earlier analyst intervention would have changed any loss. The backfill cannot answer that counterfactual by itself.
A useful handoff
Record the gap, schema change, backfill cohort, payout status, validation evidence and incident owner. Assign input contracts and volume-aware monitoring, then review effectiveness and customer impact in seven days.
Completion records participation, not competence or certification. You can change your answer and compare again.
Further reading
NIST · Information Security Continuous MonitoringReference checked 5 November 2026. The scenario, runbook and debrief are original Antifrog teaching material; the source does not endorse them.