The deceptively strong result
An offline fraud model can look excellent because its dataset contains information unavailable when the real decision would have been made. This is different from a wrong label: the outcome may be correct while the inputs accidentally reveal it. Evaluate what the model could genuinely have known at the proposed point of use.
Time belongs to each feature
Record when an event occurred and when its information became available to the system. A dispute filed weeks after checkout can be an outcome label, but it cannot be an input to a checkout-time model for that same purchase. Joining tables by transaction identifier does not, by itself, preserve that distinction.
A fictional experiment
A team predicts risky orders using a field called investigation_status from today’s database. Historical fraud cases now say “confirmed”, even though they were unreviewed at checkout. Randomly splitting rows leaves this clue in both training and testing. The impressive result measures access to later investigation work, not the proposed early detection capability.
Rebuild an honest evaluation
Construct inputs as they were available at the decision timestamp, isolate training from evaluation and choose splits that reflect the intended deployment. Keep closely related records from creating misleading overlap where relevant. Then compare offline and production feature behaviour. Google’s monitoring guidance highlights leakage and training-serving skew; neither is repaired merely by choosing a more complicated model.