Real-Time Fraud Decisioning for Instant Payments

Problem

A mid-size European retail bank joins an instant payment scheme. Fraud rules run in an overnight batch today. In the new setup, the fraud decision gets 150 ms of a 10-second payment window. Load is 300 payments per second on average and 3,000 at peak. The mainframe can only be reached through IBM MQ and cannot change. Every decision must be explainable and reproducible for seven years. Two active data centres are required, with room for cards and open banking later.

Approach

Two demands pull against each other: a tight latency budget and decisions that can be replayed years later. The design precomputes features, decides inside one process and stores every input. Flink computes 90-day behaviour features and writes them to Valkey. The decision service reads one feature vector, then runs Drools rules and an ONNX model in the same JVM. Each decision is saved as a snapshot with the feature vector, rule-set version, model hash and image digest.

Architecture

• Mainframe bridge: Kafka Connect source and sink for IBM MQ. The platform never calls the mainframe.
• Backbone: one Kafka cluster stretched across two data centres, with a KRaft witness at a third site. Schemas are Avro contracts in Apicurio Registry.
• Features and decision: Flink, Valkey, and a Quarkus decision service running Drools and ONNX Runtime.
• Evidence and learning: snapshots go to Apache Iceberg on object storage with Object Lock. MLflow tracks models. Analyst outcomes return as training labels.

Data flow: payment request on MQ, Kafka, decision, then decision and snapshot in one Kafka transaction, then the reply on MQ.

Decisions and trade-offs

• Model inside the decision process. The kata estimates 20–40 ms of the 150 ms budget for a separate model service. Inside the JVM, ONNX Runtime scores in about 2 ms. I gave up operational flexibility for p99 latency. Promotion and rollback stay a pointer change, because both models stay loaded. With a looser budget, a separate model service would be the more flexible choice.
• Store features, do not recompute them. Each snapshot holds the feature vector, rule-set version, model hash and image digest. Replaying a decision means running the same image on the same vector. Recomputing would use data that arrived later and change the outcome. The same snapshots become the training set, so training and serving see identical features. The cost is archive size, estimated at about 45 TB over seven years.
• One stretched Kafka cluster, not two with MirrorMaker 2. Idempotency rests on one compacted decisions topic where the first writer wins. That rule cannot be defined across two clusters. If a data centre fails mid-decision, the other side checks the topic before it scores. The cost is a third site for the KRaft witness, within 10 ms of both data centres.
• Kafka on the synchronous path, not a direct MQ read. Kafka gives replay from one log, fan-out to the Flink feature jobs and an audit trail. Reading MQ directly would mean building all three separately. The cost is poll and queue wait, about 8 ms, which fits inside the estimated 40 ms at p99.
• Ordered degradation. Without features: rules only, reported as a fallback rate. Without rules: the model with a conservative threshold. Without both: send to review. A timeout on the mainframe side also means review, never a blind approval. The cost is more held payments during an incident. False positives are the biggest risk, so the fallback rate sits on the dashboard.
• Challenger in the same process. The challenger scores every payment in shadow mode, and only its result is logged. Promotion moves the champion pointer after a four-eyes approval. Rollback moves it back, and nothing restarts. The cost is two models in memory on every decision pod.

What I would revisit

• Mainframe timeout default. Does the mainframe hold or approve when the reply is late? The degradation design depends on that answer.
• A third site. Without a site within 10 ms of both data centres, the topology falls back to two clusters with MirrorMaker 2. Then the first-writer-wins rule no longer works as designed.
• Card tokens. Can the card platform really send tokens, or will card numbers become visible? The PCI scope depends on it.
• Feature count and windows. I assumed 40 features over 90 days. The real numbers size Valkey and Flink.

Outcome notes (design assumptions, not measurements)

This is a design exercise, not a production system. The output is a four-page diagram set, a one-page whiteboard version and a 14-page Turkish PDF. The PDF holds assumptions, component rationale, licences, capacity arithmetic and open questions. The latency budget (about 40 ms at p99 against 150 ms) is a design estimate, not a measurement.

Sources

• Architecture PDF (Turkish, 14 pages).
• draw.io diagrams (English labels): four-page set and a one-page 30-minute version.