Architecture kata · K—06

BSS Modernisation and Real-Time Customer 360

Architecture kata · design study on a fictional brief

Sector
Telecommunications
Technologies
Debezium, Apache Kafka, Apache Flink, PostgreSQL, OpenSearch, Temporal, Unleash, Apache Iceberg, Trino, DataHub, Open Policy Agent, MLflow
Flow: change events from 14 systems pass Kafka and Flink into one read model, and new domains write back via an ACL.
The design at a glance: the one-page, 30-minute live drawing.

In short

Build a Customer 360 from change events of 14 systems without touching the mainframe, then retire billing by cohort over five years.

Problem

A telecom operator with 20 million subscribers bills on a 25-year-old mainframe application. Customer data sits in 14 systems, and a tariff change takes three days to reach all of them. The operator wants a real-time Customer 360 and a self-service app that executes orders in seconds. It also wants to retire the mainframe within five years, without a big-bang migration. The mainframe team allows at most 2% extra CPU and no schema changes.

Approach

One constraint shapes everything: no extra load and no schema change on the mainframe. Integration can therefore use change events only. The design builds the read model from events first, grows new domains behind an anti-corruption layer (ACL), and runs each domain in dual mode. A domain moves over only after a measured reconciliation gate. Debezium reads Db2 through capture tables. Systems on IMS and VSAM use the MQ events they already emit.

Architecture

  • Capture: Debezium for Db2, other CDC connectors for the satellite systems, and file or API adapters for the two nightly sources.
  • Backbone: Kafka with Avro schemas in Apicurio Registry, using TM Forum models.
  • Read model: Flink maps raw change data to a canonical model and joins 14 streams on the customer key. PostgreSQL and OpenSearch serve the apps (CQRS).
  • New domains: Temporal workflows expose TM Forum Open APIs. The ACL is the only write path to the mainframe, through an MQ transaction it already knows.
  • Data mesh and churn: Iceberg, Trino and DataHub for domain data products. Spark, MLflow and SHAP for churn scoring.
Architecture: change events from 14 systems pass Kafka and Flink into one read model; new domains write back through an ACL.
Solution overview: change capture and the anti-corruption layer (ACL), Kafka, the Customer 360 read model, new domains, the reconciliation gate and the data mesh.Open full size (opens in new tab)

Why these components

Need Choice Why
Mainframe change capture Debezium Db2 connector on Kafka Connect Reading the Db2 log through ASN capture tables is measurable and leaves the application schema alone. Querying tables directly would burn mainframe CPU. ASN Capture itself is a licensed IBM component.
Write path to the mainframe Anti-corruption layer (ACL) service with Kafka Connect IBM MQ sink The ACL turns a TM Forum order into an MQ transaction the mainframe already knows, with a correlation ID and a dedupe record before sending. The old model never leaks into the new domains.
Identity resolution Customer crosswalk owned by the CRM domain, with master data matching rules The same person lives under different IDs in 14 systems. Matching on national ID and contract number gives a confidence score. Below the threshold it shows as a possible match and is never merged.
Satellite systems Debezium for PostgreSQL, Oracle and MySQL, plus file and API adapters Twelve sources expose their logs, so change data capture gives the lowest load and best freshness. The two nightly-file sources carry their own freshness flag. No source pretends to be real time.
Event backbone Apache Kafka on Strimzi A durable, replayable log ordered by customer key. Compacted canonical topics can rebuild the read model without going back to the source. Redpanda was ruled out because of its Business Source License.
Schema governance Apicurio Registry with Avro and TM Forum schemas Canonical and business event schemas are versioned, and backward compatibility is enforced in CI. Confluent Schema Registry was ruled out because of its Community License.
Read model builder Apache Flink A stateful join of 14 streams on 20 million golden customer keys is exactly Flink’s job. Exactly once into Kafka; at least once with idempotent upserts into PostgreSQL and OpenSearch.
Read model stores PostgreSQL on CloudNativePG and OpenSearch Command and query separation: the app reads one document from PostgreSQL, agents search in OpenSearch. OpenSearch over Elasticsearch because of the SSPL and AGPL dual-licence period.
Order workflows Temporal A 10-second order includes a mainframe MQ step and compensation. Temporal keeps state durable, retries and makes timeouts visible. Camunda 8 was ruled out after its move to a source-available licence.
Cohort routing Unleash Cutover and rollback must be a flag flip. The gateway and the workflow read the same flag. LaunchDarkly is commercial. Unleash keeps SSO and RBAC in its paid edition, but cohorts are open.
Data mesh Apache Iceberg on Ceph, Trino, DataHub Domain teams own their data products, with catalogue, lineage and SLA in DataHub. The centre provides platform and standards, not the data. Ceph, because the MinIO community edition narrowed.
Churn model Apache Spark, MLflow, SHAP, ONNX serving Daily scores for 20 million subscribers are a batch job, and event-triggered rescoring uses the exported ONNX model. SHAP reason codes explain scores to agents. Pseudonymised features are still personal data under GDPR.
Gateway, identity and policy Apache APISIX, Keycloak, Open Policy Agent TM Forum Open APIs behind one gateway, with OpenID Connect for subscribers and agents. OPA enforces purpose-based access to usage data in the gateway and in Trino. APISIX, because Kong OSS cut features.
Secrets and keys OpenBao Mainframe credentials live only in the ingest namespace, and the pseudonymisation key stays in-house. HashiCorp Vault was ruled out because of its Business Source License.
Observability OpenTelemetry, Prometheus, Grafana, Loki, Tempo Mainframe CPU budget, capture lag, order step traces and the dual-run difference rate on one dashboard. It is the migration committee’s single input.

Decisions and trade-offs

  • Old system stays the source of truth during dual run. Differences count as defects in the new system until proven otherwise, and customers still get the old answer. The gate is a number: under 0.1% overall and zero on money fields for four weeks. The cost is running both systems for years. I treat that as an insurance premium, cheaper than a failed big bang.
  • Cutover by cohort with feature flags. Staff first, then 1%, 10%, 50% and 100%. Rollback is a flag flip, and data stays consistent because both systems keep processing. A failed gate stops cohort growth automatically. The cost is cohort design, which for billing must align with billing periods.
  • One Kafka cluster across two data centres, not MirrorMaker 2. In other katas an async copy is enough. Here, order commands and reconciliation differences must be confirmed in both sites. A round trip under 5 ms makes that possible. The cost is a witness controller at a third site. Where an async copy is enough, I would keep MirrorMaker 2.
  • Change events only, never a query. The mainframe allows 2% CPU and no schema change. So the 360 is built from log capture and existing MQ events, and read load on the mainframe is zero. Freshness is estimated at about 60 seconds at p99 against a five-minute target. The cost: IMS and VSAM have no log access and depend on MQ events.
  • No blind retry on mainframe writes. The ACL records the correlation ID before sending a persistent MQ message. On timeout it asks for the order status before any retry. A double charge would need a blind retry, and that path does not exist. The cost is a visible pending state with a timer in the app.
  • Purpose-based access. Usage attributes sit in a separate group. Reading them needs a declared purpose, and OPA decides in the gateway and in Trino. Every read goes to an append-only table and a monthly report to the regulator. It is a technical control, not a policy document. The cost: legal must define the list of purposes.
Flow: each domain is captured, translated, dual-run, reconciled and cut over by cohort; the old system stays the reference.
The strangler-fig migration flow: capture, translate, dual-run, reconcile and route by cohort. Every domain walks the same five stages.Open full size (opens in new tab)

What I would revisit

  • ASN capture on Db2 for z/OS. Is it allowed, and do capture tables count as a schema change? If they do, Db2 takes the same route as IMS and VSAM: MQ events or commercial log capture.
  • IMS and VSAM coverage. Do the existing MQ events carry enough fields for the 360, or is commercial log capture required?
  • Mainframe order time. If the tariff-change MQ transaction uses more than half of the 10-second budget, the app needs a designed pending experience.
  • Dual-run threshold. I assumed 0.1% and zero on money fields. Finance and the regulator have to accept a number.

Outcome notes (design assumptions, not measurements)

This is a design exercise, not a migration that happened. The output is a four-page diagram set, a one-page whiteboard version and a 16-page Turkish PDF with a five-stage strangler-fig flow. The 5-minute freshness target is a requirement, and the design offers no measured result.

Sources

  • Architecture PDF (Turkish, 16 pages).
  • draw.io diagrams (English labels): four-page set and a one-page 30-minute version.

Real projects on this topic

  1. P—07

    Completed

    Cloudera Big Data ETL Offload Program

    200+ high-load ETL processes rebuilt as ELT on Spark, Hive and Impala on a 15-node Cloudera cluster: about 12x faster on average, up to 30x.

    • Data pipelines
    • Orchestration
    • Data platform
    • Analytics-ready data

    Role

    Program lead · Big Data Management & Data Architecture Manager

  2. P—09

    Completed

    First Data Engineering Team, Hadoop Platform and Data Lake

    The company's first data engineering team, a production 15-node Cloudera CDH cluster and a multi-source data lake feeding reporting and machine learning.

    • Data platform
    • Data pipelines
    • Analytics-ready data
    • AI-ready data

    Role

    Built the team and platform · Data Engineering Team Lead

Decisions: projects, katas and publications

About the katas

Write to me about a similar design problem.