Architecture kata · K—06
BSS Modernisation and Real-Time Customer 360
Architecture kata · design study on a fictional brief
- Sector
- Telecommunications
- Topic
- Decisions
- Technologies
- Debezium, Apache Kafka, Apache Flink, PostgreSQL, OpenSearch, Temporal, Unleash, Apache Iceberg, Trino, DataHub, Open Policy Agent, MLflow

In short
Build a Customer 360 from change events of 14 systems without touching the mainframe, then retire billing by cohort over five years.
Problem
A telecom operator with 20 million subscribers bills on a 25-year-old mainframe application. Customer data sits in 14 systems, and a tariff change takes three days to reach all of them. The operator wants a real-time Customer 360 and a self-service app that executes orders in seconds. It also wants to retire the mainframe within five years, without a big-bang migration. The mainframe team allows at most 2% extra CPU and no schema changes.
Approach
One constraint shapes everything: no extra load and no schema change on the mainframe. Integration can therefore use change events only. The design builds the read model from events first, grows new domains behind an anti-corruption layer (ACL), and runs each domain in dual mode. A domain moves over only after a measured reconciliation gate. Debezium reads Db2 through capture tables. Systems on IMS and VSAM use the MQ events they already emit.
Architecture
- Capture: Debezium for Db2, other CDC connectors for the satellite systems, and file or API adapters for the two nightly sources.
- Backbone: Kafka with Avro schemas in Apicurio Registry, using TM Forum models.
- Read model: Flink maps raw change data to a canonical model and joins 14 streams on the customer key. PostgreSQL and OpenSearch serve the apps (CQRS).
- New domains: Temporal workflows expose TM Forum Open APIs. The ACL is the only write path to the mainframe, through an MQ transaction it already knows.
- Data mesh and churn: Iceberg, Trino and DataHub for domain data products. Spark, MLflow and SHAP for churn scoring.

Why these components
| Need | Choice | Why |
|---|---|---|
| Mainframe change capture | Debezium Db2 connector on Kafka Connect | Reading the Db2 log through ASN capture tables is measurable and leaves the application schema alone. Querying tables directly would burn mainframe CPU. ASN Capture itself is a licensed IBM component. |
| Write path to the mainframe | Anti-corruption layer (ACL) service with Kafka Connect IBM MQ sink | The ACL turns a TM Forum order into an MQ transaction the mainframe already knows, with a correlation ID and a dedupe record before sending. The old model never leaks into the new domains. |
| Identity resolution | Customer crosswalk owned by the CRM domain, with master data matching rules | The same person lives under different IDs in 14 systems. Matching on national ID and contract number gives a confidence score. Below the threshold it shows as a possible match and is never merged. |
| Satellite systems | Debezium for PostgreSQL, Oracle and MySQL, plus file and API adapters | Twelve sources expose their logs, so change data capture gives the lowest load and best freshness. The two nightly-file sources carry their own freshness flag. No source pretends to be real time. |
| Event backbone | Apache Kafka on Strimzi | A durable, replayable log ordered by customer key. Compacted canonical topics can rebuild the read model without going back to the source. Redpanda was ruled out because of its Business Source License. |
| Schema governance | Apicurio Registry with Avro and TM Forum schemas | Canonical and business event schemas are versioned, and backward compatibility is enforced in CI. Confluent Schema Registry was ruled out because of its Community License. |
| Read model builder | Apache Flink | A stateful join of 14 streams on 20 million golden customer keys is exactly Flink’s job. Exactly once into Kafka; at least once with idempotent upserts into PostgreSQL and OpenSearch. |
| Read model stores | PostgreSQL on CloudNativePG and OpenSearch | Command and query separation: the app reads one document from PostgreSQL, agents search in OpenSearch. OpenSearch over Elasticsearch because of the SSPL and AGPL dual-licence period. |
| Order workflows | Temporal | A 10-second order includes a mainframe MQ step and compensation. Temporal keeps state durable, retries and makes timeouts visible. Camunda 8 was ruled out after its move to a source-available licence. |
| Cohort routing | Unleash | Cutover and rollback must be a flag flip. The gateway and the workflow read the same flag. LaunchDarkly is commercial. Unleash keeps SSO and RBAC in its paid edition, but cohorts are open. |
| Data mesh | Apache Iceberg on Ceph, Trino, DataHub | Domain teams own their data products, with catalogue, lineage and SLA in DataHub. The centre provides platform and standards, not the data. Ceph, because the MinIO community edition narrowed. |
| Churn model | Apache Spark, MLflow, SHAP, ONNX serving | Daily scores for 20 million subscribers are a batch job, and event-triggered rescoring uses the exported ONNX model. SHAP reason codes explain scores to agents. Pseudonymised features are still personal data under GDPR. |
| Gateway, identity and policy | Apache APISIX, Keycloak, Open Policy Agent | TM Forum Open APIs behind one gateway, with OpenID Connect for subscribers and agents. OPA enforces purpose-based access to usage data in the gateway and in Trino. APISIX, because Kong OSS cut features. |
| Secrets and keys | OpenBao | Mainframe credentials live only in the ingest namespace, and the pseudonymisation key stays in-house. HashiCorp Vault was ruled out because of its Business Source License. |
| Observability | OpenTelemetry, Prometheus, Grafana, Loki, Tempo | Mainframe CPU budget, capture lag, order step traces and the dual-run difference rate on one dashboard. It is the migration committee’s single input. |
Decisions and trade-offs
- Old system stays the source of truth during dual run. Differences count as defects in the new system until proven otherwise, and customers still get the old answer. The gate is a number: under 0.1% overall and zero on money fields for four weeks. The cost is running both systems for years. I treat that as an insurance premium, cheaper than a failed big bang.
- Cutover by cohort with feature flags. Staff first, then 1%, 10%, 50% and 100%. Rollback is a flag flip, and data stays consistent because both systems keep processing. A failed gate stops cohort growth automatically. The cost is cohort design, which for billing must align with billing periods.
- One Kafka cluster across two data centres, not MirrorMaker 2. In other katas an async copy is enough. Here, order commands and reconciliation differences must be confirmed in both sites. A round trip under 5 ms makes that possible. The cost is a witness controller at a third site. Where an async copy is enough, I would keep MirrorMaker 2.
- Change events only, never a query. The mainframe allows 2% CPU and no schema change. So the 360 is built from log capture and existing MQ events, and read load on the mainframe is zero. Freshness is estimated at about 60 seconds at p99 against a five-minute target. The cost: IMS and VSAM have no log access and depend on MQ events.
- No blind retry on mainframe writes. The ACL records the correlation ID before sending a persistent MQ message. On timeout it asks for the order status before any retry. A double charge would need a blind retry, and that path does not exist. The cost is a visible pending state with a timer in the app.
- Purpose-based access. Usage attributes sit in a separate group. Reading them needs a declared purpose, and OPA decides in the gateway and in Trino. Every read goes to an append-only table and a monthly report to the regulator. It is a technical control, not a policy document. The cost: legal must define the list of purposes.

What I would revisit
- ASN capture on Db2 for z/OS. Is it allowed, and do capture tables count as a schema change? If they do, Db2 takes the same route as IMS and VSAM: MQ events or commercial log capture.
- IMS and VSAM coverage. Do the existing MQ events carry enough fields for the 360, or is commercial log capture required?
- Mainframe order time. If the tariff-change MQ transaction uses more than half of the 10-second budget, the app needs a designed pending experience.
- Dual-run threshold. I assumed 0.1% and zero on money fields. Finance and the regulator have to accept a number.
Outcome notes (design assumptions, not measurements)
This is a design exercise, not a migration that happened. The output is a four-page diagram set, a one-page whiteboard version and a 16-page Turkish PDF with a five-stage strangler-fig flow. The 5-minute freshness target is a requirement, and the design offers no measured result.
Sources
- Architecture PDF (Turkish, 16 pages).
- draw.io diagrams (English labels): four-page set and a one-page 30-minute version.
Real projects on this topic
P—07
Completed
Cloudera Big Data ETL Offload Program
200+ high-load ETL processes rebuilt as ELT on Spark, Hive and Impala on a 15-node Cloudera cluster: about 12x faster on average, up to 30x.
Role
Program lead · Big Data Management & Data Architecture Manager
P—09
Completed
First Data Engineering Team, Hadoop Platform and Data Lake
The company's first data engineering team, a production 15-node Cloudera CDH cluster and a multi-source data lake feeding reporting and machine learning.
Role
Built the team and platform · Data Engineering Team Lead