Architecture decisions.
Every component choice from the katas, with the reason and the alternative that lost.
Design studies, not delivered systems.
Cold-Chain Telemetry and Compliance Platform
| Need | Choice | Why |
|---|---|---|
| Device ingest | EMQX with MQTT 5 | MQTT fits devices: tens of thousands of persistent connections, small messages, QoS 1 acknowledgements and sessions for offline clients. EMQX clusters and checks an X.509 certificate per device. Mosquitto and HiveMQ CE do not cluster. |
| Bridge from broker to log | EMQX Kafka data integration | Writes to Kafka with acks=all and acknowledges the device only after Kafka confirms, so no MQTT offset can be lost. An Enterprise feature; the open alternative is Kafka Connect MQTT source, at least once. |
| Sources without MQTT | Ingestion API, HTTPS batch with an idempotency key | The third vendor only offers portal exports. A second door into the same raw topic, where the idempotency key makes a resend harmless. |
| Event backbone | Apache Kafka with KRaft on Strimzi | An MQTT broker is a distribution point, not a durable log. Kafka gives a replayable log ordered by device, and three replicas with acks=all prevent loss. Redpanda was ruled out over its Business Source License. |
| Schema contract | Apicurio Registry with Avro | A versioned contract for three vendor formats and future ones. Incompatible changes fail in CI, not in production. Confluent Schema Registry was ruled out because of its Community License. |
| Transport reference data | Debezium change data capture | Reads the shipment, vehicle and driver mapping from the transport system's log, with no query load and no change to that system. Flink uses it as broadcast state. |
| Stream processing | Apache Flink with the Kubernetes operator | Event time, watermarks, 72 hours of allowed lateness, stateful joins and rules in one place. Deduplication uses the monotonic sequence: the last number plus a small gap set per device, so 20,000 keys, not billions. |
| Hot analytics | ClickHouse with the Altinity operator | Sub-second time series queries over 90 days, with idempotent writes through ReplacingMergeTree on device and sequence. TimescaleDB was ruled out: its compression and continuous aggregates sit under the Timescale License. |
| Evidence archive | Apache Iceberg on Ceph RGW with Rook | Cheap object storage for ten years of raw data, an open table format with ACID snapshots, and write-once retention through S3 Object Lock. Ceph, because the MinIO community edition narrowed in 2025. |
| Operational state | PostgreSQL with PostGIS on CloudNativePG | Shipments, geofences, alert state and last position: relational and geographic queries in one place. A boring and correct choice for small, hot, often-updated data. |
| Cache and pub/sub | Valkey | Redis-compatible and BSD-licensed under the Linux Foundation, for the live position cache and WebSocket fan-out. Redis 8 returned to AGPL, which would also be acceptable; Valkey stays out of the licence debate. |
| Immutable ledger | immudb | Report digest, signature, Iceberg snapshot ID and report version go into an append-only ledger. Merkle proofs let an auditor verify independently that nothing changed. Codenotary offers enterprise support. |
| Secrets and keys | OpenBao | Device PKI, a tokenisation key per driver and transit signing. The Linux Foundation fork under MPL 2.0, since HashiCorp Vault moved to the Business Source License. |
| API gateway | Apache APISIX | OAuth2 validation, quotas per customer, versioning and WebSocket proxying. APISIX under the Apache licence, because Kong OSS moved enterprise features into its closed edition. |
| Identity | Keycloak | OpenID Connect federated to the corporate identity provider for staff, and client credentials for business customers. Support comes through the Red Hat build of Keycloak. |
| Map and UI | React, MapLibre GL and Martin vector tiles | An own tile server means no fee per map call. MapLibre is the BSD-licensed fork made after Mapbox GL v2 went closed. Enough for 500 dispatchers. |
| Analytical access | Trino, Apache Superset, JupyterHub | Data scientists query Iceberg with SQL through Trino, dashboards run in Superset and notebooks in JupyterHub. The evidence archive doubles as the analytics source, so no copy is needed. |
| Platform | Kubernetes or OpenShift, Argo CD, cert-manager, step-ca, Cilium | Every workload is operator-managed, and GitOps makes the setup reproducible. step-ca issues device certificates; Cilium enforces mTLS and network policy. |
| Observability | OpenTelemetry, Prometheus, Grafana, Loki, Tempo | Device liveness, consumer lag, checkpoint health, alert delay from device time to delivery, and report backlog in one stack. |
Open Lakehouse with Data Contracts and a Shared Semantic Layer
| Need | Choice | Why |
|---|---|---|
| Open table format | Apache Iceberg | All four engines read it, and the standard REST catalog protocol they implement is Iceberg's. Atomic snapshots, schema evolution, row-level deletes for erasure. Delta and Hudi lack an equal neutral catalog protocol: capability, not licence. |
| Catalog | Apache Polaris or Lakekeeper, both Iceberg REST | One catalog with a warehouse per region and short-lived, table-scoped credentials for each engine. OPA policy applies here. Per-engine catalogs like Glue, Unity or BigLake metastore would mean three permission models. |
| Regional storage | Cloud Storage in the EU, S3-compatible in the UK, Ceph RGW on premises in Türkiye | Data stays in its region, with one KMS key per region. Files are written once and never copied, so there is no seventh copy. Ceph over MinIO, given its licence and distribution doubts. |
| Cross-brand and group query engine | Trino | Iceberg connector, OPA row and column policies and a pool per region. It is also the on-premises brand's writer. Spark would work, but Trino suits interactive, federated queries better. Athena is commercial and single-cloud. |
| Data contract | Open Data Contract Standard YAML with CI | Schema, PII class, SLA, owner, version and chargeback label live with the table. CI blocks breaking changes before merge; a post-commit job tags snapshots. A catalogue entry filled in afterwards is not a contract. |
| Transformation | dbt Core or Dataform | The tools the brands already use, with model and contract in one repository and lineage for free. A new tool would be a migration in disguise. dbt Cloud and Semantic Layer stay out on purpose. |
| Semantic layer | Open metric spec in YAML, a custom compiler, a metric API | One definition outside the engines, compiled into four SQL dialects and checked by a conformance suite. Power BI models or LookML tie it to one tool; dbt Semantic Layer depends on dbt Cloud. |
| Metadata catalogue | OpenMetadata | Lineage from dbt and Iceberg snapshots, freshness against SLA, contract status and version notices. Apache 2.0. DataHub was a candidate, but OpenMetadata's quality and contract features are more mature. |
| Policy | Open Policy Agent | One policy bundle in the catalog and in Trino: the single-writer rule, masking by region and role, and aggregate-only access for the group principal. Apache Ranger is heavier and centred on Hadoop. |
| Identity | Keycloak | OpenID Connect for people and for engine service principals. One principal per brand and region, and a separate group principal. |
| Orchestration | Apache Airflow | Group jobs only: regional aggregation, consolidation, reconciliation and the conformance suite. Brand pipelines stay in their own schedulers. |
| Observability, secrets and delivery | Prometheus, Grafana, Loki, Tempo, OpenTelemetry; OpenBao; Argo CD | Contract breaches, freshness, read latency per engine, chargeback and conformance drift on one dashboard. GitOps runs three regions from one repository. OpenBao, because HashiCorp Vault moved to BSL. |
Open the kata: Open Lakehouse with Data Contracts and a Shared Semantic Layer
Regional Health Information Exchange
| Need | Choice | Why |
|---|---|---|
| Source adapters | Apache Camel with HL7, FHIR and FTP components | One stateless adapter per vendor, with HL7 v2 over MLLP, FHIR and file handling ready to use. Vendor quirks stay in the adapter. Mirth Connect moved to a closed licence in 2024. |
| Event backbone | Apache Kafka on Strimzi | A durable, replayable log ordered by source plus record number, so order holds per patient. The acknowledgement goes back only after commit. Redpanda was ruled out because of its Business Source License. |
| Schema and profile contract | Apicurio Registry and a FHIR Implementation Guide | Raw envelope schemas and regional FHIR profiles are versioned, and incompatible changes fail in CI. Confluent Schema Registry was ruled out because of its Community License. |
| Canonical clinical store | HAPI FHIR JPA Server on PostgreSQL | The FHIR R4 reference implementation, with search, versioning and interceptors ready. Consumers never see a vendor format. IBM FHIR Server is no longer maintained; Smile CDR offers commercial support. |
| Patient matching | Splink with an own matching service and a PostgreSQL link table | Probabilistic matching is required for the 15% without a national ID. A merge is a row with its evidence and can be undone. Commercial master patient indexes such as IBM Initiate and Verato are closed. |
| Consent policy | Open Policy Agent with Rego | Consent, identity, organisation and purpose of use in one policy bundle, run by the API, bulk export and research alike. Consent tables inside the data layer leak data in bulk reads. |
| Identity and authorisation | Keycloak with SMART on FHIR and OpenID Connect | Hospital identity federation, national e-ID brokering, SMART scopes and a purpose-of-use claim. Apache 2.0 with Red Hat support. Ory Hydra is the alternative. |
| API gateway | Apache APISIX with an OPA plugin | Token validation, the policy call, rate limits and audit headers. The FHIR server is reachable only through the gateway. APISIX, because Kong OSS moved features out of its open edition. |
| Audit ledger | immudb | A cryptographically verifiable, append-only record of who saw what and why, kept ten years and shown in the patient app. Amazon QLDB was ruled out: closed and tied to one cloud. |
| Cache | Valkey | Sub-millisecond reads for consent data and the precomputed patient summary. The cache key includes the consent version. Valkey over Redis because of the Redis 7.x licence change. |
| Raw archive and documents | Ceph RGW with S3 Object Lock | Raw HL7 and FHIR messages and PDFs as immutable objects for ten years, and the source for replay. Ceph, because the MinIO community edition narrowed. |
| Research pipeline | Apache Flink, Apache Iceberg, Trino | De-identification in the stream, one Iceberg table per project, and the university queries through Trino. Raw data never goes to the university. |
| Secrets and keys | OpenBao | The re-identification key is reachable only under the trusted third party's policy. Also PKI and database secrets. HashiCorp Vault was ruled out because of its Business Source License. |
| Observability | OpenTelemetry, Prometheus, Grafana, Loki, Tempo | Flow health per vendor, end-to-end freshness, matching rates, API p95 and audit write delay on one dashboard. |
Enterprise Data Warehouse Migration to BigQuery
| Need | Choice | Why |
|---|---|---|
| Data warehouse | BigQuery, commercial and allowed by the requirement | A board decision: serverless, columnar, EU multi-region. Reservations and byte caps make cost predictable; time travel and snapshots give a way back. Trino on Iceberg was not an option once the board chose BigQuery. |
| Change capture | Debezium on Kafka Connect, Apache Kafka on Strimzi | Streams SQL Server change tables by log position, with a watermark per table that reconciliation reads. Datastream is managed but lacks older SQL Server versions and schema history. Redpanda was ruled out over its licence. |
| Initial bulk load | bcp and distcp to Avro in Cloud Storage, then BigQuery load jobs | 60 TB once per wave, about two days over Interconnect. BigQuery Data Transfer Service is an alternative, but Hadoop data needs distcp anyway. One method means fewer tools. |
| Landing | Google Cloud Storage, EU, immutable | Avro files by table and day. Every BigQuery layer can be rebuilt from landing plus Dataform. A lifecycle rule moves objects to Nearline after 90 days. |
| Transformation | Dataform with SQLX | The key to SQL Server specialists being productive in three months: they write SQL, not Java. Assertions per model, Git, and CI/CD only. dbt Core is the portable alternative; Dataform needs no extra infrastructure. |
| Core model | Data Vault 2.0 with hubs, links and satellites | A source change touches only a satellite; marts and Power BI models stay put. Loads are idempotent MERGEs by business key and load time. A direct dimensional core would break marts on every source change. |
| Orchestration | Cloud Composer, managed Apache Airflow | Nightly DAGs per domain, wave DAGs, watermark checks and the 05:00 target. Airflow is open source, and the DAGs stay portable. |
| Migration control plane | Quarkus services on Cloud Run, Cloud SQL PostgreSQL | Dependency graph from SSIS XML, wave planner, reconciliation, a cutover switch per table and the close-owner flag. Commercial SSIS translators carry the packages' technical debt; dropping dead packages halves the work. |
| Reconciliation | Reconciliation service with Great Expectations | Row counts, column sums and a canonical row hash per partition, on both sides. Each difference is classified and its rule goes into the transformation. Great Expectations adds thresholds and a readable report. |
| Compatibility interface | Legacy view schema in BigQuery, reverse sync to the existing SQL Server | Forty consumers cannot change on one day. Old names live on as views, and 15 T-SQL consumers get a nightly reverse sync. No new SQL Server licence. Babelfish exists only on AWS, with partial T-SQL. |
| Catalogue, lineage, policy tags | Dataplex and BigQuery policy tags | Personal data columns are tagged in staging, access goes through policy tags, and an audit dataset logs reads. OpenMetadata is the open alternative; Dataplex gives BigQuery lineage automatically. |
| Identity | Keycloak with OpenID Connect, Google Cloud IAM groups | Keycloak for the migration cockpit and Grafana. BigQuery access goes through IAM groups per domain, never per-user grants. |
| Observability | Prometheus, Grafana, Cloud Monitoring, a Logging sink | Reconciliation differences, batch finish time, bytes per domain and legacy view use on one dashboard. Audit logs flow into BigQuery for GDPR Article 30 records. |
Open the kata: Enterprise Data Warehouse Migration to BigQuery
Real-Time Fraud Decisioning for Instant Payments
| Need | Choice | Why |
|---|---|---|
| Mainframe bridge | Kafka Connect IBM MQ source and sink | The mainframe only speaks MQ. The connector reads at least once, and a replayed message is dropped by payment ID against the compacted decisions topic. IBM publishes the connector under Apache 2.0. |
| Event backbone | Apache Kafka on Strimzi | A durable, replayable log ordered by customer key, where decision and snapshot share one transaction. One cluster stretched across both data centres, not two clusters joined by MirrorMaker 2. |
| Schema contract | Apicurio Registry with Avro | Versions the canonical PaymentRequest and snapshot schemas, and incompatible changes fail in CI. Confluent Schema Registry was ruled out because of its Community License. |
| Streaming features | Apache Flink | Keeps 90-day windows such as velocity, beneficiary history and device novelty as state. Computing them at query time would eat the 150 ms. Precomputing turns the lookup into one GET. |
| Online feature store | Valkey | Sub-millisecond GET with a local replica in each data centre. A BSD-licensed Redis fork, chosen because Redis 7.x moved to source-available licences. A fast path only, not the source of idempotency. |
| Decision service | Quarkus with Drools and ONNX Runtime | Rules and model run in one process, so there is no network hop. Drools rules hot-reload and are versioned. ONNX Runtime runs a model from any training framework on CPU in milliseconds. |
| Snapshot archive and offline store | Apache Iceberg on Ceph RGW with Object Lock | Cheap, immutable storage for seven years of snapshots, in a table format that Spark and Trino both read. Ceph, because the MinIO community edition narrowed. |
| Training and model registry | Apache Spark and MLflow | Training joins snapshots with analyst labels, so it uses the features the model actually saw. MLflow holds versions, metrics and approvals, and moves the champion pointer. |
| Case management | PostgreSQL with custom services | Alert queue, analyst outcomes and rule metadata. Small, relational and auditable. |
| Gateway and identity | Apache APISIX and Keycloak | Analyst and auditor screens use OpenID Connect, federated to the bank's identity provider, with roles and an audit log. APISIX, because Kong OSS moved features out of its open edition. |
| Secrets and keys | OpenBao | Connector credentials, encryption keys and PKI. The MPL-licensed fork of Vault, since HashiCorp Vault moved to the Business Source License. |
| Observability | OpenTelemetry, Prometheus, Grafana, Loki, Tempo | Per-stage traces show where the 150 ms goes. The Grafana stack's AGPL licence is OSI-approved and creates no obligation for internal use. |
Open the kata: Real-Time Fraud Decisioning for Instant Payments
BSS Modernisation and Real-Time Customer 360
| Need | Choice | Why |
|---|---|---|
| Mainframe change capture | Debezium Db2 connector on Kafka Connect | Reading the Db2 log through ASN capture tables is measurable and leaves the application schema alone. Querying tables directly would burn mainframe CPU. ASN Capture itself is a licensed IBM component. |
| Write path to the mainframe | Anti-corruption layer (ACL) service with Kafka Connect IBM MQ sink | The ACL turns a TM Forum order into an MQ transaction the mainframe already knows, with a correlation ID and a dedupe record before sending. The old model never leaks into the new domains. |
| Identity resolution | Customer crosswalk owned by the CRM domain, with master data matching rules | The same person lives under different IDs in 14 systems. Matching on national ID and contract number gives a confidence score. Below the threshold it shows as a possible match and is never merged. |
| Satellite systems | Debezium for PostgreSQL, Oracle and MySQL, plus file and API adapters | Twelve sources expose their logs, so change data capture gives the lowest load and best freshness. The two nightly-file sources carry their own freshness flag. No source pretends to be real time. |
| Event backbone | Apache Kafka on Strimzi | A durable, replayable log ordered by customer key. Compacted canonical topics can rebuild the read model without going back to the source. Redpanda was ruled out because of its Business Source License. |
| Schema governance | Apicurio Registry with Avro and TM Forum schemas | Canonical and business event schemas are versioned, and backward compatibility is enforced in CI. Confluent Schema Registry was ruled out because of its Community License. |
| Read model builder | Apache Flink | A stateful join of 14 streams on 20 million golden customer keys is exactly Flink's job. Exactly once into Kafka; at least once with idempotent upserts into PostgreSQL and OpenSearch. |
| Read model stores | PostgreSQL on CloudNativePG and OpenSearch | Command and query separation: the app reads one document from PostgreSQL, agents search in OpenSearch. OpenSearch over Elasticsearch because of the SSPL and AGPL dual-licence period. |
| Order workflows | Temporal | A 10-second order includes a mainframe MQ step and compensation. Temporal keeps state durable, retries and makes timeouts visible. Camunda 8 was ruled out after its move to a source-available licence. |
| Cohort routing | Unleash | Cutover and rollback must be a flag flip. The gateway and the workflow read the same flag. LaunchDarkly is commercial. Unleash keeps SSO and RBAC in its paid edition, but cohorts are open. |
| Data mesh | Apache Iceberg on Ceph, Trino, DataHub | Domain teams own their data products, with catalogue, lineage and SLA in DataHub. The centre provides platform and standards, not the data. Ceph, because the MinIO community edition narrowed. |
| Churn model | Apache Spark, MLflow, SHAP, ONNX serving | Daily scores for 20 million subscribers are a batch job, and event-triggered rescoring uses the exported ONNX model. SHAP reason codes explain scores to agents. Pseudonymised features are still personal data under GDPR. |
| Gateway, identity and policy | Apache APISIX, Keycloak, Open Policy Agent | TM Forum Open APIs behind one gateway, with OpenID Connect for subscribers and agents. OPA enforces purpose-based access to usage data in the gateway and in Trino. APISIX, because Kong OSS cut features. |
| Secrets and keys | OpenBao | Mainframe credentials live only in the ingest namespace, and the pseudonymisation key stays in-house. HashiCorp Vault was ruled out because of its Business Source License. |
| Observability | OpenTelemetry, Prometheus, Grafana, Loki, Tempo | Mainframe CPU budget, capture lag, order step traces and the dual-run difference rate on one dashboard. It is the migration committee's single input. |
Unified Marketing Intelligence on Google Cloud
| Need | Choice | Why |
|---|---|---|
| GA4 streaming ingest | Server-side Google Tag Manager collector, Pub/Sub, BigQuery subscription | GA4 has no native Pub/Sub export. A BigQuery subscription needs no code, and a streaming view meets the two-minute target. Dataflow only for real enrichment; one Dataflow job per stream is needless cost. |
| Batch file ingest | Cloud Composer, managed Airflow | Competitor files need retries, backfill and dependencies, a classic orchestration problem. Airflow is an open standard, and the managed service keeps the DAGs portable. |
| API ingest | Cloud Run connector service | Social, Ads and Search Console APIs need token handling, rate limits and a watermark per source. A small, stateless service that scales; Composer only triggers it on schedule. |
| Warehouse access | Authorised views | The warehouse already sits in BigQuery in the same organisation; no Analytics Hub listing needed. A copy adds cost and a second truth. Views read the same rows, and the data team keeps ownership. |
| Storage and compute | BigQuery with layered datasets | Every source already lands here; another engine means a second copy and a second truth. Marts are partitioned by event date. This lock-in is deliberate; Trino or ClickHouse on Iceberg is the named alternative. |
| Transformation | Dataform with SQL and assertions in Git | SQL with versions and a dependency graph, and the assertions are the publish gate's test set. The same model can be rebuilt in dbt Core, so the dependency is low. |
| Publish gate | Dataform assertions and a view swap by publish ID | Candidate tables are built and tested, then CREATE OR REPLACE VIEW points the marts at them. On failure the views stay on yesterday. Rollback is a view change, with no reload. |
| Semantic layer | Metric contract in Git and a metric API on Cloud Run | The heart of the design, and deliberately our own code. One definition feeds Power BI, the LLM and the agents. Cube or dbt Semantic Layer could replace it, but the definition stays ours. |
| BI | Power BI tabular model, import with incremental refresh | 400 users are trained, it is the company standard and the licence exists. Import mode keeps clicks away from BigQuery. Measures and row-level roles are generated from the contract. Superset is the open alternative. |
| LLM | Vertex AI Gemini on EU endpoints | The model turns a question into metric, filter and date range as structured output; the metric API produces the number. Data stays inside the perimeter. An open-weight model on vLLM would keep the same interface. |
| Agents | Vertex Agent Engine with an approval queue | The tools are the metric API, the catalogue and a proposal writer. There is no raw SQL tool and no direct Ads change. Every action is stored with evidence, approver and a rollback record. |
| ML feed | Vertex AI Feature Store | Features come from the same marts, so there is no second pipeline and the model uses the report's definitions. Feast on BigQuery follows the same pattern. |
| Catalogue and contract | Dataplex | Certified versus raw shows as a badge. Lineage is native in BigQuery, and Dataplex tags carry contract version and freshness target. Metadata can be exported to OpenMetadata. |
| Identity, privacy and secrets | IAM, policy tags, row access policies, Cloud DLP, Secret Manager | Column policy tags stop the marketing role from selecting identifiers. Row access policies filter brand and country on ad-hoc and LLM paths. API tokens rotate automatically. |
Open the kata: Unified Marketing Intelligence on Google Cloud
Competitor Price and Markdown Intelligence Platform
| Need | Choice | Why |
|---|---|---|
| Collection law and courtesy | Site registry in Git and a BigQuery table | Robots rules, rate, country legal status and selector version in one place; no worker starts without its row. Legal can stop a country without an engineer. A config file inside the worker cannot be audited. |
| Collector fleet | Cloud Run Jobs with Playwright and HTTP workers | Scales to zero when done, with one job and token bucket per site and country. An idle Kubernetes pool still bills; Cloud Functions are too limited for a browser. Plain HTTP stays the cheap path. |
| Fixed egress addresses | Cloud NAT with a registered IP per country | Legal needs to know which address we collect from. Proxy rotation counts as evading blocks and is out of scope. Each country gets a declared IP and an honest user agent. |
| Scheduling | Cloud Composer, managed Airflow | Daily plans, retries, re-parse campaigns and supplier files are classic orchestration. The DAGs stay portable. |
| Raw snapshot store | Cloud Storage bronze with a five-year retention lock | Reproducibility only comes from an untouched source, and the lock stops deletion even by admins. Raw HTML in BigQuery would be expensive and cannot be locked. The Archive class keeps five years affordable. |
| Events | Pub/Sub | Captured, parsed and matched events trigger the next stage instead of direct calls. Kafka would be too much here: no stream processing, no ordering need, a few tens of millions of small messages a day. |
| Parsing | Cloud Run service with versioned selectors | A stage separate from collection. When a selector breaks, snapshots are parsed again and the site is not visited again. Personal data such as review authors is dropped here and never reaches the bronze tables. |
| Matching | Rules, Vertex embeddings with pgvector, Gemini in the grey zone, a human queue | Rules and embeddings settle about 80% deterministically. The LLM sees only the grey zone, with a daily cap and 5% sampled review. pgvector is open and portable; Pinecone-style managed services add needless dependency. |
| Warehouse and transformation | BigQuery EU and Dataform with bronze, silver and gold | A partitioned, columnar warehouse for five years at 8 million rows a day. Running our own cluster would break the fixed budget. Dataform SQL lives in Git and stays compatible with dbt. |
| Index, elasticity and advice | BigQuery ML and a SQL rule engine | The daily run is deterministic: same input, same output. Rules draw the decision space and the model only ranks within it. A separate Vertex model service is unnecessary for a daily batch. |
| Serving and identity | Intelligence API on Cloud Run, Keycloak | A read-only API. No service account can write to the pricing system, so "advice only" is enforced by IAM. Keycloak gives open-source OpenID Connect and country roles. |
| Trend summary | Gemini, weekly, with citations required | Every sentence cites a raw snapshot and carries a "summary" label; uncited sentences are not produced. Golden, drift and injection tests run on every release. |
| LLM tracing and evaluation | Langfuse on Cloud Run and Cloud SQL | Every LLM call is traced with cost and output, and evaluation runs gate releases. The core is MIT-licensed. Managed tracing products would move data outside the perimeter. |
| Dashboard | Looker Studio with BI Engine | A free dashboard on gold, with the legal team's collection audit view in the same place. Superset is the open alternative but needs one more service to run. |
Open the kata: Competitor Price and Markdown Intelligence Platform
Governed GenAI Data Agents with LLM Evaluation
| Need | Choice | Why |
|---|---|---|
| API gateway and identity | Apache APISIX and Keycloak | OpenID Connect login, then a short-lived signed user claim that travels with every agent and tool call. Per-user rate limits and an audit log. APISIX, because Kong OSS moved features out of its open edition. |
| Agent runtime | LangGraph on Kubernetes | Agent steps form an explicit graph. Tool calls are schema-checked, each agent's tool allow-list comes from Git, and every step emits a span. Managed frameworks like Bedrock Agents hide the tool surface and traces. |
| Model gateway | LiteLLM | Two providers and vLLM behind one OpenAI-compatible interface. Provider keys live only here, with quotas per team, routing per agent and cost labels. Restricted documents route only to vLLM. The MIT core is enough. |
| Open-weight model serving | vLLM with a 32B model under Apache 2.0 | Two replicas with tensor parallelism of two cover the estimated peak of about 1,760 tokens a second. The on-premises path for sensitive document classes and for cost. The Llama community licence is not OSI-approved. |
| Metric API | Certified definitions built with Dataform, plus a thin service | The agent picks a metric ID and filters; the layer compiles SQL and applies the user's row and column policy. Free text-to-SQL caused the wrong revenue figure, so it is closed by design. |
| Catalogue and data products | OpenMetadata | Agents see only products with a contract for schema, owner, SLA and policy; lineage and owner go into the evidence. DataHub is also Apache 2.0, but OpenMetadata's contract and policy model is more direct. |
| Document store for retrieval | PostgreSQL with pgvector, and OpenSearch | Access list and class label per document come from the document system. The permission filter runs before vector ranking, so the model never sees an unauthorised chunk. OpenSearch over Elasticsearch after its licence change. |
| Policy decision | Open Policy Agent as a sidecar | Decides agent, user and resource in every tool service. Allow-lists and class policies are versioned bundles in Git, and every denial is counted. Policy is enforced on the data, not in the prompt. |
| Evidence store and archive | Own PostgreSQL evidence store, outbox, Debezium, Apache Iceberg for two years | Evidence row and outbox row in one transaction; the answer is released only after commit. Langfuse v3 cannot take an outbox row because of its ClickHouse store and async ingest, so evidence lives separately. |
| LLM observability | Langfuse v3 with ClickHouse, Valkey and blob storage | Spans, latency, tokens and evaluation scores, ingested asynchronously for observability only. Valkey because of the Redis 7.4 licence. Arize Phoenix was ruled out over its Elastic 2.0 licence. |
| Evaluation platform | Own CI runners, Langfuse evals, a policy file in Git | Golden set, tool-order score, injection and exfiltration suites, and a provider tolerance test per agent. Thresholds sit in a versioned policy file. Commercial eval services such as LangSmith were ruled out. |
| Approval flow | Flowable with BPMN | Every action with a side effect becomes a human task, recording who approved what and when, with rollback. Approval never runs automatically. Camunda 8 was ruled out after its move to a source-available licence. |
| Deterministic production | Cloud Composer and Dataform | The onboarding agent drafts code and contracts at design time; a person reviews and merges. Composer runs the merged code without an LLM. Chosen because it is already a platform component. |
| Governance register and cost | PostgreSQL with a small UI, LiteLLM labels, Grafana | Agent, risk class, owner, evaluation status, allowed models, incidents and EU AI Act duties. The register is a release precondition. Daily cost per agent and team comes from LiteLLM labels. |
Open the kata: Governed GenAI Data Agents with LLM Evaluation
Claims Intake with Document Intelligence
| Need | Choice | Why |
|---|---|---|
| Multichannel intake | Apache Camel | IMAP, REST and SFTP connectors are ready, and every channel produces the same claim.received event and storage layout. A route per channel instead of custom code. Apache 2.0 with Red Hat support. |
| Document store | Ceph RGW with S3 Object Lock | Originals and OCR text stay unchanged for seven years. GPU workers and Iceberg use the same store through the S3 API. Ceph, because the MinIO community edition narrowed. |
| Event backbone and GPU job queue | Apache Kafka on Strimzi | Claim events and GPU jobs share one backbone; the priority and batch lanes are two topics. The queue absorbs GPU outages. I reused Kafka instead of adding RabbitMQ. Redpanda was ruled out over its licence. |
| Text recognition and document classes | PaddleOCR or docTR | Fast on GPU, with layout and table extraction, under Apache 2.0. Commercial OCR APIs are excluded because no data may leave the company. Tesseract is too weak on photographed documents. |
| Language model serving | vLLM with an open-weight model | Continuous batching gets the most from 8 GPUs. Output is constrained to a JSON schema, with confidence from calibrated log-probs. Apache 2.0 weights such as Qwen or Mistral; Llama's licence is not OSI-approved. |
| Vision model | Open-weight vision model on vLLM | Damage class and duplicate-photo detection in the same serving layer, with its own model record. A separate vision platform is unnecessary. |
| Workflow | Flowable with BPMN | Every step from intake to payment is visible, with SLA timers and escalation built in. Human approval is a task type. Camunda 8 went source-available, and Camunda 7 is in maintenance mode. |
| Rule engine | Drools with DRL and DMN | Simple claims under 2,000 EUR are decided by deterministic, versioned, testable rules. The reason for a negative decision comes from the fired rule. This is where "the model does not decide" becomes concrete. |
| Fraud graph | PostgreSQL with Apache AGE | Repeat and ring patterns across repair shops, parties, invoices and accounts in Cypher, in the same database as the relational data. Neo4j Community is GPL and its clustering is commercial. JanusGraph adds operational load. |
| Decision audit record | PostgreSQL outbox, Debezium, Apache Iceberg on Ceph | The audit row and outbox row go into Flowable's PostgreSQL transaction, and Debezium publishes from the log, so there is no dual write. Iceberg keeps seven years, and auditors query it with Trino. |
| LLM tracing | Langfuse | Prompt, version, tokens and latency for every call, plus evaluation scores and shadow comparisons. The core is MIT-licensed; the commercial enterprise features are not needed. |
| Model and prompt registry | MLflow and Git | Weight hash and version in MLflow; prompt templates and the evaluation set in Git, with a nightly regression run on a GPU. Promotion needs four-eyes approval, and rollback is a pointer change. |
| Gateway and identity | Apache APISIX and Keycloak | Adjuster, investigator, auditor and customer roles through OpenID Connect, with a gateway audit log and rate limits. APISIX, because Kong OSS moved features out of its open edition. |
| Secrets and keys | OpenBao | Envelope encryption with a data key per claim. Destroying the key at the end of retention is erasure; a litigation hold keeps it. HashiCorp Vault was ruled out because of its Business Source License. |