Open Lakehouse with Data Contracts and a Shared Semantic Layer

Problem

An e-commerce holding has six brands acquired over eight years. Each brand has its own data stack: two on BigQuery, two on Snowflake, one on Databricks and one on an on-premises PostgreSQL warehouse. Group finance consolidates monthly in spreadsheets, and the numbers never match brand dashboards, because “net revenue” is defined differently in each brand. Brands will not accept a forced migration. The group will not accept a seventh copy of the data. Three brands operate in the EU, two in the UK and one in Türkiye, so personal data must stay in its region.

Approach

The design makes the table format and catalog independent of the engines, and the metric definition independent of any SQL dialect. Each shared data product is produced in the owning brand’s engine and written once to a regional bucket as an Apache Iceberg table. One REST Iceberg catalog, Apache Polaris or Lakekeeper, lets all four engines read the same table. A contract lives next to the table and blocks breaking changes in CI. Metrics are defined once in YAML and compiled for each engine.

Architecture

• Tables and catalog: Iceberg in regional object storage, with one REST catalog and short-lived credentials per engine.
• One writer per table: the owner’s engine. A policy in Open Policy Agent rejects commits from any other principal.
• Contracts: Open Data Contract Standard YAML with schema, PII class, SLA, owner and chargeback label. A post-commit job tags a snapshot as verified, and consumers read only verified snapshots.
• Semantic layer: a compiler turns the metric spec into BigQuery, Snowflake, Databricks and Trino SQL. Power BI, Looker, notebooks and an LLM assistant use one metric API.
• Regions and catalog: regional Trino pools aggregate data. OpenMetadata shows lineage, freshness and contract status.

Decisions and trade-offs

• Federate, do not migrate. Six brands built teams and tools over eight years, and they reject a migration. If a migration failed, group data would sit empty for two years. One open format and one catalog give one truth and one copy without that risk. The cost is supporting four SQL dialects and four engines.
• One open catalog, not one per engine. The REST catalog is the common denominator of the four engines. Per-engine catalogs would mean three policy engines and three permission models. The cost is a catalog the group must operate. If it goes down, no new commits land, but engines keep reading the snapshots they already have.
• Single-writer rule. Each table has one writer: the owning brand’s engine, and OPA rejects any other commit. Concurrent multi-engine writes would promise a consistency no engine guarantees, so I promise no cross-engine transactions. The cost is handling brands that run two engines during a transition.
• Pinned, verified snapshots. “Consistent across four engines” means the same snapshot, the same files, the same number. Consumers read the snapshot tagged as verified, not the latest one. If a brand renames a column and skips CI, the new snapshot stays untagged. The dashboard shows a stale but correct figure with its freshness stamp, never a wrong one.
• Conformance suite. It runs each metric on four engines against one snapshot and compares to the cent. A dialect drift fails the suite and blocks that spec version. The cost: the group may have to write and maintain its own compiler.
• Residency in the architecture, not in a policy. Row-level personal data never leaves its region. Regional Trino pools aggregate, and the group principal sees only tables classed as aggregate without PII. Row-level tables are not even visible to it. The cost is that cross-region analysis works only on aggregates.

What I would revisit

• Native writes per engine. Which engine versions can write to the Iceberg REST catalog directly? BigLake and Snowflake support depends on the version, and a brand without it needs Trino as its writer.
• Semantic spec maturity. Is there a ready open-source compiler, or does the group write its own and open it?
• Two engines in one brand. During a transition, is a temporary second writer acceptable, or is a writer change a new contract version?
• Disconnected Turkish region. How many hours can the on-premises brand’s commits wait if the EU control plane is unreachable?

Outcome notes (design assumptions, not measurements)

This is a design exercise, not a running platform. The output is a four-page diagram set, a one-page whiteboard version and a 16-page Turkish PDF. The two-day finance consolidation and the 0.1% reconciliation tolerance are targets from the brief and the design.

Sources

• Architecture PDF (Turkish, 16 pages).
• draw.io diagrams (English labels): four-page set and a one-page 30-minute version.