Case study · P—08
Data Governance, Metadata Management and MDM
- Role
- Implementation · Big Data Management & Data Architecture Manager
- Period
- August 2019 – May 2022
- Organisation
- LC Waikiki
- Stack
- Data catalog, Data lineage, Master data management (MDM), Metadata management
- Status
- Completed
In short
Data governance, metadata management and master data management (MDM) across 100+ source systems, with data catalog and lineage tooling.
Context
LC Waikiki’s enterprise data warehouse is fed by 100+ source systems. In this period the same role owned the Cloudera Hadoop platform, the SQL Server warehouse and the OLAP layers.
Role: implemented data governance, metadata management and MDM, as Big Data Management & Data Architecture Manager.
Problem
With 100+ sources, the questions are where a dataset comes from, what it means and which master record is correct.
Approach
- Governance: data governance across source systems.
- Metadata management and data catalog: a catalog of the data assets and their metadata.
- Lineage: lineage tooling to trace data from source systems onward.
- Master data management (MDM): across source systems.
Outcome
Data governance, metadata management and MDM implemented across 100+ sources, with data catalog and lineage tooling.
Lessons
- The scope was the source systems, not only the warehouse.
- Catalog and lineage carried over to the cloud platform: the current Google Cloud platform also keeps catalog and lineage across 100+ sources.