First Data Engineering Team, Hadoop Platform and Data Lake

Context

At the start of this work, LC Waikiki’s data platform was built on SQL Server and SSIS, and the company had no data engineering team.

Role: built the company’s first data engineering team (4-6 engineers) and led the move to a hybrid platform, as Data Engineering Team Lead.

Problem

Data from CRM, ERP, clickstream and third-party sources had to come together for reporting and machine learning, and heavy SSIS jobs carried pricing, supply chain and customer 360 workloads.

Approach

• Hybrid architecture: Hadoop next to the relational databases, instead of a single SQL Server and SSIS stack.
• Production cluster: a 15-node Cloudera CDH cluster with namespaces, security and resource isolation.
• First data lake: multi-source (CRM, ERP, clickstream, third-party data), feeding reporting and machine learning.
• Pipelines: heavy SSIS jobs replaced with Spark, Hive and Impala pipelines for pricing, supply chain and customer 360.
• Practices: Git, code review, automated testing and on-call introduced.

Architecture and technology

Cloudera CDH (Hadoop) with Spark, Hive and Impala, alongside SQL Server. Sources: CRM, ERP, clickstream and third-party data. Consumers: reporting and machine learning.

Outcome

• The company’s first data engineering team (4-6 engineers).
• A production 15-node Cloudera CDH cluster.
• The first multi-source data lake, feeding reporting and machine learning.

Lessons

• Engineering practice came with the platform: Git, code review, automated testing and on-call were introduced together with the new pipelines.
• The Hadoop platform was later one of the sources of the warehouse migration to BigQuery (2023–2024).