GA4 Analytics Modernization: Gold Layer, Streaming and Anomaly Detection

Context

LC Waikiki’s GA4 analytics for digital marketing and e-commerce runs on BigQuery.

Role: led the modernization program, as Data Solution Architect.

Problem

The old setup was expensive and slow, and people had stopped fully trusting the numbers. BigQuery cost $9K per month, the heaviest queries ran for 46 minutes, and 138 scheduled queries were in place.

Approach

1. Storage cleanup first: 2021–2022 data archived to Cloud Storage, billing fixed on 4 of 38 datasets, and 46 unused high-volume tables removed.
2. Processing rework: 28 data models, ETL flows and queries reworked; the 138 scheduled queries reviewed and 56 unused jobs decommissioned; idle services turned off.
3. Gold layer: one e-commerce model for cart, checkout, conversion and revenue, so funnel and campaign metrics live in a single trusted place.
4. Real time: an Apache Beam (Java) pipeline on Pub/Sub, writing to BigQuery through the Storage Write API on top of the Gold model.
5. Anomaly detection: rules and statistical checks over GA4 events and Gold metrics, to catch tracking gaps and data drift before they reach reports.

Architecture and technology

GA4 data and the Gold e-commerce model in BigQuery; archive in Cloud Storage; streaming with Pub/Sub, Apache Beam (Java) and the BigQuery Storage Write API; anomaly checks over events and Gold metrics.

Outcome

• Monthly BigQuery cost: from $9K to $3K.
• Heaviest queries: from 46 minutes to 8 seconds.
• Storage cost down 65%; processing cost down 60%.
• Scheduled queries: from 138 to 82.
• A Gold layer that shortened the decision loop for the digital team.

Lessons

• Cleanup came first. Archiving, billing fixes and removing unused tables and jobs came before the new Gold layer and the streaming work.
• Trust was part of the scope: one place for funnel and campaign metrics, and checks that catch tracking gaps before reports do.