Case study · P—05
Dataflow Development with Apache Beam (Java)
- Role
- Pipeline development · Data Solution Architect
- Period
- March 2025 – June 2025
- Organisation
- LC Waikiki
- Stack
- Apache Beam (Java), Dataflow, Pub/Sub, BigQuery Storage Write API, BigQuery, SQL Server
- Status
- Completed
In short
A real-time Pub/Sub to BigQuery pipeline via the Storage Write API, and a bulk BigQuery to SQL Server pipeline with initial-load and append modes.
Context
LC Waikiki’s enterprise data warehouse moved to BigQuery in 2023–2024. Data also has to flow the other way: operational systems that run on SQL Server need data from BigQuery.
Role: built the pipelines, as Data Solution Architect.
Problem
Two directions needed scalable, flexible pipelines: real-time events into BigQuery, and bulk data from BigQuery back to SQL Server.
Approach
- Real time: a streaming pipeline from Pub/Sub to BigQuery, written with Apache Beam (Java) on Dataflow and writing through the BigQuery Storage Write API.
- Bulk integration: a pipeline from BigQuery to SQL Server with two options, initial load and append.
- Standard: Pub/Sub → Dataflow (Beam) → BigQuery Storage Write API became the organization-wide real-time pattern, with reverse pipelines from BigQuery to SQL Server next to it.
Architecture and technology
Apache Beam (Java) on Dataflow; Pub/Sub as the event source; BigQuery Storage Write API for writes into BigQuery; SQL Server as the target of the bulk pipeline.
Outcome
- Pub/Sub, Dataflow and the BigQuery Storage Write API are the company-wide pattern for real-time streaming.
- Bulk data moves from BigQuery to SQL Server in initial-load and append modes.
- Better scalability, flexibility and performance of cross-platform pipelines.
Lessons
- The standard is backed by working code: the real-time pattern came with a running Beam pipeline on Dataflow, in line with the role’s practice of reference implementations.