Question

Your ML team needs to preprocess 8 TB of raw clickstream logs stored in Cloud Storage before training. The pipeline must join the logs against a 200 GB user-profile table in BigQuery, apply session-based windowing, and write cleaned features back to BigQuery for downstream training. Multiple teams will reuse this pipeline on a scheduled daily basis. Which approach best satisfies scalability, reusability, and maintainability requirements?

Accepted Answer

Build an Apache Beam pipeline on Dataflow that reads logs from Cloud Storage, joins with the BigQuery user-profile data as a side input or via a BigQuery source, applies session windows, and writes to BigQuery, deployed as a reusable Flex Template.

Answer

Load all data into a single BigQuery table and perform the joins and windowing with a scheduled SQL query, exporting results to a CSV in Cloud Storage.

Answer

Write a Python script in a Vertex AI notebook using pandas to load the logs and profiles into memory, perform the joins, and upload the results to BigQuery.

Answer

Use a Dataproc Spark job triggered manually each day, storing intermediate results in HDFS and copying the final output to BigQuery.