Question

Your data engineering and ML teams share a clickstream pipeline. Raw events arrive continuously on a Pub/Sub topic and must be cleaned, windowed into 5-minute session aggregates, and written to BigQuery so both teams can build features from the same processed data. The processing logic includes stateful sessionization and must handle late-arriving events with watermarks. Which approach best meets these requirements?

Accepted Answer

Build an Apache Beam streaming pipeline on Dataflow that reads from Pub/Sub, applies session windows with allowed lateness, and streams results to BigQuery.

Answer

Schedule an hourly BigQuery scheduled query that pulls raw events from an external Pub/Sub table and aggregates them into sessions.

Answer

Use a Cloud Function triggered per Pub/Sub message to compute session aggregates and insert rows into BigQuery.

Answer

Export Pub/Sub messages to Cloud Storage in batches and run a nightly Dataproc Spark batch job to compute session windows.