Ingesting and processing the data

Your team runs two independent Cloud Composer DAGs. The first DAG ingests raw sales events into BigQuery every hour. A separate downstream DAG builds hourly aggregate reports, but it must only run after the corresponding ingestion DAG run for that hour has fully completed. The teams want to keep the two DAGs separate for ownership reasons, and the downstream DAG should not start processing until the upstream data is confirmed present. What is the best way to coordinate these two DAGs?

Your team runs a daily Cloud Composer (Airflow) DAG that ingests partner sales files, transforms them in Dataproc Serverless, and loads results into BigQuery. Downstream analysts must have refreshed data by 08:00 each morning. Occasionally a partner uploads files late, causing the pipeline to finish after the deadline without anyone noticing until analysts complain. You want Airflow itself to automatically detect when a task takes longer than expected and notify the on-call engineer, without failing the DAG run. What is the most appropriate approach?

Your team runs about 300 DAGs in a Cloud Composer 2 environment. Recently, the Airflow scheduler has become sluggish: tasks are queued but slow to start, and the DAGs list in the UI takes a long time to refresh. On investigation, you find that several DAG files execute heavy top-level code — including API calls to fetch configuration and expensive pandas transformations — every time the file is parsed. What is the most effective change to improve scheduler performance?

A retail analytics team uses Cloud Composer (Apache Airflow) to orchestrate a daily batch pipeline that reads the previous day's sales files from Cloud Storage, transforms them in Dataflow, and appends results into a BigQuery table partitioned by date. Occasionally a task fails midway and Airflow automatically retries it. The team also frequently runs manual backfills for past dates. They notice that retries and backfills sometimes produce duplicate rows in BigQuery. What is the best way to redesign the pipeline so that retries and backfills remain safe?

Your team uses Cloud Composer to orchestrate ingestion DAGs. The same DAG code must run in dev, staging, and prod environments, each pointing to different BigQuery datasets, Pub/Sub topics, and service accounts. Developers currently hardcode these values, causing errors when DAGs are promoted between environments. You want a maintainable approach that keeps a single DAG codebase while allowing per-environment configuration without editing the DAG files. What should you do?

Your team uses Cloud Data Fusion to build a batch ETL pipeline that reads CSV files from Cloud Storage, applies several transformations, and writes to BigQuery. Before deploying the pipeline to production, business analysts report that certain output records have unexpected null values. You need to inspect how each transformation stage processes a small sample of the source data without incurring the cost of provisioning a full Dataproc cluster or writing to the production BigQuery table. What should you do?

A retail company runs an operational MySQL database on-premises and wants continuous, near-real-time replication of insert, update, and delete operations into BigQuery for analytics. The data engineering team prefers a fully managed, low-code solution that handles ongoing change data capture without writing custom Dataflow code, and that can automatically create and keep target BigQuery tables in sync with the source schema. Which approach best meets these requirements?

A retail analytics team includes several business analysts who understand data transformation logic but do not write code. They need to build repeatable ETL pipelines that ingest CSV files from Cloud Storage, join them with reference data, apply cleansing rules, and load the results into BigQuery. Leadership wants the analysts to be self-sufficient, wants the pipelines to be version-controlled, and prefers a managed service that provides a visual, drag-and-drop interface with prebuilt connectors and transformations. Which approach best meets these requirements?

A retail analytics team uses Cloud Data Fusion to build a batch pipeline that ingests daily CSV exports from partner SFTP servers into BigQuery. Occasionally a partner adds new columns or sends malformed rows, which currently causes the entire pipeline run to fail. The team wants the pipeline to continue processing valid records, route bad records to a separate location for later inspection, and avoid manual code changes for each new column. What is the most appropriate approach within Data Fusion?

A logistics company receives daily inventory files (about 5 GB each) dropped into a Cloud Storage bucket every night at 2 AM. The data engineering team must transform and load these files into BigQuery so that analysts have fresh data by 6 AM. There is no requirement for sub-hourly latency, and the files arrive as a predictable, bounded set each night. Which pipeline design best fits these requirements while minimizing operational cost?

Your data engineering team maintains several Dataflow pipelines whose source code lives in a Git repository. Currently, engineers manually build templates and deploy them from their laptops, which has led to inconsistent deployments and undetected regressions reaching production. You are asked to implement a CI/CD process that automatically runs unit and integration tests on every merge request, and only promotes validated pipeline artifacts to the production project after approval. Which approach best meets these requirements?

A retail analytics team runs a streaming Dataflow pipeline that must join two Pub/Sub streams: an 'orders' stream and a 'payments' stream. An order and its matching payment share an order_id but can arrive up to 10 minutes apart and out of order. The team needs to correlate each order with its payment to compute settled revenue, emitting a result once both events for an order_id are seen (or marking the order unsettled if no payment arrives within the window). Which pipeline design best meets these requirements?

A Dataflow streaming pipeline aggregates ad-impression counts per publisher using a GroupByKey followed by a Sum combiner. A handful of very large publishers generate the vast majority of events, and monitoring shows severe data skew: a few workers are overloaded processing these 'hot keys' while others sit idle, causing backlog growth. You must reduce processing latency without changing the final aggregation results. What is the most effective change?

A retail company runs a critical streaming Dataflow pipeline that reads from Pub/Sub, applies transformations, and writes aggregated sales into BigQuery. The data engineering team needs to deploy a new pipeline version that adds a field to the output schema, but they must not lose any in-flight messages or duplicate already-processed data during the transition. Which approach should they use to stop the currently running job?

A gaming company processes a Pub/Sub stream of player events. Analysts want to measure the duration of each player's continuous play session, where a session ends after a player is inactive for 30 minutes. Sessions vary widely in length, and there is no fixed period boundary. Which Dataflow windowing strategy should you use to compute per-player session metrics?

A retail company publishes order events to a single Pub/Sub topic. A Dataflow streaming pipeline must route events to different BigQuery tables based on the event's region, but the region value is only available inside the JSON message payload, not as a Pub/Sub attribute. The team wants to avoid deserializing the full payload twice and wants the routing decision to happen efficiently within the pipeline. What is the most appropriate approach?

A logistics company receives IoT telemetry from delivery trucks. Sensors publish location updates every 5 seconds to a Pub/Sub topic, and the analytics team needs a dashboard that reflects truck positions with no more than 30 seconds of delay. Separately, each night a partner uploads a large CSV file of completed-delivery records to a Cloud Storage bucket, which must be joined with the telemetry and loaded into BigQuery once per day. You must design the ingestion approach. Which combination correctly matches processing modes to these two workloads?

A retail company runs a Dataflow streaming pipeline that reads transaction events from Pub/Sub, enriches them with a REST call to an external inventory API, and writes results to BigQuery. During a promotional event, the external API begins returning intermittent HTTP 503 errors. The team observes that some elements fail permanently and cause the pipeline to stall, while transient failures should be retried without dropping data. What is the most appropriate way to build robust error handling into this transformation step?

A retail analytics team streams point-of-sale events into Pub/Sub and processes them with a Dataflow streaming pipeline. Business users need a running total of sales revenue per store, emitted every hour on the hour, so they can compare hourly performance across the day. Events can arrive slightly out of order, and the team wants each hourly figure to reflect only the transactions that occurred within that clock hour. Which windowing and triggering configuration should the pipeline use?

A retail company runs a streaming Dataflow pipeline that ingests order events from a single Pub/Sub topic. The business now wants to consolidate events from five additional regional Pub/Sub topics into the same processing logic, applying identical transformations and writing to one BigQuery table. The team wants to minimize operational overhead and avoid running six separate pipelines, while retaining the ability to identify each event's source region downstream. What is the best approach?