Designing data processing systems

A retail company is migrating a nightly on-premises ETL job to Google Cloud. During the migration cutover period, both the legacy on-premises pipeline and the new Dataflow pipeline will run in parallel against the same source data for two weeks to build confidence before decommissioning the old system. The data engineering team must prove that the new pipeline produces identical aggregated results to the legacy system before switching over. Which approach best validates output fidelity during this parallel-run period?

A retail analytics team stores a large customer transactions table in a BigQuery dataset called 'raw_finance'. A separate marketing team needs read access only to aggregated, non-PII summary data derived from that table. The finance team must retain full control of the underlying dataset and must not grant the marketing team any direct access to 'raw_finance'. What is the most appropriate way to design this data-sharing solution?

A financial services company stores critical transactional data in a BigQuery dataset located in the us-central1 region. Compliance requires that the company be able to recover the dataset with a Recovery Point Objective (RPO) of no more than 4 hours if the entire region becomes unavailable. The team wants a solution that is automated, cost-effective, and requires minimal ongoing operational overhead. Which approach should they implement?

Your team runs a transactional application backed by a Cloud SQL for PostgreSQL instance in the us-central1 region. A compliance review requires that the database withstand a full regional outage with a recovery point objective (RPO) close to zero and a recovery time objective (RTO) of minutes, not hours. Automated backups are already enabled but stored only in the primary region. Which approach best meets these disaster recovery requirements?

A financial services company must retain 7 years of transaction audit logs to satisfy regulatory requirements. The logs are queried frequently during the first 30 days for operational troubleshooting, occasionally between 30 and 90 days for reconciliation, and rarely afterward but must remain immutable and undeletable for the full retention period. The team wants to minimize storage cost while guaranteeing that no user, including project administrators, can delete or overwrite the logs before 7 years elapse. What should you implement?

A data engineering team stores critical curated datasets in a Cloud Storage bucket that serves as the source of truth for downstream BigQuery loads. Recently, an automated cleanup job misconfigured a prefix and permanently deleted several current object versions, causing a multi-hour outage while data was rebuilt from raw sources. Leadership wants a solution that allows quick recovery of individual overwritten or deleted objects without maintaining separate backup copies or scripts. What should the team implement?

A financial services company uses Customer-Managed Encryption Keys (CMEK) via Cloud KMS to encrypt data in BigQuery and Cloud Storage. A new compliance policy requires that encryption keys be rotated automatically every 90 days, and that the security team retains sole authority to disable or destroy keys while data engineers continue to run pipelines without interruption. What is the most appropriate way to meet these requirements?

A retail company stores a BigQuery table containing customer purchase records. The table includes columns such as customer_id, purchase_amount, and a sensitive credit_card_hash column. A team of financial analysts needs to run aggregate queries on purchase_amount but must never be able to read the credit_card_hash column. Compliance requires that access controls be enforced at query time without duplicating or restructuring the table into separate tables. Which approach best meets these requirements?

A media analytics company runs nightly Spark batch jobs on a self-managed Hadoop cluster on-premises. Leadership wants a cloud design that avoids vendor lock-in, so future workloads could potentially run on another cloud provider with minimal rewrites. The engineering team plans to use Dataproc but is concerned that tightly coupling their data and job logic to proprietary services will make a future move costly. Which design approach best preserves portability while running on Google Cloud?

A European financial services company must store and process customer transaction data in BigQuery. Regulatory requirements mandate that both the data and the encryption keys used to protect it remain physically within the EU, and the company must be able to permanently revoke access to the data across all copies at any time. Which approach best satisfies these requirements?

A financial services company is consolidating dozens of BigQuery datasets across multiple projects. Analysts complain they cannot easily discover which tables exist or determine which columns contain sensitive PII such as national IDs and account numbers. The compliance team requires that sensitive columns be automatically identified and tagged so that access policies can be applied consistently. Which approach best meets these data governance requirements with the least custom development?

A retail company is migrating a 4 TB self-managed PostgreSQL database from an on-premises data center to Cloud SQL for PostgreSQL. The business requires the application to remain available during the migration, with only a brief cutover window of a few minutes. The team wants a managed approach that continuously replicates changes from the source until they are ready to switch over. Which migration strategy should they use?

A healthcare analytics company runs a streaming Dataflow pipeline that reads patient event data from Pub/Sub, transforms it, and writes to BigQuery. Compliance requires that if any record fails validation (e.g., malformed PHI fields), it must not be silently dropped and must be available for later inspection without halting the pipeline. Which design best satisfies this reliability and fidelity requirement?

A retail company runs a streaming Dataflow pipeline that reads clickstream events from Pub/Sub, parses each JSON message, and writes structured rows to BigQuery. Occasionally, upstream systems emit malformed JSON or messages with schema violations. Currently, a single bad message causes the pipeline to throw an unhandled exception, stalling processing and creating a backlog. The data engineering team must ensure the pipeline keeps processing valid events while preserving the invalid ones for later investigation, without losing any data. What is the best design?

A data engineering team runs several Apache Beam pipelines on Dataflow. Analysts frequently request the same pipeline with different input paths, output tables, and window sizes. Currently, launching a job requires a developer to build the pipeline locally with the SDK and pass parameters at compile time, which creates a bottleneck. The team wants to let non-developers launch parameterized jobs from the Console, gcloud, or a REST call without rebuilding code, while keeping the pipeline logic packaged and version-controlled. Which approach best meets these flexibility and reusability requirements?

A media company runs a streaming Dataflow pipeline that ingests events from Pub/Sub, enriches them, and writes results to BigQuery. Leadership requires that if the primary region (us-central1) suffers a full regional outage, the pipeline can resume processing with minimal data loss and an RTO of under one hour. Which design best satisfies this disaster recovery requirement?

Your team is migrating a set of recurring Spark ETL jobs from an on-premises Hadoop cluster to Google Cloud. The jobs currently run on a shared, always-on cluster, but each job has different memory and CPU requirements. Leadership wants to avoid a permanently running cluster, minimize idle cost, and ensure each job runs on infrastructure sized to its needs while keeping the orchestration definition version-controlled and repeatable. Which approach best meets these goals?

A healthcare analytics company stores patient records in BigQuery and must comply with regulations requiring that the organization retain full control over the encryption keys, including the ability to revoke access and disable keys on demand. Their security team wants to manage the key rotation schedule and be able to audit all key usage. Which approach should the data engineering team implement to satisfy these requirements with the least operational overhead?

A media analytics company runs dozens of independent Spark batch jobs each day on a single long-running Dataproc cluster. The team complains about noisy-neighbor resource contention, version conflicts between jobs requiring different Spark releases, and high costs from the cluster running 24/7 even when idle. They want a design that maximizes flexibility and portability while minimizing cost. What should you recommend?

Your company runs a Dataflow streaming pipeline that reads from Pub/Sub, enriches records with lookups in a BigQuery reference dataset, and writes results to a separate BigQuery dataset. A security review requires that the pipeline's service account follow least-privilege principles and that no human users be able to assume it directly. Which IAM configuration best satisfies these requirements?