Maintaining and automating data workloads

A data engineering team runs hundreds of ETL queries every night against an on-demand BigQuery project. Occasionally, queries fail immediately with a 'resources exceeded' or concurrency-related error because too many interactive queries are submitted at once and hit the per-project concurrent query limit. These jobs are not time-sensitive as long as they complete before the morning business reports run. Which change should the team make to improve reliability without moving to a flat-rate reservation?

Your team runs BigQuery in an Enterprise edition reservation for a project that executes heavy ETL only between 01:00 and 05:00 UTC each night. Outside that window, the project runs almost no queries. Finance wants to eliminate the cost of idle capacity during the 20 idle hours per day while still guaranteeing fast throughput during the nightly batch. What is the most cost-effective reservation configuration?

A retail company runs a mix of workloads in BigQuery. Their nightly ETL jobs consume large, predictable volumes of slots, and their finance team requires stable, predictable monthly billing with no risk of surprise overages. Meanwhile, a small data science team runs occasional ad hoc exploratory queries that are sporadic and low-volume. The data engineering lead must recommend how to organize these workloads to meet the business requirements while minimizing total cost. What should they recommend?

Your company runs on-demand BigQuery for most of its ad hoc analytics, but every month-end there is a 3-day period where the finance team executes very heavy scheduled batch reports, causing costs to spike unpredictably and occasionally hitting on-demand concurrency limits. Outside of month-end, query volume is low and steady. Management wants predictable, minimized cost for the month-end workload without over-committing capacity for the rest of the month. What should you do?

Your finance team wants a weekly automated report attributing BigQuery on-demand query costs to individual users and teams so they can identify the most expensive recurring queries. You need a low-maintenance approach that does not require enabling additional export pipelines and can be scheduled to run without external tooling. What should you do?

Your organization runs BigQuery workloads for several departments (marketing, finance, and engineering) within a single billing account. Finance leadership wants a repeatable, automated way to break down monthly BigQuery costs by department for chargeback, without creating separate projects for each team. What is the most effective approach?

Your data engineering team runs an on-demand BigQuery project where analysts frequently write ad hoc queries. Last month a single mistyped query scanned 45 TB and caused an unexpected cost spike. Leadership wants a repeatable, automated guardrail that prevents any individual query from processing more than a set data volume, and rejects the query before it runs if it would exceed that limit. What should you implement?

Your data team runs ad hoc analytics on BigQuery using on-demand (per-TB) pricing. Last month a few analysts accidentally ran full-table scans on multi-terabyte tables, causing an unexpected spike in your bill. Leadership wants to prevent any single analyst from consuming an excessive amount of on-demand bytes while keeping the flexible on-demand model for exploratory work. What is the most effective way to enforce this?

Your organization uses BigQuery flat-rate reservations. You created a reservation named 'etl-res' with 500 slots and assigned it at the organization level to a folder called 'data-eng'. A new project 'nightly-loads' was placed inside the 'data-eng' folder, but you also want that single project's queries to run against a separate 'priority-res' reservation with 200 slots instead of the folder's assignment. What is the correct way to achieve this?

Your data team runs BigQuery on a slot-based reservation with editions pricing. Nightly ETL workloads require a consistent minimum of 500 slots to meet SLAs, but they occasionally spike to 1,500 slots during month-end processing. During the day, ad hoc BI queries are unpredictable and light. You want to guarantee the ETL baseline capacity at the lowest committed cost while still handling spikes automatically. How should you configure the reservation?

Your company purchased a one-year BigQuery Enterprise edition slot commitment 11 months ago to cover predictable ELT workloads. A recent architecture change moved most transformations to Dataflow, and internal reports show the committed slots are now consistently underutilized. Finance asks you to reduce ongoing BigQuery compute spend as the commitment nears expiration, while keeping some guaranteed capacity for remaining BI queries. What should you do?

Your company runs BigQuery in the Enterprise edition with a single reservation of 1,000 baseline slots assigned to a production project for scheduled ETL. During overnight batch windows the ETL reservation is fully utilized, but during the day it sits nearly idle while a separate ad-hoc analytics project (currently on-demand billing) generates unpredictable, spiky query costs. Leadership wants to reduce overall cost and improve slot utilization without degrading the ETL SLA. What should you do?

Your company runs two BigQuery Enterprise edition reservations: 'prod-etl' (500 baseline slots, autoscale to 1000) for nightly transformation jobs, and 'adhoc-analytics' (200 baseline slots, no autoscale) for data scientists' interactive queries. During business hours, the ETL reservation sits mostly idle, while analysts frequently complain that their queries queue for minutes waiting for slots. Management wants analyst query latency reduced without increasing total committed spend. What should you do?

A retail company uses BigQuery with a flat-rate reservation of 1,000 slots. Their nightly ETL jobs and interactive BI dashboard queries share the same reservation. During the day, analysts complain that dashboards are slow whenever ad-hoc data science queries run large scans, and at night the ETL jobs sometimes run long because reporting refresh jobs compete for slots. The team wants to isolate these workloads so critical dashboards always get priority, while still allowing idle capacity to be reused. What should they do?

Your team runs a critical BigQuery scheduled query every morning that populates a reporting table consumed by executive dashboards. Last week the query silently failed for three consecutive days because an upstream table was renamed, and no one noticed until executives complained about stale data. You need a low-maintenance, native solution that alerts the on-call engineer whenever any scheduled query fails, without building custom polling scripts. What should you do?

A data engineering team owns a BigQuery scheduled query that materializes a nightly reporting table. The query was originally created by an engineer who has since left the company; when her user account was deactivated, the scheduled query began failing with authentication errors. The team wants the pipeline to keep running reliably regardless of employee turnover, while following Google-recommended practices. What should they do?

A data engineering team runs a nightly transformation that reads from three staging tables and writes into a reporting table used by BI dashboards each morning. They currently trigger the SQL manually. They want a fully managed, low-maintenance way to run this single BigQuery SQL statement on a recurring schedule with built-in run history, retries, and no external infrastructure to manage. Which approach best meets these requirements with the least operational overhead?

Your team runs a nightly transformation in BigQuery that consists of a single SQL statement writing to a reporting table. It has no upstream or downstream dependencies on other systems, requires no conditional branching, and only needs to run once per day at 02:00. Currently it is a manual query someone triggers each morning. Operations wants the lowest-maintenance way to automate this while keeping failure notifications visible. Which approach best fits these requirements?

Your company recently purchased a 2,000-slot annual flat-rate commitment in BigQuery to handle the analytics workload of several teams. Three months in, the finance team reports the commitment is expensive, and you suspect it is over-provisioned. Before renewing or resizing, you need data-driven evidence about how many slots your queries actually consume during peak and off-peak periods, and how a smaller commitment would affect query performance. What should you do to right-size the commitment?

Your company purchased 2,000 baseline slots using a BigQuery Enterprise edition capacity commitment. You created two reservations: 'prod-etl' (1,200 slots) for scheduled nightly pipelines and 'adhoc-analytics' (800 slots) for interactive analyst queries during business hours. Overnight, the ETL workload frequently needs more than 1,200 slots while the analytics reservation sits idle, causing pipeline SLAs to be missed. You want to let ETL borrow the unused analytics capacity automatically without buying more slots. What should you do?