Question

A data engineering team runs hundreds of ETL queries every night against an on-demand BigQuery project. Occasionally, queries fail immediately with a 'resources exceeded' or concurrency-related error because too many interactive queries are submitted at once and hit the per-project concurrent query limit. These jobs are not time-sensitive as long as they complete before the morning business reports run. Which change should the team make to improve reliability without moving to a flat-rate reservation?

Accepted Answer

Submit the ETL jobs using BATCH priority so BigQuery queues them and starts them as capacity becomes available instead of failing on concurrency limits.

Answer

Set a higher maximum_bytes_billed value on each query so the jobs are allowed to consume more resources.

Answer

Enable BigQuery BI Engine for the project so the ETL queries are served from in-memory acceleration.

Answer

Increase the per-query timeout so queries wait longer before returning the concurrency error.