Question

A data engineering team runs a nightly production ETL pipeline that is triggered automatically by the Databricks Jobs scheduler. During code review, a teammate notices the pipeline is configured to run on an all-purpose cluster that stays running 24/7 so analysts can also attach notebooks to it interactively during the day. Management wants to reduce compute cost for the scheduled pipeline without affecting the analysts' interactive work. What is the most appropriate change?

Accepted Answer

Configure the scheduled job to use a job cluster that is created when the job starts and terminated when it finishes.

Answer

Keep using the shared all-purpose cluster but reduce its worker count so both workloads run more cheaply.

Answer

Move the analysts' notebooks to a job cluster and keep the ETL pipeline on the all-purpose cluster.

Answer

Enable autoscaling on the all-purpose cluster so it scales down when the ETL job is not running.