Question

Your data engineering team already operates an Apache Spark pool inside an Azure Synapse Analytics workspace. Data scientists in your Azure Machine Learning workspace need to run interactive feature-engineering jobs on that existing Spark pool from within Machine Learning notebooks, reusing the pool's configured node sizes. You want to expose the Synapse Spark pool as a compute target in the Machine Learning workspace with the least ongoing management overhead. What should you do?

Accepted Answer

Attach the Synapse Spark pool to the Machine Learning workspace as an attached Synapse Spark compute

Answer

Create a new Machine Learning compute cluster and install Spark libraries in a custom environment

Answer

Create a Machine Learning compute instance and configure it to submit jobs to the Synapse pool over a private endpoint

Answer

Register the Synapse workspace as a datastore so the Spark pool becomes available as a compute target