Question

A data engineering team ingests raw CSV files into a lakehouse, applies complex PySpark transformations that reuse a shared library, and then loads curated data into a warehouse. Business analysts also need a low-code way to reshape a small reference dataset from a SharePoint list before the Spark job runs. The team wants a single orchestration that: runs the reference reshaping first, then the heavy transformation, then the warehouse load, with each step only starting after the previous one succeeds. Which combination best fits these requirements?

Accepted Answer

Create a pipeline that calls a Dataflow Gen2 for the SharePoint reference reshaping, then a notebook for the PySpark transformations, then a Copy/warehouse load activity, chained with success dependencies.

Answer

Build one Dataflow Gen2 that performs the SharePoint reshaping, the PySpark transformations, and the warehouse load in sequence.

Answer

Schedule three independent notebooks with staggered start times so they run in the correct order.

Answer

Use a single notebook that reads the SharePoint list, runs the Spark transformations, and writes to the warehouse, triggered on file arrival.