Question

A data engineering team maintains a Fabric workspace containing a Data Factory pipeline that ingests raw files, a notebook that transforms them, and a semantic model refresh. The transformation notebook must run only after the ingestion pipeline completes successfully, and business users need the entire flow to run automatically every night at 2:00 AM. The team wants a single orchestration artifact that controls the end-to-end run order and can pass the ingestion date as a parameter into the notebook. What should the team implement?

Accepted Answer

Create a Data Factory pipeline that chains a Copy activity, a Notebook activity (with a base parameter for the ingestion date), and a Semantic model refresh activity, then attach a scheduled trigger for 2:00 AM.

Answer

Schedule the ingestion pipeline, the notebook, and the semantic model refresh independently, each with its own 2:00 AM schedule so they all start at the same time.

Answer

Use a Dataflow Gen2 to perform ingestion and transformation together, then rely on the semantic model's automatic refresh to complete the flow.

Answer

Add the notebook's transformation code as a cell inside the ingestion pipeline's Copy activity and enable Airflow to trigger it hourly.