Question

A data engineering team receives a nightly drop of 2 TB of raw Parquet files in an Azure Data Lake Storage Gen2 account owned by a partner organization. The team must run heavy PySpark transformations (joins, aggregations, and deduplication) over this data every night, but they want to avoid physically copying the 2 TB into their own Fabric workspace and want to minimize storage duplication and ingestion latency. Which approach should they implement?

Accepted Answer

Create a OneLake shortcut in a lakehouse pointing to the ADLS Gen2 location, then run the PySpark transformations against the shortcut.

Answer

Use a Copy activity in a Data Factory pipeline to load the full 2 TB into a lakehouse Files section each night before transforming.

Answer

Configure database mirroring from the ADLS Gen2 account into a Fabric warehouse and transform with T-SQL.

Answer

Build a Dataflow Gen2 that imports the Parquet files with staging enabled and applies the transformations in Power Query.