A retail company has a Fabric lakehouse containing raw sales files in deeply nested JSON. The data engineering team must flatten and enrich this data on a nightly schedule. The transformation logic is complex, involving multiple joins, custom Python functions for currency conversion, and reuse of libraries already published to a Spark environment. The team consists of experienced Python developers who need full control over the transformation code and expect data volumes to grow to terabytes. Which transformation approach best fits these requirements?