Question

A data engineer at a retail company ingests order data as JSON files into a Fabric lakehouse. Each JSON record represents one order and contains a nested array field named lineItems, where each element holds a product SKU, quantity, and price. The engineer must transform this data into a flat fact table where every row represents a single line item, retaining the parent order ID and order date on each row. Using PySpark in a notebook, which approach produces the required output?

Accepted Answer

Use explode() on the lineItems array column, then select the parent order fields alongside the exploded struct fields.

Answer

Use groupBy() on the order ID and apply collect_list() to the lineItems column.

Answer

Use pivot() on the lineItems column to create one column per product SKU.

Answer

Use dropDuplicates() on the order ID to keep one representative line item per order.