Question

A data engineer notices that queries against a Delta table in a Fabric lakehouse have become progressively slower over several weeks. The table is loaded every 15 minutes by a streaming pipeline that appends small batches. Investigating the table's storage folder, the engineer finds tens of thousands of tiny Parquet files. Which action will most directly resolve the query performance degradation?

Accepted Answer

Run the OPTIMIZE command on the table to compact the many small files into fewer, larger files

Answer

Increase the Spark session executor memory in the workspace environment

Answer

Enable result set caching on the lakehouse SQL analytics endpoint

Answer

Switch the table from Delta format to CSV to reduce metadata overhead