Question

A data engineer manages a Delta table that is continuously written to by a Structured Streaming job with a short micro-batch interval. Analysts complain that read queries against this table have become progressively slower over several days. In the Spark UI, they notice queries are launching thousands of tiny file-scan tasks. The table has no partitioning issues and query filters are appropriate. What is the most likely root cause and appropriate remediation?

Accepted Answer

The streaming job is producing many small files per micro-batch; schedule periodic OPTIMIZE to compact them into larger files.

Answer

The table is missing a Z-ORDER index on the join key; run ANALYZE TABLE to rebuild statistics.

Answer

Time travel history is accumulating; run VACUUM with a zero-hour retention to remove old versions and speed up reads.

Answer

The Delta transaction log is corrupted; recreate the table with CREATE TABLE AS SELECT to reset it.