Question

A data engineering team ingests data using Snowpipe with auto-ingest. Their upstream process lands data in the external stage as thousands of very small files (a few KB each) every minute. The team notices that their Snowpipe costs are surprisingly high relative to the small volume of data being loaded. What is the MOST likely cause and the best remediation?

Accepted Answer

Snowpipe charges a per-file overhead in addition to compute; batching the source data into fewer, larger files before staging will reduce cost.

Answer

Snowpipe requires a dedicated virtual warehouse that is always running; assigning a smaller warehouse will reduce cost.

Answer

Snowpipe bills based on the number of columns loaded; reducing the number of columns in the target table will reduce cost.

Answer

Snowpipe charges for each SNS/SQS notification at premium rates; disabling auto-ingest and switching to manual REST calls will eliminate the cost.