Question

A data engineering team stores CSV files in an AWS S3 bucket that is configured as an external stage. New files arrive continuously throughout the day, and the team needs them loaded into Snowflake within about a minute of arrival, without running any scheduled COPY jobs or scripts. They want the most cost-efficient, hands-off ingestion mechanism. Which approach should they implement?

Accepted Answer

Create a Snowpipe with AUTO_INGEST = TRUE and configure an S3 event notification (SQS) to trigger the pipe when files land.

Answer

Schedule a task every minute that runs a COPY INTO statement against the external stage using a dedicated warehouse.

Answer

Call the Snowpipe REST API insertFiles endpoint from a custom application each time a file is uploaded.

Answer

Use the Kafka connector with Snowpipe Streaming to write rows directly into the target table.