Question

A data engineering team collects genomic sequencing files that are each 40 GB in size and must be uploaded from an on-premises server to Amazon S3 for downstream ML training. Uploads over the corporate internet connection frequently fail partway through, forcing the team to restart the entire transfer. They want to make uploads resilient to transient network failures and improve throughput without rewriting their ingestion tooling from scratch. Which approach best addresses this requirement?

Accepted Answer

Use S3 multipart upload so the file is split into independent parts that can be uploaded in parallel and individually retried on failure.

Answer

Compress each file with gzip before a single PutObject call to reduce the payload size below the failure threshold.

Answer

Enable S3 Versioning on the bucket so failed uploads are automatically resumed from the last saved version.

Answer

Switch the destination to S3 Glacier Flexible Retrieval to take advantage of its bulk upload resilience features.