Question

A media company runs a nightly batch pipeline that transcodes thousands of independent video segments. The workload currently runs on an Amazon ECS cluster backed by On-Demand Fargate tasks and takes about 4 hours to complete. Each task processes one segment, writes output to Amazon S3, and updates a job record in DynamoDB. The tasks are stateless and any interrupted segment can simply be reprocessed. Finance has asked the team to reduce compute cost for this pipeline without materially extending the completion window or losing any work. Which approach best meets these goals?

Accepted Answer

Convert the tasks to run on a mix of Fargate Spot and On-Demand Fargate capacity providers with a base of On-Demand and a higher weight on Fargate Spot, and configure the ECS service to reprocess any segments interrupted by Spot reclamation.

Answer

Migrate the pipeline to a single large On-Demand EC2 instance with a Reserved Instance purchase, running all transcoding jobs sequentially to maximize the per-hour discount.

Answer

Keep all tasks on On-Demand Fargate but reduce the vCPU and memory of each task by half so more tasks fit into the existing budget.

Answer

Move the transcoding to AWS Lambda functions triggered per segment, ignoring the 15-minute timeout because most segments finish quickly.