Question

A media company is designing a new content-processing platform. Every night at 2 AM, a scheduled job must trigger the processing of up to 50,000 newly uploaded articles. Each article requires an independent enrichment task that calls a third-party API with strict rate limits, and processing may take anywhere from 30 seconds to 8 minutes per article. The solution must gracefully handle third-party throttling, automatically retry failed tasks, and be fully managed with minimal operational overhead. The company wants to avoid maintaining any always-on compute. Which architecture best meets these requirements?

Accepted Answer

Use Amazon EventBridge Scheduler to invoke a Lambda function at 2 AM that enqueues one message per article into an Amazon SQS queue, then have Lambda consumers process messages with reserved concurrency to respect the API rate limit, using an SQS dead-letter queue for failed tasks.

Answer

Use a cron job on a persistent Amazon EC2 Auto Scaling group that reads all articles at 2 AM and processes them in parallel threads, retrying failures in application code.

Answer

Use Amazon EventBridge Scheduler to trigger an AWS Batch array job on a large On-Demand compute environment, launching 50,000 concurrent containers that each call the third-party API directly.

Answer

Use an Amazon Kinesis Data Stream fed by a nightly Lambda, with Kinesis Data Firehose delivering records to the third-party API endpoint on a fixed schedule.