Question

A company processes clickstream events using a Kinesis Data Streams stream consumed by an application built with the Kinesis Client Library (KCL) running on EC2 instances. During a deployment, several EC2 instances were terminated mid-batch and the application restarted. The team notices that some records were processed more than once, causing duplicate rows in the downstream analytics database. The team wants to reduce duplicate processing caused by consumer restarts while keeping the KCL-based architecture. What is the MOST appropriate action?

Accepted Answer

Configure the KCL to checkpoint more frequently and make the downstream write operations idempotent so reprocessed records do not create duplicates.

Answer

Enable enhanced fan-out on the stream so each consumer gets a dedicated throughput pipe that prevents duplicates.

Answer

Increase the stream retention period to 168 hours so KCL can avoid re-reading already-processed records.

Answer

Switch the consumer to use the Kinesis Data Streams GetRecords API directly without the KCL to avoid checkpoint issues.