Question

A data engineering team uses Amazon Kinesis Data Firehose to deliver clickstream JSON records to an Amazon Redshift table. The delivery stream uses an intermediate S3 bucket before the COPY into Redshift. During a schema drift event, a subset of records fail the Redshift COPY command because of a data type mismatch. The team needs to ensure that no failed records are lost and that they can reprocess only the failed records later without re-ingesting the entire stream. What should the team do?

Accepted Answer

Enable S3 backup mode set to 'Failed data only' so that records that fail Redshift delivery are written to a separate S3 error prefix for later reprocessing.

Answer

Increase the Firehose buffer size and buffer interval so that failed records are automatically retried until the COPY succeeds.

Answer

Add a Lambda transformation to the delivery stream that drops any record that does not match the Redshift schema before delivery.

Answer

Switch the delivery destination from Redshift to Amazon S3 and rely on Redshift Spectrum to query the raw JSON directly.