Question

A data engineering team ingests clickstream events into an Amazon Kinesis Data Stream that feeds a downstream ML feature pipeline. Producers are written by multiple teams and occasionally add or rename fields, which has repeatedly broken the Apache Flink consumer that parses the records. The ML team needs to enforce that only backward-compatible schema changes are accepted before records enter the stream, while keeping serialization compact for high-throughput ingestion. What should they implement?

Accepted Answer

Register an Avro schema in the AWS Glue Schema Registry and configure producers to serialize records with compatibility set to BACKWARD, rejecting incompatible schema versions.

Answer

Convert all records to CSV format and add a header row so the consumer can dynamically detect new columns as they appear.

Answer

Run an AWS Glue crawler on the Kinesis stream every five minutes to update the Data Catalog and propagate schema changes to consumers.

Answer

Store each event as a separate JSON object in Amazon S3 and use a Lambda function to validate fields against a hard-coded list before processing.