Question

A company runs a self-managed Apache Kafka producer fleet that publishes clickstream events as Avro to an Amazon MSK cluster. Downstream consumers frequently break when producers add or rename fields without coordination. The data engineering team wants a managed way to enforce compatibility rules on message schemas before producers can publish incompatible changes, while keeping payloads compact. Which solution best meets these requirements with the least operational overhead?

Accepted Answer

Enable AWS Glue Schema Registry, register the Avro schema, configure BACKWARD compatibility, and integrate the registry serializer into the producers and consumers.

Answer

Convert all messages to JSON and add a version field to each record, then validate the version in a Lambda function triggered per message.

Answer

Store the Avro schema files in an S3 bucket and have producers download and validate against the latest schema before publishing.

Answer

Enable MSK tiered storage and use Amazon Athena to query the topic data and detect schema drift after ingestion.