Question

A company runs a self-managed Apache Kafka cluster on EC2 in its own VPC to handle real-time event ingestion for several downstream microservices. The operations team spends significant effort patching brokers, managing ZooKeeper, and rebalancing partitions. Leadership wants to reduce operational overhead by migrating to Amazon MSK while keeping the existing topic structure, consumer offsets, and message ordering intact. The migration must occur with minimal downtime, and producers/consumers should be cut over gradually rather than all at once. Which approach best meets these requirements?

Accepted Answer

Provision an Amazon MSK cluster and use MirrorMaker 2 to replicate topics, partitions, and consumer group offsets from the self-managed cluster, then progressively repoint producers and consumers to MSK before decommissioning the old cluster.

Answer

Use AWS Database Migration Service (DMS) with Kafka as both the source and target endpoint to replicate all topics and offsets to Amazon MSK, then perform a single cutover of all clients.

Answer

Export the Kafka log segment files from the EC2 brokers using AWS DataSync, copy them into the MSK broker storage, and restart the MSK cluster so it resumes from the migrated data.

Answer

Create an Amazon MSK cluster and configure a Kinesis Data Streams pipeline to buffer events during migration, then have consumers read from Kinesis until all producers are moved to MSK.