Question

A retail company operates a mobile ordering app. Order events are produced continuously to an Amazon Kinesis Data Stream. The data engineering team needs to enrich each order with customer loyalty tier data (a small reference table under 50 MB that changes daily) and write the enriched records to Amazon S3 in Parquet format. The enrichment logic requires a distributed join, and the team wants a fully managed serverless option that natively supports Apache Spark and reads directly from the Kinesis stream with checkpointing. Which approach best meets these requirements?

Accepted Answer

Use an AWS Glue streaming ETL job that consumes from the Kinesis Data Stream, broadcasts the loyalty reference table for the join, and writes Parquet output to S3.

Answer

Configure a Kinesis Data Firehose delivery stream with a Lambda transform to perform the join and enable Parquet format conversion to S3.

Answer

Deploy a long-running Amazon EMR cluster with Spark Structured Streaming reading from Kinesis, and manually manage cluster capacity and checkpointing.

Answer

Use an AWS Lambda function triggered by the Kinesis stream to look up loyalty tiers via DynamoDB and write individual Parquet files to S3.