Question

A retail company ingests clickstream events into an Amazon Kinesis Data Stream. A data engineer must build a pipeline that continuously reads the stream, removes duplicate events that arrive within a short window (caused by client retries), enriches each event with product metadata stored in a small Amazon DynamoDB table, and writes the results to Amazon S3 in Parquet format for analytics. The solution must be serverless and minimize operational overhead. Which approach best meets these requirements?

Accepted Answer

Use an AWS Glue streaming ETL job that reads from the Kinesis stream, applies dropDuplicates within a time window, joins with the DynamoDB metadata loaded as a DataFrame, and writes Parquet output to S3.

Answer

Configure Amazon Kinesis Data Firehose to deliver records directly to S3 as Parquet, then run a nightly AWS Glue batch job to deduplicate and enrich the data.

Answer

Launch a long-running Amazon EMR cluster with Spark Structured Streaming to read the Kinesis stream, deduplicate, enrich from DynamoDB, and write to S3.

Answer

Use a Lambda function triggered by the Kinesis stream to write each raw event to S3 as JSON, then use Amazon Athena CTAS queries to remove duplicates and enrich the data on demand.