Question

A data engineering team is building a training dataset for a fraud detection model. Transaction records arrive continuously in an Amazon Kinesis Data Stream, while customer profile data is stored as a nightly export in Amazon S3. The team needs to enrich each transaction with the corresponding customer attributes and land the combined records in S3 for downstream SageMaker training. They want a managed, serverless approach that minimizes operational overhead. Which solution best meets these requirements?

Accepted Answer

Use an AWS Glue streaming ETL job that consumes the Kinesis stream and joins each micro-batch with the customer profile data loaded from S3, then writes the enriched output to S3.

Answer

Configure a Kinesis Data Firehose delivery stream to write raw transactions to S3, then run a manual Amazon Athena CREATE TABLE AS SELECT query each night to join with profiles.

Answer

Launch a persistent Amazon EMR cluster running Spark Structured Streaming to read from Kinesis and perform the join against profile data cached in HDFS.

Answer

Use an AWS Glue batch ETL job triggered once per day that reads the entire Kinesis stream from the beginning and joins it with the S3 profile export.