Question

A data engineer configures Amazon Kinesis Data Firehose to receive clickstream events and wants to convert incoming records to Apache Parquet before delivery to Amazon S3 to reduce Athena scan costs. During testing, the record format conversion feature fails to activate. The incoming records are compressed with GZIP by the producer application, and each record is a nested JSON document. Which requirement must be met for Firehose record format conversion to Parquet to work correctly?

Accepted Answer

The producer must send uncompressed, single-line JSON records, and a schema must be defined in the AWS Glue Data Catalog.

Answer

The producer must GZIP-compress each record and Firehose will automatically infer the schema from the first 100 records.

Answer

The records must be sent as Avro because Firehose only converts Avro sources to Parquet.

Answer

A Lambda transformation must first convert the JSON to CSV before Firehose applies Parquet conversion.