Question

A data engineering team manages a fleet of IoT sensors that append about 500 GB of raw JSON readings to Amazon S3 every day. Business analysts run scheduled dashboards only against the last 90 days of data, but the company must retain all historical data for 7 years to satisfy regulatory audits. Auditors query the older data only a few times per year using ad hoc SQL. The team wants to minimize total storage and query cost while keeping all data queryable with SQL. Which storage and access approach best meets these requirements?

Accepted Answer

Convert incoming data to Parquet, keep the last 90 days in Redshift for dashboards, and query the older Parquet data on S3 with Amazon Athena for the infrequent audit queries.

Answer

Load all 7 years of data into an Amazon Redshift RA3 cluster and rely on managed storage to hold the full history.

Answer

Keep all data as raw JSON in S3 and use only Amazon Athena for both the daily dashboards and the audit queries.

Answer

Store all 7 years in Amazon DynamoDB with a TTL of 7 years and query historical records with PartiQL.