Data Ingestion and Transformation

A media company needs to migrate 40 TB of archived video files from an on-premises NFS file server to Amazon S3. After the initial transfer, they must automatically sync any newly added or modified files every night with minimal engineering effort, and they require data integrity verification during each transfer. Which approach best meets these requirements?

A retail company is migrating a 2 TB Oracle database running on-premises to Amazon Aurora PostgreSQL. The database is actively used by production applications during business hours, and the business requires that the source database remain fully operational with minimal downtime during the migration. After the initial data copy, all new and changed records in the source must continue to be replicated to Aurora until the final cutover. Which AWS DMS approach meets these requirements?

A data engineering team has an existing library of custom Apache Spark applications written and tuned by their engineers, including several jobs that require specific open-source Spark configurations and third-party JAR dependencies. They want to run these jobs on a fully managed, auto-scaling service without provisioning or managing clusters, and they want to pay only for the resources consumed during job execution. Which service best meets these requirements with the least migration effort?

A data engineering team runs a nightly Spark job on Amazon EMR that joins a 2 TB fact table of transactions with a 40 MB reference table containing store metadata. The job is slow because Spark performs a shuffle-based sort-merge join, redistributing the massive fact table across the cluster over the network. The reference table easily fits in each executor's memory. What is the most effective change to accelerate this join with minimal code effort?

A data engineering team runs a nightly batch job that processes 8 TB of raw JSON logs stored in Amazon S3. The transformation logic is written in custom Apache Spark and Scala code that relies on several third-party JAR libraries and requires fine-grained tuning of Spark executor memory and shuffle configurations. The job runs once per day for about two hours and cost efficiency is a priority. Which approach best meets these requirements?

A data engineering team ingests high-volume clickstream events as JSON through Amazon Kinesis Data Firehose delivering to Amazon S3. Analysts query the data with Amazon Athena and complain about slow queries and high scan costs. The team wants Firehose to reduce storage size and improve Athena query performance with minimal custom code. What should they configure?

A retail company runs a daily AWS Glue ETL job that reads new CSV files landing in an S3 prefix, converts them to Parquet, and writes to a curated bucket. As data volume grows, the team notices the job reprocesses all files in the source prefix every run, increasing runtime and cost. They want the job to process only files that have not been processed in previous runs, without manually tracking file names or maintaining a separate state store. Which approach best meets this requirement?

A data engineering team lands compressed Avro files in an S3 bucket every hour from an upstream application. The Avro schema occasionally adds new optional fields over time. The team wants Amazon Athena to query the latest data without manual schema updates each time a new field appears, while minimizing operational overhead. Which approach best meets these requirements?

A data engineering team runs an AWS Glue Spark ETL job that processes a large daily fact table (approximately 500 GB) of clickstream events stored in S3. Each event record must be enriched with product category and region attributes from a small reference dataset (about 40 MB) also stored in S3. The current job performs a standard join between the two DynamicFrames/DataFrames, but the job is slow and produces heavy shuffle activity across executors. Which approach will most effectively reduce shuffle and speed up the enrichment join?

A data engineer is building an AWS Glue Spark ETL job that reads semi-structured JSON files from S3. The same field 'amount' appears as a string in some records and as a double in others, causing the job to fail when writing to a Parquet target with a fixed schema. The engineer wants to handle the ambiguous column so that all values are cast to a single numeric type without dropping records. Which approach best resolves this within the Glue job?

A data engineering team runs an AWS Glue Spark ETL job that must read from an Amazon RDS for PostgreSQL database located in a private subnet with no public IP. The Glue job also needs to write the transformed results to an Amazon S3 bucket. When the job runs, it fails with a connection timeout while attempting to reach the RDS instance. What is the correct way to configure Glue so it can reach both RDS and S3?

A data engineering team runs a daily AWS Glue Spark ETL job that loads customer records from multiple upstream systems into a curated S3 dataset. Due to overlapping source feeds, the same customer_id can appear multiple times per run, and only the record with the most recent last_updated timestamp should be kept. The team needs to implement deduplication within the Glue job with minimal custom code while ensuring the surviving row is always the latest version. Which approach best meets these requirements?

A data engineering team runs an AWS Glue Spark ETL job that reads clickstream events and writes the results to Amazon S3 in Parquet format. Downstream Athena queries filter heavily by event_date and country. Currently the job writes all output into a single S3 prefix, causing Athena to scan the entire dataset on every query. The team wants the Glue job itself to physically organize the output so that Athena can prune data by these two columns without a separate reprocessing step. What is the MOST appropriate way to configure the Glue write operation?

A data engineering team ingests daily nested JSON logs (~500 GB/day) into S3. Analysts query the data with Amazon Athena, filtering mostly on a small subset of columns. The team wants to reduce Athena scan costs and improve query performance while keeping storage efficient. Which AWS Glue ETL output configuration best meets these goals?

A data engineer is building an AWS Glue Spark ETL job to process raw event logs. Each incoming JSON record contains an array field named 'items' with a variable number of nested purchase objects. The business requires that each element of the 'items' array become its own separate row in the output Parquet dataset, with the parent record's metadata (order_id, customer_id, timestamp) repeated on every resulting row. Which transformation approach correctly produces the desired output?

A retail company ingests raw customer order data into Amazon S3 as CSV files. A data engineer must build an AWS Glue ETL job that cleanses the data before loading it into a curated zone. The requirements are: replace empty strings and the literal text 'N/A' in the customer_email column with null, drop any row where order_id is missing, and standardize the country column so that values like 'usa', 'USA', and 'Usa' all become 'US'. Which approach implements all three cleansing requirements most efficiently within a single Glue Spark ETL job?

A data engineering team runs a daily AWS Glue Spark ETL job that writes a large analytical dataset to Amazon S3 in Parquet format. The data is queried heavily via Amazon Athena, and queries frequently filter and aggregate across many columns. The team wants to minimize storage costs while achieving the best query performance, and they are willing to accept slightly higher write-time CPU usage during the ETL job. Which compression configuration should they apply to the Parquet output?

A data engineer receives deeply nested JSON documents from a SaaS API, landed in Amazon S3. Each document contains arrays of order line items nested inside customer objects. The team needs to load this data into an Amazon Redshift data warehouse where analysts run SQL joins across flattened relational tables. The engineer is building an AWS Glue ETL job using DynamicFrames. Which approach most efficiently converts the nested structure into normalized, relational tables suitable for Redshift?

A data engineering team maintains a customer dimension table in an S3-based data lake. The business requires that whenever a customer's attributes (such as address or loyalty tier) change, the previous values must be preserved as historical records rather than overwritten, so analysts can query the state of a customer at any point in time. The team uses AWS Glue Spark ETL jobs to process incoming daily batches of customer updates. Which transformation approach should the team implement in the Glue job?

A media company drops variable numbers of raw JSON log files into an S3 landing bucket at unpredictable times throughout the day. A data engineer must run an AWS Glue ETL job to clean and convert each new batch to Parquet within a few minutes of arrival, without incurring cost when no files are present. Which approach best meets these requirements?