AWS Certified Data Engineer - Associate

Question 1

A data engineering team runs a nightly reporting pipeline that executes several Amazon Athena queries orchestrated by AWS Step Functions. Recently, some queries intermittently fail, and the team needs to programmatically detect which specific queries failed and capture the error reason so they can automatically retry or alert. What is the MOST effective way to identify the exact failure cause for each Athena query execution?

Accepted Answer

Call the Athena GetQueryExecution API using the QueryExecutionId and inspect the Status.StateChangeReason field for failed executions

Answer

Enable AWS CloudTrail data events on the Athena workgroup and parse the event logs for query errors

Answer

Enable VPC Flow Logs on the subnet hosting the Athena queries to trace network-related failures

Answer

Query the AWS Glue Data Catalog metrics in CloudWatch to determine why the tables returned errors

Question 2

A data engineering team runs ad hoc Athena queries against a large S3 data lake. Finance reports that Athena costs have spiked because analysts frequently run SELECT * queries that scan entire multi-terabyte tables. The team wants to enforce a hard limit on how much data any single query can scan, and automatically cancel queries that exceed that limit, without modifying the underlying tables or query patterns. Which approach meets this requirement with the least operational effort?

Accepted Answer

Configure a per-query data usage control in the Athena workgroup that cancels any query scanning more than the specified amount of data.

Answer

Create a CloudWatch alarm on the Athena BytesScannedCutoffPerQuery metric and use a Lambda function to cancel running queries.

Answer

Convert all tables to Parquet with Snappy compression so that queries scan less data automatically.

Answer

Enable an S3 bucket policy that denies GetObject requests once a daily byte-transfer threshold is reached.

Question 3

A data engineer maintains a nightly Athena CREATE TABLE AS SELECT (CTAS) job that transforms raw event logs into a curated Parquet dataset in an S3 prefix, registered as a table in the Glue Data Catalog. The job is orchestrated by Step Functions. Occasionally the workflow retries a failed step, and the engineer notices that the CTAS statement fails on rerun with an error indicating the target location already exists. The team wants the pipeline to be safely re-runnable without manual cleanup. Which approach best resolves this while keeping the transformation logic in Athena?

Accepted Answer

Before running the CTAS, drop the target table and delete the existing S3 output prefix (or write to a new time-partitioned prefix each run), so each execution starts from a clean location.

Answer

Increase the Athena workgroup's query timeout so the CTAS has enough time to overwrite the existing files in place.

Answer

Switch the query to an INSERT INTO statement against the same table so Athena appends results and ignores the existing location.

Answer

Enable Athena query result reuse in the workgroup so repeated runs return cached results instead of rewriting the output.

Question 4

A data engineer stores clickstream data in S3 under the prefix s3://analytics-lake/events/year=2024/month=03/day=15/ using Hive-style partition folders. A Glue Data Catalog table was created by a crawler two months ago. New daily folders are being written continuously by a Firehose stream, but Athena queries that filter on recent dates return zero rows even though the files exist in S3. The engineer confirms the schema and file format are correct. What is the MOST cost-effective way to ensure Athena can query the newly added partitions going forward with minimal ongoing operational effort?

Accepted Answer

Enable partition projection on the table by defining the partition key ranges and formats in the table properties, so Athena calculates partitions at query time.

Answer

Run the Glue crawler on a manual schedule every hour to detect and register the new partition folders in the Data Catalog.

Answer

Run MSCK REPAIR TABLE in Athena after each new file is delivered by Firehose to synchronize the partition metadata.

Answer

Recreate the table as an unpartitioned external table so Athena scans all files and never misses new data.

Question 5

A data engineer manages an Athena-based reporting pipeline that queries a QuickSight dashboard backed by S3 data in Parquet format. New data arrives hourly into date-partitioned S3 prefixes (s3://logs/year=/month=/day=/hour=/). Analysts complain that queries frequently miss the most recent hour of data even though the files exist in S3. The team currently runs a Glue crawler once per day to update partitions. The engineer needs the most operationally efficient way to ensure Athena always recognizes newly arrived partitions without manual intervention or frequent crawler runs. What should the engineer do?

Accepted Answer

Enable partition projection on the Athena table by configuring projection properties for the date and hour columns in the table's properties.

Answer

Increase the Glue crawler schedule to run every 5 minutes so partitions are discovered faster.

Answer

Run the MSCK REPAIR TABLE command manually before each dashboard refresh.

Answer

Switch the table storage format from Parquet to CSV so Athena can scan the new files without partition metadata.

Question 6

A data engineering team runs ad-hoc analytics on a data lake using Amazon Athena. Analysts have complained about unexpectedly high monthly Athena costs. The team needs to identify which specific queries and users are scanning the most data so they can take corrective action, without deploying additional infrastructure. What is the MOST operationally efficient way to gain this visibility?

Accepted Answer

Query the Athena query history metadata (e.g., via CloudWatch metrics and the Athena DataScannedInBytes metric grouped by workgroup) and enforce per-query data scan limits using workgroups

Answer

Enable AWS CloudTrail data events on the S3 buckets and manually correlate each GetObject call to an Athena query

Answer

Configure a Glue crawler to scan the query result location in S3 and build a report of query sizes

Answer

Migrate all analysts to Redshift Spectrum so that scan costs are eliminated

Question 7

A data engineering team maintains a daily reporting pipeline. A Glue job writes curated Parquet data to Amazon S3 each morning at 06:00 UTC, and a Glue crawler updates the Data Catalog afterward. Business analysts use an Amazon QuickSight dashboard backed by a SPICE dataset built from an Athena query. Analysts report that the dashboard shows yesterday's numbers even though the morning pipeline completed successfully. The team wants the dashboard to reflect the latest data automatically each day with minimal ongoing management. What should the team do?

Accepted Answer

Schedule a SPICE dataset refresh in QuickSight to run after the pipeline completes, ensuring the query re-reads the latest Athena data each morning.

Answer

Switch the QuickSight dataset from SPICE to direct query mode so it always reads the latest S3 data without any refresh configuration.

Answer

Increase the SPICE capacity allocated to the account so that the dataset holds more recent data.

Answer

Re-run the Glue crawler more frequently throughout the day to keep the Data Catalog partitions current.

Question 8

A data engineering team runs ad hoc Athena queries against a data lake containing regulated financial data. A compliance requirement states that all Athena query result sets written to Amazon S3 must be encrypted with a customer-managed KMS key, and this encryption must be enforced consistently for every analyst regardless of individual query settings. What is the most effective way to satisfy this requirement?

Accepted Answer

Configure the Athena workgroup's result configuration to use SSE-KMS with the customer-managed key and enable the setting that overrides client-side result configuration.

Answer

Attach an S3 bucket policy to the query results bucket that denies any PutObject request, forcing analysts to encrypt manually before writing.

Answer

Instruct each analyst to select the customer-managed KMS key in the Athena console encryption settings before running queries.

Answer

Enable default SSE-S3 encryption on the query results bucket so all objects are automatically encrypted at rest.

Question 9

A retail company is building a product catalog service for its e-commerce platform. Each product record has a different set of attributes depending on category (electronics have specifications like resolution and ports, while apparel have size and material). Products are looked up by product ID at very high request rates with single-digit millisecond latency requirements, and the schema evolves frequently as new categories are added. Which data store best fits these requirements?

Accepted Answer

Amazon DynamoDB with the product ID as the partition key

Answer

Amazon Redshift with a wide fact table containing all possible attribute columns

Answer

Amazon RDS for PostgreSQL with a normalized relational schema

Answer

Amazon OpenSearch Service with a single index across all categories

Question 10

A gaming company runs a multiplayer platform that must store player session state, such as current score, position, and inventory. The application requires single-digit millisecond read and write latency at any scale, and access is always by a known player ID. Traffic is highly unpredictable, spiking during tournaments to millions of requests per second. The data engineering team wants a fully managed data store that automatically scales without capacity planning and minimizes operational overhead. Which AWS data store best meets these requirements?

Accepted Answer

Amazon DynamoDB with on-demand capacity mode

Answer

Amazon Aurora PostgreSQL with read replicas

Answer

Amazon Redshift with a KEY distribution style on player ID

Answer

Amazon OpenSearch Service with a dedicated master node

Question 11

An IoT company ingests billions of sensor readings per day. Each reading has a device ID, timestamp, and metric value. The primary access pattern is to retrieve the most recent readings for a specific device by device ID, and single-digit millisecond read/write latency is required at massive scale. The data volume grows continuously and the team wants a fully managed, serverless data store that scales automatically without capacity planning for servers. Which data store best fits this use case?

Accepted Answer

Amazon DynamoDB with a partition key of device ID and a sort key of timestamp

Answer

Amazon Redshift with the device ID as the distribution key and timestamp as the sort key

Answer

Amazon RDS for PostgreSQL with an index on device ID and timestamp

Answer

Amazon OpenSearch Service with a rolling index per day

Question 12

A financial services company must produce an audit trail showing every individual GetObject and PutObject API call made against a specific S3 bucket that stores regulated customer data. Their existing CloudTrail trail already captures management events across all regions, but auditors report they cannot see object-level read and write activity for the bucket. What is the most appropriate way to meet this requirement?

Accepted Answer

Configure CloudTrail data events for the S3 bucket to log object-level (S3 data plane) API activity.

Answer

Enable S3 server access logging on the bucket and deliver logs to a separate logging bucket.

Answer

Enable S3 Event Notifications to send object create and delete events to an SQS queue for review.

Answer

Turn on CloudTrail Insights events on the existing trail to detect unusual API call rates.

Question 13

A financial services company runs a data lake on Amazon S3 governed by AWS Lake Formation, with the Glue Data Catalog holding table definitions. Compliance auditors require a complete, queryable record of every user and role that has read specific sensitive tables, including which columns were accessed and when. The data engineering team needs to enable this auditing with the least custom development. Which approach satisfies the requirement?

Accepted Answer

Enable AWS CloudTrail data events and rely on Lake Formation's integration to log GetDataAccess and related events, then query the CloudTrail logs (or use Lake Formation's audit query capability) to review who accessed which tables and columns.

Answer

Turn on S3 server access logging on the data lake buckets and parse the logs with Athena to determine which principals read each object.

Answer

Enable Glue Data Catalog encryption with KMS and review KMS key usage in CloudTrail to identify who decrypted the catalog metadata.

Answer

Configure Amazon Macie to scan the data lake and generate findings whenever sensitive data is accessed by a principal.

Question 14

A data engineering team runs a nightly AWS Glue ETL job that populates a reporting table. Recently, the job has intermittently failed, but the team only discovers the failures the next morning when business users complain about missing dashboard data. The team wants to be notified automatically within minutes whenever the Glue job enters a failed state, with the least operational overhead. Which approach should they implement?

Accepted Answer

Create an Amazon EventBridge rule that matches the Glue Job State Change event for the FAILED state and targets an Amazon SNS topic that emails the team.

Answer

Enable Glue job bookmarks and configure the job to retry three times, then write failure messages to an S3 bucket that the team reviews each morning.

Answer

Schedule a Lambda function to run every hour that queries the Glue GetJobRuns API and sends an email if any run failed in the past 24 hours.

Answer

Configure Amazon Athena to query the Glue job's CloudWatch logs on a nightly schedule and export the results to a QuickSight dashboard.

Question 15

A data engineering team runs a nightly pipeline that has three sequential stages: a Glue ingestion job, a Glue transform job, and a Redshift COPY load. Each stage currently has its own CloudWatch alarm that sends an SNS notification on failure. On-call engineers complain that a single upstream failure produces a cascade of three separate pages, because when the ingestion job fails the downstream stages also alarm. The team wants a single actionable notification that fires only when the pipeline as a whole is in a failed state, while keeping the individual alarms for dashboards. What is the MOST operationally efficient way to achieve this?

Accepted Answer

Create a CloudWatch composite alarm that combines the three stage alarms with an alarm rule, and attach the SNS notification to the composite alarm instead of the individual alarms.

Answer

Delete the two downstream alarms and keep only the ingestion job alarm, since a failure there always blocks the rest of the pipeline.

Answer

Increase the evaluation period and datapoints-to-alarm on each individual alarm so they take longer to trigger and overlap less.

Answer

Route all three SNS notifications to an SQS queue and write a Lambda function that de-duplicates messages within a time window before paging on-call.

Question 16

A data engineering team runs a nightly Glue ETL pipeline orchestrated by AWS Step Functions. Business rules require that after the transformation step completes and Glue Data Quality checks pass, a data steward must manually review a summary report before the curated data is published to the production S3 prefix that feeds Athena and QuickSight. The team wants the workflow to pause and wait for the steward's explicit approval, then continue automatically, without polling or wasting compute. What is the MOST appropriate way to implement this pause-and-approve behavior?

Accepted Answer

Use a Step Functions callback pattern with the .waitForTaskToken integration, sending an SNS/email notification containing the task token, and have the steward's approval action invoke SendTaskSuccess to resume the workflow

Answer

Add a Wait state configured for 12 hours in the state machine to give the steward time to review, then automatically proceed to the publish step

Answer

Split the pipeline into two separate Step Functions state machines and have the steward manually start the second (publish) machine after review

Answer

Configure a Lambda function that polls a DynamoDB approval table every minute inside a loop until an approval flag is set, then continue the workflow

Question 17

A data engineering team runs a nightly AWS Glue ETL job that normally completes in about 25 minutes. Over the past two weeks, the job's runtime has gradually crept up to over 90 minutes, though it still succeeds. Leadership wants proactive alerting when future jobs deviate significantly from their historical baseline, without hard-coding a fixed threshold that must be manually adjusted as data volume grows. Which approach best meets this requirement?

Accepted Answer

Create a CloudWatch alarm on the Glue job's execution duration metric using anomaly detection, and configure the alarm to trigger an SNS notification when the metric falls outside the expected band.

Answer

Create a static CloudWatch alarm on the glue.driver.aggregate.elapsedTime metric with a fixed threshold of 30 minutes and update it manually each quarter.

Answer

Enable Glue job bookmarks so the job processes less data and finishes faster, eliminating the need for duration monitoring.

Answer

Configure a CloudWatch Logs metric filter on the Glue job logs to count log lines and alarm when the count exceeds a fixed value.

Question 18

A data engineering team runs a nightly ingestion pipeline. A custom application writes structured log lines to Amazon CloudWatch Logs, including a line formatted as "REJECTED_RECORDS count=<n>" whenever malformed records are skipped. The team wants to be automatically alerted whenever the number of rejected records exceeds 500 in any single run, without modifying the application or building a separate log-processing service. What is the most operationally efficient way to achieve this?

Accepted Answer

Create a CloudWatch Logs metric filter that extracts the count value from matching log events, publish it as a custom metric, and configure a CloudWatch alarm with a threshold of 500 that notifies an SNS topic.

Answer

Stream the CloudWatch log group to Amazon Athena and schedule a query every hour to count rejected records, sending results by email.

Answer

Enable CloudWatch Logs Insights and run a scheduled query that filters for REJECTED_RECORDS, then manually review the dashboard each morning.

Answer

Create a Lambda function subscribed to the log group that parses every event and writes the count to DynamoDB, then poll DynamoDB with a second Lambda to send alerts.

Question 19

A media company needs to migrate 40 TB of archived video files from an on-premises NFS file server to Amazon S3. After the initial transfer, they must automatically sync any newly added or modified files every night with minimal engineering effort, and they require data integrity verification during each transfer. Which approach best meets these requirements?

Accepted Answer

Deploy an AWS DataSync agent on-premises, create a task from the NFS location to an S3 bucket, and schedule the task to run nightly.

Answer

Write a custom script using the AWS CLI 's3 sync' command and run it on a cron schedule from the on-premises server.

Answer

Use AWS Database Migration Service (DMS) with full load plus CDC to continuously replicate the NFS files to S3.

Answer

Configure Amazon Kinesis Data Firehose to stream the video files from the NFS server into the S3 bucket each night.

Question 20

A retail company is migrating a 2 TB Oracle database running on-premises to Amazon Aurora PostgreSQL. The database is actively used by production applications during business hours, and the business requires that the source database remain fully operational with minimal downtime during the migration. After the initial data copy, all new and changed records in the source must continue to be replicated to Aurora until the final cutover. Which AWS DMS approach meets these requirements?

Accepted Answer

Create a DMS task with migration type 'Migrate existing data and replicate ongoing changes' (full load plus CDC).

Answer

Create a DMS task with migration type 'Migrate existing data only' (full load) and schedule it to rerun nightly.

Answer

Create a DMS task with migration type 'Replicate data changes only' (CDC only) starting from the migration begin time.

Answer

Use AWS DataSync to copy the database files to Aurora and enable continuous synchronization.

Question 21

A retail company stores order records in a DynamoDB table using OrderId as the partition key. Business analysts increasingly need to query orders by CustomerId and by OrderStatus, but these attributes are not part of the table's primary key. The queries must return results with low latency and must not require scanning the entire table. What is the MOST appropriate way to enable these access patterns?

Accepted Answer

Create global secondary indexes (GSIs) on CustomerId and OrderStatus so those attributes can be queried directly with low latency.

Answer

Enable DynamoDB adaptive capacity so the table can automatically handle the new query patterns without changes.

Answer

Run a Scan operation with filter expressions on CustomerId and OrderStatus each time analysts need results.

Answer

Create local secondary indexes (LSIs) on CustomerId and OrderStatus to add the alternate query keys.

Question 22

A gaming company stores player match records in an Amazon DynamoDB table. Business rules require that each match record be automatically deleted 90 days after it is created to comply with a data retention policy, and the deletion should incur no additional write capacity charges. The team wants the least operational overhead. Which approach best meets these requirements?

Accepted Answer

Enable DynamoDB Time to Live (TTL) on an attribute that stores the record's expiration epoch timestamp.

Answer

Create a scheduled AWS Lambda function that scans the table daily and issues DeleteItem calls for records older than 90 days.

Answer

Configure an S3 Lifecycle rule to expire objects after 90 days and export the table to S3 nightly.

Answer

Enable DynamoDB point-in-time recovery and restore the table every 90 days to drop old records.

Question 23

A data engineering team has an existing library of custom Apache Spark applications written and tuned by their engineers, including several jobs that require specific open-source Spark configurations and third-party JAR dependencies. They want to run these jobs on a fully managed, auto-scaling service without provisioning or managing clusters, and they want to pay only for the resources consumed during job execution. Which service best meets these requirements with the least migration effort?

Accepted Answer

Run the existing Spark applications on Amazon EMR Serverless

Answer

Rewrite the jobs using AWS Glue DynamicFrames and run them as Glue ETL jobs

Answer

Deploy a persistent Amazon EMR cluster on EC2 sized for peak load

Answer

Convert the Spark logic into AWS Lambda functions triggered by EventBridge

Question 24

A data engineering team runs a nightly Spark job on Amazon EMR that joins a 2 TB fact table of transactions with a 40 MB reference table containing store metadata. The job is slow because Spark performs a shuffle-based sort-merge join, redistributing the massive fact table across the cluster over the network. The reference table easily fits in each executor's memory. What is the most effective change to accelerate this join with minimal code effort?

Accepted Answer

Apply a broadcast hint to the 40 MB reference table so it is sent to all executors, enabling a broadcast hash join and avoiding shuffling the fact table

Answer

Increase the number of shuffle partitions with spark.sql.shuffle.partitions to spread the sort-merge join across more tasks

Answer

Repartition the fact table by the join key before the join to co-locate matching records

Answer

Convert both tables to bucketed tables on the join key to eliminate the shuffle

Question 25

A data engineering team runs a nightly batch job that processes 8 TB of raw JSON logs stored in Amazon S3. The transformation logic is written in custom Apache Spark and Scala code that relies on several third-party JAR libraries and requires fine-grained tuning of Spark executor memory and shuffle configurations. The job runs once per day for about two hours and cost efficiency is a priority. Which approach best meets these requirements?

Accepted Answer

Run the Spark job on an Amazon EMR transient cluster launched via a scheduled Step Functions workflow, using Spot Instances for task nodes and terminating the cluster after job completion.

Answer

Use AWS Glue ETL with a standard worker type, importing the third-party JARs and Scala code as job dependencies, running on a persistent development endpoint.

Answer

Launch a long-running Amazon EMR cluster with On-Demand Instances so the Spark configurations remain consistent across nightly runs.

Answer

Convert the Spark code into AWS Lambda functions triggered by an S3 event to process the JSON logs incrementally.

Question 26

A data engineering team runs a nightly AWS Glue ETL job triggered by an Amazon EventBridge cron rule at 02:00 UTC. Due to a multi-hour AWS maintenance window in a downstream dependency, the source data was not available for three consecutive nights, so the job ran but produced empty output files that were written to the curated S3 zone. Analysts querying the data with Athena now see gaps and empty partitions. The team wants to prevent the pipeline from writing output when the expected upstream data has not landed, while keeping the automated schedule. Which approach BEST addresses this requirement?

Accepted Answer

Add a preliminary validation step (for example, a Lambda function or an initial Glue job) that checks for the presence and minimum record count of the source data, and only proceed to the transformation job if the check passes.

Answer

Increase the EventBridge cron schedule frequency to run the Glue job every hour so that it eventually picks up the source data when it lands.

Answer

Enable Glue job bookmarks so the job automatically skips runs where no new source data is present.

Answer

Configure a CloudWatch alarm on the Glue job's completion metric to notify the team after each empty run so they can manually delete the empty partitions.

Question 27

A data engineering team maintains an inventory system backed by a DynamoDB table. Whenever an item's stock level changes, they must trigger a downstream data enrichment process running as an AWS Lambda function, while also optionally filtering out changes that are not stock-related to avoid unnecessary invocations. The solution should be event-driven, require minimal custom code, and support built-in filtering without polling logic. Which approach best meets these requirements?

Accepted Answer

Enable a DynamoDB stream and use an EventBridge Pipe with the stream as the source, a content filter, and the Lambda function as the target.

Answer

Schedule an EventBridge rule to run every minute that invokes a Lambda function to scan the DynamoDB table for changed items.

Answer

Enable DynamoDB TTL and configure a Lambda function to process expired items as they are deleted.

Answer

Write a Kinesis Data Streams consumer application that continuously polls the DynamoDB table and applies filtering in custom code before calling Lambda.

Question 28

A data engineering team runs 40 independent Glue ETL jobs that must each start at specific times throughout the day. They currently manage schedules using cron expressions in individual Glue triggers, but they want a single, centralized service to manage all schedules, support one-time and recurring runs, apply flexible time windows, and automatically retry failed invocations with a dead-letter queue for troubleshooting. Which approach best meets these requirements with the least operational overhead?

Accepted Answer

Use Amazon EventBridge Scheduler with schedules that target the Glue StartJobRun API, configuring flexible time windows, retry policies, and an SQS dead-letter queue

Answer

Continue using individual Glue scheduled triggers and add CloudWatch alarms on each job to detect failures

Answer

Create a single Amazon EventBridge (default event bus) rule with a cron schedule that invokes all 40 jobs through one Lambda function

Answer

Configure an AWS Step Functions state machine with a Wait state for each job and schedule it with a cron-based EventBridge rule

Question 29

A data engineering team ingests high-volume clickstream events as JSON through Amazon Kinesis Data Firehose delivering to Amazon S3. Analysts query the data with Amazon Athena and complain about slow queries and high scan costs. The team wants Firehose to reduce storage size and improve Athena query performance with minimal custom code. What should they configure?

Accepted Answer

Enable Firehose record format conversion to Apache Parquet using an AWS Glue Data Catalog table as the schema source, and enable dynamic partitioning by event date.

Answer

Attach a Lambda transformation to Firehose that compresses each JSON record with GZIP before delivery to S3.

Answer

Increase the Firehose buffer size to 128 MB and buffer interval to 900 seconds so fewer, larger JSON files are written to S3.

Answer

Deliver the raw JSON to S3 and schedule a nightly AWS Glue ETL job to rewrite the data into Avro format partitioned by hour.

Question 30

A retail company runs a daily AWS Glue ETL job that reads new CSV files landing in an S3 prefix, converts them to Parquet, and writes to a curated bucket. As data volume grows, the team notices the job reprocesses all files in the source prefix every run, increasing runtime and cost. They want the job to process only files that have not been processed in previous runs, without manually tracking file names or maintaining a separate state store. Which approach best meets this requirement?

Accepted Answer

Enable AWS Glue job bookmarks so the job persists processed-object state and processes only new files on each run.

Answer

Add a Lambda function triggered by S3 events that records processed file keys in a DynamoDB table for the Glue job to filter against.

Answer

Configure the Glue job to use a pushdown predicate on the ingestion timestamp partition to skip older partitions.

Answer

Enable S3 Lifecycle rules to move already-processed files to a different prefix after each Glue run.

Question 31

A data engineering team lands compressed Avro files in an S3 bucket every hour from an upstream application. The Avro schema occasionally adds new optional fields over time. The team wants Amazon Athena to query the latest data without manual schema updates each time a new field appears, while minimizing operational overhead. Which approach best meets these requirements?

Accepted Answer

Configure an AWS Glue crawler to run on a schedule against the S3 path and enable the option to update the table definition in the Data Catalog when new columns are detected.

Answer

Manually define the Athena table with all current columns using a CREATE EXTERNAL TABLE statement and update the DDL each time a schema change occurs.

Answer

Convert the Avro files to CSV with a Lambda function before querying, so Athena can infer the schema automatically at query time.

Answer

Load the Avro files into an Amazon Redshift table using COPY and query them through Redshift Spectrum without cataloging the data.

Question 32

A data engineering team stores JDBC connection definitions in the AWS Glue Data Catalog to allow Glue jobs to reach an on-premises database. A security audit requires that the database credentials embedded in these connections not be stored in plaintext within Glue, and that any encryption keys be centrally managed and auditable. Which action best satisfies this requirement?

Accepted Answer

Enable connection password encryption in the Glue Data Catalog settings using a customer managed AWS KMS key

Answer

Enable server-side encryption on the S3 bucket that stores Glue job scripts using SSE-S3

Answer

Attach an IAM policy to the Glue job role that denies read access to the connection object

Answer

Store the connection credentials in a Glue job argument passed at runtime and enable job bookmarks

Question 33

A data engineering team stores clickstream data in Amazon S3 using a Hive-style layout: s3://logs/year=2024/month=05/day=12/. New daily partitions are added continuously. Analysts query the data through Amazon Athena, but query planning has become slow because millions of partitions must be fetched from the AWS Glue Data Catalog, and running a crawler after every load adds operational overhead. The team wants to eliminate partition-management overhead while keeping fast query planning. What should they do?

Accepted Answer

Enable partition projection on the Athena table by defining the partition key types and ranges in the table properties.

Answer

Increase the Glue crawler schedule to run every hour so partitions are always current.

Answer

Convert the data to a single unpartitioned Parquet file to remove the need for partition metadata.

Answer

Migrate the table metadata from the Glue Data Catalog to a self-managed Hive metastore on Amazon EMR.

Question 34

A company runs a central Glue Data Catalog in a shared services account. A data science team in a separate AWS account needs to run Athena queries against several databases and tables registered in that central catalog. The security team requires that access be granted at the Data Catalog level without duplicating table definitions and without using Lake Formation. Which approach should the data engineer implement to enable the cross-account access?

Accepted Answer

Attach a Glue Data Catalog resource policy in the shared services account granting the data science account glue:GetDatabase and glue:GetTable actions, and grant matching IAM permissions in the consumer account.

Answer

Enable a Glue crawler in the data science account that points at the shared services account's S3 buckets to recreate the table definitions locally.

Answer

Create an S3 bucket policy on the underlying data buckets that grants the data science account read access, which automatically shares the catalog metadata.

Answer

Export the databases from the central catalog and import them into a new Glue Data Catalog in the data science account using AWS Backup.

Question 35

A data engineering team runs an AWS Glue Spark ETL job that processes a large daily fact table (approximately 500 GB) of clickstream events stored in S3. Each event record must be enriched with product category and region attributes from a small reference dataset (about 40 MB) also stored in S3. The current job performs a standard join between the two DynamicFrames/DataFrames, but the job is slow and produces heavy shuffle activity across executors. Which approach will most effectively reduce shuffle and speed up the enrichment join?

Accepted Answer

Broadcast the small 40 MB reference dataset so it is replicated to all executors, allowing a map-side (broadcast) join with the large fact table

Answer

Repartition both datasets on the join key with a very high partition count before joining to force parallel processing

Answer

Convert both datasets to JSON before the join to reduce serialization overhead across the shuffle

Answer

Increase the number of DPUs (workers) so the shuffle completes faster across more executors

Question 36

A data engineering team recently onboarded a new upstream vendor feed that lands as CSV files in Amazon S3 and is processed by an AWS Glue ETL job. Because the vendor's schema and value ranges are not well documented, the team wants a fast way to establish an initial baseline set of data quality checks (such as completeness, uniqueness, and column value ranges) without hand-authoring every rule from scratch. Which approach requires the least manual effort to produce a starting DQDL ruleset?

Accepted Answer

Run AWS Glue Data Quality rule recommendations against the dataset in the Data Catalog to generate a suggested DQDL ruleset that the team can review and adjust.

Answer

Write a custom PySpark script in the Glue job that computes column statistics and manually translate the results into DQDL rules.

Answer

Enable Amazon CloudWatch anomaly detection on the Glue job's numFailedTasks metric to infer data quality thresholds.

Answer

Configure an AWS Glue crawler with schema inference and rely on the crawler to emit data quality rules for each column.

Question 37

A data engineering team runs a nightly AWS Glue ETL job that loads customer transaction data into an S3-based data lake. Recently, downstream analysts reported that some batches contained duplicate transaction IDs and null values in the required 'amount' column. The team wants to automatically validate incoming data against defined rules (such as uniqueness and completeness) and halt the pipeline before publishing bad data, while capturing quality scores in CloudWatch for monitoring. Which approach best meets these requirements with the least custom code?

Accepted Answer

Add an AWS Glue Data Quality ruleset using DQDL within the Glue job, configure it to evaluate rules and fail the job when the quality score falls below the threshold, and publish results to CloudWatch.

Answer

Write a custom PySpark validation function inside the Glue job that counts nulls and duplicates, raises an exception on failure, and manually pushes metrics to CloudWatch using the boto3 put_metric_data API.

Answer

Enable an Amazon Athena scheduled query after the load completes to count nulls and duplicates, then use an EventBridge rule to alert the team if results exceed thresholds.

Answer

Configure an AWS Glue crawler to run before the job and reject files that do not match the expected schema, then rely on the schema check to prevent duplicates and nulls.

Question 38

A data engineering team runs a daily AWS Glue ETL job that includes an AWS Glue Data Quality ruleset. Business stakeholders want an ongoing, automated way to observe whether the overall data quality of the pipeline is degrading over several weeks, without failing the job or blocking downstream processing. They want to visualize the trend and receive a notification only when quality drops below a defined threshold. Which approach best meets these requirements?

Accepted Answer

Configure the Glue Data Quality evaluation to publish result metrics to Amazon CloudWatch, then create a CloudWatch dashboard for the score trend and a CloudWatch alarm on the metric threshold that notifies an SNS topic.

Answer

Set the Glue Data Quality ruleset action to stop the job on failure, and rely on the job-failure notification to indicate quality degradation over time.

Answer

Write the Data Quality results to Amazon S3 and run a weekly Athena query manually to check whether the average score has dropped.

Answer

Enable Glue job bookmarks and inspect the CloudWatch job duration metric, treating longer runtimes as a proxy for declining data quality.

Question 39

A data engineer is building an AWS Glue Spark ETL job that reads semi-structured JSON files from S3. The same field 'amount' appears as a string in some records and as a double in others, causing the job to fail when writing to a Parquet target with a fixed schema. The engineer wants to handle the ambiguous column so that all values are cast to a single numeric type without dropping records. Which approach best resolves this within the Glue job?

Accepted Answer

Apply the ResolveChoice transform on the DynamicFrame with a cast to 'double' for the 'amount' column before writing to Parquet

Answer

Convert the DynamicFrame to a Spark DataFrame immediately and rely on Spark's automatic schema inference to unify the types

Answer

Use the ApplyMapping transform to rename the 'amount' column so the conflicting types are stored separately

Answer

Enable a Glue job bookmark so that only records with consistent types are processed on each run

Question 40

A data engineering team runs an AWS Glue Spark ETL job that must read from an Amazon RDS for PostgreSQL database located in a private subnet with no public IP. The Glue job also needs to write the transformed results to an Amazon S3 bucket. When the job runs, it fails with a connection timeout while attempting to reach the RDS instance. What is the correct way to configure Glue so it can reach both RDS and S3?

Accepted Answer

Attach a Glue connection that references the RDS VPC, subnet, and security group, and create an S3 VPC gateway endpoint in that VPC route table.

Answer

Enable a public IP on the Glue job and open the RDS security group to 0.0.0.0/0 on port 5432.

Answer

Move the RDS instance to a public subnet and assign it an Elastic IP so Glue can reach it over the internet.

Answer

Add an internet gateway to the RDS VPC and route all Glue traffic through a NAT gateway to access S3 and RDS.

Question 41

A data engineering team runs a daily AWS Glue Spark ETL job that loads customer records from multiple upstream systems into a curated S3 dataset. Due to overlapping source feeds, the same customer_id can appear multiple times per run, and only the record with the most recent last_updated timestamp should be kept. The team needs to implement deduplication within the Glue job with minimal custom code while ensuring the surviving row is always the latest version. Which approach best meets these requirements?

Accepted Answer

Convert the DynamicFrame to a Spark DataFrame, use a window function partitioned by customer_id ordered by last_updated descending, and keep only the first row per partition.

Answer

Apply the DynamicFrame dropDuplicates() transform with no arguments to remove exact duplicate rows across all columns.

Answer

Enable Glue job bookmarks so that previously processed records are automatically excluded from subsequent runs.

Answer

Use the FindMatches ML transform to identify fuzzy duplicate customer records and merge them automatically.

Question 42

A data engineering team runs an AWS Glue Spark ETL job that reads clickstream events and writes the results to Amazon S3 in Parquet format. Downstream Athena queries filter heavily by event_date and country. Currently the job writes all output into a single S3 prefix, causing Athena to scan the entire dataset on every query. The team wants the Glue job itself to physically organize the output so that Athena can prune data by these two columns without a separate reprocessing step. What is the MOST appropriate way to configure the Glue write operation?

Accepted Answer

Write the DynamicFrame using the getSink API with partitionKeys set to ["event_date", "country"] so Glue writes Hive-style partitioned prefixes in S3.

Answer

Enable Glue job bookmarks so that only new records are written on each run, which forces partition creation automatically.

Answer

Increase the number of Glue DPUs so the job produces more output files, improving Athena partition pruning.

Answer

Run a Glue crawler after the job completes and set the crawler to create partitions based on the event_date and country columns.

Question 43

A data engineering team ingests daily nested JSON logs (~500 GB/day) into S3. Analysts query the data with Amazon Athena, filtering mostly on a small subset of columns. The team wants to reduce Athena scan costs and improve query performance while keeping storage efficient. Which AWS Glue ETL output configuration best meets these goals?

Accepted Answer

Write the output as ORC files with Snappy compression, partitioned by the columns most used in query filters.

Answer

Write the output as gzip-compressed JSON files, partitioned by ingestion date only.

Answer

Write the output as uncompressed CSV files to maximize compatibility with Athena.

Answer

Write the output as Avro files with Deflate compression to optimize columnar reads.

Question 44

A data engineer is building an AWS Glue Spark ETL job to process raw event logs. Each incoming JSON record contains an array field named 'items' with a variable number of nested purchase objects. The business requires that each element of the 'items' array become its own separate row in the output Parquet dataset, with the parent record's metadata (order_id, customer_id, timestamp) repeated on every resulting row. Which transformation approach correctly produces the desired output?

Accepted Answer

Use the Spark DataFrame explode() function on the 'items' column after selecting the parent metadata columns, so each array element generates a distinct row.

Answer

Apply the Glue DynamicFrame ResolveChoice transform to cast the 'items' array to a struct and write directly to Parquet.

Answer

Use the Glue DropFields transform to remove the 'items' array and write only the parent metadata to Parquet.

Answer

Apply the Spark distinct() function to the DataFrame to deduplicate the nested items before writing to Parquet.

Question 45

A retail company ingests raw customer order data into Amazon S3 as CSV files. A data engineer must build an AWS Glue ETL job that cleanses the data before loading it into a curated zone. The requirements are: replace empty strings and the literal text 'N/A' in the customer_email column with null, drop any row where order_id is missing, and standardize the country column so that values like 'usa', 'USA', and 'Usa' all become 'US'. Which approach implements all three cleansing requirements most efficiently within a single Glue Spark ETL job?

Accepted Answer

Convert the DynamicFrame to a Spark DataFrame, use withColumn with when/otherwise and regexp/upper functions to normalize country and null-out email values, and use dropna(subset=['order_id']) before writing back out.

Answer

Use the Glue DynamicFrame ResolveChoice transform to cast all columns to strings, then rely on the Glue Data Catalog crawler to automatically remove null order_id rows during the next crawl.

Answer

Write the raw data to S3 unchanged and create an Athena view with CASE expressions so the transformations are applied only at query time.

Answer

Enable a Glue job bookmark and configure the job's transformation_ctx so that Glue automatically standardizes casing and removes null-key records on incremental runs.

Question 46

A data engineering team runs a daily AWS Glue Spark ETL job that writes a large analytical dataset to Amazon S3 in Parquet format. The data is queried heavily via Amazon Athena, and queries frequently filter and aggregate across many columns. The team wants to minimize storage costs while achieving the best query performance, and they are willing to accept slightly higher write-time CPU usage during the ETL job. Which compression configuration should they apply to the Parquet output?

Accepted Answer

Write the Parquet files using GZIP compression to maximize the compression ratio and reduce storage footprint the most

Answer

Write the Parquet files using Snappy compression because it is the only codec Athena can read for Parquet

Answer

Write the Parquet files uncompressed so Athena does not spend time decompressing data during scans

Answer

Convert the output to CSV with GZIP compression to reduce both storage and Athena scan costs

Question 47

A data engineer receives deeply nested JSON documents from a SaaS API, landed in Amazon S3. Each document contains arrays of order line items nested inside customer objects. The team needs to load this data into an Amazon Redshift data warehouse where analysts run SQL joins across flattened relational tables. The engineer is building an AWS Glue ETL job using DynamicFrames. Which approach most efficiently converts the nested structure into normalized, relational tables suitable for Redshift?

Accepted Answer

Use the Relationalize transform on the DynamicFrame to flatten nested structures and unnest arrays into separate tables linked by generated keys

Answer

Apply the ResolveChoice transform to cast the nested arrays into string columns and load them as-is into Redshift

Answer

Use the ApplyMapping transform to rename each nested field individually into a single wide flat table

Answer

Convert the DynamicFrame to a Spark DataFrame and use the SelectFields transform to drop the nested arrays before loading

Question 48

A data engineering team maintains a customer dimension table in an S3-based data lake. The business requires that whenever a customer's attributes (such as address or loyalty tier) change, the previous values must be preserved as historical records rather than overwritten, so analysts can query the state of a customer at any point in time. The team uses AWS Glue Spark ETL jobs to process incoming daily batches of customer updates. Which transformation approach should the team implement in the Glue job?

Accepted Answer

Implement a Slowly Changing Dimension (SCD) Type 2 pattern: compare incoming records to current records, expire the old version by setting an end-date and active flag, and insert the new version with a new effective-date.

Answer

Use dropDuplicates on the customer ID to keep only the most recent record for each customer in every batch.

Answer

Perform an in-place overwrite of matching customer records using an SCD Type 1 upsert so the table always reflects the latest values.

Answer

Append every incoming record to the table without any comparison, relying on the ingestion timestamp column alone.

Question 49

A media company drops variable numbers of raw JSON log files into an S3 landing bucket at unpredictable times throughout the day. A data engineer must run an AWS Glue ETL job to clean and convert each new batch to Parquet within a few minutes of arrival, without incurring cost when no files are present. Which approach best meets these requirements?

Accepted Answer

Configure an EventBridge rule that matches S3 'Object Created' events and targets a Step Functions state machine that starts the Glue ETL job.

Answer

Schedule the Glue ETL job to run every 5 minutes using a Glue time-based trigger so new files are always picked up quickly.

Answer

Enable a Glue crawler on a 5-minute schedule and use an on-demand Glue trigger conditioned on crawler completion to start the ETL job.

Answer

Configure S3 to send a notification to an SQS queue and poll the queue continuously from a long-running EMR cluster that launches the Glue job.

Question 50

A data engineering team runs a daily AWS Glue ETL job that ingests data from a third-party REST API. Roughly once a week the job fails because the API returns an intermittent HTTP 503 error during the first few minutes of execution; re-running the job manually always succeeds. The team wants to reduce operational overhead and recover automatically from these transient failures without rewriting the job logic. What is the MOST operationally efficient solution?

Accepted Answer

Configure the maximum retries property on the Glue job so that it automatically re-executes when a run fails.

Answer

Create a CloudWatch alarm on the Glue job failure metric that publishes to an SNS topic so an on-call engineer can manually re-run the job.

Answer

Increase the number of DPUs allocated to the Glue job to make the API calls complete faster.

Answer

Enable Glue job bookmarks so the job skips previously processed records after a failure.

Question 51

A data engineer runs a Glue Spark ETL job that processes a few large, unsplittable gzip files. The job takes several hours and often finishes with the maximum number of DPUs allocated, yet costs are high. In the Glue job metrics (CloudWatch), the engineer observes that only one or two executors show high CPU while glue.driver.aggregate.numCompletedStages stays low and most executors show near-zero activity for long stretches. What is the MOST likely cause and the best corrective action?

Accepted Answer

The input gzip files cannot be split, so parallelism is limited to a few tasks; recompress the source data into a splittable format (e.g., bzip2 or many smaller files/Parquet) so Spark can distribute work across executors.

Answer

The job has too few DPUs allocated; increase the maximum capacity so more executors can process the same files in parallel.

Answer

Job bookmarks are disabled, causing reprocessing of the entire dataset; enable bookmarks to reduce runtime and executor idle time.

Answer

The Glue Data Catalog partitions are not registered; run a crawler so Spark can read all partitions in parallel.

Question 52

A data engineering team schedules an AWS Glue ETL job to run every 15 minutes to process incoming files. During heavy traffic periods, upstream events sometimes trigger the job while a previous run is still executing. The team notices that some job runs fail immediately with a 'ConcurrentRunsExceededException' error, and downstream data is occasionally incomplete. They want to allow overlapping runs when needed but prevent runaway parallelism, without rewriting the pipeline logic. What is the MOST appropriate action?

Accepted Answer

Increase the job's 'Maximum concurrency' setting in the Glue job properties to a value that accommodates expected overlapping runs.

Answer

Reduce the job's worker count (DPUs) so each run completes faster and fewer runs overlap.

Answer

Enable Glue job bookmarks so overlapping runs do not reprocess the same data.

Answer

Convert the Glue job to an EMR Serverless application, which has no concurrency limits.

Question 53

A data engineer runs a nightly AWS Glue Spark ETL job that has recently started failing after processing about 80% of its input. The CloudWatch job metrics show glue.driver.jvm.heap.usage climbing steadily toward 1.0 and glue.ExecutorAllocationManager.executors.numberMaxNeededExecutors well above the number of allocated executors, while glue.driver.aggregate.shuffleBytesWritten is very high. The job code performs a large groupBy followed by a collect() to build a lookup dictionary on the driver. What is the MOST likely root cause and appropriate fix?

Accepted Answer

The driver is running out of memory because collect() pulls the full grouped dataset onto the driver; refactor the logic to keep the aggregation distributed (e.g., use a join instead of collecting to the driver).

Answer

The job lacks enough DPUs for the crawler; enable a scheduled crawler to run before the job so partitions are cached.

Answer

Glue bookmarks are disabled, causing reprocessing of all data; enable job bookmarks to reduce the driver heap pressure.

Answer

The S3 source uses too many small files; convert the source to a single large CSV file to reduce shuffle.

Question 54

A data engineer runs several AWS Glue Spark ETL jobs nightly. Occasionally a job succeeds but produces fewer output records than expected, and the team wants a way to retroactively investigate the root cause (skewed partitions, data movement, executor behavior) without re-running the job or adding custom logging to each script. What is the MOST operationally efficient way to capture this detailed diagnostic information?

Accepted Answer

Enable Glue job observability metrics (Amazon CloudWatch observability metrics) on the jobs so per-job performance and data metrics are automatically emitted to CloudWatch.

Answer

Add print statements throughout each Spark script and review the driver output in CloudWatch Logs after each run.

Answer

Enable AWS CloudTrail data events for the Glue jobs and analyze the API calls in Athena.

Answer

Increase the number of DPUs allocated to each job so Spark automatically records more detailed execution statistics.

Question 55

A data engineering team runs a daily AWS Glue Spark ETL job that reads from a large S3-based data lake containing several years of clickstream data partitioned by year, month, and day. The job only needs to process the most recent day of data, but it currently takes over an hour because the job scans the entire dataset before filtering. Which approach will most effectively reduce the amount of data read and improve job performance?

Accepted Answer

Apply a pushdown predicate on the partition columns when creating the DynamicFrame from the Data Catalog so only the relevant partitions are read from S3.

Answer

Increase the number of Glue DPUs allocated to the job so the full dataset is scanned faster in parallel.

Answer

Convert the source data from Parquet to JSON to make partition filtering easier for the Spark engine.

Answer

Add a Spark filter() transformation on the date column after loading the entire dataset into a DynamicFrame.

Question 56

A retail company ingests customer records daily from multiple acquired subsidiaries into Amazon S3. The records contain the same customers but with inconsistent formatting: names with typos, abbreviated addresses, and varying capitalization. A data engineer must deduplicate these records where there is no reliable common key, and matches must account for minor spelling and formatting variations. Which AWS Glue capability best addresses this requirement with the least custom development?

Accepted Answer

Use the Glue FindMatches ML transform to train a model that identifies fuzzy duplicate records across the datasets

Answer

Use the DynamicFrame dropDuplicates method to remove exact duplicate rows in the Glue ETL job

Answer

Use a Glue crawler to catalog the data and enable schema-based deduplication in the Data Catalog

Answer

Use the Glue Relationalize transform to flatten the records and remove duplicates by primary key

Question 57

A data engineering team maintains a data-cleansing job written entirely in Python using pandas and NumPy. The logic performs record-level normalization on datasets that have grown to several hundred gigabytes, and the single-node script now runs out of memory and takes hours. The team wants to scale the existing Python/pandas-style code across multiple nodes with minimal rewriting, staying within AWS Glue. Which approach best meets these requirements?

Accepted Answer

Use an AWS Glue for Ray job, which distributes Python and pandas-style workloads across a cluster of worker nodes.

Answer

Rewrite the entire job in PySpark DataFrame APIs and run it as a standard AWS Glue Spark ETL job.

Answer

Run the existing script as an AWS Glue Python shell job and increase the allocated DPUs to the maximum.

Answer

Migrate the pandas code to an AWS Lambda function with the maximum memory allocation and process the data in chunks.

Question 58

A retail company ingests clickstream events into an Amazon Kinesis Data Stream. A data engineer must build a pipeline that continuously reads the stream, removes duplicate events that arrive within a short window (caused by client retries), enriches each event with product metadata stored in a small Amazon DynamoDB table, and writes the results to Amazon S3 in Parquet format for analytics. The solution must be serverless and minimize operational overhead. Which approach best meets these requirements?

Accepted Answer

Use an AWS Glue streaming ETL job that reads from the Kinesis stream, applies dropDuplicates within a time window, joins with the DynamoDB metadata loaded as a DataFrame, and writes Parquet output to S3.

Answer

Configure Amazon Kinesis Data Firehose to deliver records directly to S3 as Parquet, then run a nightly AWS Glue batch job to deduplicate and enrich the data.

Answer

Launch a long-running Amazon EMR cluster with Spark Structured Streaming to read the Kinesis stream, deduplicate, enrich from DynamoDB, and write to S3.

Answer

Use a Lambda function triggered by the Kinesis stream to write each raw event to S3 as JSON, then use Amazon Athena CTAS queries to remove duplicates and enrich the data on demand.

Question 59

A data engineering team runs a nightly AWS Glue workflow consisting of a crawler, followed by three sequential ETL jobs (Job A, Job B, Job C). Last night, Job A and Job B succeeded, but Job C failed due to a transient downstream S3 throttling error. The team wants to complete last night's processing as quickly and cost-effectively as possible without reprocessing data that was already successfully transformed. What should they do?

Accepted Answer

Manually run only Job C from the AWS Glue console using the same run parameters, then verify the workflow output before the next scheduled run.

Answer

Restart the entire Glue workflow from the beginning so that the crawler and all three jobs run again to guarantee consistency.

Answer

Delete the S3 output written by Job A and Job B, then rerun the full workflow to avoid duplicate records.

Answer

Disable the workflow schedule and migrate the pipeline to a single monolithic Glue job that performs all three transformations in one run.

Question 60

A data engineering team runs a nightly Glue workflow with three sequential jobs. The first job computes the exact S3 partition path that was processed (based on the latest ingested date) and the downstream jobs must read from that same computed path. Currently the team hardcodes the date, which causes failures when late data shifts the target partition. They want a native way to pass the computed value from the first job to the subsequent jobs within the same workflow run without using an external store. What should they do?

Accepted Answer

Have the first job set workflow run properties using the Glue API, and have downstream jobs retrieve those run properties at the start of each job.

Answer

Write the computed path to a fixed key in Amazon S3 and have each downstream job read that object before processing.

Answer

Store the computed path in an SSM Parameter Store parameter and reference it as a job argument in the downstream jobs.

Answer

Pass the computed path by enabling Glue job bookmarks so the state is automatically shared across the jobs in the workflow.

Question 61

A data engineering team runs a nightly Glue workflow that ingests transactional files, transforms them, and updates the Data Catalog. The workflow must start automatically at 02:00 UTC every day. During a recent incident, an upstream system delivered files late, and the team wants a way to also trigger the same workflow manually when they know files have arrived early, without duplicating the workflow definition. Which approach meets both requirements with the least operational overhead?

Accepted Answer

Configure a scheduled trigger on the workflow for 02:00 UTC and start the same workflow on demand via the StartWorkflowRun API or console when needed.

Answer

Create two separate Glue workflows with identical job definitions, one with a scheduled trigger and one with an on-demand trigger.

Answer

Use an EventBridge rule with a cron expression as the only trigger, and disable Glue triggers entirely so the workflow can only run on a schedule.

Answer

Convert the workflow jobs into a single Glue job with a cron schedule, and re-run the job manually by editing its schedule to the current time each morning.

Question 62

A data engineering team runs a nightly batch pipeline consisting of three sequential AWS Glue jobs: a raw ingestion job, a cleansing job, and an aggregation job. The team wants the pipeline to start automatically when a new file lands in an S3 bucket, and each Glue job should run only after the previous one completes successfully, with no code to manage between jobs. They prefer to keep the orchestration entirely within AWS Glue's native tooling rather than adding another service. Which approach best meets these requirements?

Accepted Answer

Create an AWS Glue workflow with an EventBridge trigger that starts on S3 object creation, followed by conditional triggers that fire each subsequent job on the SUCCEEDED state of the prior job.

Answer

Schedule all three Glue jobs with a cron-based Glue trigger set to the same time so they run together each night.

Answer

Use a single Glue job with three separate DynamicFrame transformations and enable job bookmarks to chain the steps automatically.

Answer

Configure an S3 event notification to invoke a Lambda function that calls StartJobRun three times in a loop for the three Glue jobs.

Question 63

A financial services company allows senior data engineers to create IAM roles for their own Glue jobs and Lambda functions. The security team is concerned that engineers could create roles with broader permissions than intended (for example, granting full S3 or KMS access to production buckets). The team wants to guarantee that any role an engineer creates can never exceed a defined maximum set of permissions, regardless of the policies the engineer attaches. Which approach meets this requirement?

Accepted Answer

Attach a permissions boundary to the engineers' IAM roles that constrains the maximum permissions of any roles they create, and require the boundary to be attached to newly created roles

Answer

Attach a resource-based S3 bucket policy that denies access to any role created by the data engineers

Answer

Enable IAM Access Analyzer to automatically remove excessive permissions from roles the engineers create

Answer

Grant the engineers only the iam:CreateRole permission and rely on the least-privilege default of new roles

Question 64

A company runs hundreds of on-premises web servers that continuously append records to local Apache log files. The data engineering team must stream these log records into Amazon Kinesis Data Streams for near real-time analytics. The solution should require minimal custom code, automatically monitor log files for new lines, handle file rotation, and retry on delivery failures. Which approach best meets these requirements?

Accepted Answer

Install the Amazon Kinesis Agent on each web server and configure it to tail the log files and forward records to the Kinesis data stream.

Answer

Write a custom application using the Kinesis Producer Library (KPL) that reads the log files and calls PutRecords, then deploy it to every server.

Answer

Configure Amazon Kinesis Data Firehose to poll the log files on each server and deliver them to the Kinesis data stream.

Answer

Use AWS DataSync to transfer the log files to Amazon S3 and trigger a Lambda function to publish each record to the Kinesis data stream.

Exam domain	Exam weight	Practice
Data Ingestion and Transformation	34%	Practice this topic
Data Store Management	26%	Practice this topic
Data Operations and Support	22%	Practice this topic
Data Security and Governance	18%	Practice this topic

AWS Certified Data Engineer - Associate (DEA-C01) practice exam & study guide

Start studying DEA-C01

All DEA-C01 study resources

DEA-C01 exam domains

Sample DEA-C01 questions

Key DEA-C01 terms

DEA-C01 frequently asked questions

Data Ingestion and Transformation (34%)

Data Store Management (26%)

Data Operations and Support (22%)

Data Security and Governance (18%)