Hard — DEA-C01 Practice Quiz

Question 1

A data engineer runs a Glue Spark ETL job that processes a few large, unsplittable gzip files. The job takes several hours and often finishes with the maximum number of DPUs allocated, yet costs are high. In the Glue job metrics (CloudWatch), the engineer observes that only one or two executors show high CPU while glue.driver.aggregate.numCompletedStages stays low and most executors show near-zero activity for long stretches. What is the MOST likely cause and the best corrective action?

Accepted Answer

The input gzip files cannot be split, so parallelism is limited to a few tasks; recompress the source data into a splittable format (e.g., bzip2 or many smaller files/Parquet) so Spark can distribute work across executors.

Answer

The job has too few DPUs allocated; increase the maximum capacity so more executors can process the same files in parallel.

Answer

Job bookmarks are disabled, causing reprocessing of the entire dataset; enable bookmarks to reduce runtime and executor idle time.

Answer

The Glue Data Catalog partitions are not registered; run a crawler so Spark can read all partitions in parallel.

Question 2

A data engineer runs a nightly AWS Glue Spark ETL job that has recently started failing after processing about 80% of its input. The CloudWatch job metrics show glue.driver.jvm.heap.usage climbing steadily toward 1.0 and glue.ExecutorAllocationManager.executors.numberMaxNeededExecutors well above the number of allocated executors, while glue.driver.aggregate.shuffleBytesWritten is very high. The job code performs a large groupBy followed by a collect() to build a lookup dictionary on the driver. What is the MOST likely root cause and appropriate fix?

Accepted Answer

The driver is running out of memory because collect() pulls the full grouped dataset onto the driver; refactor the logic to keep the aggregation distributed (e.g., use a join instead of collecting to the driver).

Answer

The job lacks enough DPUs for the crawler; enable a scheduled crawler to run before the job so partitions are cached.

Answer

Glue bookmarks are disabled, causing reprocessing of all data; enable job bookmarks to reduce the driver heap pressure.

Answer

The S3 source uses too many small files; convert the source to a single large CSV file to reduce shuffle.

Question 3

A retail company ingests clickstream events into an Amazon Kinesis Data Stream. A data engineer must build a pipeline that continuously reads the stream, removes duplicate events that arrive within a short window (caused by client retries), enriches each event with product metadata stored in a small Amazon DynamoDB table, and writes the results to Amazon S3 in Parquet format for analytics. The solution must be serverless and minimize operational overhead. Which approach best meets these requirements?

Accepted Answer

Use an AWS Glue streaming ETL job that reads from the Kinesis stream, applies dropDuplicates within a time window, joins with the DynamoDB metadata loaded as a DataFrame, and writes Parquet output to S3.

Answer

Configure Amazon Kinesis Data Firehose to deliver records directly to S3 as Parquet, then run a nightly AWS Glue batch job to deduplicate and enrich the data.

Answer

Launch a long-running Amazon EMR cluster with Spark Structured Streaming to read the Kinesis stream, deduplicate, enrich from DynamoDB, and write to S3.

Answer

Use a Lambda function triggered by the Kinesis stream to write each raw event to S3 as JSON, then use Amazon Athena CTAS queries to remove duplicates and enrich the data on demand.

Question 4

A financial services company allows senior data engineers to create IAM roles for their own Glue jobs and Lambda functions. The security team is concerned that engineers could create roles with broader permissions than intended (for example, granting full S3 or KMS access to production buckets). The team wants to guarantee that any role an engineer creates can never exceed a defined maximum set of permissions, regardless of the policies the engineer attaches. Which approach meets this requirement?

Accepted Answer

Attach a permissions boundary to the engineers' IAM roles that constrains the maximum permissions of any roles they create, and require the boundary to be attached to newly created roles

Answer

Attach a resource-based S3 bucket policy that denies access to any role created by the data engineers

Answer

Enable IAM Access Analyzer to automatically remove excessive permissions from roles the engineers create

Answer

Grant the engineers only the iam:CreateRole permission and rely on the least-privilege default of new roles

Question 5

A company processes clickstream events using a Kinesis Data Streams stream consumed by an application built with the Kinesis Client Library (KCL) running on EC2 instances. During a deployment, several EC2 instances were terminated mid-batch and the application restarted. The team notices that some records were processed more than once, causing duplicate rows in the downstream analytics database. The team wants to reduce duplicate processing caused by consumer restarts while keeping the KCL-based architecture. What is the MOST appropriate action?

Accepted Answer

Configure the KCL to checkpoint more frequently and make the downstream write operations idempotent so reprocessed records do not create duplicates.

Answer

Enable enhanced fan-out on the stream so each consumer gets a dedicated throughput pipe that prevents duplicates.

Answer

Increase the stream retention period to 168 hours so KCL can avoid re-reading already-processed records.

Answer

Switch the consumer to use the Kinesis Data Streams GetRecords API directly without the KCL to avoid checkpoint issues.

Question 6

A data engineering team runs a Glue ETL job under an IAM role that must decrypt objects in an S3 bucket encrypted with a customer managed KMS key. The KMS key is owned by a separate security team who manages the key policy centrally and refuses to add every new consumer role directly into the key policy. The security team wants to allow the Glue role to perform decrypt operations while retaining full control over the key and enabling the permission to be revoked programmatically at any time without editing the key policy. Which approach best meets these requirements?

Accepted Answer

Create a KMS grant on the key that permits the Glue role the Decrypt and GenerateDataKey operations, and revoke the grant when access is no longer needed.

Answer

Add the Glue role's ARN as a principal in the KMS key policy with kms:Decrypt permission.

Answer

Attach an IAM policy to the Glue role granting kms:Decrypt on the key, with no changes to the key policy.

Answer

Enable automatic key rotation on the KMS key so the Glue role inherits decrypt access on each rotation.

Question 7

A data engineer registers a new S3 data lake location with AWS Lake Formation and creates several databases and tables in the Glue Data Catalog. The team wants Lake Formation permissions to be the sole mechanism controlling access to these tables. However, after granting fine-grained SELECT permissions to specific analyst roles, they discover that IAM principals with broad Glue and S3 permissions can still access the tables directly. What is the root cause and correct fix?

Accepted Answer

The IAMAllowedPrincipals group still has Super permission on the databases and tables; revoke it so only explicit Lake Formation grants apply.

Answer

Lake Formation cannot govern Glue Catalog tables unless a resource link is created; create resource links for each table.

Answer

The S3 bucket policy is overriding Lake Formation; remove all S3 bucket policies to let Lake Formation take effect.

Answer

KMS key policies are granting access; update the CMK policy to deny the analyst roles direct decrypt access.

Question 8

A data engineer grants a data analyst SELECT permission on a Glue Data Catalog table using AWS Lake Formation. The underlying data is stored in an S3 location that Lake Formation manages (the S3 location is registered with Lake Formation). When the analyst runs an Athena query against the table, the query fails with an access-denied error on the S3 objects, even though Lake Formation shows the SELECT grant. The analyst's IAM role has no direct S3 bucket policy or IAM permission granting access to the S3 path. What is the correct explanation and fix?

Accepted Answer

The S3 location must be registered with Lake Formation using a service role, allowing Lake Formation to vend temporary credentials so the analyst does not need direct S3 IAM permissions; verify the location registration and that Lake Formation credential vending is being used.

Answer

Lake Formation permissions never control S3 object access, so the analyst's IAM role must always be granted explicit s3:GetObject permissions on the bucket path in addition to the Lake Formation SELECT grant.

Answer

The analyst needs a KMS key policy update because Lake Formation cannot decrypt the S3 objects on the analyst's behalf under any circumstances.

Answer

SELECT permission in Lake Formation only applies to metadata in the Glue Data Catalog; a separate Lake Formation DATA_LOCATION_ACCESS grant on the S3 path is required for the analyst to read the objects.

Question 9

A data engineering team is adopting AWS Lake Formation to govern a data lake stored in Amazon S3. The team's Glue ETL jobs run under an IAM role that already has S3 bucket permissions granted through IAM policies, but analysts querying the same tables through Athena report 'Insufficient Lake Formation permission' errors even though they have been granted SELECT via Lake Formation. What must the team do so that Lake Formation permissions are actually enforced and honored for the S3 data location?

Accepted Answer

Register the S3 location with Lake Formation using a registered role, and remove the broad IAM S3 permissions that bypass Lake Formation enforcement.

Answer

Attach an S3 bucket policy that grants s3:GetObject to the analysts' IAM roles so Athena can read the underlying objects.

Answer

Enable default encryption with a KMS key on the S3 bucket and grant the analysts kms:Decrypt permission.

Answer

Add the analysts to the Lake Formation data lake administrators list so their SELECT grants take effect.

Question 10

A data engineering team has been managing access to their S3-based data lake entirely through IAM policies and S3 bucket policies. As the number of analysts and Glue Data Catalog tables grows, they want to centralize fine-grained (database, table, and column-level) permissions and maintain a single audit trail of who granted access to what. They enable AWS Lake Formation and register the S3 location. However, after granting a data analyst SELECT on a specific table through Lake Formation, the analyst can still query all tables in the S3 location via Athena. What is the MOST likely cause?

Accepted Answer

The IAMAllowedPrincipals default permission is still granted on the databases and tables, so Lake Formation is deferring to the analyst's existing IAM/S3 permissions.

Answer

Lake Formation column-level permissions do not take effect in Athena and require Redshift Spectrum instead.

Answer

The S3 location must be encrypted with a KMS key before Lake Formation permissions are enforced.

Answer

Athena does not support Lake Formation permissions and always uses the query executor's IAM role directly.

Question 11

A data engineering team ingests order events into Amazon MSK using Kafka Connect with a source connector, and serializes messages in Avro using AWS Glue Schema Registry. Downstream consumers include a long-running Flink application that cannot be redeployed frequently. Product teams will soon add several new optional fields to the order event schema. The team must ensure that existing consumers continue reading messages without deserialization failures after producers begin sending the new schema version. Which schema compatibility setting should the team configure in the Glue Schema Registry?

Accepted Answer

FORWARD compatibility, so consumers using an older schema can read data produced with the new schema

Answer

BACKWARD compatibility, so consumers using the new schema can read data produced with older schemas

Answer

NONE compatibility, so any schema change is accepted without validation

Answer

FULL_ALL compatibility, which requires renaming existing fields on each schema change

Question 12

A data engineering team runs a nightly Glue ETL job that writes curated sales data to Amazon S3, cataloged in the Glue Data Catalog. Business analysts consume this data through an Amazon QuickSight dashboard built on a direct Athena query dataset (not SPICE). After a recent change, analysts report that the dashboard sometimes shows partial data for the current day, appearing before the nightly job completes. The team wants QuickSight to reliably reflect only fully processed data and to refresh automatically once each night's ETL finishes. Which approach best meets this requirement?

Accepted Answer

Add an EventBridge rule triggered on Glue job SUCCEEDED state that invokes a Lambda function to publish a partition-complete marker, and point the QuickSight Athena dataset at a table that only exposes fully committed partitions.

Answer

Switch the QuickSight dataset to SPICE and schedule an hourly SPICE refresh so the dashboard picks up new data throughout the day.

Answer

Increase the Glue job DPU allocation so the ETL completes faster and analysts are less likely to query the table mid-run.

Answer

Enable QuickSight anomaly detection on the dataset so partially loaded days are automatically flagged and hidden from viewers.

Question 13

A data engineering team is designing a Redshift data warehouse for retail sales analytics. The fact table (sales_transactions) contains over 5 billion rows and grows daily. It is frequently joined against several small dimension tables: dim_store (500 rows), dim_product (50,000 rows), and dim_date (10,000 rows). Queries commonly aggregate sales measures grouped by store, product category, and date. The team wants to minimize data redistribution during joins while keeping large fact data evenly spread across compute node slices. Which distribution strategy should they apply?

Accepted Answer

Use DISTSTYLE EVEN on the fact table and DISTSTYLE ALL on the small dimension tables.

Answer

Use DISTSTYLE KEY on the fact table's product_id and set the same KEY distribution on all dimension tables.

Answer

Use DISTSTYLE ALL on the fact table and DISTSTYLE KEY on each dimension table.

Answer

Use DISTSTYLE AUTO on all tables and rely on automatic sort key selection to eliminate join redistribution.

Question 14

A data engineering team stores curated datasets in an S3 data lake organized by department prefixes (e.g., s3://corp-lake/finance/, s3://corp-lake/marketing/). The company has thousands of employees synchronized from an external corporate identity provider into IAM Identity Center. Business analysts should be granted read access to only their department's prefix, and access should map directly to their corporate directory group membership. The current approach of editing a single S3 bucket policy has grown unmanageable and is approaching the bucket policy size limit. Which solution provides scalable, identity-based access mapping with the LEAST ongoing policy maintenance?

Accepted Answer

Use S3 Access Grants to map corporate directory groups from IAM Identity Center to specific S3 prefixes, issuing temporary credentials scoped to each grant.

Answer

Create one IAM role per department with an inline policy limiting access to the department prefix, and have analysts assume the appropriate role.

Answer

Continue using the bucket policy but split it across multiple buckets, one per department, each with its own policy referencing directory groups.

Answer

Register the data lake with Lake Formation and grant table-level permissions to each analyst individually based on their department.

Question 15

A data engineering team is building an orchestration layer for a high-volume event pipeline. For each incoming clickstream event, a short-lived workflow validates the payload, invokes a Lambda function for enrichment, and writes the result to a Kinesis Data Stream. The pipeline must handle roughly 50,000 workflow executions per second, each completing in under 3 seconds, and the team wants the lowest possible cost while requiring at-least-once execution semantics. Which AWS Step Functions configuration best meets these requirements?

Accepted Answer

Use an Express workflow with the asynchronous (ASYNC) invocation type to achieve high throughput at low cost with at-least-once execution semantics.

Answer

Use a Standard workflow because it guarantees exactly-once execution and full visual execution history for every clickstream event.

Answer

Use a Standard workflow with a Map state to batch multiple events into a single execution and reduce per-execution charges.

Answer

Use an Express workflow with the synchronous (SYNC) invocation type to guarantee exactly-once processing for each event.