Question

A data engineering team runs a nightly AWS Glue workflow consisting of a crawler, followed by three sequential ETL jobs (Job A, Job B, Job C). Last night, Job A and Job B succeeded, but Job C failed due to a transient downstream S3 throttling error. The team wants to complete last night's processing as quickly and cost-effectively as possible without reprocessing data that was already successfully transformed. What should they do?

Accepted Answer

Manually run only Job C from the AWS Glue console using the same run parameters, then verify the workflow output before the next scheduled run.

Answer

Restart the entire Glue workflow from the beginning so that the crawler and all three jobs run again to guarantee consistency.

Answer

Delete the S3 output written by Job A and Job B, then rerun the full workflow to avoid duplicate records.

Answer

Disable the workflow schedule and migrate the pipeline to a single monolithic Glue job that performs all three transformations in one run.