Question

A data engineering team runs a nightly Glue workflow with three sequential jobs. The first job computes the exact S3 partition path that was processed (based on the latest ingested date) and the downstream jobs must read from that same computed path. Currently the team hardcodes the date, which causes failures when late data shifts the target partition. They want a native way to pass the computed value from the first job to the subsequent jobs within the same workflow run without using an external store. What should they do?

Accepted Answer

Have the first job set workflow run properties using the Glue API, and have downstream jobs retrieve those run properties at the start of each job.

Answer

Write the computed path to a fixed key in Amazon S3 and have each downstream job read that object before processing.

Answer

Store the computed path in an SSM Parameter Store parameter and reference it as a job argument in the downstream jobs.

Answer

Pass the computed path by enabling Glue job bookmarks so the state is automatically shared across the jobs in the workflow.