Question

A data engineering team orchestrates a nightly pipeline using AWS Step Functions (Standard workflow). One step submits a long-running Apache Spark job to a persistent Amazon EMR cluster via the EMR Steps API. The job can run for up to 3 hours. The team notices that the state machine transitions to the next step immediately after the EMR step is submitted, rather than waiting for the Spark job to actually finish, causing downstream steps to read incomplete data. What is the MOST appropriate way to make the workflow wait for the EMR job to complete before proceeding?

Accepted Answer

Use the .sync (Run a Job) integration pattern for the EMR AddStep task so Step Functions polls the job and waits until it reaches a terminal state before continuing.

Answer

Add a Wait state configured for 3 hours after the EMR step task so the downstream steps run only after the job's maximum runtime elapses.

Answer

Switch the state machine to an Express workflow, which automatically blocks on all service integration tasks until they complete.

Answer

Wrap the EMR submission in a Lambda function and increase the Lambda timeout to 3 hours so it blocks until the Spark job finishes.