Question

Your team runs a Vertex AI Pipeline (built with the Kubeflow Pipelines SDK v2) that trains a model, uploads artifacts to Cloud Storage, and spins up a temporary Dataproc cluster for a feature-engineering step. Occasionally a downstream component fails partway through the run. When this happens, the temporary Dataproc cluster is left running (accruing cost) and no one is notified of the failure. You need to guarantee that a cleanup-and-notify task always executes at the end of the pipeline, whether the run succeeds or fails. What is the best way to implement this?

Accepted Answer

Wrap the pipeline's main steps inside a dsl.ExitHandler and attach a cleanup-and-notify component as the exit task so it runs regardless of upstream success or failure.

Answer

Add the cleanup-and-notify component as the last step in the pipeline DAG with an explicit .after() dependency on the final training component.

Answer

Configure component-level retries with dsl.RetryPolicy on every task so failures are automatically retried and cleanup never becomes necessary.

Answer

Enable component caching on all tasks so that on a re-run the pipeline skips completed steps and reaches the cleanup step faster.