Question

Your team runs a Vertex AI Pipeline (compiled from KFP) that includes a component calling an external REST API to enrich features. Intermittently this component fails due to transient network timeouts, causing the entire pipeline run to fail hours into execution. You want the pipeline to automatically recover from these transient failures without re-executing already-completed upstream components. What is the most appropriate approach?

Accepted Answer

Configure a retry policy on the enrichment component using the task's set_retry() method with a backoff, and ensure upstream component outputs are cached so they are not recomputed.

Answer

Wrap the entire pipeline in an ExitHandler that re-submits the whole pipeline run whenever any component fails.

Answer

Increase the machine size for the enrichment component so the API calls complete faster and avoid timeouts.

Answer

Disable caching globally so that on restart every component runs again from a clean state, guaranteeing consistency.