Question

Your team builds a Vertex AI Pipeline (KFP v2) that must train the same model architecture on 12 different regional datasets. Each regional training run is independent, and the datasets are known at compile time from a list. Downstream, a single aggregation component must collect the evaluation metrics from all 12 runs to select the best regional model. You want to express this fan-out/fan-in pattern efficiently in the pipeline DSL without hardcoding 12 separate component calls. Which approach should you use?

Accepted Answer

Use a dsl.ParallelFor loop over the list of regions to fan out the training component, then use dsl.Collected to gather the outputs into the aggregation component.

Answer

Compile 12 separate pipelines and trigger them concurrently, then run a manual script afterward to compare the metrics files in Cloud Storage.

Answer

Write a single training component that internally loops over all 12 regions sequentially, then pass its combined output to the aggregation component.

Answer

Use a dsl.Condition block for each region and rely on component caching to reuse results across the branches.