Automating and orchestrating ML pipelines

You maintain a Vertex AI Pipeline built with the Kubeflow Pipelines SDK. It contains several lightweight Python function-based components that each import pandas and scikit-learn. Recently, a pipeline run failed because one component silently pulled a newer scikit-learn version than the one used during development, causing an incompatible model artifact. You want to guarantee that every future run of these components executes with the exact same library versions, while keeping component authoring simple and reproducible. What is the best approach?

Your team builds a Vertex AI Pipeline (KFP v2) that must train the same model architecture on 12 different regional datasets. Each regional training run is independent, and the datasets are known at compile time from a list. Downstream, a single aggregation component must collect the evaluation metrics from all 12 runs to select the best regional model. You want to express this fan-out/fan-in pattern efficiently in the pipeline DSL without hardcoding 12 separate component calls. Which approach should you use?

Your team runs a Vertex AI Pipeline (built with the KFP SDK) that trains a model and produces a metrics artifact for each run. Product managers want a self-service way to compare the accuracy and AUC of models across many pipeline runs over time, without manually inspecting logs. You want to use built-in capabilities with minimal custom code. What should you do?

Your team runs a Vertex AI Pipeline (compiled from KFP) that includes a component calling an external REST API to enrich features. Intermittently this component fails due to transient network timeouts, causing the entire pipeline run to fail hours into execution. You want the pipeline to automatically recover from these transient failures without re-executing already-completed upstream components. What is the most appropriate approach?

Your team is building a Vertex AI Pipeline using the Kubeflow Pipelines SDK v2. A preprocessing component produces a large transformed dataset (several GB) that must be consumed by a downstream training component. You want the pipeline to pass this data between components in a way that supports lineage tracking, caching, and avoids embedding the data directly in the pipeline's parameter payload. How should you define the data flow between the two components?

You are building a Vertex AI Pipeline using the Kubeflow Pipelines (KFP) v2 SDK. The pipeline has three components: a lightweight data validation step, a CPU-heavy feature engineering step, and a GPU-intensive model training step. During test runs, all steps are provisioned with the same default machine type, causing the training step to run out of memory while the validation step wastes expensive resources. You want each component to request appropriate compute resources without changing pipeline logic. What is the recommended approach?

Your team runs a Vertex AI Pipeline (built with the Kubeflow Pipelines SDK v2) that trains a model, uploads artifacts to Cloud Storage, and spins up a temporary Dataproc cluster for a feature-engineering step. Occasionally a downstream component fails partway through the run. When this happens, the temporary Dataproc cluster is left running (accruing cost) and no one is notified of the failure. You need to guarantee that a cleanup-and-notify task always executes at the end of the pipeline, whether the run succeeds or fails. What is the best way to implement this?

Your ML platform team maintains several Vertex AI Pipelines across different product teams. Multiple pipelines all need an identical data-validation step that your team authored as a Kubeflow (KFP) component. Currently each team copies the component's Python source into their own repository, causing drift and inconsistent behavior when the validation logic is updated. You want teams to consume a single, centrally-maintained, versioned copy of the component with minimal duplication. What should you do?

Your ML platform team runs a Vertex AI Pipeline that trains a model, evaluates it, and pushes it to the Vertex AI Model Registry. Compliance requires that no model version reach the production endpoint until a designated reviewer manually inspects the evaluation metrics and explicitly approves the release. You want to keep the training-through-registration steps fully automated while inserting this human approval before deployment, without rebuilding or retraining the model after approval. What is the best way to design this?

Your team runs a Vertex AI Pipeline daily. A preprocessing component reads a static reference table from Cloud Storage and produces a transformed dataset artifact. You enabled execution caching to avoid recomputing unchanged steps. This week, the reference file in Cloud Storage was updated, but the pipeline continued reusing the cached output from the preprocessing component and never reprocessed the new data. What is the most likely cause?

Your ML team develops Vertex AI Pipelines using the KFP SDK. You want a CI/CD process where the pipeline definition is validated and versioned in a build step, then the identical, immutable artifact is promoted through dev, staging, and production projects without recompiling in each environment. What is the best approach?

Your team runs a Vertex AI Pipeline daily. The pipeline has a data-validation step, a heavy preprocessing step (transforming a static reference dataset that rarely changes), and a training step whose hyperparameters you tune frequently. During tuning iterations, the input data and preprocessing code are unchanged, but each full pipeline run re-executes the expensive preprocessing, wasting time and cost. You want to minimize redundant computation across runs while ensuring the step re-runs automatically if its inputs or code change. What is the most appropriate approach?

Your ML platform team maintains a shared Vertex AI Pipelines training pipeline. The training component runs in project 'ml-training-prod' but must read TFRecord data from a GCS bucket owned by the data engineering team in a separate project, 'de-datalake-prod', and write model artifacts back to a bucket in 'ml-training-prod'. During pipeline execution, the training step fails with a 403 error when accessing the source data. The pipeline runs under a custom service account 'pipeline-runner@ml-training-prod.iam.gserviceaccount.com'. What is the correct way to resolve this while following least-privilege principles?

Your team runs a Vertex AI Pipeline that trains a fraud-detection model nightly. Business stakeholders require that a newly trained model is only deployed to the production endpoint if its evaluation AUC exceeds the currently deployed model's AUC by at least 0.02; otherwise the pipeline should skip deployment and leave the existing model serving. You want to implement this entirely within the pipeline definition using the KFP v2 SDK. Which approach best meets this requirement?

Your team runs a fraud-detection model on a Vertex AI endpoint with model monitoring enabled for feature drift. You want production drift alerts to automatically kick off your existing Vertex AI Pipelines retraining workflow, without a human manually starting the run. You also want to avoid standing up and maintaining any always-on infrastructure. Which approach best achieves this automated, event-driven retraining trigger?

You have a Vertex AI Pipeline built with the Kubeflow Pipelines (KFP v2) SDK. The pipeline trains a model, evaluates it, and conditionally uploads it to the Model Registry only if accuracy exceeds a threshold. Occasionally the training component fails partway through, leaving temporary artifacts in a staging Cloud Storage bucket. The team wants two behaviors: (1) skip the upload step when accuracy is below the threshold, and (2) always run a cleanup step that deletes the staging artifacts, regardless of whether any upstream component succeeds or fails. Which combination of KFP constructs correctly implements both requirements?

Your team maintains a Vertex AI Pipeline (KFP v2) that trains and deploys a model. For a new pipeline, you need to reuse a model artifact that was produced outside of any pipeline run — it currently exists only as a set of files in a Cloud Storage bucket and is not tracked in Vertex ML Metadata. You want the downstream evaluation and deployment components to consume it as a proper Model artifact with lineage, without writing a custom component that re-uploads the files. What is the most appropriate approach?

Your team runs a Vertex AI Pipeline that retrains a model. Business analysts periodically upload a new curated training CSV to a specific Cloud Storage bucket, and the pipeline must run automatically each time a new file lands in that bucket—without any manual intervention or a fixed schedule. Which approach best implements this event-driven triggering?

Your ML team has built a Vertex AI Pipeline (compiled to a KFP pipeline spec) that trains and evaluates a model. Data scientists frequently update the training component code in a GitHub repository. You need a CI/CD process that automatically recompiles the pipeline and triggers a new pipeline run whenever code is merged into the main branch, while keeping the pipeline definition versioned and auditable. Which approach best meets these requirements?

Your ML platform team maintains a legacy data-validation tool that is already packaged as a Docker image with a well-defined command-line interface (it reads an input GCS path flag and writes a validation report to an output GCS path flag). You want to reuse this exact container inside a new Vertex AI Pipeline authored with the KFP SDK v2, without rewriting the logic in Python or rebuilding the image. What is the most appropriate way to incorporate this tool as a pipeline step?