Question

Your ML team develops Vertex AI Pipelines using the KFP SDK. You want a CI/CD process where the pipeline definition is validated and versioned in a build step, then the identical, immutable artifact is promoted through dev, staging, and production projects without recompiling in each environment. What is the best approach?

Accepted Answer

Compile the pipeline once into a JSON pipeline spec in the CI build, store it as a versioned artifact (e.g., in Artifact Registry or GCS), and submit that same spec file to each environment while passing environment-specific values as runtime pipeline parameters.

Answer

Store the Python pipeline source in Git and have each environment's deployment job re-run the KFP compiler with environment-specific hardcoded constants before submitting the run.

Answer

Use a single shared pipeline run across all three projects and rely on IAM conditions to isolate the environments.

Answer

Bake all environment-specific settings (project IDs, buckets, service accounts) directly into the component definitions and maintain a separate branch per environment.