Question

A machine learning platform team maintains a single SageMaker Pipeline definition used by several data science teams. Each team needs to run the same pipeline but with different training instance types, input dataset S3 URIs, and hyperparameter values at execution time, without editing or redefining the pipeline. What is the most appropriate way to design the pipeline to support this requirement?

Accepted Answer

Define pipeline Parameters (e.g., ParameterString, ParameterInteger) for the variable values and pass different argument values when each team calls the StartPipelineExecution API.

Answer

Create a separate hardcoded pipeline definition for each team and store each as its own pipeline in SageMaker.

Answer

Use SageMaker Pipeline step caching so that each team's execution reuses the previous team's cached step outputs with new values.

Answer

Store the per-team values in environment variables on the SageMaker Studio domain and read them inside each step's container at runtime.