Question

An ML engineering team runs a SageMaker Pipeline daily that includes a data-processing step, a training step, and an evaluation step. The raw input data only changes about once a week, but the training code and hyperparameters change frequently as data scientists experiment. Each full pipeline run is expensive because the processing step reprocesses the same unchanged data every day. The team wants to reduce cost and run time by avoiding reprocessing when neither the processing inputs nor the step configuration have changed, while still guaranteeing that any change to code or parameters triggers a fresh run. What is the most appropriate approach?

Accepted Answer

Enable step caching on the pipeline steps by configuring a CacheConfig with an expiration period, so SageMaker reuses prior step results when the inputs and step arguments are unchanged.

Answer

Split the pipeline into two separate pipelines and manually trigger the processing pipeline only on days when the data changes.

Answer

Add a Lambda step at the start that compares S3 object timestamps and stops the pipeline execution if the data has not changed since the last run.

Answer

Convert the processing step to a SageMaker Processing job with Managed Spot Instances to lower the cost of reprocessing each day.