Question

Your team is building a Vertex AI Pipeline using the Kubeflow Pipelines SDK v2. A preprocessing component produces a large transformed dataset (several GB) that must be consumed by a downstream training component. You want the pipeline to pass this data between components in a way that supports lineage tracking, caching, and avoids embedding the data directly in the pipeline's parameter payload. How should you define the data flow between the two components?

Accepted Answer

Define the preprocessing component with an Output[Dataset] artifact and have the training component accept an Input[Dataset] artifact, letting the SDK handle GCS staging and metadata.

Answer

Have the preprocessing component return the dataset as a string parameter output and pass it as an input parameter to the training component.

Answer

Write the dataset to a hardcoded GCS bucket path inside the preprocessing component and read from the same hardcoded path in the training component.

Answer

Store the dataset in a global Python variable in the preprocessing component so the training component can reference it directly at runtime.