Question

Your team runs a Vertex AI Pipeline for model retraining. The retraining should be triggered automatically whenever an upstream data-engineering team finishes loading a new curated dataset into BigQuery. That team already publishes a completion message to a Pub/Sub topic at the end of their nightly Airflow DAG. You do not control their DAG and cannot modify it. You need a low-maintenance way to launch the pipeline run in response to their signal. What should you implement?

Accepted Answer

Deploy a Cloud Function subscribed to the existing Pub/Sub topic that calls the Vertex AI Pipelines API to submit a pipeline run using the compiled pipeline template.

Answer

Configure a Vertex AI Pipelines recurring schedule with a cron expression set to run shortly after the upstream DAG typically finishes.

Answer

Add a new task to the upstream Airflow DAG that submits the Vertex AI Pipeline run after the BigQuery load completes.

Answer

Create an Eventarc trigger on the BigQuery dataset that starts the pipeline whenever any table in the dataset is modified.