Question

A Generative AI Engineer at an insurance company is building a RAG application that answers agent questions using a corpus of 50,000 policy documents stored as Delta tables in Unity Catalog. The documents are updated frequently as policies change, and the team wants the retrieval layer to automatically stay in sync with the source Delta table without manual re-embedding jobs. They also want to reuse Databricks-managed governance and lineage. Which retrieval component design best satisfies these requirements?

Accepted Answer

Create a Databricks Vector Search Delta Sync Index backed by the source Delta table, with embeddings computed by a managed embedding endpoint

Answer

Export the documents to a self-managed FAISS index stored on DBFS and refresh it nightly via a scheduled job

Answer

Create a Databricks Vector Search Direct Vector Access Index and write a custom pipeline that pushes new embeddings whenever documents change

Answer

Store precomputed embeddings in a Delta table column and perform brute-force cosine similarity in a Spark UDF at query time