Question

A Generative AI engineer is building a RAG application that ingests thousands of PDF and DOCX files supplied by various business units. Before chunking and embedding, the raw unstructured files must be stored in a governed location that supports Unity Catalog access controls, lineage, and auditing, while allowing the ingestion pipeline to read the files as objects. Which storage approach should the engineer use?

Accepted Answer

Store the raw files in a Unity Catalog managed Volume and grant READ VOLUME privileges to the ingestion pipeline's service principal

Answer

Base64-encode each file and insert it as a string column into a Unity Catalog managed Delta table

Answer

Upload the files to the driver node's local /tmp directory and reference them by absolute path in the notebook

Answer

Store the files in a personal DBFS home directory and share the DBFS path with the ingestion job