Question

A healthcare analytics team ingests raw CSV files into an S3 bucket that will be used to build a training dataset for a readmission-prediction model. Before the data can be used, compliance requires that any personally identifiable information (PII) such as names, phone numbers, and Social Security numbers be automatically discovered and redacted at scale during the ETL process. The team already uses AWS Glue for their transformation jobs. Which approach best meets this requirement with the least custom development?

Accepted Answer

Use the AWS Glue PII detection transform (Detect Sensitive Data) within the Glue ETL job to identify and mask or redact PII columns and values during processing.

Answer

Write a custom PySpark UDF in the Glue job that applies regular expressions to every column to find and remove PII patterns.

Answer

Enable S3 default encryption with SSE-KMS on the bucket so that PII data is protected before the Glue job reads it.

Answer

Run an Amazon Macie classification job on the bucket and rely on Macie to redact the sensitive values in place before Glue reads the files.