Question

A data engineering team is preparing a proprietary corpus of 2 million customer support transcripts to fine-tune a foundation model on Amazon Bedrock. During exploratory analysis they discover that roughly 30% of the transcripts are near-duplicates (repeated boilerplate greetings, canned responses, and copy-pasted templates), and a subset contains full credit card numbers embedded in free-text fields. The team wants the fine-tuned model to generalize well and wants to minimize compliance risk before the customization job runs. Which data preparation approach best addresses both concerns?

Accepted Answer

Deduplicate the transcripts before training and apply PII detection/redaction (e.g., Amazon Comprehend or Macie-driven workflows) to remove or mask credit card numbers, then use the cleaned corpus for the customization job.

Answer

Run the fine-tuning job on the raw dataset and rely on Bedrock Guardrails at inference time to block credit card numbers in outputs, since deduplication is handled automatically by the training service.

Answer

Keep all near-duplicates because higher record volume always improves fine-tuning quality, and encrypt the S3 bucket with SSE-KMS so the credit card numbers are protected at rest during training.

Answer

Split the dataset so credit card records are used only for validation rather than training, and leave the duplicates in the training split to preserve realistic frequency distributions.