Question

A data engineer is preparing a tabular dataset of manufacturing sensor readings for a regression model. Exploratory analysis shows that a small number of temperature readings are physically impossible (e.g., 9999 degrees) due to sensor malfunctions, and these extreme values are skewing the model. The team wants a low-code, visual way to profile the data, identify these anomalies, and apply cleaning transformations before writing the results back to Amazon S3 for training. Which approach best meets these requirements with the least custom code?

Accepted Answer

Use AWS Glue DataBrew to run a data profile job that surfaces outliers and column statistics, then build a DataBrew recipe with transformations to filter or replace the invalid values, and publish the cleaned dataset to S3.

Answer

Write a custom PySpark AWS Glue ETL script that computes z-scores for each numeric column and drops rows exceeding a hardcoded threshold before writing to S3.

Answer

Enable a SageMaker Ground Truth labeling job so human reviewers can flag each impossible temperature reading individually before training.

Answer

Configure an AWS Glue crawler to scan the data and automatically remove statistical outliers based on the inferred schema.