Hard MLA-C01 practice questions

A machine learning engineer is training a deep neural network image classifier on SageMaker. Using a very large batch size to maximize GPU throughput, they observe that training loss converges quickly but validation accuracy is noticeably lower than a prior run that used a smaller batch size. Compute budget allows either approach. Which adjustment is MOST likely to improve generalization while keeping training stable?

A financial services team trains a gradient boosting classifier to predict loan default. Downstream systems consume the model's predicted probabilities directly as risk scores to set interest rates, so the numeric probabilities must reflect true likelihoods (e.g., a predicted 0.30 should default about 30% of the time). During evaluation, the model shows strong ranking performance with AUC of 0.88, but the business team reports the predicted probabilities appear systematically too high in the mid-range. Which evaluation step best addresses whether the probabilities can be trusted as risk scores?

A healthcare company runs a SageMaker real-time endpoint that processes protected health information (PHI). A compliance auditor requires that all model prediction logs and captured inference data be encrypted at rest with a customer-managed key, and that any attempt to disable logging or delete the key be centrally recorded for audit. The ML engineer has already enabled SageMaker Data Capture writing to an S3 bucket. Which combination of actions BEST satisfies these requirements with the least custom code?

A data science team trains two versions of a binary classifier on the same small validation set of 400 records. Version A reports 88.0% accuracy and Version B reports 88.5% accuracy across a single evaluation run. The lead engineer must decide whether Version B is genuinely better before promoting it in the SageMaker Model Registry. What is the most statistically sound next step?

A financial services company trains models in the us-east-1 account and copies the resulting model artifacts to an S3 bucket in a disaster-recovery account located in us-west-2. Both buckets use SSE-KMS with customer managed keys. When a SageMaker training job in us-east-1 tries to write artifacts that are then replicated, and when the DR account's SageMaker attempts to deploy the model, the team encounters AccessDenied errors on decryption. What is the MOST likely root cause and correct remediation?

A machine learning team has trained a large transformer-based text classifier that achieves 94% accuracy on the validation set. When deployed to a real-time SageMaker endpoint, the model exceeds the required p99 latency budget and cannot be served on the available CPU instance family within cost constraints. The team wants to significantly reduce inference latency while retaining as much of the original accuracy as possible, and they are willing to run an additional training phase. Which modeling approach best meets these goals?

A data engineering team ingests semi-structured JSON event logs into an S3 bucket daily. The schema of these events evolves frequently — new optional fields appear and some fields change from single values to arrays. The team uses an AWS Glue crawler to populate a Data Catalog table that a downstream Glue ETL job reads to build ML training features. Recently, the crawler has been creating multiple separate tables and marking fields as incompatible, breaking the ETL job. What is the most appropriate way to handle this evolving schema so the ETL job continues to run reliably?

A data engineering team is building a training dataset for a fraud detection model. Transaction records arrive continuously in an Amazon Kinesis Data Stream, while customer profile data is stored as a nightly export in Amazon S3. The team needs to enrich each transaction with the corresponding customer attributes and land the combined records in S3 for downstream SageMaker training. They want a managed, serverless approach that minimizes operational overhead. Which solution best meets these requirements?

A data engineering team ingests clickstream events into an Amazon Kinesis Data Stream that feeds a downstream ML feature pipeline. Producers are written by multiple teams and occasionally add or rename fields, which has repeatedly broken the Apache Flink consumer that parses the records. The ML team needs to enforce that only backward-compatible schema changes are accepted before records enter the stream, while keeping serialization compact for high-throughput ingestion. What should they implement?

A data scientist is preparing a churn prediction dataset with AWS Glue. One categorical feature, 'sales_region', has a strong relationship with the target label, and the team wants to replace each region with the mean churn rate for that region (target encoding). During initial experiments, cross-validation accuracy is excellent but the model performs poorly on held-out production data. What is the MOST likely cause and the correct way to fix it?

A data science team in Account A trains models and stores the resulting model artifacts in an S3 bucket in a shared Account B, which is encrypted with a customer managed KMS key owned by Account B. A SageMaker training job's execution role in Account A repeatedly fails to write artifacts, returning an AccessDenied error even though the S3 bucket policy grants the role s3:PutObject. What is the MOST likely cause and correct remediation?

A retail company runs a SageMaker real-time endpoint for a credit approval model. Over recent weeks, the overall accuracy on ground-truth data remains stable, but the business team suspects the model is relying on different input features than during training, which could signal changing customer behavior. The ML engineer wants an automated way to detect when the relative importance of input features shifts significantly compared to the training baseline, and to alert the team without manually recomputing feature importance. Which approach best meets this requirement?

A team hosts 300 customer-specific fraud models behind a single SageMaker multi-model endpoint (MME) to save cost. Most models are invoked a few times per hour, but three high-traffic customers generate the majority of requests. Operators notice that these three heavily-used models occasionally see elevated p99 latency spikes that coincide with invocations of rarely-used models. What is the most likely cause and the best remedy?

A retail company runs a real-time SageMaker endpoint for product recommendations. Business analysts report that recommendation click-through rates have gradually declined over the past three months, though the endpoint shows no errors and stable latency. The ML team confirms input feature distributions have shifted as customer behavior evolved seasonally. The team wants an automated, low-maintenance approach to keep the deployed model accurate over time without manually inspecting metrics each week. Which approach best addresses this requirement?

A healthcare company runs a real-time SageMaker endpoint that serves patient risk scores. Compliance requires that all inference request and response payloads captured to Amazon S3 be encrypted with a customer-managed KMS key, and that data captured from this specific endpoint can be cryptographically distinguished from data captured by other endpoints in audit logs. The security team also wants CloudTrail records to show which endpoint triggered each KMS decrypt operation without decrypting the data itself. What is the MOST appropriate way to meet these requirements?

A healthcare company deploys a SageMaker real-time endpoint that processes protected health information. Compliance requires that the inference container must not be able to make any outbound network calls (including to the internet or third-party APIs), while the endpoint itself must still be reachable by the company's internal applications running in a private VPC. Which configuration meets both requirements?

A data science team stores transaction features in a SageMaker Feature Store offline store, continuously ingesting new feature records with event timestamps. To train a fraud model, they must build a training dataset that reflects each labeled event's feature values as they existed at the moment of that event, avoiding leakage of future feature updates. Which approach correctly produces this training dataset?

A healthcare company runs SageMaker training jobs that process protected health information (PHI). Security auditors require that all inter-node communication during distributed training be encrypted, that model artifacts and training data at rest be encrypted with a customer managed key, and that no training traffic traverse the public internet. The ML engineer has already configured a private VPC with no internet gateway. Which additional set of configurations satisfies ALL of the auditors' requirements?

A healthcare company runs SageMaker real-time endpoints inside a VPC with no internet access. They want to enable SageMaker Model Monitor to run scheduled data quality monitoring jobs against captured inference data stored in S3. During testing, the monitoring jobs remain stuck in a pending state and eventually fail to reach S3 and the SageMaker service. What is the MOST likely cause and correct remediation?

A healthcare company runs SageMaker training and processing jobs from notebook instances that process protected health information (PHI). Security requires that these workloads can reach only approved S3 buckets and SageMaker APIs, must never traverse the public internet, and must be blocked from writing PHI to any S3 bucket outside the company's account. Which combination of controls best enforces these requirements?

A team has a SageMaker Pipeline in a development account that trains a model, evaluates it, and registers approved versions to the SageMaker Model Registry. The company mandates that all production endpoints run in a separate production account, and that the deployment to production must be triggered automatically whenever a model version is approved in the registry. The team wants to minimize custom code and avoid manual copying of model artifacts between accounts. Which approach best meets these requirements?

A data scientist is running a SageMaker Automatic Model Tuning job for an XGBoost model. To finish the tuning as fast as possible, they want to run many training jobs concurrently (high parallelism). They notice that when they set the strategy to a method that learns from prior results, increasing parallelism does not improve the quality of the hyperparameters chosen. Which explanation and adjustment is most accurate?

A data scientist builds an XGBoost model to predict customer churn. During training, cross-validation reports 96% accuracy, but when the model is deployed and evaluated on genuinely new customers, accuracy drops to 71%. Investigation reveals that the training pipeline computed feature scaling statistics (mean and standard deviation) using the entire dataset before splitting into folds, and one input feature was derived from a value only known after churn occurs. What is the MOST likely root cause of the inflated cross-validation score?