Microsoft Machine Learning Operations Engineer Associate

Question 1

You manage several registered models in an Azure Machine Learning workspace. A fraud-detection model has 12 versions, and versions 1 through 8 are obsolete and should no longer appear in default listings or be selectable for new deployments. However, compliance requires that these older versions remain retrievable for audit purposes and are not permanently deleted. What should you do?

Accepted Answer

Archive versions 1 through 8 of the registered model so they are hidden from default listings but remain stored.

Answer

Delete versions 1 through 8 of the registered model using the Azure ML CLI.

Answer

Set the stage property of versions 1 through 8 to 'Production' so they are grouped separately.

Answer

Export versions 1 through 8 to a blob container and unregister them from the workspace.

Question 2

Your team's security policy prohibits storing storage account access keys or SAS tokens anywhere in the Azure Machine Learning workspace. Data scientists must still be able to read training data from an existing Azure Data Lake Storage Gen2 container through a registered datastore. Users each have their own Microsoft Entra ID identity. What should you configure when registering the datastore so that data access is authorized using the user's own identity rather than stored credentials?

Accepted Answer

Register a credential-less datastore and grant each user's Entra ID the Storage Blob Data Reader role on the storage account

Answer

Register the datastore with an account key stored in Azure Key Vault and reference it via a secret

Answer

Register the datastore using a service principal with a client secret shared by the team

Answer

Register the datastore with a SAS token scoped to the container with read-only permissions

Question 3

Your team provisions a new Azure Machine Learning workspace inside a resource group named 'rg-ml-prod' that also contains the associated storage account, key vault, and container registry. A group of ML engineers needs Contributor-level permissions on the workspace and every dependent resource, and you want to minimize the number of role assignments you have to manage as more supporting resources are added to the group over time. Which approach should you take?

Accepted Answer

Assign the Contributor role to the ML engineers group at the 'rg-ml-prod' resource group scope.

Answer

Assign the Contributor role to each ML engineer individually on the workspace, storage account, key vault, and container registry.

Answer

Assign the AzureML Data Scientist role to the ML engineers group at the workspace scope only.

Answer

Assign the Contributor role to the ML engineers group at the subscription scope.

Question 4

Your team runs Azure Machine Learning training jobs that read from an Azure Blob Storage account. Security policy prohibits storing storage account keys or SAS tokens in the workspace. You register the storage as a datastore and need the compute cluster to authenticate to the blob container without embedded credentials. What should you do?

Accepted Answer

Enable a system-assigned managed identity on the compute cluster and grant it the Storage Blob Data Reader role on the storage account, then register a credential-less datastore using identity-based access.

Answer

Create the datastore using the storage account key and store the key in Azure Key Vault referenced by the workspace.

Answer

Generate a short-lived SAS token for the container and configure the datastore to refresh it automatically through a scheduled pipeline.

Answer

Set the workspace to public network access and use the storage account's connection string in the job's environment variables.

Question 5

Your team stores large training datasets in an Azure Data Lake Storage Gen2 account that has hierarchical namespace enabled. You need to register this storage as a datastore in your Azure Machine Learning workspace using the Azure CLI (v2) so data scientists can reference paths in their jobs. Which datastore type should you specify in the YAML definition?

Accepted Answer

AzureDataLakeGen2

Answer

AzureBlob

Answer

AzureFile

Answer

AzureDataLakeGen1

Question 6

Your team has an existing Azure Blob Storage account that already contains curated training data in a container named 'training-data'. You need to make this data accessible from your Azure Machine Learning workspace so data scientists can reference it by name in jobs, without moving or copying the data. Credentials must be stored securely in the workspace and reused across jobs. What should you do?

Accepted Answer

Create a new datastore in the workspace that points to the existing blob container, using an account key or SAS token for authentication

Answer

Upload the container contents to the workspace's default 'workspaceblobstore' datastore using the CLI

Answer

Register the container as a data asset of type uri_folder without configuring a datastore

Answer

Mount the storage account directly to each compute cluster node using a startup script that supplies the account key

Question 7

You are training a classification model in Azure Machine Learning using a command job that logs a validation metric called 'val_auc' with MLflow. You want to automate hyperparameter tuning to maximize this metric across combinations of learning rate and batch size, sampling values randomly and stopping poorly performing trials early. Which approach should you use to configure the tuning job?

Accepted Answer

Convert the command job into a sweep job, define the search space with random sampling, set the primary_metric to 'val_auc' with goal 'maximize', and attach a bandit early termination policy.

Answer

Submit multiple independent command jobs with different fixed hyperparameter values and manually compare their logged 'val_auc' values in the studio.

Answer

Create a sweep job with grid sampling, set the primary_metric to 'val_auc' with goal 'minimize', and omit an early termination policy so all trials complete.

Answer

Register the command job as a component in a pipeline and configure parallel steps that each use a different learning rate hard-coded in the script.

Question 8

Your team maintains Azure Machine Learning infrastructure using Bicep templates stored in a GitHub repository. You need to configure a GitHub Actions workflow that deploys the workspace and its resources to Azure automatically on every merge to the main branch. Company security policy prohibits storing long-lived service principal secrets in GitHub. What should you configure to allow the workflow to authenticate to Azure?

Accepted Answer

Register an app in Microsoft Entra ID, configure a federated identity credential for the GitHub repository, and use workload identity federation (OIDC) with the azure/login action

Answer

Create a service principal, generate a client secret, and store it as a GitHub Actions repository secret referenced by the azure/login action

Answer

Store the Azure account username and password as encrypted GitHub secrets and pass them to the Azure CLI in the workflow

Answer

Enable a system-assigned managed identity on the GitHub Actions runner and grant it Contributor on the subscription

Question 9

You maintain a fraud-detection model deployed to a managed online endpoint in Azure Machine Learning. A model monitor already computes prediction drift and feature-attribution drift on a recurring schedule. The business requires that when the monitored metric exceeds a defined threshold, a retraining pipeline is automatically kicked off without a data scientist manually reviewing the results first. Which approach best implements this automated retraining trigger?

Accepted Answer

Configure an Azure Event Grid subscription on the workspace that reacts to the model monitor's threshold-exceeded event and triggers the training pipeline via an Azure Logic App or Function.

Answer

Increase the model monitor's scheduling frequency to hourly so that retraining occurs automatically on every monitoring run.

Answer

Enable the responsible AI dashboard on the endpoint so that it retrains the model whenever fairness metrics degrade.

Answer

Set the deployment's traffic percentage to 0 when drift is detected so the endpoint automatically falls back to the previous model version.

Question 10

You are configuring an automated machine learning classification job in Azure Machine Learning to predict fraudulent transactions. Only about 2% of the transactions in your training dataset are labeled as fraud. Your business goal is to correctly identify as many fraudulent transactions as possible while keeping false positives manageable. Which primary metric should you configure for the AutoML job to best guide model selection?

Accepted Answer

average_precision_score_weighted

Answer

accuracy

Answer

AUC_weighted

Answer

norm_macro_recall

Question 11

Your team has an existing Python training script that trains a scikit-learn regression model and logs metrics with MLflow. You need to run this script on an Azure Machine Learning compute cluster as a reproducible job, passing a registered data asset and a learning-rate argument, while ensuring the run appears in a specific experiment for later comparison. Which Azure Machine Learning job type should you configure?

Accepted Answer

A command job that specifies the script, compute target, environment, inputs, and experiment_name

Answer

An AutoML job configured for the regression task with the data asset as training data

Answer

A sweep job referencing the script with a search space defined over the learning-rate argument

Answer

A pipeline job containing a single parallel component that maps over the data asset

Question 12

You are running a hyperparameter tuning sweep job in Azure Machine Learning to optimize a deep learning model. The sweep uses random sampling over a large search space, and each trial is expensive to run on GPU compute. You want to automatically stop poorly performing trials early to reduce cost, while still allowing promising trials to complete. Which configuration should you add to the sweep job?

Accepted Answer

Configure a bandit early termination policy with a slack factor and evaluation interval

Answer

Set the sampling algorithm to grid sampling instead of random sampling

Answer

Increase max_total_trials so more trials run in parallel

Answer

Set the primary metric goal to minimize and lower the timeout on the compute cluster

Question 13

You are an ML engineer at a logistics company. A data scientist has trained a model and logged it with MLflow, and you now need to deploy it to a managed online endpoint for real-time scoring. The endpoint must serve two model versions simultaneously so you can compare responses before committing. You want to minimize the amount of custom code required for the deployment. What should you do?

Accepted Answer

Deploy each model version as a separate deployment under the same managed online endpoint, relying on MLflow's no-code deployment since the model has a logged signature.

Answer

Create two separate managed online endpoints, one per model version, and configure a load balancer to route between them.

Answer

Author a custom scoring script and environment for each version and deploy both as batch endpoints under a shared batch deployment.

Answer

Register both models in a workspace registry and invoke them directly from the registry without creating any endpoint.

Question 14

Your team has a registered MLflow model that must score a nightly file drop of roughly 5 million records stored in a Blob datastore. Latency is not important, but the job must finish within a maintenance window and produce a single consolidated output file. You need to create the most cost-effective managed inference solution in Azure Machine Learning. What should you do?

Accepted Answer

Create a batch endpoint with a batch deployment, set mini_batch_size and instance_count to parallelize scoring, and configure output_action to append_row.

Answer

Deploy the model to a managed online endpoint and send the records in a loop of REST requests.

Answer

Deploy the model to a managed online endpoint with autoscaling enabled and increase the max instance count during the maintenance window.

Answer

Create a batch endpoint but set instance_count to 1 and output_action to summary_only to reduce compute usage.

Question 15

You deploy an MLflow model to an Azure Machine Learning batch endpoint to score a folder of 500,000 small CSV files. The initial scoring job runs very slowly, and monitoring shows the compute cluster nodes are underutilized with frequent overhead between individual file reads. You need to improve throughput without changing the scoring logic. Which batch deployment setting should you adjust?

Accepted Answer

Increase the mini_batch_size so more files are grouped and passed to each run() invocation.

Answer

Set the output_action to append_row so results are written to a single file.

Answer

Reduce the instance_count on the compute cluster to consolidate work on fewer nodes.

Answer

Lower the max_concurrency_per_instance to 1 to avoid resource contention.

Question 16

You are deploying a scikit-learn model to an Azure Machine Learning batch endpoint using a custom scoring script. During execution, the endpoint must process thousands of CSV files, and each mini-batch should return one row of predictions per input row. Which function in your scoring script is responsible for processing each mini-batch of files and returning the results?

Accepted Answer

The run(mini_batch) function, which receives a list of file paths for each mini-batch and returns the scored results

Answer

The init() function, which loads the model once per worker and returns predictions for each file

Answer

The main() function, which is invoked automatically by the batch endpoint for every input file

Answer

The score(input_data) function, which is the required entry point for real-time managed online endpoints

Question 17

Your team trained a demand-forecasting model that must score a 40 GB CSV file dropped into Azure Blob Storage every night. The job has no latency requirement and results are written back to a data lake for downstream reporting. You want to minimize compute cost by only provisioning resources when the scoring job runs. Which Azure Machine Learning deployment approach best fits this workload?

Accepted Answer

Deploy a batch endpoint backed by a compute cluster that scales from zero, and invoke it on a nightly schedule.

Answer

Deploy a managed online endpoint with a single instance and autoscale rules based on request volume.

Answer

Deploy a managed online endpoint with two deployments and split traffic 50/50 for redundancy.

Answer

Deploy the model to a Kubernetes online endpoint kept always running to guarantee immediate scoring.

Question 18

Your team must provision an Azure Machine Learning workspace using Bicep as part of a GitHub Actions pipeline. Company security policy requires that the workspace and its associated resources (storage account, key vault) not be reachable from the public internet, and that all traffic flow through the corporate virtual network. Which combination should you configure in the Bicep template to meet this requirement?

Accepted Answer

Set the workspace 'publicNetworkAccess' property to 'Disabled' and deploy a private endpoint for the workspace and its dependent resources into the corporate VNet subnet.

Answer

Set the workspace SKU to 'Basic' and assign a system-assigned managed identity to the workspace.

Answer

Enable a service endpoint for Microsoft.MachineLearningServices on the subnet and leave 'publicNetworkAccess' at its default value.

Answer

Configure IP firewall rules on the storage account to allow only the GitHub Actions runner IP ranges.

Question 19

Your team built a generative AI summarization assistant in Azure AI Foundry. Reviewers complain that some generated summaries are grammatically awkward and read poorly, while others jump between ideas without logical flow. You want to set up an automated evaluation run that specifically quantifies (1) how grammatically correct and natural the language is, and (2) how well sentences connect and follow a logical structure. Which two built-in AI quality metrics should you configure?

Accepted Answer

Fluency for grammatical correctness and Coherence for logical flow between sentences

Answer

Groundedness for grammatical correctness and Relevance for logical flow between sentences

Answer

Relevance for grammatical correctness and Fluency for logical flow between sentences

Answer

Coherence for grammatical correctness and Groundedness for logical flow between sentences

Question 20

You are training a classification model in Azure Machine Learning using several experiment jobs, each logging accuracy, F1 score, and AUC through MLflow autologging. Your team lead asks you to identify which job produced the best-performing model and to visually compare the logged metrics side by side across all runs within the same experiment. Which approach should you use in the Azure Machine Learning studio?

Accepted Answer

Open the experiment, select multiple jobs, and use the Compare view to display their logged metrics and parameters together.

Answer

Register each job's model to the model registry and compare the version numbers in the Models page.

Answer

Download the outputs of each job individually and manually inspect the logged metric files.

Answer

Create a separate experiment for each job so that the studio automatically ranks them by accuracy.

Question 21

Your team ran 15 separate training jobs in Azure Machine Learning, each logged to the same MLflow experiment with a logged metric named 'val_f1_score'. A stakeholder asks you to identify which single job produced the highest validation F1 score so it can be promoted for registration. You want the fastest approach that requires no re-execution of the jobs. What should you do?

Accepted Answer

Open the experiment in Azure Machine Learning studio, add the jobs to a comparison view, and sort or chart the runs by the logged 'val_f1_score' metric.

Answer

Re-run all 15 jobs as a single hyperparameter sweep job so the best model is automatically selected by the primary metric.

Answer

Download the output artifacts of every job and manually inspect each scoring log file to find the highest value.

Answer

Deploy each of the 15 models to a separate online endpoint and send test traffic to measure which returns the best F1.

Question 22

You maintain an Azure Machine Learning training pipeline that trains a model and then registers it. The business requires that the model is only registered when its validation accuracy exceeds 0.85; otherwise the pipeline should skip registration and finish successfully without failing. You want to implement this within the pipeline itself using components. What is the most appropriate approach?

Accepted Answer

Add a conditional control-flow node that evaluates the training component's accuracy output and only executes the registration component when the threshold is met.

Answer

Configure a data drift monitor on the training output so that registration is triggered only when drift stays below 0.85.

Answer

Set the registration component's retry policy to fail after the accuracy metric drops, causing Azure ML to skip that step.

Answer

Deploy the model to a batch endpoint first and let the endpoint reject models below 0.85 accuracy before registration.

Question 23

Your team runs intermittent training jobs on an Azure Machine Learning workspace. Costs are high because a dedicated compute cluster stays allocated even when no jobs are queued. You must minimize cost while ensuring capacity is available automatically when jobs are submitted. Which compute cluster configuration should you apply?

Accepted Answer

Set the minimum node count to 0 and configure a suitable maximum node count with an idle-seconds-before-scaledown value.

Answer

Provision a compute instance instead of a compute cluster and leave it running for the team.

Answer

Set both the minimum and maximum node count to the same fixed value so capacity is always reserved.

Answer

Attach a self-managed Azure Kubernetes Service cluster and run all training jobs on it continuously.

Question 24

Your team runs training jobs in an Azure Machine Learning workspace using the Python SDK v2 from a compute instance. Compliance requires that every submitted job be traceable to the exact commit of the training code. The code lives in an Azure DevOps Git repository that is already cloned onto the compute instance. What is the simplest way to ensure the Git commit hash and repository details are automatically captured with each job?

Accepted Answer

Submit the job from the directory that contains the cloned Git repository so that the SDK detects the .git metadata and logs the repository URL and commit hash automatically.

Answer

Manually pass the commit hash as a job tag by calling git rev-parse in a shell before every submission.

Answer

Enable a diagnostic setting on the workspace to stream Git metadata to a Log Analytics workspace.

Answer

Register the training code as a versioned data asset before each run and reference the asset version in the job.

Question 25

Your data science team needs a reproducible training environment in Azure Machine Learning that includes scikit-learn plus two internal Python packages hosted on a private Azure Artifacts feed. The environment must be versioned as a reusable asset so that all training jobs reference the exact same dependency set. Which approach should you use to create and register this environment?

Accepted Answer

Define a custom environment from a base Docker image plus a conda specification that installs the internal packages from the private feed, then register it as a versioned environment asset in the workspace.

Answer

Select an Azure ML curated environment for scikit-learn and reference it directly in each job, since curated environments are automatically versioned.

Answer

Install the internal packages manually on the compute cluster nodes via a startup script so every job inherits them without defining an environment.

Answer

Create a data asset that packages the wheel files for the internal packages and mount it into each job at runtime.

Question 26

Your team runs an Azure AI Foundry customer-support agent that must always include a legal disclaimer phrase in responses that mention refund policies. The built-in AI-assisted quality metrics (groundedness, relevance, coherence, fluency) do not capture this business rule. You need the automated batch evaluation workflow to flag every response missing the required phrase, in addition to the built-in metrics. What should you do?

Accepted Answer

Author a custom code-based evaluator that programmatically checks each output for the required phrase and register it alongside the built-in evaluators in the evaluation run.

Answer

Increase the groundedness metric threshold so responses missing the disclaimer score lower and are automatically flagged.

Answer

Enable the risk and safety evaluations for harmful content, which will detect the missing disclaimer as a compliance violation.

Answer

Add the disclaimer requirement as a system prompt instruction and rely on the coherence metric to confirm it was followed.

Question 27

Your team runs automated evaluation on a customer-support agent in Azure AI Foundry. The built-in groundedness, relevance, and coherence evaluators are already configured, but the business requires a domain-specific check: each response must include a valid support ticket ID in a fixed format, and the evaluation must report the percentage of responses that pass this check across the test dataset. You need to add this measurement to the existing automated evaluation workflow with the least ongoing maintenance. What should you do?

Accepted Answer

Author a custom evaluator that implements the ticket-ID format check and returns a pass/fail score, then include it alongside the built-in evaluators in the evaluation run.

Answer

Switch the relevance evaluator to a higher threshold so that responses without a ticket ID are scored as failures.

Answer

Add a content safety risk evaluator to detect responses that omit the ticket ID.

Answer

Manually review the evaluation output CSV after each run and tally the responses that contain a valid ticket ID.

Question 28

Your team wants to standardize a data preprocessing step so it can be reused across multiple Azure Machine Learning pipelines built by different data scientists. You need to define this step as a versioned, shareable asset in the workspace using the Azure ML CLI (v2). Which asset type should you create and register for the preprocessing step?

Accepted Answer

A command component defined in a YAML spec file, registered with 'az ml component create'

Answer

An environment defined in a YAML spec file, registered with 'az ml environment create'

Answer

A data asset defined as an MLTable, registered with 'az ml data create'

Answer

A managed online endpoint deployed with 'az ml online-endpoint create'

Question 29

You operate a fraud-detection model on an Azure Machine Learning managed online endpoint. Confirmed fraud labels arrive weeks after scoring, so ground-truth accuracy cannot be evaluated in near real time. Leadership wants an early warning when the model's behavior shifts, without waiting for labels. Which monitoring signal should you configure in the Azure ML model monitor?

Accepted Answer

A prediction drift signal that compares the distribution of the model's output predictions against a baseline reference dataset

Answer

A model performance signal computing precision and recall against the incoming scoring requests

Answer

A data quality signal that checks null rates and type mismatches on input features

Answer

A feature attribution drift signal comparing SHAP importance values on the labeled test set

Question 30

You are training a large deep learning model in Azure Machine Learning using a PyTorch script that already implements DistributedDataParallel. You want to run the job across 4 nodes with 4 GPUs each on a GPU compute cluster, and you need Azure Machine Learning to launch one process per GPU and set the correct environment variables (RANK, WORLD_SIZE, MASTER_ADDR) for the distributed process group. How should you configure the command job?

Accepted Answer

Add a PyTorch distribution configuration with process_count_per_instance set to 4 and set the job's instance_count (resources) to 4.

Answer

Set instance_count to 16 with no distribution block, relying on the cluster to auto-detect the GPUs and spawn processes.

Answer

Add an MPI distribution configuration with process_count_per_instance set to 16 and set instance_count to 1.

Answer

Wrap the script in a ParallelRunStep with mini_batch_size tuned to the number of GPUs and set node_count to 4.

Question 31

You are training a large deep learning model on Azure Machine Learning using a compute cluster of 8 GPU nodes. The model no longer fits into the memory of a single GPU, so you need to shard the model parameters, gradients, and optimizer states across all available GPUs to reduce per-device memory consumption. Which approach should you configure for the training job?

Accepted Answer

Use a model-parallel/sharded approach such as DeepSpeed ZeRO to partition parameters, gradients, and optimizer states across GPUs.

Answer

Use data-parallel training where each GPU holds a full copy of the model and processes a different mini-batch.

Answer

Reduce the batch size to 1 and enable gradient accumulation on a single GPU node.

Answer

Submit the job as a parameter sweep so each node trains an independent copy with different hyperparameters.

Question 32

Your team's Azure Machine Learning workspace is configured with a private endpoint and public network access disabled. When you submit a job that references a custom environment defined by a Dockerfile, the environment image build fails because the default build process cannot reach the workspace's container registry over the public network. You need to allow custom environment images to build successfully while keeping the workspace isolated from the public internet. What should you do?

Accepted Answer

Configure the workspace to use a compute cluster as the image build compute so the build runs inside the virtual network.

Answer

Re-enable public network access on the workspace only during the image build and disable it afterward.

Answer

Switch the environment to a curated environment so that no image build is required.

Answer

Grant the AcrPush role to your user identity on the workspace's Azure Container Registry.

Question 33

Your team is building an automated evaluation workflow in Azure AI Foundry for a customer-support chat agent. You have a test dataset containing the user query, the retrieved context passages, and the model's generated response, but you do NOT have any human-written ground-truth answers. Leadership wants a built-in AI quality metric that assesses whether the generated responses are grammatically correct and read naturally, independent of any reference answer. Which built-in metric should you configure?

Accepted Answer

Fluency

Answer

Similarity

Answer

Groundedness

Answer

F1 score

Question 34

Your team runs a RAG-based customer support agent in Azure AI Foundry. During evaluation, stakeholders report that the agent frequently produces answers that are well-written and directly address the user's question, but sometimes include facts that are NOT present in the retrieved documents. You need to configure an automated evaluation that specifically detects this problem while ignoring writing style. Which built-in AI quality metric should you prioritize?

Accepted Answer

Groundedness, which measures whether the response is supported by the provided context

Answer

Fluency, which measures the grammatical and linguistic quality of the response

Answer

Relevance, which measures how well the response addresses the user's query

Answer

Coherence, which measures the logical flow and consistency of the response

Question 35

Your team is preparing to evaluate a customer-support RAG agent in Azure AI Foundry before production release. You have only a small set of real user questions and need a larger, diverse test dataset that includes adversarial and edge-case queries so that safety and quality evaluators produce statistically meaningful results. You want to minimize the manual effort of hand-writing hundreds of prompts. Which approach best generates the required test dataset?

Accepted Answer

Use the Azure AI Evaluation SDK simulator (adversarial and non-adversarial simulators) to synthesize a diverse set of query/response pairs, then run the evaluators against the generated dataset.

Answer

Enable production traffic sampling first, then wait to collect enough real conversations before running any evaluation.

Answer

Duplicate the small set of real questions multiple times until you reach several hundred rows and run the evaluators on the expanded file.

Answer

Deploy the agent to a provisioned throughput (PTU) endpoint so the higher throughput automatically increases the size of the evaluation dataset.

Question 36

Your team is configuring an automated evaluation run in Azure AI Foundry for a RAG-based customer support agent. You have a JSONL test dataset whose columns are named 'user_question', 'retrieved_docs', and 'model_reply'. You want to run the built-in groundedness and relevance evaluators, but the evaluation run fails because the evaluators cannot find the expected inputs. What should you do to resolve this without renaming the dataset columns?

Accepted Answer

Configure data mapping to bind the evaluator inputs (query, context, response) to the dataset columns 'user_question', 'retrieved_docs', and 'model_reply'.

Answer

Convert the JSONL file to CSV format because built-in evaluators only accept tabular CSV test datasets.

Answer

Add a custom Python evaluator that reads the raw columns directly, since built-in groundedness and relevance evaluators cannot be reused with custom schemas.

Answer

Enable the content safety evaluators, which automatically infer input columns from any dataset regardless of column names.

Question 37

You are fine-tuning a GPT-4o model in Azure OpenAI to classify customer support tickets into 12 categories. Your training file contains 3,000 curated JSONL examples. After the first fine-tuning job, the training loss drops steadily but you have no way to tell whether the model will generalize to unseen tickets before you deploy it. You want to detect overfitting during the job itself and compare candidate hyperparameter configurations objectively. What should you do?

Accepted Answer

Supply a separate validation file to the fine-tuning job so Azure OpenAI reports validation loss and validation token accuracy alongside training metrics.

Answer

Increase the number of epochs so the training loss converges to zero, which guarantees the model has learned the categories.

Answer

Add all 3,000 examples to a single training file and rely on the final training loss value to judge generalization.

Answer

Deploy the fine-tuned model to a serverless endpoint and manually inspect a handful of live predictions to decide if it generalizes.

Question 38

You are managing an Azure OpenAI fine-tuning job for a customer-support classification assistant. After the run completes, you review the training metrics and observe that training loss continues to decrease steadily across all 8 epochs, but validation loss stops improving after epoch 3 and begins to rise slightly. The deployed fine-tuned model performs noticeably worse on held-out real support tickets than a checkpoint you evaluated earlier. What is the most appropriate corrective action for the next fine-tuning run?

Accepted Answer

Reduce the number of epochs so training stops near the point where validation loss was lowest.

Answer

Increase the number of epochs to give the model more time to generalize to the validation set.

Answer

Remove the validation split entirely so the model trains on all available data each epoch.

Answer

Raise the learning rate multiplier to accelerate convergence on the training data.

Question 39

You fine-tune a GPT-4o model in Azure AI Foundry on a proprietary dataset of 800 curated examples. After training, the model performs excellently on the training set but produces rigid, low-quality responses on held-out validation prompts, and the training loss curve dropped rapidly while validation loss increased after the first epoch. You must adjust the fine-tuning hyperparameters to improve generalization. Which change is the most appropriate first step?

Accepted Answer

Reduce the number of training epochs and lower the learning rate multiplier.

Answer

Increase the number of epochs so the model sees the data more times.

Answer

Increase the batch size to the maximum allowed to speed convergence.

Answer

Switch from supervised fine-tuning to direct preference optimization (DPO) using the same dataset.

Question 40

Your team must fine-tune a large open-source language model for a specialized legal-summarization task. You have a single GPU with limited VRAM and a modest labeled dataset. Full fine-tuning of all model weights repeatedly runs out of memory, and you also need to keep the fine-tuning artifact small so that multiple task-specific adapters can be swapped in and out on the same base model in production. Which fine-tuning approach best meets these requirements?

Accepted Answer

Parameter-efficient fine-tuning with LoRA, training only low-rank adapter matrices while keeping the base model weights frozen

Answer

Full fine-tuning of all model parameters using gradient checkpointing to fit the model in memory

Answer

Few-shot prompting with retrieved examples instead of updating any model weights

Answer

Continued pre-training on a large unlabeled legal corpus before task training

Question 41

You are preparing a synthetic dataset to fine-tune a domain-specific chat model in Azure AI Foundry. The synthetic examples were generated by a larger teacher model using prompts derived from your internal knowledge base. Before training, you split the data into training and evaluation sets. After fine-tuning, the model reports excellent evaluation scores, but real-world performance on held-out user queries is noticeably worse. Which action should you take first to make the evaluation scores trustworthy?

Accepted Answer

Deduplicate near-identical synthetic examples across the training and evaluation splits to prevent data leakage between them

Answer

Increase the learning rate multiplier so the model generalizes better to unseen queries

Answer

Add more synthetic examples generated from the same prompts to expand the training set

Answer

Switch the evaluation metric from exact match to a coherence metric

Question 42

You are fine-tuning a GPT-4o model for a legal contract-review assistant. Your production traffic contains many rare clause types (indemnification, force majeure, arbitration) that are underrepresented in your labeled training set of 800 examples. Human annotation is expensive and slow. You must expand and balance the training data before fine-tuning while minimizing the risk of degraded generalization. Which approach best addresses this?

Accepted Answer

Use a capable teacher model to generate synthetic examples targeted at the underrepresented clause types, apply automated and human validation to filter low-quality samples, then blend the validated synthetic data with the real labeled examples.

Answer

Duplicate the existing 800 examples several times so that rare clause types appear more frequently in each fine-tuning epoch.

Answer

Generate a fully synthetic dataset from the teacher model and discard the 800 human-labeled examples to ensure consistent formatting across the training set.

Answer

Increase the number of fine-tuning epochs on the original 800 examples so the model memorizes the rare clause patterns it has already seen.

Question 43

You are fine-tuning a domain-specific model for a legal-tech company. To expand a small labeled dataset, you generate synthetic training examples using a larger teacher model. After the first fine-tuning run, evaluation shows the model reproduces factual errors and inconsistent formatting that were present in some synthetic examples. You must improve the fine-tuned model's quality before the next production candidate. What should you do FIRST with the synthetic data?

Accepted Answer

Apply an automated quality-filtering and validation pass that scores each synthetic example for correctness and consistency, then remove or correct low-scoring examples before retraining.

Answer

Increase the learning rate multiplier so the model relies less on the noisy synthetic examples during training.

Answer

Merge all synthetic examples with the genuine examples and increase the number of training epochs to let the model average out the errors.

Answer

Discard the synthetic data entirely and fine-tune only on the small labeled dataset to guarantee accuracy.

Question 44

You fine-tuned a base language model on a large domain-specific dataset for a legal-document assistant. After deployment, evaluation shows the model now performs well on legal tasks but has significantly degraded on general reasoning and instruction-following tasks it previously handled. You must re-run the customization to preserve general capabilities while still gaining domain accuracy, using minimal compute. Which approach best addresses this problem?

Accepted Answer

Use a parameter-efficient fine-tuning method such as LoRA with a lower learning rate and fewer epochs, keeping the base model weights frozen.

Answer

Increase the number of training epochs on the full-parameter fine-tune to force the model to relearn general tasks.

Answer

Raise the learning rate on the full-parameter fine-tune so the model converges faster on the legal dataset.

Answer

Discard the domain dataset and rely solely on retrieval-augmented generation with the untuned base model.

Question 45

Your team built a multi-step agent in Azure AI Foundry that calls several tools (a search API, a calculator, and a database query function) to answer user requests. During evaluation, users report that the agent frequently selects the wrong tool or passes malformed arguments, even when the final text answer sounds plausible. You need to add an automated evaluation that specifically measures whether the agent chooses appropriate tools and invokes them with correct parameters, so you can gate deployment on this dimension. Which built-in evaluator should you add to your evaluation run?

Accepted Answer

The Intent Resolution and Tool Call Accuracy evaluators, which assess whether the agent selected the right tools and passed valid parameters.

Answer

The Groundedness evaluator, which verifies that the response is supported by the retrieved context.

Answer

The Fluency evaluator, which scores the grammatical quality of the final natural-language answer.

Answer

The Coherence evaluator, which measures logical consistency across the response sentences.

Question 46

Your team ships a customer-support agent in Azure AI Foundry that uses multiple tools to resolve billing questions. QA reports that the agent frequently misunderstands what the customer is actually asking for before it even selects a tool, leading to irrelevant tool calls. You must add an agent-specific evaluator to your automated evaluation run that directly measures how well the agent identifies the user's request from the conversation. Which built-in evaluator should you add?

Accepted Answer

Intent Resolution evaluator

Answer

Groundedness evaluator

Answer

Fluency evaluator

Answer

Task Adherence evaluator

Question 47

Your team manages an Azure AI Foundry project using infrastructure-as-code. You author a Bicep template that provisions the Foundry resource, a project, and a model deployment. During CI/CD you must be able to re-run the same deployment repeatedly (for example after adding a new model deployment) without recreating existing resources or causing errors, and the deployment name for each model must be supplied at build time per environment. Which approach best meets these requirements?

Accepted Answer

Deploy the Bicep template with 'az deployment group create' in Incremental mode and pass the model deployment name as a template parameter.

Answer

Deploy the Bicep template with 'az deployment group create' using Complete mode so any drift is corrected on each run.

Answer

Hardcode the model deployment name in the Bicep resource and run 'az deployment group create --confirm-with-what-if' each time.

Answer

Convert the Bicep file to an ARM template and deploy it with 'az group create' passing environment values as tags.

Question 48

Your team maintains a production Azure AI Foundry environment defined entirely in Bicep, including the hub, projects, and model deployments. Before applying an updated Bicep template that changes a serverless model deployment's SKU capacity, the platform lead requires a preview of exactly which resources will be created, modified, or deleted—without actually applying any changes to the live environment. Which Azure CLI command should you run against the resource group deployment?

Accepted Answer

az deployment group what-if --resource-group rg-foundry --template-file main.bicep --parameters main.bicepparam

Answer

az deployment group validate --resource-group rg-foundry --template-file main.bicep --parameters main.bicepparam

Answer

az deployment group create --resource-group rg-foundry --template-file main.bicep --confirm-with-what-if --mode Incremental

Answer

az bicep build --file main.bicep --stdout

Question 49

Your team maintains a non-interactive Azure DevOps pipeline that uses the Azure CLI to deploy Azure AI Foundry resources and model deployments into a subscription. The pipeline currently fails at the deployment step because it cannot authenticate. Security policy forbids storing user credentials or interactive browser sign-in in the pipeline. Which authentication approach should you configure for the Azure CLI in the pipeline?

Accepted Answer

Authenticate 'az login' with a service principal using a workload identity federation (OIDC) service connection

Answer

Run 'az login' interactively and cache the token in the pipeline agent's user profile

Answer

Embed a personal access token for a named engineer in a pipeline variable and use it with 'az login'

Answer

Assign a system-assigned managed identity to the Azure DevOps organization and rely on it for CLI calls

Question 50

Your team runs an Azure AI Foundry project that serves a chat application. The current production deployment uses version 3 of a foundation model on a serverless API endpoint. A newer version 4 has been released, and stakeholders want to validate its response quality against version 3 using the same evaluation dataset before committing to a full cutover, while keeping production traffic entirely on version 3. What is the most appropriate approach?

Accepted Answer

Create a separate deployment for version 4 under the same project and route evaluation traffic to it, leaving the version 3 deployment serving all production requests.

Answer

Update the existing version 3 deployment in place to point to version 4, then roll back if the evaluation results are unsatisfactory.

Answer

Delete the version 3 deployment and create a new deployment for version 4, since only one model version can exist per project.

Answer

Provision reserved provisioned throughput units for version 4 to guarantee that production traffic is unaffected during evaluation.

Question 51

Your team develops prompt flows in Azure AI Foundry and stores the flow definitions (including the flow.dag.yaml and prompt templates) in a Git repository. The team lead wants a repeatable process so that when a validated prompt change is merged into the main branch, the flow is automatically evaluated and deployed to the staging Foundry project without manual portal steps. Which approach best meets this requirement?

Accepted Answer

Configure a CI/CD pipeline (e.g., GitHub Actions) that, on merge to main, uses the Azure CLI/az ml commands to run the flow evaluation and deploy the flow to the staging project

Answer

Have each developer manually export the flow from the Foundry portal and re-import it into the staging project after every merge

Answer

Store the prompt templates as secrets in Azure Key Vault so that the staging project automatically picks up the latest version on merge

Answer

Enable autosave on the authoring flow so changes replicate directly to the staging project in real time

Question 52

Your team deploys a fine-tuned open-source language model to a managed compute endpoint in Azure AI Foundry. During load testing, the endpoint sustains high GPU utilization but throughput plateaus and request latency climbs sharply as concurrent traffic increases. The model must remain always-on for an internal application with steady, predictable demand. You need to increase throughput without changing the model. What should you do first?

Accepted Answer

Scale out the managed compute deployment by increasing the instance count on the same GPU SKU

Answer

Migrate the model to a serverless API endpoint to remove capacity limits

Answer

Purchase provisioned throughput units (PTUs) for the deployment

Answer

Reduce the instance count to force requests to queue and process sequentially

Question 53

You are deploying an Azure AI Foundry project that must call a downstream Azure Key Vault to retrieve API secrets at runtime. Security policy prohibits storing any credentials in application code or configuration files, and the Foundry resource was created with a system-assigned managed identity. What is the correct way to grant the Foundry resource access to the Key Vault secrets?

Accepted Answer

Assign the Foundry resource's system-assigned managed identity the 'Key Vault Secrets User' RBAC role scoped to the Key Vault.

Answer

Generate a service principal client secret and store it as an environment variable in the Foundry project.

Answer

Create a shared access signature (SAS) token for the Key Vault and embed it in the application connection string.

Answer

Add the Key Vault access key to the Foundry project's Bicep parameters file during deployment.

Question 54

Your team deploys a foundation model to a serverless API endpoint in an Azure AI Foundry project. A downstream App Service must call this endpoint. Security policy prohibits storing any long-lived secrets or API keys in application configuration, and all access must be auditable through Microsoft Entra ID. What is the recommended way to configure authentication between the App Service and the Foundry endpoint?

Accepted Answer

Enable a system-assigned managed identity on the App Service, grant it the appropriate Azure RBAC role on the Foundry resource, and authenticate using Microsoft Entra tokens.

Answer

Retrieve the endpoint's primary key and store it in the App Service application settings, rotating it manually every 90 days.

Answer

Store the endpoint API key in Azure Key Vault and reference it from App Service using a Key Vault reference in application settings.

Answer

Configure the App Service to use a shared access signature (SAS) token generated from the Foundry project storage account.

Question 55

Your team is building a document-summarization feature in Azure AI Foundry. Source documents routinely exceed 60,000 tokens and must be summarized in a single pass. The team wants the lowest-cost model available in the model catalog that can accept the full document without chunking. Which criterion should drive your model selection?

Accepted Answer

Choose the model with the largest maximum context window that can hold the full document in one request, then compare per-token pricing among qualifying models.

Answer

Choose the model with the highest MMLU benchmark score regardless of its context window size.

Answer

Choose any chat model and enable provisioned throughput units to expand its context window.

Answer

Choose the model with the fastest time-to-first-token latency, since summarization is latency sensitive.

Question 56

Your team runs a production chat application on an Azure AI Foundry deployment of a GPT model. Compliance requires that any model behavior change be validated in a staging environment before it reaches production. During a quarterly review you discover that the production deployment's underlying model version silently changed, causing subtly different responses that were never tested. When you inspect the deployment, its version update policy is set to allow automatic upgrades. What should you change to ensure the production deployment stays on a validated model version until you explicitly approve an upgrade?

Accepted Answer

Set the deployment's version update policy to 'No auto update' so it stays pinned to the specific model version you selected

Answer

Set the deployment's version update policy to 'Once new default version is available' (auto-update to default)

Answer

Reduce the deployment's provisioned throughput units so upgrades cannot be applied during high load

Answer

Enable a content safety filter on the deployment to block responses from newer model versions

Question 57

Your team is deploying an Azure AI Foundry hub and project for a regulated financial workload. Security policy requires that no data traverses the public internet: both inbound access to the hub and the hub's outbound connections to its dependent Azure resources (Storage, Key Vault, Container Registry) must stay on the Microsoft backbone. You need to configure the Foundry hub's networking so these requirements are met with the least ongoing management overhead. Which configuration should you choose?

Accepted Answer

Configure the hub with a managed virtual network in 'Allow Only Approved Outbound' mode, add private endpoints for inbound access, and create private endpoint outbound rules to the dependent resources.

Answer

Enable a managed virtual network on the hub in 'Allow Internet Outbound' mode and add a private endpoint only for inbound access.

Answer

Leave the hub on public network access, but restrict access using a resource firewall with IP allow-lists for the corporate network ranges.

Answer

Deploy the hub into a customer-managed virtual network and rely on network security group (NSG) rules to block all outbound public traffic.

Question 58

Your team has deployed an Azure AI Foundry hub with a managed virtual network and configured a private endpoint so that all inbound access is restricted to your corporate network. On-premises developers connected through ExpressRoute report that they can reach the Foundry portal by IP address but consistently get name-resolution failures when using the resource's fully qualified domain name. What is the most appropriate action to resolve the connectivity issue?

Accepted Answer

Configure conditional forwarders on the on-premises DNS servers to forward the Foundry public DNS zones to Azure DNS, and ensure the corresponding Azure Private DNS zones are linked to the virtual network.

Answer

Add an inbound network security group (NSG) rule on the private endpoint subnet to allow TCP port 53 from the on-premises network.

Answer

Switch the hub from a private endpoint to a service endpoint so that public DNS resolution is used for the resource.

Answer

Assign a user-assigned managed identity to the hub and grant it the DNS Zone Contributor role on the Private DNS zone.

Question 59

Your organization has an Azure AI Foundry hub that centralizes an Azure OpenAI connection, an Azure AI Search connection, and a storage account. Three separate teams each need their own isolated Foundry project for developing distinct GenAI applications, but all three teams must reuse the same Azure OpenAI connection defined at the hub without redefining credentials in each project. What is the most appropriate way to structure this so that connections are inherited while keeping project-level work isolated?

Accepted Answer

Create three projects under the shared hub; hub-level connections are automatically available to all projects, while each project maintains its own isolated assets and access boundaries.

Answer

Create three separate hubs, each with its own duplicate Azure OpenAI connection, and link them together with a connection group.

Answer

Create a single project and use folders inside it to separate each team's work, since connections cannot be shared across multiple projects.

Answer

Define the Azure OpenAI connection at the project scope in the first project and grant the other two projects Reader access to that project's resources.

Question 60

Your organization is standardizing on Azure AI Foundry. The platform team wants a central place to configure shared networking, connections to Azure OpenAI, and a common security and governance boundary, while individual data science teams each need isolated workspaces for their own model deployments, prompt flows, and code assets. You must implement this with the fewest configuration steps and maximize resource reuse. Which approach should you take when creating the Foundry resources?

Accepted Answer

Create one Foundry hub for the organization and create a separate Foundry project under that hub for each data science team.

Answer

Create one Foundry hub and a single shared project, and separate the teams using folders inside the project.

Answer

Create a separate Foundry hub for each team and one shared project that all hubs reference.

Answer

Create one Foundry project for the organization and add a child hub for each data science team.

Microsoft Machine Learning Operations Engineer Associate (AI-300) full-length practice exam

How this timed AI-300 mock exam works

What your AI-300 results show

How to get the most out of it

Ready for the full AI-300 mock exam?