Hard — CCAO-F Practice Quiz

Question 1

Your team has built a test set of 500 prompts to evaluate a customer-support summarization feature before each release. Manually reading and scoring every model output takes two engineers a full day, which is delaying releases. The outputs are free-form summaries where correctness is somewhat subjective. Which approach best lets the team scale their evaluation while keeping quality judgments trustworthy?

Accepted Answer

Use an LLM-based grader with a clear rubric to score all 500 outputs automatically, and have humans spot-check a sample to validate the grader's alignment with human judgment.

Answer

Reduce the test set to 20 prompts so the two engineers can finish grading in under an hour each release.

Answer

Skip the evaluation for minor releases and rely on user complaints to surface any quality regressions after deployment.

Answer

Switch entirely to exact-string matching against a single approved summary for each prompt to make grading fully automatic.

Question 2

A team iterating on a customer-support classification prompt has been tuning it repeatedly against the same 50-example test set. Their scores keep improving, but when they deploy, real-world accuracy is noticeably lower than their eval suggested. What is the most likely cause and best corrective action?

Accepted Answer

The prompt has been overfit to the small test set; they should maintain a separate held-out set that they do not tune against and use it to validate generalization.

Answer

The model temperature is too high; they should set it to 0 and the eval scores will match production.

Answer

The test set is too large to be reliable; they should shrink it to 10 examples for faster iteration.

Answer

Automated grading is inherently inaccurate; they should abandon the eval and rely on manual spot-checks instead.

Question 3

A team building a customer-support summarization feature achieves 95% on their automated eval, which grades summaries by ROUGE overlap against reference summaries. However, support agents complain that the deployed summaries frequently omit the customer's actual requested action, making them useless in practice. What is the most likely root cause and the best corrective step?

Accepted Answer

The eval metric measures surface word overlap rather than whether the required outcome (capturing the requested action) is present; redefine the success criteria and add a targeted check that verifies the requested action is captured.

Answer

The model temperature is too high, causing inconsistent summaries; lower it to 0 and rerun the same eval.

Answer

The test set is too small; add more examples and the ROUGE score will better reflect real usefulness.

Answer

The prompt lacks a role assignment; give Claude a 'senior support analyst' persona and the omission problem will resolve.

Question 4

A customer-facing chatbot built on Claude includes a guardrail that refuses to provide instructions for defeating the company's own fraud-detection system. During a security review, a tester discovers that by starting with harmless questions about 'how fraud detection generally works,' then slowly asking increasingly specific follow-up questions across many turns, they can eventually get the bot to reveal sensitive detection thresholds it would refuse to give if asked directly. Which mitigation most directly addresses this specific attack pattern?

Accepted Answer

Add output validation that screens each response for sensitive content regardless of how the conversation reached that point

Answer

Lower the model's temperature so responses become more deterministic and predictable

Answer

Increase max_tokens so the model has room to fully explain its refusals

Answer

Instruct the model in the system prompt to always be maximally helpful to paying customers

Question 5

A financial analyst is building a prompt that asks Claude to evaluate whether a loan application should be approved based on several criteria. The analyst wants Claude to reason through each criterion before committing to a final decision, so they add 'Think step by step' to the prompt. However, in testing, Claude often states its approval decision in the first sentence and then writes reasoning that merely justifies that snap decision. What is the most effective adjustment to genuinely improve the quality of Claude's reasoning?

Accepted Answer

Explicitly instruct Claude to write out its analysis inside tags first, and to place its final decision only afterward inside tags.

Answer

Increase the temperature setting so Claude explores more reasoning paths before deciding.

Answer

Add the phrase 'Think step by step' a second time at the end of the prompt for emphasis.

Answer

Prefill the assistant response with the word 'Approved:' to guide the output format.

Question 6

A developer prefills the assistant response with an opening brace to force Claude to return only a JSON object. However, the task also requires Claude to reason through several conditional pricing rules before deciding the final values. After adding the prefill, output quality drops noticeably — Claude picks wrong values more often than before. What is the most likely cause, and what should the developer do?

Accepted Answer

Prefilling with an open brace suppresses Claude's ability to reason step by step first; move the reasoning into a chain-of-thought step before the JSON, or use prompt chaining so reasoning happens in a separate call.

Answer

Prefilling always reduces accuracy, so the developer should remove the prefill entirely and rely on instructions asking for JSON-only output.

Answer

The temperature is too low; raising it will let Claude explore the pricing rules more thoroughly before answering.

Answer

The model is too small for the task; switching to a larger model is the only reliable fix for reasoning-heavy JSON generation.

Question 7

You are building an agent that lets Claude call a set of tools in a loop. In testing, you notice that for some ambiguous requests Claude keeps requesting tools repeatedly (stop_reason 'tool_use' every turn) and never produces a final text answer, causing runaway API costs. Which change to your application code best addresses this while keeping the tool-use loop functional for normal requests?

Accepted Answer

Set a maximum number of tool-call iterations in your loop, and after the limit is reached stop returning tool results and prompt Claude to answer with the information gathered so far.

Answer

Set tool_choice to 'none' on every request so Claude can never call a tool and always returns text immediately.

Answer

Remove the tool descriptions from the tool definitions so Claude is less likely to know when to invoke them.

Answer

Lower max_tokens to a very small value so each tool-use response is truncated before Claude can request another tool.