4.1 System Prompts with Explicit Criteria

What You Need to Know

The single biggest mistake in production prompt engineering is relying on vague instructions. “Be conservative.” “Only report high-confidence findings.” “Use your best judgement.” None of these give the model an actionable decision boundary. They sound reasonable, which is exactly why the exam uses them as distractors.

The correct approach is explicit categorical criteria that define precisely what the model should flag and what it should skip. Compare these two system prompts for a CI/CD code review pipeline:

Wrong approach:

Review this code. Be conservative. Only report high-confidence findings.

Correct approach:

Flag comments only when claimed behaviour contradicts actual code behaviour.
Report bugs and security vulnerabilities.
Skip minor style preferences and local patterns.

The first gives the model no criteria to apply. “Conservative” means different things in different contexts, and “high-confidence” is a subjective threshold the model cannot calibrate. The second provides concrete categories: what to report (bugs, security), what to skip (style, local patterns), and a specific trigger for comment flags (claimed vs actual behaviour contradiction).

The False Positive Trust Problem

High false positive rates in one category destroy developer trust in all categories. The exam leans on this hard. If your “documentation mismatch” findings are wrong 40% of the time, developers stop reading your “security vulnerability” findings too, even when those run at 98% accuracy. Trust isn’t category-specific. It bleeds across the whole output.

The fix feels backwards: temporarily disable the high false-positive categories while you rework their prompts. Trust in the categories that already work comes back straight away. Then you iterate on the broken category with concrete code examples, switching it back on only once precision improves.

You’re not abandoning the category. You’re putting system-wide trust ahead of category completeness.

Severity Calibration with Code Examples

Defining severity levels requires concrete code examples, not prose descriptions. Compare:

Prose description (insufficient):

Critical: Issues that could cause system failures or data loss
Minor: Issues that affect code readability but not functionality

Code example approach (correct):

Critical — Unsanitised user input in SQL query:
  query = f"SELECT * FROM users WHERE id = {user_input}"

Minor — Inconsistent variable naming:
  userName vs user_name in the same module

The prose description forces the model to interpret what “could cause system failures” means. The code example removes ambiguity entirely. When the model sees actual code patterns classified at each severity level, it produces consistent classification across invocations.

Why Confidence-Based Filtering Fails

The exam frequently presents “only report high-confidence findings” as a tempting answer. It sounds like good engineering: filter by confidence, keep only the strong signals. But LLM self-reported confidence is poorly calibrated. The model is often sure about wrong findings and hesitant about right ones. Confidence scores earn their keep in routing (sending low-confidence findings to human review, as covered in Task Statement 4.6), but they’re no substitute for explicit criteria that define what counts as a valid finding in the first place.

The hierarchy is: explicit criteria first, confidence-based routing second. Never skip the first step.

Exam Traps

Practice Scenario

Your CI/CD code review pipeline has a 40% false positive rate on ‘documentation mismatch’ findings, causing developers to ignore ALL review categories including accurate security findings. What is the most effective fix?

A. Add “only report high-confidence documentation issues” to the system prompt so the model filters its own weaker findings
B. Add a second model pass that re-examines each documentation finding and discards any it cannot verify before the report reaches developers
C. Increase the model temperature to produce more varied review runs, then filter out findings that appear only once
D. Temporarily disable the documentation mismatch category while refining its prompts with explicit criteria and code examples

Answer & explanation

Correct: D

A — Vague confidence instructions do not improve precision. The model has no concrete criteria for what high-confidence means in this context.
B — A second pass without better criteria will have the same false positive problem. Fix the root cause — the criteria — before adding verification layers.
C — Temperature affects randomness, not precision. Higher temperature would likely increase false positives rather than reduce them.
D — This restores trust in all other categories immediately while you iterate on the problematic category with specific, concrete criteria. Trust recovery across all categories is the priority.

Sources

Claude Certified Architect Foundations Exam Guide — Task Statement 4.1 — Anthropic
Prompt Engineering Overview — Anthropic
Building with Claude API (Skilljar) — Anthropic

Exam Simulator

Six exam-style multiple-choice questions on System Prompts with Explicit Criteria. Pick an answer, then open the explanation.

Question 1

Your CI/CD code review pipeline has a 40% false positive rate on “documentation mismatch” findings, causing developers to ignore ALL review categories including accurate security findings. What is the most effective fix?

A. Add “only report high-confidence documentation issues” to the system prompt
B. Increase the model temperature to get more varied results and filter outliers
C. Disable the documentation mismatch category while its prompts are refined
D. Add a second model pass to verify each documentation finding before reporting

Answer & explanation

Correct: C

A is wrong because vague confidence instructions do not improve precision. The model has no concrete criteria for what “high-confidence” means.
B is wrong because temperature affects randomness, not precision. Higher temperature would increase false positives.
C is correct because it restores trust in all other categories immediately while you iterate on the problematic category with specific criteria. Trust recovery across all categories is the priority.
D is wrong because a second pass without better criteria will have the same false positive problem. Fix the criteria first.

Question 2

Which of the following system prompt instructions gives the model the most actionable decision boundary for a code review task?

A. “Flag only where claimed behaviour contradicts the code.”
B. “Use your best judgement to identify important code issues”
C. “Be conservative and only flag issues you are highly confident about”
D. “Try to minimise false positives while maintaining good coverage of real issues”

Answer & explanation

Correct: A

A is correct because it defines specific trigger conditions (claimed vs actual behaviour contradiction), categories to report (bugs, security), and categories to skip (style, local patterns).
B is wrong because “best judgement” and “important” give no concrete criteria.
C is wrong because “conservative” and “highly confident” are subjective thresholds with no actionable interpretation.
D is wrong because it describes desired outcomes without specifying the criteria to achieve them.

Question 3

Your team defines severity levels for a code review system. Which approach produces the most consistent classification across invocations?

A. “Critical: issues that could cause system failures or data loss. Minor: issues that affect code readability.”
B. A decision tree that categorises by file type first, then by issue type
C. A confidence threshold where findings above 0.9 are critical and below 0.5 are minor
D. Concrete code examples for each severity level, critical through minor

Answer & explanation

Correct: D

A is wrong because prose descriptions like “could cause system failures” force the model to interpret ambiguous language, producing inconsistent classification.
B is wrong because file type is not a reliable proxy for severity. A critical SQL injection can appear in any file type.
C is wrong because confidence scores are poorly calibrated and do not correlate reliably with actual severity.
D is correct because concrete code examples remove ambiguity entirely. When the model sees actual code patterns classified at each severity, it produces consistent results.

Question 4

A developer argues that adding “only report findings with confidence above 0.85” to the system prompt will reduce false positives. Why is this approach insufficient?

A. 0.85 is too high a threshold; 0.70 would be more appropriate
B. LLM self-reported confidence is poorly calibrated and unreliable
C. Confidence filtering only works with the Batches API, not synchronous calls
D. The model ignores confidence threshold instructions in system prompts

Answer & explanation

Correct: B

A is wrong because the issue is not the threshold value but that confidence scores themselves are unreliable for this purpose.
B is correct because LLM self-reported confidence is poorly calibrated. Explicit categorical criteria that define what to flag and skip are the correct first step; confidence-based routing is a secondary technique.
C is wrong because confidence filtering is not API-dependent. The issue is calibration quality.
D is wrong because the model does not ignore these instructions; it applies them inconsistently because confidence is poorly calibrated.

Question 5

Your code review system has five categories. “Unused imports” has a 45% false positive rate, while the other four categories average 8%. Developers have stopped reading any review output. What should you do first?

A. Disable “unused imports” while its criteria are refined with real code examples
B. Add a confidence threshold to filter low-confidence findings across all categories
C. Retrain the system on more code examples to improve overall accuracy
D. Reduce the number of categories to only the three most reliable ones permanently

Answer & explanation

Correct: A

A is correct because high false positives in one category destroy trust in ALL categories. Disabling the problematic category restores trust in the four working categories immediately while you iterate on “unused imports” criteria.
B is wrong because confidence thresholds are poorly calibrated and would suppress valid findings in the working categories.
C is wrong because the problem is a single noisy category destroying trust, not overall model capability.
D is wrong because permanently removing categories loses functionality. The strategy is to temporarily disable, improve, and re-enable.

Question 6

The hierarchy for improving precision in a production code review system is:

A. Confidence thresholds first, then explicit criteria if thresholds are insufficient
B. Few-shot examples first, then explicit criteria, then confidence thresholds
C. Explicit categorical criteria first, then confidence-based routing as a secondary technique
D. Temperature adjustment first, then criteria refinement, then confidence routing

Answer & explanation

Correct: C

A is wrong because it reverses the hierarchy. Confidence thresholds without explicit criteria are poorly calibrated.
B is wrong because few-shot examples address consistency, not precision criteria. Explicit criteria come first for precision.
C is correct because explicit criteria define what constitutes a valid finding. Confidence routing is useful but only after criteria are established.
D is wrong because temperature affects randomness, not precision. It is not part of the precision improvement hierarchy.