4.4 Validation, Retry, and Feedback Loops

What You Need to Know

Production extraction systems fail. Documents arrive in unexpected formats, numerical values don’t add up, and fields end up in the wrong places. The question isn’t whether failures happen but how your system responds. The validation-retry pattern turns those failures into self-correcting workflows.

Retry-with-Error-Feedback

The correct retry pattern sends three pieces of information back to the model:

The original document — so the model has the source to re-examine
The failed extraction — so the model can see what it produced
The specific validation error — so the model knows exactly what went wrong

// Retry with error feedback
const retryMessages = [
  {
    role: "user",
    content: `Original document:\n${originalDocument}\n\n` +
      `Your extraction:\n${JSON.stringify(failedExtraction)}\n\n` +
      `Validation error: Line items sum to £450 but stated_total is £500. ` +
      `Please re-extract, ensuring all line items are captured.`
  }
];

This beats a naive retry by a wide margin. Without the specific error, the model has no guidance for what to fix and usually reproduces the same mistake. With it, the model can target its self-correction: re-examining the document for missed line items, checking field placement, recalculating the total.

The Retry Effectiveness Boundary

This is the concept the exam tests most aggressively in this task statement. Retries have a clear effectiveness boundary:

Retries ARE effective for:

Format mismatches (wrong date format, inconsistent currency notation)
Structural output errors (values in wrong fields, incorrect nesting)
Misplaced values (data that exists in the document but was extracted into the wrong field)
Mathematical errors (the model missed a line item affecting the total)

Retries are NOT effective for:

Information genuinely absent from the source document
Data that exists only in an external document not provided to the model
Fields requiring knowledge the model does not have

The exam presents both scenarios and expects you to identify which is fixable. If a document genuinely doesn’t contain a department name, no amount of retrying will produce a correct value. Flag the extraction for human review, or return null if the schema allows it.

Self-Correction Flow Design

Rather than relying solely on external validation logic, you can build self-correction into the extraction schema itself:

calculated_total vs stated_total: Extract both the sum the model calculates from individual line items and the total stated in the document. When these differ, you have an automatic discrepancy flag without external logic.

{
  "line_items": [
    { "description": "Widget A", "amount": 150.00 },
    { "description": "Widget B", "amount": 300.00 }
  ],
  "calculated_total": 450.00,
  "stated_total": 500.00,
  "total_discrepancy": true
}

conflict_detected booleans: Add boolean fields that flag when the source document contains contradictory information. For example, if a document states “payment due: 30 days” in one section but “payment terms: net 60” in another, the model should extract both and set conflict_detected: true rather than silently picking one.

detected_pattern Fields

For code review and analysis pipelines, add detected_pattern fields to structured findings. This tracks which specific code construct triggered each finding.

{
  "finding": "Potential SQL injection vulnerability",
  "severity": "critical",
  "detected_pattern": "string concatenation in SQL query",
  "file": "user_service.py",
  "line": 42
}

When developers dismiss findings, you can analyse dismissal patterns by detected_pattern. If developers consistently dismiss findings triggered by “variable shadowing in nested scope,” that pattern likely needs prompt refinement. This creates a systematic improvement loop: extract, validate, collect dismissal data, refine prompts, repeat.

Schema Syntax Errors vs Semantic Validation Errors

The exam distinguishes between these two error categories:

Schema syntax errors — Malformed JSON, missing required fields, wrong data types. Eliminated entirely by tool_use with JSON schemas (covered in Task Statement 4.3).

Semantic validation errors — Correct JSON structure but incorrect values. Line items that do not sum, dates that precede each other incorrectly, values in wrong fields. These require validation logic outside the schema and are the focus of retry loops.

The overlap between these task statements is intentional. The exam tests whether you understand that tool_use solves the first category but not the second.

Pydantic as the Validation Layer

The exam guide names Pydantic alongside JSON Schema in its hands-on exercise for this task statement: “when Pydantic or JSON schema validation fails, send a follow-up request including the document, the failed extraction, and the specific validation error.” In a Python pipeline, Pydantic is the layer that turns “validation failed” into the specific, per-field error messages the retry loop needs.

A Pydantic model does two jobs at once. Parsing enforces structure — types, required fields, enums. Validators enforce semantics — the rules a JSON schema cannot express, like cross-field arithmetic or date ordering. Both failure kinds surface through one ValidationError, with machine-readable errors naming the field and the broken rule:

import json
from pydantic import BaseModel, ValidationError, model_validator

class LineItem(BaseModel):
    description: str
    amount: float

class Invoice(BaseModel):
    line_items: list[LineItem]
    stated_total: float

    @model_validator(mode="after")
    def totals_must_match(self):
        calculated = round(sum(i.amount for i in self.line_items), 2)
        if abs(calculated - self.stated_total) > 0.01:
            raise ValueError(
                f"line items sum to {calculated} but stated_total is {self.stated_total}"
            )
        return self

try:
    invoice = Invoice.model_validate(tool_input)  # the tool_use input from the response
except ValidationError as e:
    errors = "\n".join(
        f"{'.'.join(map(str, err['loc'])) or 'invoice'}: {err['msg']}" for err in e.errors()
    )
    retry_message = (
        f"Original document:\n{original_document}\n\n"
        f"Your extraction:\n{json.dumps(tool_input)}\n\n"
        f"Validation errors:\n{errors}\n\n"
        f"Please re-extract, fixing the identified errors."
    )

The except branch is the retry-with-error-feedback pattern from the top of this lesson — Pydantic simply supplies the third ingredient (the specific error) in a form you can format straight into the prompt. The retry-effectiveness boundary applies unchanged: a validator that fails because information is absent from the source document still means human review, not a retry.

Exam Traps

Practice Scenario

Your extraction pipeline validates that line item amounts sum to the stated total. For Document A, the calculated sum is £450 but the stated total is £500. For Document B, the ‘department’ field is missing entirely from the source text. Which retry strategy is correct?

A. Retry both documents with the validation errors, instructing the model to re-extract all fields
B. Skip retries for both documents and flag them all for human review to ensure accuracy
C. Retry both documents with the same prompt, since extraction is non-deterministic and may succeed on a second attempt
D. Retry Document A with the discrepancy error; flag Document B for human review since the information is absent from the source

Answer & explanation

Correct: D

A — Document B cannot be fixed by retrying — the department information does not exist in the source. Retrying wastes tokens and will likely produce a fabricated value.
B — Document A has a likely fixable discrepancy. Skipping the retry wastes the model’s self-correction capability for errors it can actually fix.
C — Non-determinism does not create information that does not exist. Document A may benefit from targeted retry; Document B will not benefit regardless of how many attempts you make.
D — Document A has a fixable discrepancy — the model likely missed a line item. Document B has genuinely absent information, so retries are ineffective. Flag it for human review or accept null.

Sources

Claude Certified Architect Foundations Exam Guide — Task Statement 4.4 — Anthropic
Tool Use (Function Calling) — Anthropic
Structured Outputs — Anthropic
Pydantic documentation — Pydantic
Building with Claude API (Skilljar) — Anthropic

Exam Simulator

Six exam-style multiple-choice questions on Validation, Retry, and Feedback Loops. Pick an answer, then open the explanation.

Question 1

Your extraction pipeline validates that line item amounts sum to the stated total. For Document A, the calculated sum is £450 but the stated total is £500. For Document B, the “department” field is missing entirely from the source text. Which retry strategy is correct?

A. Retry Document A with the discrepancy error, and flag Document B for human review as the data is absent from source
B. Retry both documents with the validation errors, instructing the model to re-extract all fields
C. Retry both documents with the same prompt, since extraction is non-deterministic and may succeed on a second attempt
D. Skip retries for both documents and flag them all for human review to ensure accuracy

Answer & explanation

Correct: A

A is correct because Document A has a fixable discrepancy (the model likely missed a line item) while Document B has genuinely absent information making retries ineffective.
B is wrong because Document B cannot be fixed by retrying — the department information does not exist in the source. Retrying wastes tokens and will likely produce a fabricated value.
C is wrong because non-determinism does not create information that does not exist. Document A may benefit from targeted retry; Document B will not.
D is wrong because Document A has a likely fixable discrepancy. Skipping the retry wastes the model’s self-correction capability.

Question 2

What three pieces of information should a retry message include for maximum self-correction effectiveness?

A. The original prompt, the model’s confidence score, and a request to try again
B. The original document, the failed extraction, and the specific validation error
C. The failed extraction, a corrected example, and instructions to match the example
D. The original document, a list of all possible errors, and a higher temperature setting

Answer & explanation

Correct: B

A is wrong because confidence scores are poorly calibrated and the original prompt alone does not show the model what went wrong.
B is correct because the model needs the source to re-examine, its failed output to see what it produced, and the specific error to know exactly what to fix.
C is wrong because providing a corrected example removes the model’s need to self-correct and may introduce errors if your example is wrong.
D is wrong because listing all possible errors instead of the specific error gives the model no targeted guidance, and temperature does not improve accuracy.

Question 3

Your extraction schema includes separate calculated_total and stated_total fields. The model extracts line items summing to £720 but the document states a total of £720. The total_discrepancy field is set to true. What does this indicate?

A. The extraction is correct but the model made an error setting the discrepancy flag
B. The model correctly identified a discrepancy between line items and total
C. The schema is misconfigured and both total fields should be combined into one
D. There is a semantic error — the totals match so total_discrepancy should be false

Answer & explanation

Correct: D

A is wrong because this is not an extraction error but a flag-setting error, which is itself a semantic error.
B is wrong because there is no discrepancy — the totals match. The model incorrectly set the flag.
C is wrong because separate fields are essential for automatic discrepancy detection. Combining them loses this capability.
D is correct because the calculated_total and stated_total match (both £720), so total_discrepancy should be false. This is a semantic validation error that requires a retry with the specific error: “total_discrepancy set to true but calculated_total equals stated_total.”

Question 4

Developers consistently dismiss code review findings triggered by the “variable shadowing in nested scope” pattern. What is the correct response?

A. Remove variable shadowing detection from the review system entirely
B. Add a confidence threshold to filter low-confidence shadowing findings
C. Refine the shadowing criteria and add distinguishing examples
D. Require developers to provide justification for each dismissal

Answer & explanation

Correct: C

A is wrong because some variable shadowing cases are genuine bugs. Removing the entire category loses valid detections.
B is wrong because confidence thresholds are poorly calibrated and do not address the root cause of false positives.
C is correct because detected_pattern tracking enables systematic analysis of which cases cause false positives, and refining criteria with code examples improves precision for that pattern.
D is wrong because adding friction to the developer workflow does not improve detection quality.

Question 5

Which of the following error types can be fixed by a retry-with-error-feedback loop?

A. A document genuinely lacking a vendor tax identification number
B. Information existing only in a separate document not provided to the model
C. A field requiring external database lookup not available to the model
D. Line items summing to £450 when the stated total reads £500 instead

Answer & explanation

Correct: D

A is wrong because information genuinely absent from the source cannot be created by retrying.
B is wrong because data in a separate document is not accessible, making retries ineffective.
C is wrong because external data not available to the model cannot be produced by retrying.
D is correct because a sum discrepancy suggests the model missed a line item that exists in the document. The retry with the specific discrepancy error guides the model to re-examine the source.

Question 6

What is the distinction between schema syntax errors and semantic validation errors in the context of tool_use extraction?

A. Schema syntax errors are caught at compile time; semantic errors are caught at runtime
B. tool_use removes syntax errors, but not the semantic ones
C. Both error types are eliminated by tool_use with strict JSON schemas
D. Schema syntax errors are minor; semantic errors are critical

Answer & explanation

Correct: B

A is wrong because this is not about compile vs runtime. tool_use prevents syntax errors at the API level.
B is correct because tool_use guarantees schema-compliant structure (no malformed JSON) but cannot verify semantic correctness (whether values are accurate, sums are correct, or data is fabricated).
C is wrong because tool_use only eliminates syntax errors. Semantic errors persist and require separate validation.
D is wrong because the distinction is about prevention mechanism, not severity. Both can be critical.