Very different failures can look the same at the surface.
A bad answer might come from retrieval, missing context, interpretation, authority handling, deterministic logic, the eval expectation, or the final presentation. Reaching for the prompt first can fix the wrong layer.
Diagnose before changing anything.
I use the failed case as evidence, then classify the problem before changing anything: product behavior, the eval contract, missing evidence, deterministic logic, or presentation. The goal is to change the product only when the product is actually the problem, then rerun the same case to verify the behavior changed for the intended reason.
The agent found the right evidence and still gave the wrong answer.
In a controlled State agent exercise, the investigator correctly found an open Review proposing an owner. Its final answer then presented that proposed owner as established fact.
The agent found the relevant files and recognized that the Review was still open.
The final synthesis promoted a pending proposal into project truth.
Open Reviews were made explicitly pending, then the same case was rerun and behaved correctly.
This example used a controlled fake State dataset. It is evidence of the debugging method, not production performance.
Sometimes satisfying the test would make the product worse.
In later State eval work, one scenario expected an answer-only action but observed an answer plus a proposed state update supported by the evidence. Product review found that the broader behavior could be acceptable when the evidence justified it.
The observed action did not match the original expected action exactly.
The evidence supported proposing the additional update for a person to approve, so forcing answer-only behavior would have made the test more important than the product contract.
The expectation was updated rather than changing acceptable product behavior just to make the check pass.
A failing eval is evidence, not an oracle.
Separate retrieval, context, interpretation, authority, validation, and presentation before choosing a fix.
Rerun the exact scenario, then add nearby regression cases so a narrow fix does not simply move the problem.
Authority and uncertainty failures matter even when the answer sounds plausible.
If the expected behavior no longer matches the product contract, the eval should change.
How I split the work.
I defined expected behavior, judged severity, classified where the failure belonged, and decided what should change. The same scenario then had to pass again before I treated the fix as real. AI supported case drafting, trace summaries, implementation, and regression coverage.