Normal
Representative expected behavior.
Library / 04 Quality, Evals & Reliability
Reviewed Oct 2026AI Product Evals
Define expected behavior first; test realistic risk and use the results to make product decisions.
This sheet covers what to measure and where to set release bars. For which method to use (tests, evals or QA), see Testing & Evaluating AI Systems →
01 / table
Tests and evals are different
| Deterministic software tests | AI evals |
|---|---|
| Did rules execute as programmed? | Was probabilistic behavior acceptable across real cases? |
| Schemas, permissions, state transitions, validation | Interpretation, groundedness, uncertainty, completeness |
| Automated pass/fail invariants | Rubrics, severity, repeated and human-scored cases |
02 / cards
Build a balanced eval set
Paraphrase
Same intent, different wording.
Edge
Plausible boundary case.
Uncertainty
Tentative, conflicting, or unknown facts.
Known failure
A regression that must stay fixed.
Negative / no-action
The correct response is to refrain.
03 / table
Score what matters
| Dimension | Question |
|---|---|
| Supported | Is the answer grounded in available evidence? |
| Complete | Did it catch important information? |
| Uncertainty preserved | Did unknown remain unknown? |
| Human burden | Did it avoid needless reviews or alerts? |
04 / table
Boundary examples
| Evidence | Expected |
|---|---|
| Target unknown | Keep target unknown, never assume zero |
| Explicit 0 | Allow 0 only when established |
| Tentative 10% | Do not treat as approved |
| Observed 8%, target unknown | Keep actual separate from target |
05 / flow
Failure → fix → regression
Define expected behaviorBaseline + severityTrace first wrong layerTargeted fixOriginal + neighbors + holdoutsMonitor
Check that score gains mean better work: An eval score is a proxy for the outcome you want. It can rise while usefulness falls if the rubric rewards the wrong behavior or tuning concentrates on known cases. A better general benchmark also does not show a model is better for your workflow. Before treating an improvement as progress, check unseen examples and actual user work, and inspect whether results are more useful and whether review or correction effort changed. Further reading: What actually is AI research? (Technically Speaking, Ep 2) (interview with Jonathan Frankle, Oct 2026).
06 / note
Severity beats averages
A high aggregate pass rate never excuses an unacceptable high-impact failure. Repeat high-risk and unstable cases more often.
| Level | Meaning |
|---|---|
| Low | Minor inconvenience; easy to notice and correct |
| Medium | Misleading or wasteful; generally detectable before material harm |
| High | Can materially distort product decisions or undermine trust |
| Critical | Can directly cause unsafe, irreversible or externally consequential action; treat as a stop condition unless explicitly resolved |
GOING DEEPER
Decision-boundary regression cases
| Evidence | Expected behavior |
|---|---|
| Unknown value | Keep it unknown; never silently fill with zero |
| Explicit 0 | Use 0 only when the value is positively established |
| Tentative 10% | Do not present as approved or authoritative |
| Approved 10% | Represent as a concrete approved target, with evidence |
| Observed 8%, target unknown | Keep observed performance separate from a target |
| Previously proposed target withdrawn | Return the target to unknown unless a replacement is established |
Product-oriented failure measures
| Measure | What it detects |
|---|---|
| Miss rate | Material changes or expected actions the system silently failed to identify |
| Unsupported-claim rate | Invented, exaggerated or poorly sourced claims |
| Uncertainty-loss rate | Pending or unknown information upgraded to certainty |
| Unnecessary-review rate | Avoidable human workload from duplicate or low-value requests |
Silent failures matter: A visible wrong answer is easier to notice than a consequential case the AI never flagged. Sample no-action outputs and ask humans whether the inaction was appropriate; track confirmed false negatives and their severity.
Ownership and troubleshooting: PMs set expected behavior, severity and release bars. Engineering implements repeatable runs and instrumentation; AI can propose cases and analyses; accountable humans approve expectations and acceptable risk. Trace evidence → context/retrieval → interpretation → deterministic validation → classification → action before blaming the prompt.