Production Reliability & Observability

Trace the request forward to the first observable divergence.

01 / flow

The production path

Browser + UIAPI validationContext + modelDeterministic checksDatabase + response

02 / table

Where failures hide

Boundary	Typical failure	First evidence
Frontend	Stale view, wrong API base, duplicate submit	Network, console, rendered state
API	Bad validation, unsafe retry	Request ID, route logs, response
Model	Timeout, malformed or unsupported output	Provider timing, model ID, tokens
Database	Constraint failure, stale version	Transaction, schema, migration
Cross-boundary	Local differs from deployed	Environment, commit, target, CORS

03 / cards

Trace / log / metric

Trace

What happened across this specific request, in order?

Log

Which technical event, warning, or error occurred?

Metric

How often, how slow, how costly, how widespread?

04 / table

Safer recovery

Situation	Avoid	Prefer
Model fails after evidence saved	Ask for duplicate submission	Retry against same durable evidence
Stale review races newer state	Overwrite newer facts	Version conflict and re-review
Malformed references	Write partial records	Reject deterministically
Provider unavailable	Pretend success	Safe failure without changing authority

05 / list

Production checklist

Record deployed commit and model ID
Correlate request IDs across boundaries
Measure latency by component, not guesswork
Retain authoritative data across provider outages
Test retries and idempotency
Turn meaningful incidents into regression coverage

GOING DEEPER

Latency, durability and safe recovery

Where time goes

Segment	Look for
Frontend / network	Request duplication, slow connection, stale client render
API orchestration	Validation and coordination, serialization overhead
Model provider	Inference latency, token volume, model/version changes
Validation / persistence	Schema checks, failed transactions, version conflicts
Rehydration	Authoritative response and deterministic UI refresh, not arbitrary sleeps

Safe recovery examples

Failure	Better pattern
Evidence saved; model call fails	Keep durable Evidence and retry interpretation only
Old review races with newer state	Reject stale proposals using version checks
Shape-valid output has bad references	Reject deterministically rather than persist partial records
Frontend waits for update	Use server response and authoritative revalidation, not sleep(300ms)
Provider unavailable	Safe failure; preserve truth and offer controlled retry

Production checklist: Record deployment commit, model identifier, safe request/correlation IDs, timings, categorized errors, token usage, API base, provider keys, CORS, database target, schema/migrations, and durable records that must survive restarts. Protect idempotency so retries do not duplicate side effects.

PRACTICAL FRAMEWORK / AI PRODUCT WORK

Behavior contracts and upstream changes

Define a behavior contract before selecting a model: task-level quality bar, maximum critical-error severity, p95 latency, cost per successful outcome, required tool support, output structure, permission boundary, and recovery behavior.

Change to watch	What to verify
Model alias or provider preset changes	Record the resolved model/configuration; rerun critical evals, tool calls, refusal/uncertainty cases and latency/cost checks.
Retrieval or data updates	Test freshness, provenance, access filtering and contradictory sources.
No known deployment or announcement	Run small scheduled behavioral canaries; compare outcomes against a known baseline and alert on meaningful drift.

Release rule: A green server status is not evidence that AI behavior is unchanged. Pin what can be pinned, measure the rest, and have a named owner for rollback or constrained fallback.