AI Product Metrics & Experimentation

Separate model quality from user outcome and business impact.

01 / table

Three levels of measurement

Level	Question	Examples
AI quality	Did the system behave correctly?	Task success, groundedness, severe failures
User outcome	Did it make the job easier?	Resolution time, correction effort, completion
Business outcome	Did it create net value?	Cost per resolved case, capacity, avoided rework

02 / cards

Metric distinctions

Primary vs guardrail

A main goal plus cost, latency, severe failures and review burden.

Leading vs lagging

Early use signals versus later outcomes.

Rate vs count

Always name the denominator and time window.

Segments

Task complexity, source coverage, risk and user group.

03 / table

Evidence methods

Method	Strength / caution
A/B test	Randomized, but watch assignment and spillover
Holdout	Compare outcomes against a continuing baseline
Before / after	Useful, but confounded by seasonal and task shifts
Qualitative	Explains mechanisms; not causal proof alone

04 / note

Worked example

90% acceptable support drafts do not prove value. Compare resolution, corrections, customer satisfaction, escalation and all-in cost to baseline.

05 / list

Ask before claiming impact

What is the baseline and denominator?
What change counts as meaningful?
Which severe outcome must not trade for speed?
What else could explain the same movement?

GOING DEEPER

When a quality score misleads

90% drafts, no proven value: A support assistant produces 90% acceptable drafts. Compare assisted and baseline cases on actual resolution, total agent effort including corrections, escalation, customer satisfaction and cost per case. If resolution is unchanged and correction time rises, the quality score has not established business value.

Causal evidence cautions

Method	Guardrail
A/B test	Randomize eligible users or tasks; inspect assignment, contamination/spillover, sample size and measurement
Holdout	Maintain a representative baseline to estimate incremental impact
Before vs after	Allow for seasonality, case-mix changes and other concurrent interventions
Qualitative interviews	Explain mechanism and frustration, but not causal magnitude on their own

Metric definition: Always document baseline, denominator, eligibility, exclusions, segment, window, main success measure and guardrails. Watch severe failures, review burden, cost, latency and accessibility so faster output does not conceal worse outcomes.

PRACTICAL FRAMEWORK / AI PRODUCT WORK

Quality × value: decide what earns investment

Use separate evidence for AI task quality and real workflow value. The matrix is a decision aid, not permission to waive safety, privacy or compliance requirements.

AI quality	Product value	Likely next decision
Strong	Strong	Expand carefully; monitor quality, cost and adoption by segment.
Weak	Strong	Fix quality and constrain exposure; block launch or expansion for critical failures.
Strong	Weak	Revisit the job, workflow fit, discovery and adoption rather than optimizing eval scores.
Weak	Weak	Stop or fundamentally redesign the proposed feature.

Non-negotiable: Strong usage never excuses critical safety failures; strong evals never prove a feature is needed.

PRACTICAL FRAMEWORK / BUSINESS CASE

Is it worth it? A simple AI business case

Start from what the problem costs today, not from what the model can do. Every number below is an assumption until measured: label it as one, and keep the case simple enough that someone can check it.

Line	What to estimate	Watch for
Baseline pain	Volume × time per case × loaded cost, plus the cost of errors and rework in today's workflow.	No baseline, no case. Measure it before choosing a technology.
Value	Time saved that becomes useful capacity, errors avoided, faster cycle time, revenue or retention effects.	Saved time that only turns into more work is not value.
Run cost	Model and API, retrieval and infrastructure, priced per successful outcome.	Retries, failures and long inputs; the cost per call understates it.
Hidden cost	Human review, exception handling, evals and monitoring, upkeep as models and prompts change.	Review burden often erases the headline saving.
Risk	Impact × likelihood of a wrong answer or action, compliance exposure, loss of trust.	One critical failure can outweigh a year of savings.

Net value / month = value − run cost − hidden costPayback = build cost ÷ net value per monthCompare with the simplest alternative: rules, automation or a process change

Make the recommendation falsifiable: state the decision (build, limited pilot, buy, or don't), the two or three assumptions it is most sensitive to, and the evidence that would change it. A case that cannot be wrong cannot guide a decision.