Durable AI Knowledge
AI product principles that hold as models, tools and headlines change.
For current patterns and vocabulary, see AI Professional Edge →
Start with the problem, not the model
AI should earn its place in the workflow
Start with the person’s goal, the current workflow, and evidence of where it breaks. Compare AI with simpler options: clearer rules, conventional software, deterministic automation, or process changes.
AI adds variability, evaluation work, security exposure, and operating cost. It should solve a problem those alternatives cannot solve as well.
Durable question: What is the smallest reliable improvement, and does it actually require AI?
The system matters more than the demo
The model is only one part of the product
A stronger model can improve an AI product, but reliability still depends on context, retrieval, tools, permissions, state, application logic, checks, and recovery paths.
Durable question: Where does the model end and the system begin?
More agents are not automatically a better system
A difficult task may benefit from splitting work across specialized or isolated agents, but orchestration adds cost, latency, coordination failures, and more behavior to evaluate. Use the simplest architecture that reliably completes the job, and make each additional agent earn its complexity through measurable improvement.
Durable question: What does orchestration improve enough to justify the extra cost and failure surface?
Flexible reasoning belongs inside firm boundaries
AI is useful when the right path cannot be fully written as rules. Important limits should still be enforced outside the model when possible.
Budgets, permissions, approval requirements, access scopes, and irreversible actions should not depend on the model remembering them.
Automation can move the bottleneck instead of removing it
When AI makes one step much faster, inspect what happens immediately afterward. Review, exception handling, approvals, investigation, or follow-up can become the new constraint.
Measure the whole workflow, not just the automated step.
Durable question: What work becomes the new bottleneck if this step gets dramatically faster?
Keep requirements separate from providers
AI products may increasingly specify the behavior a task requires and treat the model as the current implementation of that requirement. Reliability, speed, cost, tool support, and risk tolerance can outlast any particular model family. Keeping the authoritative workflow, permissions, and user experience separable from the provider makes replacements less disruptive.
Durable question: What does this workflow need to be reliably true, regardless of which model provides it?
Evidence matters more than confidence
Grounding beats polished wording
AI can sound certain while using incomplete, stale, or wrong information. Real work needs clear source quality, provenance, and a way to distinguish authoritative information from everything else.
Durable question: What is authoritative, and how does the AI know?
Preserve uncertainty
Unknown, inferred, proposed, and confirmed information are not interchangeable. Products should keep those distinctions visible when they affect a decision.
Data meaning and access are product requirements
Correct data can still produce wrong conclusions when important definitions are inconsistent. Establish what business terms mean, who owns the data, who may access it, and how freshness and provenance are checked.
Retrieved content is not instruction authority
Documents, websites, and tool responses can contain malicious or mistaken instructions. Treat them as untrusted input, not permission to override the product’s rules. Sensitive information also needs deliberate consent, minimization, retention, deletion, and logging boundaries.
Durable question: What data may this system read, remember, reveal, or act on, and who authorized that use?
Economic authority is still authority
Agents consume resources, not just tokens
Agents may use model calls, retrieval, compute, human review, paid APIs, or machine payments to complete a goal. Once money becomes another tool, spending behavior becomes part of product quality.
A task can succeed and still be badly designed if it costs far more than necessary.
Durable question: What does one successful outcome cost?
Count the entire cost of ownership
Include integration, evaluation, monitoring, review, error recovery, vendor dependency, and support costs when comparing AI with simpler alternatives. Optimize for value delivered, not tokens consumed or activity produced.
Machine payments need the same governance as other consequential tools
Useful controls include total budgets, per-action limits, approved services, human approval thresholds, and transaction logs. The agent can optimize inside those constraints without receiving unlimited financial authority.
Quality needs repeatable evidence
Evals turn failures into learning
One impressive answer proves very little. Representative scenarios, known failures, edge cases, and expected behaviors make AI quality repeatable enough to improve.
Product habit: Save the failure. Find where it started. Change one thing. Run it again.
Define the quality bar before release
Task success alone is not enough. Choose measures that reflect the job: factual accuracy, correct actions, safety, latency, consistency, recovery, and user effort. Evaluate ordinary tasks alongside difficult cases, and break results down by failure severity and affected user groups rather than hiding them in one average.
Set explicit release thresholds and escalation rules. A high aggregate pass rate cannot excuse a small number of severe failures. Include accessibility and uneven performance across languages or contexts where relevant.
Durable question: Which failures would make this unacceptable to ship, even if most tests pass?
Human review is a control only if it works
Human-in-the-loop is not automatically safe. Reviewers need the time, context, and authority to disagree, and too much review turns into fatigue and rubber-stamping. Check whether review actually catches problems before counting it as a safeguard.
Measure whether AI preserves human capability
AI can improve a person’s output on a bounded creative task while making outputs across people more alike. In one story-writing experiment, AI-generated ideas improved ratings of individual stories but increased similarity between them. That is a task-specific result, not proof that all AI use narrows creativity. Read the study.
Also check whether AI reduces chances to practice, weakens independent review, shifts cleanup onto people, or turns saved time into more workload. Long-term skill effects remain uncertain, so monitor them rather than assume them.
Durable question: What does AI help people produce—and what might it displace them from practicing?
Stable software does not imply stable AI behavior
An AI product can change meaningfully even when its application code, prompt, and API call stay the same. Models, routing, provider defaults, retrieval systems, and other upstream dependencies can alter behavior underneath the product.
Use event-triggered testing for changes you know about and periodic behavioral checks for changes you may not know happened.
Durable question: What could change upstream without us deploying anything, and how would we detect its effect?
People and adoption matter as much as the technology
Make uncertainty and control understandable
People generally need to know what happened, what evidence was used, what the system actually did, what remains uncertain, and how to correct or challenge it. Ask for clarification when ambiguity changes the outcome. Offer review, recovery, and meaningful override instead of relying on polished language to create confidence.
Transparency works best in layers rather than all-or-nothing: status → understandable activity → evidence and actions → specialist traces and logs. Most people need only the outcome and current status. Someone investigating a surprising result needs the evidence, actions, and sources. Specialists may need the full trace. Raw model reasoning is rarely the right default form of transparency.
Keep generated experiences inside understandable structure
AI can personalize content, recommendations, status, or task-specific controls without regenerating the whole interface unpredictably. Stable interaction patterns help people form a mental model of the product while the content inside those patterns adapts.
Durable question: What should remain predictable even when AI-generated content changes?
Measure adoption as workflow change
A pilot can pass technical tests and still fail in practice. Training, incentives, role clarity, support, user trust, and downstream capacity determine whether people can use it consistently. Measure sustained outcomes and burden, not just clicks or generated outputs.
Durable question: Is this making people more capable and the workflow better, or merely adding another system to manage?
State and observability keep agents trustworthy
Memory is not automatically a source of truth
Conversation history, retrieved facts, inferred state, and authoritative state are different things. Important facts may need explicit ownership, validation, timestamps, history, or human authorization.
Production systems need accountable owners
Name who monitors behavior, handles incidents, approves changes, responds to user reports, and decides when to pause or roll back. Use staged rollout and feedback-to-evaluation loops so learning continues after launch.
AI can fail without a software error
Traditional telemetry shows whether an API returned, a tool ran, or latency stayed within bounds. An AI system can still fail semantically: misunderstand the task, rely on weak evidence, choose the wrong tool, frustrate the user, expose sensitive information, cross an authority boundary, stop too early, or successfully complete the wrong job.
Durable question: Which important failures could happen while every service still reports healthy?
Observe behavior, not only system health
Where practical, make important behaviors searchable, measurable, or evaluable from production traces rather than relying only on user complaints or the model’s own explanation.
Three practices answer different questions. Evals deliberately test expected behavior. Observability shows what actually happened in individual or aggregated runs. Monitoring watches production over time for unacceptable change or recurring patterns. What observation and monitoring find should become new evals.
Durable question: What can we observe directly instead of trusting the model to explain itself?
Design an ongoing worker as a complete workflow
A useful way to scope an agent is to walk through eight linked decisions:
- Goal: What outcome is the worker responsible for?
- Context: What facts, memory, and sources may it use, and which are authoritative?
- Tools: What systems can it read or act through?
- Authority: What may it decide, and what requires human approval?
- Execution: How does it start, persist progress, stop, and recover after interruption?
- Observation: What can a person inspect about its evidence and actions?
- Evaluation: How will we test both outcomes and the path taken to reach them?
- Intervention: How can a person correct, pause, or take over?
Together, these decisions make the worker’s boundary and operating behavior reviewable. They also connect the AI feature to the surrounding workflow, state, permissions, and recovery paths.
Questions I keep coming back to
- What job are we actually trying to improve, and is AI necessary?
- What information is authoritative?
- What should the AI reason about, and what should software enforce?
- What can the AI see, decide, and do?
- What happens when it is uncertain or wrong?
- How will we notice and recover?
- What does one successful outcome cost?
- What evidence would justify giving the system more autonomy?
- What behavior must remain stable even if the underlying model changes?
- Who owns the system once it is live, and what would make us stop it?
- Can people recognize, challenge, and correct an uncertain answer or action?
The point
Models and products will keep changing. The durable skill is learning how to make good decisions about evidence, authority, reliability, cost, and value while they do.