Making AI useful takes human judgment.

Before changing a workflow, I map how it works today, where it breaks down, who feels the pain, and what would count as a real improvement.

Data readiness: Before adding AI, check whether the needed records exist, are accurate and current, share consistent identifiers, have clear owners, and can be accessed lawfully through maintained integrations. If the sources disagree or access is unreliable, a better model will not repair the underlying problem.

A real problem does not automatically need AI. I compare AI with simpler options, make the tradeoffs visible, and add autonomy only when it earns the extra complexity.

Vendor decision: Compare options against the actual workflow and a non-AI baseline. Alongside quality and price, check privacy and data terms, service reliability, integration effort, auditability, support, portability, and how hard it would be to leave. A cheaper demo is not always the lower-cost product.

Seen in: State: “Where AI Earned Its Place”

When scoping or debugging, I check what the model sees, how it gets information and tools, where state lives, and what the surrounding software controls.

Diagnose the layer: When output is wrong, ask whether the model lacked facts (retrieval), followed the wrong procedure (instructions or skills), or lacked a safe action (tools and permissions). A different model rarely fixes the first two.

Seen in: State: how it works

I define good behavior before launch, test normal and high-risk cases, and trace failures to the right layer. Reliability also means planning for bad model output, missing retrieval, broken tools, and other predictable failure modes.

Severity over averages: A high pass rate never excuses an unacceptable high-impact failure. Set release bars by severity, and turn every production failure into a regression case.

Seen in: Project Health: AI evals

For consequential work, I make decision rights explicit. AI can interpret, software can enforce rules, and people can authorize the decisions that need human judgment. Higher-impact automation should require stronger evidence and clearer recovery paths.

Evidence bar: The more an action affects people, money or records, the more evidence it needs before it is automated, and the easier it must be to reverse.

Seen in: State: authority boundary

I design so users can see what the AI is doing, what they still own, and how to correct or challenge it. Good AI UX should make review and recovery easier, not hide uncertainty or quietly push extra work onto people.

Human capability: AI can help individuals produce stronger work while nudging a group toward more similar ideas. Design for useful assistance without making AI the first or only source of ideas: preserve room for independent thinking, meaningful review, and skill practice. In pilots, check output quality and speed alongside variety of ideas, reviewer effort, people’s ability to challenge suggestions, and whether saved time becomes useful capacity or simply more work.

Seen in: Tastemake: editable taste model

A working prototype only matters if people can actually use and sustain it. I treat adoption problems as signals about the product, workflow, ownership or role change, not as something training alone will fix.

Pilot to production: A pilot shows whether an approach has promise; scaling requires a named owner, acceptable quality thresholds, a rollout and rollback plan, user support, budget, training, and a way to turn incidents into better tests. Check real usage and review workload after launch, not just pilot success.

I connect AI quality to normal product outcomes: useful metrics, instrumentation of the right behavior, checks on what users actually do, and verification of AI-assisted analysis before it drives decisions.

Experimentation: An eval score says how the system performs on tested tasks, not whether it caused better outcomes. Define a baseline and primary outcome, compare groups or time periods carefully, and watch guardrails such as cost, latency, errors, and human review. A/B tests and holdouts help when practical; otherwise, state the uncertainty and possible confounders.

Seen in: Project Health

Client work starts with the business problem, not an AI pitch. I explain tradeoffs plainly, surface missing evidence, set realistic expectations, and turn technical uncertainty into decisions a client can act on.

Evaluation as a client capability: For client work, I would turn eval evidence into launch and scale recommendations, then help the client maintain a quality loop after launch: production failures become new eval cases, which inform regression testing and updated thresholds and decisions.

Seen in: Meridian; Legal AI opportunity

I keep up with changes when they affect what is feasible, reliable, affordable, or practical to adopt. The goal is not to follow every launch. It is to know when new information should change a product recommendation.

I use AI differently across exploration, building, and review, then check the output against the product rules and evidence that matter for the task.

AI Power-User Playbook: Practical techniques for researching, scoping, evaluating, and working with AI. Includes reusable prompts, examples, and checklists.