Library / 02 AI Opportunity & Product Judgment
Reviewed Oct 2026Evaluating AI Vendors & Platforms
Compare fit to a real workflow, not how polished the demonstration looks.
01 / flow
Fair selection process
Baseline workflowNeeds + constraintsSame trial casesRisk + operating reviewCost per outcomeDecision + exit plan
02 / table
Option fit
| Approach | Best case |
|---|---|
| Buy | Common workflows where a product already covers most needs |
| Boost | A useful current platform needs one bounded AI capability |
| Build | Distinctive workflow, controls or data justify ownership |
03 / table
Compare on equivalent evidence
| Dimension | Inspect |
|---|---|
| Capability | Representative tasks, edge cases, groundedness and integrations |
| Security | Training use, retention, subprocessors, access and audit |
| Operations | SLA, support, incidents, availability and exit |
| Economics | Seats, setup, usage, review and maintenance |
| Portability | Data export, prompts, contracts and replacement effort |
04 / note
Worked example
Cheaper calls may lose to a more expensive vendor with audit logs and reliable escalation for high-impact record changes.
05 / list
POC rules
- Set acceptance criteria before demonstrations
- Use the same realistic sources and high-risk cases
- Record missing features instead of changing the test
- Count human review and support costs
- Plan replacement if terms, models or pricing change
GOING DEEPER
Make the vendor tradeoff visible
Vendor A vs Vendor B: Vendor A is cheaper per call. Vendor B has similar measured accuracy but stronger audit logs and escalation support. For low-risk drafting, A may offer better value. For consequential record changes, B’s controls and support may justify the premium. The best model is not necessarily the best vendor or product.
Fair comparison checklist
| Dimension | Require comparable proof |
|---|---|
| Workflow and edge cases | Same real tasks, source inputs, acceptance bar and failure conditions |
| Authority and privacy | Data use, retention, subprocessors, user scopes, audit, deletion and contractual commitments |
| Operations | Uptime/support terms, incident notification, region availability, human escalation |
| Cost per successful outcome | Licenses, model usage, implementation, integration, review, cleanup and support |
| Portability | Export formats, ownership of data/prompts, vendor-model switching and exit plan |
Do not move the goalposts: Write the quality and risk acceptance thresholds before seeing a polished demo. Record unsupported requirements; compare real workflow improvement and operational burden, not marketing-feature counts.