A failed release, a lower eval score, and a runtime warning all lived in different places.
As I built and maintained several prototypes, checking on them meant opening too many separate tools and pages. I wanted one place to see failures across projects and jump into the relevant tool when something needed attention.
The information already existed in deployments, evals, analytics, and logs. The product work was bringing it together without losing where each signal came from.
One starting point for checking my projects.
I chose which signals mattered and how to order them. The overview leads with what needs attention, so I can decide where to look first, and it answers two questions.
What needs attention?
See the project’s current goal, known concerns, next decision, and changes since your last visit.
Is anything blocking release?
Review quality failures, delivery status, whether users are likely affected, and missing checks before deciding whether to ship.
Keep different kinds of health separate. Product quality, delivery, incidents, and usage answer different questions. Project Health keeps them distinct instead of collapsing them into one score.
Unknown is not healthy. If Project Health cannot evaluate something, it shows that gap instead of treating missing information as a healthy result. Signals link back to the tools they came from.
What does the failure mean?
A failed AI check does not always mean the same thing. It may point to a real product bug, a test expectation that is wrong, or incomplete evidence. Project Health shows what was observed and what was expected; deciding what should change is still my job.
What failed. One saved run of State’s answer-quality checks passed 9 of 10. The miss: State knew the pilot in its fictional example project, Northstar, was paused, but its answer didn’t say that the reason for the pause wasn’t on record. A user could have mistaken an inferred explanation for maintained project truth.
What Project Health showed. It put the failed scenario at the top of the quality view, with the failed check, the expected behavior, and the scenario evidence. It did not diagnose the cause.
What it left to me. Whether State’s behavior or the test expectation needed to change.
What happened next. The same scenario had passed in the three earlier saved runs, and it passed again in the next two. One failure among passing runs is exactly when a dashboard should show the evidence and leave the judgment to a person, rather than declare a regression or wave it away.
Delivery and quality are judged separately. If a release fails to deploy while the previous version stays live, Project Health flags the release for engineering and reports no confirmed impact on users, without marking product quality as failing.
Investigate starts with the specific problem
When a signal needs more context, Investigate drafts a handoff: what happened, what is known, what is still uncertain, and what should happen next. It takes one of two paths:
- A failed eval. Rules use the recorded results to assemble the explanation, without another model call.
- A release or deployment problem. A bounded, read-only AI investigation examines the available evidence, such as deployment status and logs, and drafts the handoff.
Neither path changes code, configuration, or deployments. A person makes the decision.
QA changed the product
The dashboard said “Healthy” when it didn’t know.
What QA found. In an October production check, two projects read “Everything looks healthy right now” without the evidence to back it:
- A slow first load timed out before State’s quality data arrived.
- Authority Lab, another of my prototypes, had no quality checks connected at all.
What changed. A signal that fails to load now marks the project Incomplete, names what couldn’t be checked, and offers a retry. A project with nothing measuring quality says so instead of calling itself healthy. Authority Lab later gained automated tests, and its status now reads “Healthy on what’s measured”, naming the one thing still unmeasured: the full lifecycle walkthrough.
What it taught me. A dashboard shouldn’t reassure someone when it lacks the evidence to justify reassurance.
| Before | After |
|---|---|
| Each failed deployment attempt appeared as its own “Deployment failed” entry. | Failed attempts a few minutes apart are grouped into one incident that shows how many attempts failed. |
What held up
Bringing project information into one dashboard was only part of the problem. The harder part was making sure that missing information, outdated results, and failed checks didn’t create a misleading picture of how a project was doing. Project Health became as much about knowing when not to trust a status as knowing where to investigate next.