A failed release, a lower eval score, and a runtime warning all lived in different places.
The information already existed. It was just spread across deployments, commits, evals, analytics, logs, and the live products themselves. I built Project Health around my own projects as a practical test bed for a pattern I would expect to matter inside a product team: bring the signals together, then help someone decide whether a signal is a real product problem, something to watch, or simply missing information.
A dashboard that brings product signals into one place so teams can quickly see what needs attention first.
Project Health brings together product quality, delivery, user impact, and missing signals to help answer whether anything should block or delay a release.
Three product decisions shaped it.
Keep different kinds of health separate
Product quality, delivery, incidents, and usage answer different questions. Project Health keeps them distinct instead of collapsing them into one score.
Lead with what needs attention
The overview brings important failures, changes, and open questions forward so a team can decide where to look first.
Unknown is not healthy
If Project Health cannot evaluate something, it shows that gap instead of quietly treating missing data as healthy.
What does the failure mean?
A failed AI check does not always mean the same thing. One failure may point to a real product bug. Another may show that the test expectation is wrong or that the available evidence is incomplete. Project Health shows what was observed, what was expected, and why the difference matters before I decide what should change.
One scenario failed in a way that mattered
Most of State's AI quality checks passed, but one scenario failed in a way that could change what a user believed was true.
State knew the Northstar pilot was paused, but its answer did not clearly preserve that the reason for the pause was still unknown.
A user could mistake an inferred explanation for maintained project truth. Project Health put the failed scenario at the top of the quality view and pointed the next review at the scenario evidence.
Review whether the product behavior or the eval contract is wrong, make the smallest justified change, then rerun the affected suite.
Delivery and quality can also diverge. When one portfolio deployment failed while the previous version stayed live, Project Health flagged the release for engineering attention without marking the product's quality as failing.
Investigate starts with the specific problem
When a signal needs more context, Investigate gathers the relevant evidence and produces a handoff that shows what happened, what is known, what is still uncertain, and what should happen next. The AI helps investigate; it does not make the decision.
QA changed the product
Activity became less noisy
Repeated failures are grouped into one incident, so the activity feed is easier to scan.
Missing data stopped looking healthy
The dashboard now calls out monitoring gaps, stale quality results, and contradictions between signals instead of smoothing them over.
How I split the work
- framed the problem and chose which signals mattered
- designed the information hierarchy
- reviewed failed evals and decided whether product behavior or the test should change
- used QA findings to reshape the product
- implementation and code iteration
- testing and regression coverage
- drafting and interface iteration
What held up
The dashboard became useful when it stopped trying to summarize health and started helping me decide where to look next.