Context Switch

Where AI Earned Its Place in State

Before AI integration, I worked through where a model would genuinely add value, where ordinary software should stay in control, and what the tradeoffs would cost in latency, reliability, and complexity. Building State changed which constraints mattered most.

Before Architecture and cost-planning exerciseAfter Working AI-integrated productFocus Complexity, model choice, latency & cost

In shortBuilding State moved my question from what an LLM call would cost to which parts of the workflow need a model at all, which led to a simpler system.

Before I built it

I deliberately avoided pricing an architecture I hadn't chosen yet.

The original exercise started with the product, not model pricing. I expected State to keep its data and rules in the application, and use AI only where interpretation or synthesis added value. For the first build, I used SQLite rather than adding infrastructure I didn't need yet.

Original cost decisionMeasure real usage before setting limits or estimating monthly cost.

The pre-build work focused on what to measure rather than trying to predict exact costs too early.

What actually got built

The working State product kept more work deterministic than I initially expected.

Kept deterministicTruth, authority, and context selection

Evidence, Current State, Reviews, Questions, History, versions, and authority checks live outside the model. State also chooses the bounded context Ask receives. Software and people control consequential transitions.

Used AI forInterpretation and synthesis

Claude Haiku 4.5 interprets new evidence and generates Ask responses from the context State prepares. I kept the model behind an application interface so the product was not tied to one provider.

InfrastructureSimple learning-build stack

The build used a FastAPI backend and SQLite initially, with the frontend deployed separately. The goal was enough real infrastructure to test the product thesis without pretending it was an enterprise architecture.

What changed

The biggest practical constraint was not the one I could model most neatly on a spreadsheet.

BeforeCost looked like the obvious quantity to model.

I planned to track calls, input/output tokens, model choice, retries, and non-model infrastructure, then stress-test low/base/high usage rather than optimize a single estimate.

AfterLatency became painfully visible.

An early Ask could take roughly 35–38 seconds, and instrumentation showed provider inference dominated the request. That changed the work from abstract architecture planning into model, context, output-budget, and interaction-design tradeoffs.

Model choiceThe better product choice was the one that fit the tested workflow.

Haiku 4.5 remained the better fit for this Ask path because the GPT-4.1 mini experiment was slower in the workflow I actually measured. I kept the model behind an application interface so that decision stayed reversible instead of turning a provider choice into product architecture.

Context + outputOptimize waste, not the room the model needs to succeed.

I reduced unnecessary context and kept Ask focused on the records it needed. A lower interpretation cap looked like an easy cost and latency win, but dense notes started truncating and failing, so I raised the budget again. The tradeoff became clearer: remove context the model does not need, but preserve enough output room for reliable interpretation.

Estimate vs. reality

Some assumptions held. Others became more nuanced once the product existed.

Keep the architecture simpleHeld up. SQLite and a straightforward application layer were enough to learn the important product lessons before introducing heavier infrastructure.
Use AI selectivelyHeld up. Deterministic rules became more important after AI was added, not less.
Measure model usage before claiming costStill the right call. I do not have enough representative production usage to present an observed monthly State cost as if it were real operating data.
Cost would be the main optimization pressureChanged. In the working State product, latency and reliability were more immediately damaging to the user experience, while many of the same fixes also reduce token use and unnecessary calls.
What I learned

Architecture, cost, latency, and reliability are one product decision, not four separate exercises.

The biggest shift was moving from “What will an LLM call cost?” to “Which parts of this workflow deserve an LLM call at all, what context does it need, how reliable must it be, and what will the user experience while it runs?” That framing led to a simpler system and made the cost model more useful.