The 300 Second Write

A write that normally takes two seconds took three hundred.

Same code.

Same request.

Same everything.

The GPU box on the other side of my network had wedged. It still answered health checks, so my router kept sending it work, and every request sat there for five minutes waiting on a machine that had quietly stopped doing its job. The fix was a fail-fast: if the embedding backend takes too long to answer, give up immediately and fall back to a local model. Writing it took an evening. Realizing I needed it took a wedged GPU and a lot of staring at a progress bar.

I have been working through an AI product management playbook, and the chapters on model selection, launch metrics, and production keep describing incidents I already paid for at home. So this issue teaches all three, with my receipts attached.

There is no best model

The model selection chapter opens with the whole lesson in its first line: there is no universally best model. Every model that tops one leaderboard loses somewhere that matters to somebody.

The subtler point comes right after. The unit under test is bigger than a model. It is the whole configuration: the model plus the prompt, the context strategy, the retrieval setup, the reasoning budget, the tools, the retry policy. I have watched the same model swing from useless to solid because the harness around it changed. If you swap any piece, you are testing a new thing, and your old numbers are dead.

So you score configurations, on one eval suite, across the axes that actually bite: task quality, reliability across repeated trials, latency, cost, tool-use success, structured-output compliance, and whatever privacy constraints you carry. The sneaky one is cost per successful task. A cheap model that fails a third of the time charges you for every retry, plus the human minutes someone spends noticing the failure.

The chapter's worked example puts three configurations on the same suite. Configuration A succeeds 95 percent of the time at 74 cents and 18 seconds. B succeeds 93 percent at 21 cents and 5 seconds. C succeeds 84 percent at 4 cents and 2 seconds.

None of these is the right answer.

A pick only becomes right next to a user outcome.

If a wrong answer costs your user an afternoon, you pay for A. If they can eyeball the output in two seconds, C prints money. The PM job is deciding which tradeoff serves the person on the other end, then writing that decision down so nobody relitigates it every sprint.

My answer at home is a portfolio. A frontier model gets the judgment work: planning, review, anything with tradeoffs in it. Cheap local models on a used GPU get retrieval, classification, and the routine summarization that would bankrupt me at frontier prices. A router classifies each turn and decides which side handles it, and when the turn is ambiguous or something is failing, it falls back to local, because on my stack the local side is also the private side.

The model can be great while the product dies

The launch chapter has one job: separate AI quality from product success. A model can score 95 on your evals while the feature quietly dies in the workflow, and a mediocre model can carry a product if the loop around it is right.

So you instrument four layers. System metrics at the bottom: latency, cost, uptime, error rate. AI quality above that: task success, groundedness, hallucination rate, tool success. Then product: activation, repeat usage, retention, task completion. Then business: revenue, cost savings, cycle time. Most teams wire up the bottom two layers and declare victory, because those are the layers their own dashboards already speak.

The chapter's best tool is a value chain. Before AI creates business impact, a request has to survive eight stages: the system is available, the user attempts it, the call technically completes, the output is actually good, the user accepts it, the surrounding workflow finishes, the user comes back, and the business feels it.

Every stage leaks.

You only find the leak you measure.

Two metrics in this chapter are brutal enough to memorize. Redo rate: how often does the user redo work the AI already did? Human minutes per successful task: how much hidden labor remains after the automation? On my own tooling, redo rate is the honest one. When a generation lands close but wrong, I regenerate and move on. The system logs a completed task. What actually happened is that I paid twice.

The distance between demo works and product works

The production chapter opens by naming the gap everyone falls into: the distance between demo works and product works. Everything in it is a tax you pay to close that distance. Retries, fallbacks, degraded modes. Logs and traces for every model and tool call. Versioning for models, prompts, datasets, and schemas, because a prompt edit is a release whether you treat it like one or not. Regression evals in CI so a quality drop gets caught by a machine instead of a customer.

The chapter's readiness review is just a list of questions, and it reads like an interrogation. What happens when the model provider is unavailable? What happens when retrieval fails? What cannot be logged? Can you reproduce a bad result? Can you roll back? Who receives the alert?

My 300 second write was a failing grade on question one. The router had a happy path and a health check, with nothing in between, and a health check only proves a process is breathing. It says nothing about whether the work comes back. The fail-fast turned an outage I kept discovering into a degraded mode I had designed, and that swap is most of what production AI means.

A health check proves the process is breathing. It says nothing about whether the work comes back.

The playbook arranges all of this as one loop: problem, value hypothesis, task definition, system design, evaluation, product experience, production, outcomes, failures, new evals, improvement. The turn people skip is the last one. Failures are supposed to flow back in as new eval cases, so the same failure cannot cost you twice.

My wedged GPU is a regression test now. The router gets exercised against a backend that accepts the connection and never answers, and if the fail-fast ever rots, a machine tells me before a five minute progress bar does.

Pick the configuration on purpose.

Measure past the click.

Feed every incident back into the loop.

That is three chapters of an AI PM playbook, and one evening of my life I am never getting back.