Own the Definition of Good

For about three years I built AI things by feel. Write a prompt, watch it nail the demo, watch it fall over in week two, patch it, repeat. Every fix was a vibe and every regression was a surprise.

Then I started running everything through evals, and the job changed shape. This is the part of AI product work nobody demos on stage, and it is the closest thing the field has to a center of gravity. If you are learning AI product management, this is the chapter to over-invest in.

The core move is simple to say: own the definition of good, then turn that definition into a system that can be run again and again.

Strip the jargon and an eval is a test with opinions. A task is one test case. A trial is one attempt at that task. A dataset is a pile of tasks. A grader is the logic that decides whether the attempt was any good. The trace is the path the system took, the outcome is where it actually ended up, and the harness is all the plumbing around the task. That is nearly the whole vocabulary.

The dataset is where the leverage lives. The good ones get called golden sets, and they are assembled from real user cases, historical examples, known failures, edge cases, adversarial inputs, and the scenarios where a miss actually costs you money or trust. A golden set built from imagined usage grades a product that does not exist.

Graders come in a hierarchy, cheapest first. Deterministic checks lead, because string equality never hallucinates. Rules-based checks come next. Then an LLM judging another LLM, which sounds circular until you calibrate the judge against human labels and find it agrees with you most of the time. Humans sit at the top, because human review is the scarce resource you want spent on judgment.

You also run evals at three altitudes. Capability evals ask whether the system can do a class of task at all. Regression evals ask whether yesterday’s change broke last month’s win. Production evals ask how the thing performs on real traffic, which is always weirder than your dataset.

One rule from the playbook I now apply to everything: every meaningful eval should be wired to a decision. Launch, rollback, model choice, routing, scope, permissions, escalation, roadmap. If no decision would change based on the result, the eval is a dashboard ornament and you can delete it without guilt.

Here is where it got personal for me. I run a personal agent harness, the system that manages my notes, my tasks, my whole operating layer. The standing rule inside it: every failure becomes a permanent regression case. I call it the ratchet, because the case count only moves in one direction.

The agent mangles a date format. New case.

It clobbers a ledger during a merge. New case.

It full-reads a file it should have grepped. New case.

Fourteen months of this and the boring failures simply stopped recurring. The suite never forgets, so I get to. That is the entire trade.

Then agents showed up and my tidy single-turn evals stopped being enough.

An agent is a whole system: model plus instructions plus context plus tools plus state plus environment plus orchestration plus retries plus permissions plus budget. Grade only the final chat message and you are grading the press release.

The failure mode that teaches this fastest: an agent can tell you it succeeded while the file it promised never landed on disk.

So whenever you can, grade the actual end state. The database row. The reservation. The transaction. The file that either exists or does not. Take a refund agent: the correct customer identified, eligibility checked, the right amount calculated, the right tool invoked, the database actually changed, the user told the truth about all of it. Six checks, and the transcript can read perfectly while any one of them quietly fails.

The metric list widens too. Past raw task success you start tracking tool-call accuracy, unauthorized actions, turns per task, tokens per task, retries, latency, cost, human interventions, recovery rate, critical failures. An agent that succeeds after eleven retries and four dollars of tokens has a different problem than one that fails cleanly.

And then there is the arithmetic that humbles everybody. Chain five required steps that each succeed ninety percent of the time, with independent failures, and the whole run lands near fifty-nine percent. Ninety percent felt solid at every step, and the pipeline still hands you a coin flip and change.

That one line of math rewired my reliability instincts. A ninety percent step feels great in a spreadsheet. Multiply a few of them and you understand why multi-step agents need a different reliability posture than single-turn chat, and why recovery behavior matters as much as first-try accuracy.

The suite never forgets, so I get to.

Which brings the whole thing around into a loop. The playbook’s operating model is a circle: problem, value hypothesis, task definition, system design, evaluation, product experience, production, outcomes, failures, new evals, improvement. The failure step is load-bearing. Production is where the system finally meets inputs you never imagined, and every miss it produces is free curriculum.

In my harness that loop runs nightly. A critic pass reads the day’s transcripts and flags failure classes that keep recurring, which is how a one-off annoyance gets promoted to a cluster worth fixing. The cluster becomes a hypothesis, the hypothesis becomes new eval cases, the fix reruns the whole suite before it ships. Same flywheel the playbook draws, just small enough to live on one machine.

The weekly rhythm matters as much as the nightly one. The playbook calls it the operating review: product numbers like adoption and completion, quality numbers like eval score and regression count, system numbers like latency and cost, and a learning column for new failure clusters and missing eval coverage. Every review ends in decisions. What changes next, why, and which metric should move because of it.

One trap I only saw after falling into it: anchor your eval cases to answers the user ratified as correct. Early on I anchored some of mine to what the agent had done before, and I had quietly frozen yesterday’s mediocre behavior into a test suite that defended it. The definition of good has to come from the person the system serves. Mine now trace back to answers I signed off on, which means the suite argues for me even when I am asleep.

If you keep one sentence from all of this, keep the first one. Own the definition of good. Models will keep changing under you, prices will drop, capabilities will jump sideways. The golden set, the graders, and the loop that feeds them are the part of the product you actually own, and they compound while everything else churns.

Mine is fourteen months old and still tightening.