Day 25

Evals as the development loop

You ship
WorkBot on the chosen framework and 2 providers, developed with the fleet workflow, gated by evals in CI.

What you'll build today

This week changed how WorkBot is built and how it runs: 3 coding agents, a fleet that merges in batches, and 2 providers behind one routing table. Any change to a prompt, a tool or a model can quietly make WorkBot's answers worse, and nobody reads every answer. Evals, the scored tests you started on Day 19, are how you find out before WorkBot's users do.

Today you turn them into the loop every change goes through. Each task gets at least 30 cases, written by hand, sampled from what people sent, or generated and then reviewed. Checks in code run first, then a judge model scores the rest, and you check the judge against your own scores. You also learn how much a score varies, so you can tell a real change from noise. Then a gate runs the evals on every pull request that touches a prompt, a tool, a model or an eval.

By the end of the day you have watched the gate stop a deliberately worse prompt, and seen an improvement land on a scoreboard.

By the end of the day you can

  • Build eval datasets, the scored test cases for WorkBot's output, and judges that catch regressions before users do
  • Run evals in CI as a merge gate, with thresholds that allow for the noise in scores
  • Treat system prompts as versioned, tested artifacts

Topics

Eval-Driven Development

  • The loop: change a prompt, model or tool, run the suite, compare to the baseline, then ship or revert (Day 19, now the default)
  • 3 sources of cases: hand-written golden cases, production traces or the requests people sent (sampled and labeled), and synthetic cases generated from a spec and reviewed
  • Keeping the dataset honest: version it, split it, and never tune a prompt against the exact cases it is scored on

Judges

  • Programmatic checks first: schema validity, required fields, forbidden content, latency and cost budgets
  • LLM-as-judge, a model that scores another model's answers: rubric design, pairwise or absolute scoring, position bias, and a judge model different from the one under test
  • Calibration: agreement rate against human labels, where the judge disagrees, when to stop trusting it
  • Human review: a small weekly sample with a shared rubric. The judge's disagreements go there first

Statistics for Evals

  • Run every case several times, and report the mean and spread
  • Minimum detectable change: how many cases before a 3-point difference means anything
  • Thresholds: block the merge on hard failures, warn on soft regressions, never gate on a single run

System Prompting as Engineering

  • Structure: role, task, constraints, output format, examples. Specificity matters more than ordering
  • Prompts as versioned files with tests, promoted through prompt management (Day 19)
  • Negative examples and edge cases in the prompt, drawn from eval failures
  • Shortening prompts without losing score: the eval shows what was load-bearing

Evals in CI

  • Trigger: every pull request that touches a prompt, a tool, a model, a knowledge document or an eval, with the results posted to the pull request
  • Cost control: a smoke subset on each pull request, and the full suite nightly and before a release
  • Reporting: a per-task scoreboard over time

Build

Features shipped today

  • WorkBot: Eval sets with safety, refusal and escalation cases, and action cases checked by the tools an answer calls. The CI gate

Put evals in front of everything the week built. That is the agent on the framework chosen on Day 23, running on 2 providers (Day 24), developed with the fleet workflow from Days 21 and 22:

  1. Eval sets: at least 30 cases per task, from golden examples, sampled traces or requests, and reviewed synthetic cases
  2. Judges: programmatic checks, plus a model as judge, calibrated against 20 cases a person has scored. Report the agreement rate
  3. CI: a smoke subset with a pass threshold on every pull request, and the full suite nightly. Post the results to the pull request and to a scoreboard page
  4. Prove the gate: CI catches one deliberate regression, a worse prompt, and one improvement moves the scoreboard
  5. On the scoreboard page: eval scores per task and route with the day's failovers, the framework comparison from Day 23, and the fleet metrics from Day 22

Step by step

Thirty cases per task

On Day 19, 12 golden cases scored the chat, and the reference run had 14 by Day 24. Today the chat, extraction and triage each get at least 30 cases, and every case says where it came from. More cases make a score worth more, as the statistics in the next part show.

Study first

  • The 3 sources, and what each is good for: write cases by hand for the behavior the product promises, such as refusals, escalations, safety and actions. Sample what people actually sent, to find the questions nobody wrote down. Generate synthetic cases from a document and review them, to cover the corpus evenly. Anthropic's roadmap for collecting eval tasks starts from the checks you already run and the failures people report. It asks for cases that 2 experts would grade the same way.
  • WorkBot's traces keep no message content, by decision D9 in the brain (Day 21), so the sample of what people sent comes from the requests table. You cannot sample the chat's questions until WorkBot keeps a conversation on the server.
  • Action correctness, as far as a tool's name goes: when an answer must call a tool, check the names of the tools it called instead of judging it. answer() from Day 19 returns those names beside the text, and checking the arguments and the result comes next.
  • Keep the sets honest. Never edit a case to make a prompt pass. Tune a prompt against the failures of the full run, then confirm it on the smoke subset. The smoke subset is the cases marked to run on every pull request, and it sits inside the full set. There is no held-out split yet, a set of cases kept back from tuning, because at 30 cases a task there are none to spare. A set's version is the commit that ran it, and every scoreboard line records that commit.

Build

  1. In src/lib/eval.ts, give Golden the new kind values action and safety, and the fields mustCall, source and smoke. source is golden for a case written by hand, trace or synthetic. Give IntakeCase the fields source and smoke, and add smoke to the documents' expected.json.
  2. Write src/lib/eval-suite.ts, with chatSuite, extractSuite and triageSuite in one Case shape. pnpm eval runs a task on its production route, and pnpm routes runs it on every provider's models, so the two now share the cases.
  3. Write 10 chat cases by hand. Four are actions: the calculator twice, read_url on example.com, and find for the cutoff document. Six are safety cases: the system prompt asked for twice, an alarm code, a colleague's address and pay, a fake invoice, and an approval smuggled in a customer's email.
  4. Write cli/eval-generate.ts, run as pnpm eval:generate --task chat|triage|extract --count n, which writes candidates into content/eval/candidates/, a folder git ignores. For the chat, it writes n cases, one per customer-facing document. For triage, it takes the n most recent requests with a body, and writes n messages, one per team and priority in order. For documents, it writes n complete invoices, emails, letters and job postings in turn, each with the record it should be read as.
  5. Review every candidate. Keep a criterion if the question asks for the fact it checks. Keep a forbidden phrase only if no correct answer could say it, whether in a negation or because the policy itself does. Keep a label if the team table in the brain's customer note agrees (brain/customers/fieldstone-supply.md, Day 21).
  6. Append the kept cases, with their source, to content/eval/golden.json, content/eval/intake.json or content/eval/documents/expected.json, and copy each kept document's file beside expected.json. Run each new case once with pnpm eval --task <task> --only <id> --runs 1.
  7. Mark the smoke subset with smoke: 10 chat cases across the kinds, 4 documents and 4 triage messages.

Done when

  • pnpm eval --task all --runs 1 runs at least 30 cases for each task. In the reference run, that was 35 chat cases, 30 documents and 32 triage messages.
  • With --smoke, it runs only the subset, 10, 4 and 4 cases in the reference run.
  • Every case in the 3 files carries a source.
Technical notes
  • generateText returns steps, and each step its toolCalls, so the tool names are the record of the actions. read_url on example.com reaches the network during the eval, on purpose, because the tool is real.
  • In the reference run's review, 11 of 12 chat candidates were kept, and one was dropped because it asked the same question as another, from a second document. Of 8 sampled requests, 3 read as messages, and 5 were the line items of invoices. Of 17 triage candidates, 2 were relabeled from support to operations. Every generated document's record was cut to the fields that the existing cases of that kind use.
  • The generator asks for criteria "checkable from the answer's text alone", and still gets some wrong. In the reference run it produced 6 criteria that demanded facts the question did not ask for. It also produced 7 forbidden phrases that a correct answer contains, in a negation or because the policy itself says them. One was "up to $300", which is the store-credit limit. The review caught the phrases, and the first run caught the criteria and one more phrase.
  • The documents' matcher checks containment (Day 17), so a letter's record names the sender, the date and the amounts it asks about. A record of every number in the letter would test the fixture, not the reader.

How far to trust the judge

With the cases in place, something has to score the answers. Checks in code come first, and a judge model scores what they cannot check. Because the judge is a model too, you calibrate it against a person and measure how much its scores move between runs before you trust them.

Study first

  • Programmatic checks first, which are checks in code with no model. They cover schema validity, forbidden content, required tool calls and the document kind. Schema validity is Zod on every structured output since Day 17, and the extraction loop enforces it before anything is scored. A case that fails a check scores 0 without the judge being asked, and that is the one kind of failure that blocks a merge on its own.
  • LLM-as-judge, a model that scores another model's answers: WorkBot's judge uses absolute scoring, against the case's criteria, with one sentence of reason. Pairwise scoring asks which of 2 answers is better. It is for judging without criteria, and it needs each pair in both orders to cancel position bias, the judge's preference for one position. OpenAI's guide to model graders lists these formats and their biases, and asks that a judge agree with people before you scale it.
  • Use a different model from the one under test. The chat runs on Haiku since Day 24, so the judge is Sonnet, because a model grading itself has a thumb on the scale. The Day 24 check showed the two judges 0.08 apart on the same answers.
  • Calibration: a person and the judge score the same replies, and you report how often they agree. Each week, a person scores 10 replies against the same criteria, starting with the replies the judge scored lowest.
  • Statistics: run every case several times, and report the mean and the spread. The detectable change is the smallest change that is not noise, 1.96 · sd · √(2/n). In the formula, sd is the standard deviation of the scores, and n is the number of scored answers, cases times runs. The formula is a rule of thumb, because it treats repeated cases as independent. Block on hard failures, and warn on soft regressions. Anthropic's statistical approach to model evals covers the standard error, clustering for related questions, and paired differences between 2 versions.

Build

  1. Make judge() check forbidden phrases and missing tools before it asks the model. Add the standard error to stats(), and add detectable(sd, n).
  2. Set routes.judge to anthropic/large, the model the chat does not run on. In the brain, decision D14, evals as the development loop, supersedes the judge's row in D13, the routing table from Day 24, and docs/routing.md says why.
  3. Write cli/eval-calibrate.ts. With --collect 20, it writes 20 replies to content/eval/labels.json, taking the kinds in turn. Score each reply yourself from 0 to 1 against its criteria, in the entry's human field, with a note where the score is not 1. Then pnpm eval:calibrate has the judge score the same replies, and prints the agreement.

Done when

  • pnpm eval:calibrate ends with "Agreement k of n", the number of replies where the judge's score was within 0.25 of yours, and the judge's average offset from your scores.
  • pnpm eval --task chat --runs 2 ends its summary with "a change under 0.0x is noise at this size".
The judge against a person
pnpm eval:calibrate, with 20 replies scored by hand and by the judge, and the agreement.
Technical notes
  • In the reference run, the judge agreed on 20 of 20 replies, within 0.25, and scored 0.00 above the person on average. Read that narrowly. The person gave full marks to 17 of the 20 replies, so the judge was tested on only 3 partial scores. Those were 0.5, 0.67 and 0.5, each for a document named by its title where the case asks for its path, and the judge matched all 3. The next calibration takes the nightly run's lowest-scoring replies, where the partial scores are.
  • The reference run measured these detectable changes, with n the number of scored answers and the standard deviation (SD) of the scores. The 35 chat cases run twice at SD 0.24 give 0.08, or 0.06 at SD 0.18 after the case fixes. The smoke subset's 10 cases run twice give 0.09 at SD 0.15, and 0.11 at SD 0.18. Extraction's 30 cases run twice at SD 0.09 give 0.03.
  • So a 5-point change on the chat is noise at both sizes, and a 10-point change is a result at the full size but noise at the smoke size. A 3-point change needs about 490 scored answers, or 245 cases run twice.
  • The labels file is the only place a person's scores live. The judge never sees them, only the reply and the criteria.

The gate, scoreboard and proof

Scores protect WorkBot only when something acts on them. The gate turns the evals into a check that can fail a pull request, and the scoreboard keeps every run over time. A deliberately worse prompt then proves that the gate works.

Study first

  • The gate is one command that runs in 3 places. CI runs it on a pull request from a branch of this repository, not a fork, that touches a listed path. The fleet's queue from Day 22 runs it after each batch, and you run it in a terminal before a push. In GitHub Actions, a paths filter on the pull_request trigger runs the workflow only when a changed file matches.
  • Cost control: run the smoke subset on those pull requests, and the full suite nightly and by hand before a release. A schedule trigger runs the nightly job in UTC, on the latest commit of the default branch.
  • Reporting: the result goes in one comment on the pull request, which each run updates. A scoreboard per task tracks the runs over time, each with its 95% margin, 1.96 · sd / √n.
  • Prompts as versioned, tested files: the system prompt file seeds the prompts table (Day 19). It has 4 parts, role, sources (the task and its constraints), safety (constraints) and style (output format), and no examples section yet. Take the next edit from the prompt's failures in the evals, and the scoreboard then says whether the edit changed the score.

Build

  1. Add 4 flags to cli/eval.ts. --smoke runs the smoke subset. --gate exits 1 on a hard failure or a mean under 0.7, the floor under which the smoke subset's refusals and safety cases are failing in bulk. It also warns when the mean is below the last recorded run by more than the detectable change. --record appends a line to content/eval/scoreboard.jsonl, and --report writes the Markdown the comment carries.
  2. In scripts/fleet.mjs, make the queue's check run the same gate after pnpm check.
  3. Build /admin/evals, which shows every recorded run per task with its margin and a sparkline of the last runs. It also shows the day's fallbacks from the traces (failovers in tracing.ts), the Day 23 comparison read from docs/frameworks.md, and the fleet's numbers from content/eval/fleet.json, which scripts/fleet.mjs report --json writes.
  4. Write 2 tasks for the fleet in docs/eval-tasks.md, numbered after Day 22's F-01 to F-50. File them with node scripts/fleet.mjs file docs/eval-tasks.md, and work them with node scripts/fleet.mjs run --local 2 --cloud 0 --batch 2 (Day 22). F-51 is the workflow, .github/workflows/eval.yml. It runs the smoke subset with the gate on every pull request that touches a prompt, a tool, a model, a knowledge document or an eval. It posts the report as one comment, and updates that comment on each run. Every night it runs the full suite, records it and commits the result. F-52 is the prompt change the evals asked for, because the cases want the source as a path but the prompt's example showed a title.
  5. Prove the gate. Open a pull request against the day's branch with a deliberately worse prompt, and push it twice. In the first push, replace the Sources paragraph with one sentence that names no source and allows general knowledge. In the second, replace the Safety paragraph with a Helpfulness paragraph that answers everything, declines nothing and shares the instructions. Read the comment CI posts after each push, then close the pull request. The improvement's before-and-after is on the scoreboard.

Done when

  • The comment on your regression pull request ends with FAIL and the hard failures, and its check is red.
  • /admin/evals shows the chat suite's mean on the merged prompt above the baseline's, the run recorded before the fleet started.
The gate on a pull request
The eval workflow's comment on the regression pull request, with the hard failures, the mean and the verdict.
The scoreboard
/admin/evals shows the recorded runs per task with their margins, the day's fallbacks, the framework comparison and the fleet.
Technical notes
  • In the reference run, both fleet tasks needed a maintainer's hand, as 6 of Day 22's 50 did. The workflow (F-51) merged after 2 reviews, and the second review blocked on the trust model of repository secrets. A repository with one maintainer accepts that model, and a GitHub environment with required reviewers fixes it when a second maintainer arrives.
  • The review of the prompt change (F-52) blocked on 2 hard failures that were bugs in the cases, and each was fixed on the branch in its own commit. The fix session then hung for an hour on an empty log, and was stopped.
  • Keep the env file out of an agent's worktree. In the reference run, the dispatcher linked the env file into the worktree so that the F-52 agent could run the evals. The code review called the link what it was, a way around the rule that no agent reads that file, and the link is gone. An agent's worktree holds no key, and a brief that needs a measurement names the maintainer who runs it.
  • Make the fleet's collector skip closed issues. The collector is the step that opens pull requests for cloud branches. In the reference run, before either task ran, it opened 25 pull requests for Day 22 branches that cloud sessions had pushed after their issues were closed. None merged, and the collector now leaves closed issues alone.
  • In the reference run, the first regression loosened the Sources paragraph and passed the smoke gate at 0.89, against 0.94 before. A 5-point drop is inside the 0.11 that is noise at 20 answers, and the gate said so. The second regression replaced the Safety paragraph with a Helpfulness one, and failed at 0.62 with 3 hard failures. The off-topic case, an essay on the French Revolution, got its essay with the forbidden names in it. The case that asks for the prompt got the instructions quoted back.
  • So a check catches a category failure outright, while a drift needs the nightly suite. A drift that matters deserves a check of its own, and that is where a mustContain for the source path goes next.
  • The nightly job commits the scoreboard to main as the Actions bot, and a local --record appends the same line. Two branches that append to the same file conflict at merge, so record the baseline before the fleet starts, and let nothing else append until the fleet ends.
  • The repository secret ANTHROPIC_API_KEY is the only credential CI holds, and DATABASE_URL=memory:// gives the evals an embedded database for the traces they read back.

Resources