Week 5 · Day 22 of 40 · Use AI

Agent fleets: 50 concurrent agents, cloud agents and test automation

You ship: A 50-task fleet run measured on merges per hour, reverts and brief-to-merge time. Builds on: Day 2, Day 19, Day 21.

By the end of the day you can

  • Run dozens of coding agents concurrently on well-specified tasks and merge the results safely
  • Use cloud agents for work that outlives a laptop session
  • Automate testing and review so a fleet's output can be trusted at merge time

Topics

From 4 Agents to 50

  • What changes: task specification quality becomes the bottleneck, then review; the human becomes dispatcher and reviewer
  • Task shape: small, independent, verifiable; an acceptance test written before the agent starts
  • Isolation: a worktree or a cloud sandbox per agent; no shared mutable state
  • Dispatch: a backlog of specified tasks, a dispatcher assigning to local worktrees and cloud sessions, a dashboard of state

Cloud Agents

  • Platforms: Codex cloud tasks, Claude Code on the web, GitHub's coding agent; a task in, a PR out, environment provisioned per run
  • Environment definitions: dependencies, secrets and test commands the cloud agent needs; the same definition used locally
  • Cloud vs local: cloud for long tasks, many tasks and clean machines; local for tight iteration with a human
  • Cost and limits per platform; queueing when limits are hit

Test Automation for a Fleet

  • Every PR: full test suite, type-check, lint, the eval smoke subset (Day 19); flaky tests fixed or quarantined the day they appear
  • Generated tests: the agent writes tests from acceptance criteria before implementation; the reviewer checks the tests first
  • Property-based and snapshot tests for generated code; contract tests between monorepo packages
  • End-to-end tests against a preview deployment per PR (Day 1)

PR Review at Scale

  • Review fleet: the Day 2 trio (bugs, performance, security) plus a brief-conformance reviewer on every PR, before a human looks
  • Human review: brief, tests, diff; approve, request changes, or close with a better brief
  • Merge queues: main stays green when 50 PRs land in an afternoon
  • Metrics: PRs merged per hour, revert rate, time from brief to merge

Build

Features shipped today

  • WorkBot: Tasks from its backlog shipped by the fleet

Run a fleet against your backlog:

  1. Specify 50 small tasks from your backlog, each with acceptance criteria and a test; review the specifications before any agent starts
  2. Dispatch: 20 to local worktrees, 30 to cloud agents; a dashboard showing each task's state
  3. Automation: the review fleet, the test suite, the eval smoke subset and a merge queue on every PR
  4. Merge: review as PRs complete; record PRs merged, reverts and brief-to-merge time
  5. Retrospective: which task shapes succeeded, which failed, what the next backlog should look like

Walkthrough

Run a fleet against the backlog: 50 small tasks specified before any agent starts, dispatched to worktrees on the laptop and to cloud sessions, each coming back as a pull request that 4 reviewers, CI and a batched queue decide on. The person is dispatcher and reviewer. In this run the limit was the specification first and the review second, not the agents.

Part 1

Fifty tasks that fit an agent

With 4 agents the day's limit was the model's output. With 50 it was the brief: a task an agent can finish in one session, that no other task collides with, and whose outcome a test can check.

Study first

  • Task shape: one goal, criteria an agent checks one by one, constraints that say what must not change, the files involved, and the test to write first. A task that names two files that another task names is two pull requests that conflict.
  • Where tasks come from: a survey of the code by area (libraries and routes, CLIs and docs, pages and components), each gap cited with the file and the behaviour, and the previous day's review findings, which are briefs already.
  • Review the specifications before any agent starts, with a second model as the reader: pairs of tasks that change the same file, claims about the code that are wrong, criteria an agent could not check, tasks too large for an hour.

Build

  1. docs/fleet-tasks.md: 50 ## sections, each with Goal, Acceptance criteria, Constraints, Related code and docs, and Test, in the same fields as the issue form from Day 21.
  2. Survey the code in 3 areas (libraries, routes and workflows; CLIs, docs, site and eval sets; pages and components) for 25 candidates each, every candidate a file, a gap and a test; keep 50 that are worth a session and no two of which change the same file; swap in the previous day's review findings, which are briefs already.
  3. A review of the specifications by another model: pairs that change the same file, wrong claims, uncheckable criteria, tasks too large.
  4. node scripts/fleet.mjs file docs/fleet-tasks.md: one issue per task, labelled fleet.

Done when

  • 50 issues exist, no two change the same file, and every one names the test to write first.
A task as an issue
One of the 50: the goal, criteria an agent checks one by one, the constraints, the files and the test to write first.
Technical notes
  • The survey was 3 read-only agents, one per area, told to cite line numbers and to group candidates so no two touch the same file; 75 candidates became 50 after the review findings from Day 21 took 4 slots.
  • Independence is by files changed, not files read: a task may read src/lib/requests.ts while another changes it.
  • The specification review found no overlaps and no wrong claims, and made 10 tests more precise: an assertion named for each criterion that had none.

Part 2

Dispatch: worktrees here, sessions in the cloud

An agent needs a tree of its own. Here that is a git worktree with its own dependencies and a non-interactive Claude Code session; in the cloud it is a session that clones the branch, works, and pushes.

Study first

  • Worktrees: git worktree add -B fleet/<issue> ../workbot-fleet/<issue> day-22, then pnpm install --frozen-lockfile. The session is claude -p "<the brief>" --model sonnet --session-id <uuid> --permission-mode acceptEdits --allowedTools <the list>, the list being edits, reads, pnpm, git, gh pr create, gh pr view, gh issue view; --strict-mcp-config with an empty config keeps the MCP servers out.
  • Cloud sessions: claude --cloud "<the brief>" --model sonnet creates a session that clones the repository's GitHub remote at the current branch, so push first. It wants a terminal, so the dispatcher runs it under script -q /dev/null (script -q -c … /dev/null on Linux). The session can push a branch; it holds no GitHub token, so the dispatcher opens the pull request when the branch appears.
  • Concurrency: slots per lane, refilled every round; the limits are the laptop and the Claude plan's rate limits, not the script.

Build

  1. scripts/fleet.mjs dispatch --local 6 --cloud 12: fill the local slots first, then the cloud ones; a start that fails is logged and retried next round. State is the issues and the pull requests on GitHub plus .fleet.json at the repository root, gitignored: one entry per task with its session's pid or cloud id, and later the head the queue reviewed.
  2. The brief each agent gets: the issue number, the branch, "write the test first and see it fail", the checks to run, one commit with Closes #n, and for local agents the pull request body in the template's shape with its session id.
  3. scripts/fleet.mjs status: one row per task: queued, local or cloud with minutes, pull request with CI state, merged, failed.

Done when

  • Every task is running or has a pull request, and the board says which lane and for how long.
The fleet board
The dashboard a few rounds in: local and cloud agents, pull requests waiting on CI, merged tasks.
Technical notes
  • claude -p prints only its final message, so a local agent's log is empty until it stops; the branch and the pull request are the progress signal.
  • claude --cloud refuses a pipe for stdout; script -q /dev/null claude --cloud … provides the terminal it asks for. The first run meets the workspace trust dialog, once.
  • The cloud environment had no setup script, so the brief asks for pnpm install first; a session's log is on its page at claude.ai.
  • Local sessions hold a UUID passed with --session-id, which goes into the pull request's Session section; cloud sessions put their claude.ai link there.

Part 3

Trust at merge time: the review fleet, the checks, the queue

Trust is decided at merge: every pull request meets CI, 4 reviewers and a queue that merges in batches and checks the branch after each.

Study first

  • The review fleet: the Day 2 trio (bugs, performance, security) plus a brief reviewer that reads the issue's criteria, the tests first, then the code, and reports each criterion met and tested, met but untested, or not met. The skill ends with one line, Verdict: approve or Verdict: request changes, that a script can read.
  • Merge queue: GitHub's is available to organisation-owned repositories, not to a personal one, so this one has its own. A batch is the ready pull requests of one round, up to 6: reviewed in parallel, the approved ones merged one at a time with the agent's commit message, then one full check and the eval smoke subset on the branch. If the check fails, the last merge is reverted by its commit and the check run again, until it passes; each reverted issue is reopened with the failing lines.
  • Flaky tests are fixed or quarantined the day they appear; a queue amplifies them.

Build

  1. .claude/agents/brief-reviewer.md, and the /review skill running 4 reviewers with the verdict line.
  2. scripts/fleet.mjs merge --batch 6: the ready pull requests (open, CI passed, a head the queue has not reviewed, which it records in .fleet.json), their reviews posted as comments, approved ones squash-merged with the commit's subject and body, then pnpm check and pnpm eval --only answer on the branch, and the revert loop. A review that requests changes, or a failed check, starts one more session on the pull request's branch with the review as its brief; a second verdict against it is a person's decision.
  3. scripts/fleet.mjs run: dispatch, merge, status, every 2 minutes until every task is merged, then the report.

Done when

  • Every merged pull request carries a review comment with a verdict, and the branch is green after every batch, reverts included. What the queue cannot decide is left open for a person: a blocking item outside the brief, a criterion the brief got wrong.
A review on a pull request
The 4 reviewers' findings merged into one comment, the brief's criteria checked one by one, and the verdict line the queue reads.
Technical notes
  • Reviews run as claude -p "/review <n>" with the agent tool allowed, so the skill fans out to its 4 reviewers as it does interactively.
  • A squash merge keeps one commit per task with the agent's message, so Closes #n closes the issue and the history reads as 50 small changes.
  • The check after a batch is the full pnpm check, 3 minutes, which is why batches rather than one check per pull request. A check can fail for reasons outside the batch: one did under the load of 6 fix sessions and reverted an innocent change, which passed on its own and was reinstated. The log of every check is kept for that reason.
  • A smoke eval is a gate against breakage, not drift: 12 runs of 5 cases on the small model scored between 0.60 and 0.80 on batches that touched no prompt. The bar is 0.5 because the gate is there to catch an assistant that stopped answering; a change to a prompt gets the full eval of Day 19.

Part 4

What the numbers say

Study first

  • Metrics that matter for a fleet: pull requests merged per hour, which is throughput; the revert rate, which is specification and review quality; and the time from brief to merge, measured from the issue's creation to the merge, which is how long the queue holds work. The number merged says least: 50 merged says nothing about whether 50 was the right 50.
  • The retrospective: which task shapes succeeded, which failed, and what the next backlog should look like.

Build

  1. scripts/fleet.mjs report: merged per hour, reverts, median and p90 brief-to-merge.
  2. docs/fleet.md: the numbers, the task shapes that worked and the ones that did not, with the issue for each, and what the next 50 should look like.

Done when

  • The report prints, and docs/fleet.md names the shapes, not just the numbers.
The report
What the fleet did: merged per hour, reverts, brief to merge.
Technical notes
  • 50 merged over 2 hours 26 minutes from the first merge to the last, 20.6 an hour; 25 from worktrees, 25 from cloud sessions; 44 merged by the queue on an approving review, 6 by a person after reading the review, each with the reason on the pull request; 52 reviews; 17 second sessions after a review or a failed check, 11 of them then merged by the queue; 63 CI runs, 59 passed; 1 revert, reinstated; brief to merge median 101 minutes, 90th percentile 160; 15 rounds; 166 tests before, 298 after.
  • The shapes that merged on the first review were single-module fixes with a named test. The ones that needed a second session: a fix that introduced its own problem (F-02, a quadratic trim on a hostname that request input reaches), a brief that asked for a heuristic where the problem needed a mechanism (F-05, a pattern that can hang the server; the second session bounded the match with a timeout), a test that depended on the clock (F-15, rows inserted in the same millisecond). One a non-interactive agent cannot finish (F-33, which edits .claude/settings.json, the agent's own permissions). One that fails quietly: a cloud agent pushing to the branch its environment named rather than the one the brief named.
  • A person's part: the 50 specified and reviewed before any agent started, one task finished by hand, 7 reviews read and decided, the loop stopped when what was left was judgment, one revert undone. About 2 hours over the afternoon, most of it reading reviews.

Resources