You ship: WorkBot's web chat on OpenAI and Anthropic with a routing table, fallbacks and eval scores for both. Builds on:Day 10, Day 19.
What you'll build today
Until today, one setting chose the chat's provider and model, and every other place in WorkBot that calls a model chose a size for itself. That has two weaknesses. When a provider has an outage, WorkBot goes down with it. And nobody has measured which model each job needs.
Today you put both providers, Anthropic and OpenAI, behind one layer, the AI SDK, and route every model call by its task from one table. Each route gets a fallback model behind it. A flag lets you rehearse an outage on your own machine, so you can see exactly what a person chatting with WorkBot sees when a model fails. Then you run the evals, the scored tests of WorkBot's output from Day 19, for each task on each provider's models. The scores, the cost and the latency choose the routing.
By the end of the day, WorkBot stays up when a model fails, and its routing rests on measurements you can run again.
By the end of the day you can
Run the same application against OpenAI and Anthropic models through one provider layer
Route requests by task, cost and latency, with fallbacks that keep the product up
Measure provider differences with evals, the scored tests of WorkBot's output (Day 19)
Topics
Where Providers Differ
Request shape: message roles, system prompt placement, tool-call and tool-result blocks, streaming event names, stop reasons, token accounting
Structured output: a tool's input used as the schema (Day 17), or native structured outputs with JSON Schema. Both providers offer both, with different strictness guarantees
Behavior: instruction following, refusal boundaries, verbosity defaults and long-context handling, measured by evals
The Provider Layer
Vercel AI SDK: generateText, streamText, generateObject and tools, with provider packages for OpenAI, Anthropic, Google and open-weight models
What the abstraction covers: streaming, tool loops, structured output
What leaks through: provider-specific options, caching controls, model-specific features
When to call a provider SDK directly: for a feature the abstraction does not expose, or in a single-provider product
Model identifiers in configuration, never as literals in application code
Routing + Fallbacks
Routing by task: small models for classification and extraction, large models for reasoning and writing. The router is a config table
Fallback chain: a provider outage, then a rate limit, then a content refusal, and what the person sees at each step
Timeouts and hedging: sending to 2 providers at once and taking the first response
Budgets: cost and latency per route, logged on every call (Day 19 tracing)
Open-Weight Models
Where they fit: on-premise requirements, data residency, cost at volume
Serving: inference providers behind the same AI SDK interface
The same evals decide whether a route can use them
Build
Features shipped today
WorkBot: 2 providers behind one layer, routed per workflow step. A small model triages and extracts, a large model lays out pages and writes reports, and the evals decide the chat's model. A fallback keeps WorkBot up. Cost and latency are logged per tenant, the customer or run each call belongs to
Take WorkBot's web chat (Day 10) and make it provider-agnostic:
Replace direct SDK calls with the AI SDK. Behavior stays identical on the Anthropic route
Add an OpenAI route, and a config table that maps each task (chat, extraction, summarization, title generation) to a model, with a fallback model per task
Add a fault-injection flag that simulates a provider outage. WorkBot stays up and tells the person what happened
Run the evals for each task against both providers. The chat uses the Day 19 golden set, extraction the Day 17 documents, and triage a new set. Put the scores, cost and 95th percentile (p95) latency in one table
Pick the production routing from the table. Write the 2 sentences that justify each row
Walkthrough
Part 1
One layer, routed by task
The web chat has used the AI SDK since Day 10, while each of the other call sites chose a model size for itself. Today every call site names a task instead, and one table chooses the model. The curriculum names 4 tasks: chat, extraction, summarization and title generation. WorkBot has no titles to generate, but it does have triage, reading, research, page generation and a judge, so its table has 8 rows.
Study first
Where the providers differ: read the 2 stream functions of src/lib/providers.ts side by side, anthropic() and openai(), each about 60 lines against the raw SDK. Anthropic puts the system prompt beside messages, and OpenAI puts it beside input. In Anthropic's API, a tool call is a tool_use block, after which the model stops with stop_reason: "tool_use" and the caller runs the tool. In OpenAI's, a tool call is a function_call item, and the loop ends when a completed response holds none. Anthropic takes a tool result as a block in the next user message, while OpenAI takes it as an item appended after the model's own items are replayed. The token fields and the reasoning effort setting have different names, and docs/routing.md puts all of these differences in a table.
What the AI SDK covers, and what leaks through: generateText and streamText hide the request shape, the tool loop and structured output. Both providers constrain structured output to a schema, each with its own strictness. Caching controls stay provider-specific, such as providerOptions.anthropic.cacheControl on the chat route's system prompt, and so does reasoning effort. The terminal chat from Day 6 uses the raw SDKs, so the wire format stays visible.
Measure behavior differences, such as instruction following, refusal boundaries and verbosity, instead of reading about them. The golden set from Day 19 holds refusal and escalation cases, so a provider's refusal boundary shows in its chat score.
An open-weight model served by an inference provider is one more provider behind the same interface, and one more column in the table. The same evals answer whether a route can use it, while where it fits is the business's question, such as on-premise hosting, data residency or cost at volume.
Keep model identifiers in configuration, never in application code. A route is a provider and a size, such as anthropic/small, resolved against the provider's model table, and the ids and prices stay in providers.ts.
Build
Write src/lib/routing.ts with 8 tasks: chat, generate, research, summarize, extract, triage, read and judge. routes maps each task to its routes, in order. Add modelOf(route) and modelFor(task), and move aiModel(provider, id) here from the chat module, still traced.
Make every call site name its task. In the chat route, that is the summary, the answer and the memory extraction. Elsewhere it is the generate, intake, voice and documents routes, the approval workflow, the request actions and the CLIs. Then delete the size-picking function, chatModel.
Add ROUTES to config.ts, as task=provider/size pairs. The routing module checks them when it loads, which in next dev is the first request that needs a model. A pair that names no task or no route throws, with the list of both, instead of routing to a default, and an empty ROUTES is no override. Remove CHAT_PROVIDER and CHAT_MODEL.
Write tests that every route in the table resolves to a provider's model, and that a bad pair throws. Test that an override replaces a task's first model and keeps the rest behind it.
Done when
grep -rn chatModel src cli finds nothing, and pnpm test passes.
ROUTES=chat=anthropic/medium pnpm dev fails the first request that needs a model, and names the tasks and routes it accepts.
After a chat turn and a dashboard generated at /dashboards, /admin/traces shows the small model for the chat and the large one for the generation.
Technical notes
A route is ${ProviderName}/${Size}, a template literal type. The compiler checks the table, and allRoutes is built from the 2 lists at run time for the ROUTES check.
A route's models form a chain, in which each model is the fallback of the one before (next part). The chain's modelId is its first model's, and the traces page groups by it. The model that answered is in the response, as response.modelId, and each model in the chain writes its own trace row.
Two call sites are not routed. The Claude Agent SDK pipeline (Day 12) runs on the Claude Code sign-in, and the framework comparison (Day 23) compares one model on purpose.
Part 2
Fallbacks and a rehearsed outage
With every call routed, a route can hold more than one model, and that is what makes a fallback possible. A provider outage, a rate limit and a refusal are 3 different failures, and the person chatting should see something different for each. The fallback covers the first 2. A refusal is the model's answer, so WorkBot shows it as one.
Study first
The fallback chain: know what the person sees at each step, from an outage to a rate limit to a refusal. Some failures must never be retried on another model. A bad request (400) or a bad key (401) fails anywhere, and a stream that has started has already put part of a reply on the page.
The AI SDK's error model: the provider adapter sets APICallError.isRetryable from the status. The Anthropic and OpenAI adapters set it to true for 5xx, 429, 408 and a connection that could not be made. Before the stream starts, generateText and streamText retry a retryable error with backoff. maxRetries defaults to 2, so there are 3 attempts before they throw RetryError. After the stream has started, an error is retried only as streamRetries says, and it defaults to 0.
Middleware: the AI SDK's model middleware wraps a model's calls, and its wrapGenerate and wrapStream receive the operation and the call's params. That is what lets one model's call be handed to another.
Fault injection: a flag fails a call before it reaches the network. You can then exercise the chain on a development machine, and the failed call appears in the traces like a real one.
Build
Write chain(models), which puts middleware on each model that catches a transient failure and calls the next model's doGenerate or doStream with the same params. reduceRight folds the list.
Add OUTAGE to config.ts, holding a provider name or a model id. aiModel wraps the named model in middleware that throws a 503 APICallError before the network, and the trace records the failure with its reason.
In the chat route, make onError on the UI stream return one sentence: "WorkBot cannot reach its model right now. Try again in a minute." By default, toUIMessageStream masks errors as "An error occurred." If the model could not write a summary, leave the summary out instead of failing the turn, and do not send a turn with no reply to memory extraction.
Write tests that the chain falls back on a 503 and a 429 but not on a 400, and that a stream falls back before it starts. Test that the outage is traced and the chain recovers, and that the route sends the sentence, not the provider's error, and calls no extraction.
Done when
With OUTAGE=claude-haiku-5-5 pnpm dev, the chat answers. /admin/traces shows a failed call on the small model with "Simulated outage of claude-haiku-5-5" as its reason, then the large model's call that answered.
With OUTAGE=anthropic pnpm dev, the chat shows the sentence and a Retry button, with the message still on the page.
Every model of the chat's route is down. WorkBot shows one sentence and a Retry button, and the message stays on the page.The traces after a rehearsed outage of the small model. Each step shows a failed call with its reason, then the large model's call that answered it.Technical notes
The SDK's retries wrap the whole chain, so when every model is down, the SDK tries the chain 3 times with backoff before the sentence goes out. In the reference run that took 6 s, and the traces show the attempts at 0, 2 and 6 s. The first model's own failure costs no retry, because the chain moves on at once.
streamText calls the model when the response body is read, not when the route returns. In a route test, keep the outage flag set until the body has been read, because a flag flipped back too early makes the mock answer normally.
toUIMessageStream's onError decides what the stream tells the browser about an error, while the traced error keeps the provider's message for the operator. A failure before the stream exists, such as the summary's, is the route's to handle.
Fallback is per model call, not per turn, so a turn of 3 steps pays for the failed call 3 times. Each costs 0 ms when rehearsed, and the provider's time to refuse when real. A circuit breaker would remember the failure, and it is on the architecture document's list of what is missing.
Hedging sends the same call to 2 providers and takes the first answer. It doubles the bill, for a latency gain WorkBot's table does not ask for. A timeout per call is abortSignal on the call. Neither hedging nor timeouts are built today.
Part 3
Choose the routing from evals
The table now decides which model answers each task, so its rows should rest on measurements. Run each of the 3 evals on both sizes from each provider, and read the cost per case and the 95th percentile latency (p95) back from the run's traces. Mark the routes that cannot be measured.
Study first
Budgets per route: since Day 19, WorkBot logs cost and latency on every call, by tenant, the customer or run the call belongs to. The p95 is what the slowest call in 20 feels like, and that is the call people report. A judge's score has noise, so each row carries its standard deviation (SD).
Which tasks have an eval: the chat has the golden cases with the judge (Day 19), and extraction has the 18 documents of content/eval/documents (Day 17). Triage needs one, so write 12 messages from the customer note in the brain (Day 21), 4 per team across the 3 priorities. Give each message the team and priority it belongs to. Generation, research, summaries and reading have no eval yet, so their rows follow the nearest measured task, and docs/routing.md says so.
A provider without a key gives no number. Mark its rows unmeasured, and choose no route on them.
Build
Write content/eval/intake.json, with id, message, team and priority, validated by IntakeCase. Add triageScore to src/lib/eval.ts, which gives 0.5 for the right team and 0.5 for the right priority. Move the document loading and scoring of pnpm extract:eval to src/lib/eval-documents.ts, so the command keeps its output and the new CLI shares the code.
Add spend(ctx, since) to tracing.ts, which returns the calls, dollars and p95 under a context since a given moment. Run the judge under its own name, so that its calls are not on the bill of the model it judges.
Write cli/routes.ts, run as pnpm routes [--task] [--provider] [--runs], which prints one Markdown table of tasks by providers by sizes. It needs ANTHROPIC_API_KEY and DATABASE_URL, since it reads the traces back from the database, and OPENAI_API_KEY too if the OpenAI rows should fill. A provider whose key is missing gets a row that says "no key", and any other failure fails the run.
Write docs/routing.md, with the differences table, the layer, what the person sees, and the measured table. Give it a row per task, with the reason it is routed as it is, and say what an OpenAI key would change.
Add decision 13, one routing table per task, to brain/architecture/decisions.md, the brain's numbered decisions (Day 21). Mark decision 10, small model first, as superseded in part, and update the models and reliability sections of the architecture document.
Done when
pnpm routes --runs 2 prints the table and exits 0, with the OpenAI rows reading "no key". With a key, they fill.
Each task in routes has a row in the document's routing table.
/admin/traces shows the run's bill under the routes tenant, beside the visitors' bills.
pnpm routes --runs 2 shows the 3 tasks on Anthropic's 2 models, with score, noise, cost per case, p95 and calls. The OpenAI rows had no key in the reference run.Technical notes
In the reference run, the chat scored 0.83 on the small model against 0.87 on the large. It cost $0.0008 a case against $0.0171, with a p95 of 3.4 s against 4.8 s. Extraction scored 0.96 against 0.97, at $0.0006 against $0.0096, and triage scored 0.96 on both, at $0.0001 against $0.0019. The 2 runs cost $1.26, judge included.
The judge's variance limits what the scores establish. With an SD of 0.25 over 28 scored answers, the standard error of a mean, the typical error in the mean itself, is about 0.05. So 0.87 and 0.83 are the same number. Re-judged by Sonnet (ROUTES=judge=anthropic/large pnpm routes --task chat --runs 1), the chat rows came out 0.91 for the small model and 0.88 for the large. The order flips and the gap stays under that error, so the chat moves to the small model, with the large behind it. Revisit that row when the set can tell the two apart.
Extraction's score is the share of expected fields that came out right. It is 0 when the document kind was wrong, because the wrong kind means the wrong schema in the pipeline. The 36 documents took 72 calls, a classification and an extraction each.
percentile_cont(0.95) within group (order by ms) computes the p95 in both Postgres and PGlite, as the Postgres aggregate functions describe. prompt is not distinct from $3 is how a null prompt matches.
