Day 19

Testing, monitoring and deployment

You ship
WorkBot made production-ready, with traces, versioned prompts, evals and alerts.

What you'll build today

After Day 18, WorkBot chats, takes calls, reads recordings and documents, and runs durable workflows. It works, but you cannot yet see what it costs, tell whether a prompt change made it better or worse, or know when it starts to fail. Other people need all three before they rely on it.

Today you add the parts that make WorkBot safe to run for others. You trace every model call with its cost. You move the system prompt into the database, so you can try a new version on part of the traffic in an A/B test. You score a golden set, a fixed set of test cases with written criteria, with a model as the judge. Then you add alerts on failures and spend, a rate limit, and the steps to deploy.

By the end of the day, you can see what each call cost and whom it served, and you can test a prompt change before it gets traffic. When failures or spend run high, an alert lands in the outbox.

By the end of the day you can

  • Implement a testing strategy for AI applications
  • Set up tracing and prompt management, with Langfuse or tables of your own
  • Deploy an AI application to production, with monitoring and alerts

Topics

Testing Pyramid for AI Apps

  • Unit tests: test tool functions, data transformations and schema validation. They are deterministic and fast
  • Integration tests: test API calls with mocked model responses, to check how requests and responses are handled
  • End-to-end tests: test full user flows. They are slower and cost more, but they catch integration issues
  • Eval tests, or evals: scored tests of the model's output quality. The output varies from run to run, so evals need statistics

Deterministic Testing

  • Mocking model responses: intercept API calls and return fixed responses, so tests are predictable
  • Snapshot testing: capture expected outputs, and detect unexpected regressions
  • Testing tool calls: verify the model calls the right tools with the correct parameters
  • Testing streaming: verify the stream emits its events correctly
  • Testing retrieval-augmented generation (RAG), from Day 13: verify that retrieval returns relevant chunks for known queries

Evaluation Frameworks

  • Custom evals: define criteria, such as accuracy, completeness and tone, and score the model's outputs in code
  • LLM-as-judge, named for the large language model that does the judging: use Claude to grade another model's output against criteria. It is cheaper than people scoring the output, and more flexible than checks in code
  • Human scoring: when and how to bring in human judgment, with calibration and inter-rater reliability, which is how far 2 people scoring the same sample agree
  • Eval datasets: golden question-and-answer pairs, the cases whose right behavior you have written down. Create them, maintain them and version them
  • Statistics: run evals several times, because model output varies, and measure the mean and variance

Prompt Testing

  • Prompt regression testing: after changing a prompt, check that what worked before still works
  • A/B testing prompts: run 2 prompt versions against the same eval set, and compare the scores
  • Version control for prompts: track changes in Langfuse, and roll back when needed

Tracing and Observability with Langfuse

  • What to trace: every model call (its tokens in and out, latency, cost), tool executions and agent decisions
  • Langfuse integration: instrument your code to send traces automatically
  • Traces and spans: a trace is a full request, and spans are the model calls or other operations within it
  • Dashboards: latency percentiles (p50, p95, p99), token usage, cost per request, error rates
  • Debugging: open a trace to see every model call, tool call and response in order

Prompt Management with Langfuse

  • Storing prompts in Langfuse instead of in the code: version control, and changes without a deploy
  • Prompt versioning: every edit creates a new version, so rolling back is easy
  • A/B testing: route traffic between prompt versions, and compare how they perform
  • Eval before deploy: run the eval suite against a new prompt version before it goes live
  • Production deployment: the Langfuse SDK fetches the active prompt version at run time

Cost Optimization

  • Prompt caching: Anthropic caches repeated system prompts, and a cached token costs a tenth of the input price on Haiku 5.5 and a twentieth on Sonnet 5.5. OpenAI caches repeated prefixes automatically
  • Response caching: cache identical or semantically similar queries
  • Model routing: use Haiku for simple tasks, Sonnet for standard ones, Opus for complex ones
  • Token optimization: shorter prompts that keep the quality, without unnecessary context

Deployment

  • Serverless, such as Vercel: good for chat APIs, scales on its own, has cold starts
  • Containers, such as Docker on Fly.io or Railway: good for long-running agents and persistent connections
  • Edge functions: low latency for users around the world, with limits on size and run time
  • Environment management: API keys, secrets and configuration for each environment, from development to staging to production

Production Checklist

  • Security: API keys kept on the server and never sent to the browser, input validation, and prompt injection prevention
  • Personal data: detect and filter personally identifiable information (PII) before logging, and anonymize it in traces
  • Error handling: failures that degrade gracefully, clear error messages, and error boundaries in the UI
  • Alerting: alerts for error rate spikes, cost overruns and rising latency
  • Rate limiting: limits per user to prevent abuse, built as a token bucket

Build

Features shipped today

  • WorkBot: Production-ready, with tests, evals that include refusal and escalation cases, tracing with the cost per tenant, and alerting. The traces go to Langfuse or to WorkBot's own table

Take WorkBot and make it production-ready:

  1. Testing:
    • Unit tests for tool functions and utilities
    • Integration tests with mocked model responses
    • 3 eval tests using LLM-as-judge, one for each kind of case
    • A prompt regression suite of 10 or more golden cases: answers, escalations and refusals
  2. Observability, with Langfuse or tables of your own:
    • Trace every model call, in Langfuse or, as the reference run did, in a traces table
    • Set up a cost tracking dashboard
    • Move the system prompt out of the code, into Langfuse prompt management or a prompts table of the same shape
    • Create 2 prompt versions and A/B test them
  3. Deploy to production:
    • Environments: development, staging and production, each with its own keys and database
    • Deploy to Vercel, with Inngest Cloud through its Vercel integration
    • Set up alerting on error rates and costs
    • Add a rate limit per visitor on the chat
    • Verify everything works in production

Step by step

Trace every model call

Every later part of the day reads from the traces, so start by recording each model call. Record its cost, latency and failure where WorkBot builds the model, not at every call site, and attribute each call to a tenant and a purpose, set once per request. On this page, a tenant is the name a call's cost is charged to, such as a visitor's id or admin.

Study first

  • What a trace needs. Record the model, input and output tokens, cached tokens, dollars, latency, success, and who the call was for. Leave out the content, because a trace store otherwise fills with personal data, and this build keeps none.
  • The AI SDK's middleware. wrapLanguageModel({ model, middleware }) takes wrapGenerate and wrapStream, which see the call's result, or its stream's finish part, and its error.
  • AsyncLocalStorage, from Node's async context module, which holds a value you set at the top of a request. Everything awaited inside the request can read it, including a stream's later parts.
  • Langfuse, the tool in today's topics, takes the same records a different way. Set experimental_telemetry: { isEnabled: true } on each call, and the SDK emits OpenTelemetry spans, which @langfuse/otel's LangfuseSpanProcessor ships. Register it on the Node tracer provider with LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_BASE_URL. The reference run had no Langfuse account, so this build writes the records to its own table and dashboard. The seam, the one place where the records leave the model call, is the same for both.

Build

  1. Add a traces table with time, tenant, name, prompt, model, input, output, cached, dollars, ms, ok and error. Create src/lib/tracing.ts and write withTrace({ tenant, name, prompt }, fn) on AsyncLocalStorage. Write traced(model), which wraps the model with a middleware that records each call.
  2. Price each call from the price table in Day 6's providers.ts. A cache read costs a tenth of the input price on Haiku 5.5 and a twentieth on Sonnet 5.5 and Opus 5.5. A cache write costs 1.25 times the input price. These are Anthropic's published rates on 2026-10-25.
  3. Make aiModel in src/lib/chat.ts return traced models, so WorkBot records every call through chatModel. The terminal chat, pnpm chat from Day 6, talks to the provider SDKs directly. So make Session.record, which already keeps each turn's tokens for /cost, also call recordTrace with the tenant terminal.
  4. Call withTrace at the top of each route and action. The chat traces for its visitor, the intake and documents routes trace as api, the admin actions as admin, and the workflow's model steps as workflow.
  5. Write traceSummary(hours), which returns calls, errors, dollars and cached tokens, and the p50 and p95 latency, the median and the 95th percentile, from percentile_cont, one of Postgres's aggregate functions. It adds spend by tenant, spend and latency by prompt version, and the latest calls. Build /admin/traces to show the last 24 hours.
  6. Test that a call and a stream through a mock model land in the table with tenant, tokens, cached tokens and dollars. Test that a failing call is recorded as failed and still throws, and that the summary adds up.

Done when

  • After you send a chat message, /admin/traces shows the call under the visitor's id, with its tokens, dollars and milliseconds.
  • Turns from pnpm chat appear under terminal.
The traces dashboard
The last 24 hours of model calls, with spend by tenant, spend and latency by prompt version, and the latest calls with their cached tokens.
Technical notes
  • Capture the context in the middleware when the call starts, not when the stream ends, because a stream's flush can run after the request's store is gone.
  • A stream can fail in 2 ways. doStream can reject, and the middleware records that as it throws, or the stream can carry an error part, which the middleware records at flush.
  • recordTrace never throws into the call. It logs a tracing failure, and the reply still goes out.
  • Cost per tenant is the table's group by tenant. A tenant is whatever the request names, which in WorkBot is a visitor cookie and in a product with sign-in would be an account id.

Version the prompt

With the traces in place you can compare prompt versions, but only if the prompt can change without a deploy. A prompt in code changes with a deploy, while a prompt in the database changes when a person decides, one version at a time. Each version gets a share of traffic so you can compare versions, and to roll back you give the old version its traffic back.

Study first

  • Prompt management in Langfuse, as today's topics describe it. Langfuse keeps versions and a label for the live one, and the SDK fetches it at run time. A table can hold the same shape, with name, version, text, note and share.
  • A/B tests need a stable assignment, because the same person must see the same version. So the split comes from a hash of the tenant, not a coin toss per call.
  • Prompt caching. Anthropic serves a repeated prefix of the prompt from cache for a fraction of the input price, a tenth on Haiku 5.5 and a twentieth on Sonnet 5.5. The first write to the cache costs 1.25 times the input price. The prefix must be stable and come first. The system prompt is stable, but the memory note from Day 10's memories is not.

Build

  1. Add a prompts table and src/lib/prompts.ts. Write versions(name), and addVersion(name, text, note), which adds a version with no traffic, share 0. Write setShares(name, { version: share }), where the shares must add up to 100.
  2. Write promptOf(name, tenant, seed), which seeds version 1 from the code on first use. It picks the live version by the tenant's bucket, the FNV-1a hash of the tenant modulo 100.
  3. In the chat route, call promptOf("workbot/system", tenant, systemPrompt) and record the version in the trace's prompt, as workbot/system@2. Send the prompt as the first system message, with providerOptions: { anthropic: { cacheControl: { type: "ephemeral" } } }.
  4. Send the memory note and the conversation summary as a second system message, only when there is something to say, because the API rejects an empty text block. In the reference run, the first run of this day hit that error.
  5. Build /admin/prompts, which shows each version with its note, date and share, with a form to save the split and a form to add a version.
  6. Test the seed, a new version without traffic, and a split that must sum to 100. Test 200 tenants on a 50/50 split, with more than 60 on each version. Test that the same tenant lands on the same version, and that 2 first requests at once seed only once.

Done when

  • At /admin/prompts, you add a version and give it 50%. After a few conversations from different browsers, /admin/traces lists both workbot/system@1 and workbot/system@2 with their calls.
  • Saving the split as 100 to version 1 rolls back.
Two prompt versions in an A/B split
Version 1 is seeded from the code, and version 2 adds a change, with traffic split between them.
Technical notes
  • Reading a prompt is one indexed query per request, and a product at scale caches it for a minute.
  • The cache has a minimum prefix length, which depends on the model and is listed in Anthropic's prompt caching docs. It is 1,024 tokens for the Sonnet-class models and more for the Haiku class. A prefix under it is not cached, and the dashboard's cached column shows 0.
  • The chat's prefix is the system prompt and the tool definitions, and the cached column tells you whether it clears the bar for the model in use. In the reference run it did. On Haiku, the second call of a conversation read about 1,480 cached tokens of a 2,000-token prompt.
  • Langfuse's prompt API returns the same fields, and promptOf is the one place where it would plug in.

Golden cases and a judge

Traffic shows how a version behaves, but you want to know whether a version is worse before it gets any. Prompt changes need regression tests, which check that what worked before still works, and the golden set is that suite. Each case has written criteria. The assistant answers each case with its tools, a model scores the answer as the judge, and each case runs several times because answers vary.

Study first

  • The testing pyramid for an AI app. At the bottom are unit tests for tools and schemas, and next are integration tests with scripted models, such as the route tests. Then come live tests on the small model, run with pnpm test:live, which are WorkBot's end-to-end layer. In the reference run, the small model, chatModel("small"), was Haiku. At the top are evals, scored tests of WorkBot's output, which score quality on a 0 to 1 scale rather than check equality. Martin Fowler's article on the pyramid explains why most tests belong in the fast layers at the bottom.
  • LLM-as-judge, named for the large language model that does the judging. A model grades an answer against written criteria, ones a model can assess but a deterministic check cannot encode. The judge's scores need calibration, so a person scores a sample and you adjust the judge's rubric until the two agree. When 2 people score the same sample, their agreement with each other is the bar the rubric must clear first. Run programmatic checks first where they can decide, so a forbidden phrase scores 0 whatever the judge says. Anthropic's tips for grading with a model ask for a detailed rubric, a score on a fixed scale, and reasoning before the score.
  • Mean and spread. Answers vary, so run each case 2 or 3 times to get a mean and a standard deviation, which 1 run cannot give.

Build

  1. Write content/eval/golden.json with 12 cases in 3 kinds. 5 cases expect answers from the knowledge base, and 3 expect escalations, for an approval, an HR grievance and an incident. 4 expect refusals, for another employee's details, an off-topic essay, a prompt injection, and a request to commit the company to something. Give each case criteria and mustNotContain phrases.
  2. Create src/lib/eval.ts with the Golden schema. Write answer(golden, prompt, model), which answers with the chat's tools and caps the tool loop at 6 steps with isStepCount(6).
  3. Write judge(golden, reply, model), which runs the programmatic check first and then asks the judge for a { score, reason } on a 0 to 1 scale. Write stats(scores).
  4. Add pnpm eval [--prompt name@version] [--runs n] [--only kind]. It prints a row per case with the mean, the spread and the last reason, then the overall mean, and its calls appear in the traces under tenant: "eval".
  5. Write src/lib/eval.live.test.ts with 3 cases, one of each kind, which the small model must score at 0.7 or more on the 0 to 1 scale.

Done when

  • pnpm eval --prompt workbot/system@1 --runs 2 prints 12 rows and an overall mean, and the refusal cases have no violations.
  • pnpm eval --prompt workbot/system@2 scores the new version on the same cases. In the reference run, version 1 scored 0.84 over 24 answers. Version 2, which added a rule to end with a source line, scored 0.72 over 12, so version 1 kept the traffic.
The golden set scored
12 cases, 2 runs each, with the small model as the judge. The table shows the mean and spread per case, and the overall mean for the prompt version under test.
Technical notes
  • The judge is the small model, and the answers come from the chat model, because a judge that is the model under test grades its own homework. In the reference run both were Haiku, so read the judge's scores beside its reasons, not alone.
  • Use this set as the regression suite. Run it before you give a prompt version traffic, and compare its mean with the live version's.
  • Day 15's investigation log already records agent decisions, and you would trace a tool's executions the same way, by name, through withTrace.
  • The live test's threshold of 0.7 is a floor, because model answers vary from run to run, not a target. A case under it on 2 runs is a regression to look at.

Alerts, rate limit and deploy

The last part prepares WorkBot to run while nobody watches it. Add failure and spend alerts that read the traces, a rate limit per visitor, and the deployment steps.

Study first

  • What to alert on. Alert on an error rate over a time window, with a minimum count of calls so that 1 failure in 2 calls does not page anyone. Alert on spend over a budget too. Alerting lives beside the traces, because that is where the numbers are. Google's site reliability engineering workbook compares alert windows in its chapter on alerting on service level objectives. Its section on low-traffic services explains why a few calls make an error rate noisy.
  • A token bucket. Each key gets limit tokens, refilled over per milliseconds, and each request takes one. In memory, the buckets live in one process, while a serverless platform runs many processes, so a shared store is what makes the limit global.
  • Environments. Give each environment its own project, keys and database, and never put secrets in the JavaScript sent to the browser. Keep .env.example as the list of what each environment needs. Vercel's page on environments covers local, preview and production, each with its own variables.

Build

  1. Create src/inngest/watch.ts, which runs on a cron every hour and on demand, with a check step that reads the last hour and the last day. A failure rate over 10% across at least 10 calls is an alert, and so is a day's spend over ALERT_DAILY_DOLLARS, which defaults to 5. Each alert is one alert-<n> step that writes to Day 18's outbox, the notifications table, for ops.
  2. Write a test that seeds 11 traces and expects 2 alerts.
  3. Create src/lib/rate-limit.ts with allow(key, { limit: 20, per: 5 minutes }). Call it in the chat route, keyed by the visitor, so the route answers 429 past the limit.
  4. Write the Deploy section of the README. Vercel hosts the app, and Inngest Cloud connects through its Vercel integration, which sets the 2 Inngest keys and syncs the functions on each deploy. The database is a Postgres with pgvector.
  5. In the same section, give each environment its own project, keys and database, with preview deployments as staging and the production branch as production. .env.example lists the keys. Run /pre-deploy, the skill from Day 2, before a release.

Done when

  • pnpm test src/inngest/watch passes.
  • 21 chat messages in 5 minutes from one browser get a 429 on the 21st.
  • If you have the accounts, vercel --prod from a project with the keys set serves the app, and the Inngest dashboard shows the functions synced from it.
Technical notes
  • The deploy needs a Vercel account and a hosted database, so the reference run did not run it from its repository. pnpm check builds what a deploy would.
  • Personal data, or personally identifiable information (PII). The traces hold no content, the analytics events hold ids, and the memory store is per visitor and erasable. A hosted trace tool would need the same restraint, or a filter in front of it.
  • The rate limit is the simplest that works. A product behind several instances moves the buckets to the database or a cache.

Resources