Week 8 · Day 37 of 40 · Build AI

User research, and ship to a customer cloud

You ship: A findings page from interviews and usability sessions, and a customer-style deployment of WorkBot proven by someone else from the runbook. Builds on: Day 33, Day 34, Day 35, Day 36.

By the end of the day you can

  • Deliver WorkBot into a fresh cloud account as a customer's platform team would receive it
  • Prove reproducibility, isolation, observability and rollback end to end
  • Plan and run user research inside a customer deployment: discovery interviews, observation of real work, usability sessions on the deployed product
  • Turn sessions into findings with evidence, and findings into configuration changes and backlog items, with consent and data handling in order

Topics

Why FDEs Do User Research

  • A deployment succeeds when the customer's people use it: adoption is the outcome, uptime is the precondition
  • 3 questions the first week answers: who does the work today, where it breaks, and what "done" looks like to them
  • The FDE's position: on site, with the people doing the work; the research happens in the deployment

Discovery Interviews

  • Who: the people who do the work (operators, support agents, finance clerks), their managers, and the admins who will run the product; 5 to 8 per role shows the patterns
  • Ask about the past and the specific: "walk me through the last ticket you handled", never "would you use this"
  • Follow the pain: what they do when the system fails, what lives in spreadsheets, what they explain to every new hire
  • Notes: one interviewer, one note-taker, verbatim quotes, no interpretation during the session; record only with consent

Observation and Contextual Inquiry

  • Watch the work as it happens, at their desk, in their tools, and ask about it as it happens
  • Capture steps, handoffs, workarounds, time per step, interruptions, and which tabs are open
  • The runbook test is contextual inquiry: someone else follows the runbook while you observe, record every stall, and do not help

Usability Sessions

  • Task-based: 5 realistic tasks written as goals; the participant thinks aloud; the observer does not help
  • 5 participants per round surface most of the serious problems; run rounds between fixes
  • For agents, record what the person expected the agent to do, what it did, whether they trusted the result, and where they checked its work
  • Severity per task: blocks it, slows it, or cosmetic; count occurrences across participants

Synthesis

  • From notes to themes: cluster observations across sessions; each theme carries a count and a quote
  • A finding names the behavior, the evidence and the consequence; an opinion names none of them
  • Agents help with the first pass: transcripts to candidate themes with quotes attached; a human merges, names and decides
  • Output: a findings page with severity, evidence and a proposed change per finding; most changes are configuration (Day 29) or backlog items; tell participants what changed

Consent, Privacy and Personal Data

  • Consent before any recording; participants can stop at any point; raw recordings deleted on a schedule
  • Usability sessions use scripted scenarios, never a participant's own records; notes stored without identifiers
  • Measuring after: the analytics stack (Day 39) shows whether the change moved the behavior

Build

Features shipped today

  • WorkBot: A production release checklist run against the neo-cloud deployment; delivered into a fresh customer account by someone else from the runbook; the runbook test observed as a usability study, interviews with them as the customer's operations team, 5 usability sessions on scripted scenarios, findings routed into configuration and the backlog

Deploy WorkBot into a new account as if it were a customer's.

Deliverables

  1. A fresh AWS or GCP account (or project) provisioned entirely by the Day 35 Terraform, with the Day 36 isolation
  2. The full system running: web, API, workers, queue, database, object storage, secrets, tracing
  3. Evals and load tests passing against the deployment; the Day 25 scoreboard pointed at it
  4. A deployment runbook a customer platform engineer can follow without you present, tested by someone else following it
  5. A security note and an architecture diagram suitable for a customer's review
  6. A research plan: who is interviewed (whoever runs the runbook plays the customer's operations and platform teams), 5 usability tasks, the consent text, and where notes are stored
  7. A findings page: themes with counts and quotes, severity per task, and a proposed change per finding marked configuration or code; the top 3 shipped or ticketed the same day

Delivery test: someone else runs the runbook, deploys WorkBot and runs the smoke evals. You observe without speaking and keep timestamped notes of every stall and question; those notes are the first input to the findings page.

Walkthrough

Get WorkBot ready for a customer to take over, and test what they will receive on the people who receive it. The release checklist is run against the last full deployment, and a one-page review goes to their security team. A usability study covers the runbook, the admin pages, the incident runbooks, the security documents and the cost estimate. Its most severe findings ship the same day. No customer account or customer staff were available, so this round's participants were agents, each given one role, one task and only the repository. They test whether the documents can be followed by someone who did not write them, not what real people need. The plan is written for the real sessions.

Part 1

The release checklist

Study first

  • What must hold before a customer is served: evals and load on the release, rollback, recovery from code, secrets, encryption, backups, monitoring, spend alerts, isolation, runbooks, retention, a review pack, an owner on call.

Build

  1. docs/release-checklist.md: each item done, planned or open, with where it was checked. Items that need a cloud account say so.
  2. Close the gaps the checklist finds in code: an uptime check on /api/health and a monthly budget with alerts at 50, 90 and 100%, both in infra/gcp and asserted in its tests.

Done when

  • Every item is done or names exactly what it waits for.

Part 2

A page for the customer's reviewers

Build

  1. docs/review.md: the architecture as one diagram, where the data goes, who can read it, what happens when the AI is tricked, how long data is kept and how it ends, and what the page does not cover.

Done when

  • A reviewer can find, for each claim, where to check it.
Review
The architecture for a customer's review: one host, a private database, secrets and data under the customer's key, every outbound call through the egress proxy.

Part 3

The research plan

Study first

  • Usability testing: scripted tasks, think-aloud, one facilitator who only asks "what would you do if I weren't here?". Outcome per task, every point of difficulty with its severity, quotes word for word.
  • Who receives the system: the engineer who deploys it, the lead who runs it, the reviewer who approves it, the on-call engineer, and the person who pays for it.

Build

  1. docs/research.md: the five roles and five tasks, the session format, the consent text read at the start, and where the notes live.

Done when

  • The plan could be run with the customer's people tomorrow.

Part 4

Five sessions

Build

  1. Give each participant one role, one task and the repository, and nothing else: stand up staging from an empty project, trace an invoice that was never filed, approve WorkBot for a dental group, stop spend running away at 2 a.m., set the monthly budget.
  2. Collect each one's outcome, sticking points with severity, and quotes.

Done when

  • Every task has an outcome and every sticking point a place in the documents or the product.
Sessions
Two completed with difficulty, two approved with conditions, one plan with gaps; five themes across them.
Technical notes
  • The platform engineer put together a working sequence, but only by merging Terraform, a release script and parts of the hand-written runbook. Three APIs, two service agents, the secret values before the services, and the DNS record before the release were missing or out of order.
  • The security reviewer found the questionnaire answering "yes" in the present tense for controls that are written and tested but not applied in a cloud account. The finding was about tense, not code.
  • The on-call engineer found that no switch holds spend without taking the assistant down. That is code, so it went to the backlog.

Part 5

Findings, and the top three shipped

Build

  1. Themes with the number of sessions that met each, a quote for each, and every finding's change marked documents, configuration or code.
  2. Ship the three most severe the same day. First, an ordered sequence from an empty project, with the missing APIs added to Terraform. Second, a path for a document that was never filed, with dates in the inbox and the document's name on a dead letter. Third, a questionnaire that says which controls run and which are only written.
  3. File the rest in the backlog.

Done when

  • The top three are merged, and every other finding has a ticket or a reason it has none.
Technical notes
  • The dead letter's document name first read the id from the wrong place in the stored event, so the label was empty for exactly the failures it was meant to name. A review caught it, and a test of the stored shape now guards it.

Resources