Week 7 · Day 33 of 40 · Build AI

Process lifecycle: queues, pub/sub, SSE, WebSockets and serverless

You ship: A processing layer with a fair queue, dead letters, versioned events and streams across 2 instances, load tested. Builds on: Day 16, Day 18, Day 32.

By the end of the day you can

  • Choose the execution model for each part of an agent system: request, stream, job or workflow
  • Build queue-backed and pub/sub-backed processing with backpressure, retries and dead letters
  • Know the limits of serverless runtimes and design around them

Topics

4 Kinds of Work

  • Request-response: under a few seconds, synchronous, cacheable
  • Streams: SSE for server-to-client token streams (Days 6 and 8); WebSockets for bidirectional audio and shared state (Day 16)
  • Jobs: seconds to minutes, retried, run by workers off a queue
  • Workflows: minutes to days, durable, multi-step, waiting on humans and events (Day 18)
  • Mapping: every operation of an agent onto the 4; most products need all of them

Queues

  • Mechanics: producers, consumers, visibility timeouts, acknowledgements; at-least-once delivery, so handlers must be idempotent
  • Backpressure: bounded concurrency per worker, per tenant and per provider rate limit; behavior when the queue grows
  • Retries: exponential backoff with jitter; dead-letter queues and who looks at them
  • Fairness: priorities across tenants; one noisy tenant must not starve the rest
  • Options: Postgres-backed queues at small scale, Redis-backed (BullMQ), managed (SQS, QStash, Pub/Sub)

Pub/Sub + Events

  • Topics and subscribers; fan-out of one event to several consumers; ordering guarantees per key
  • Event schemas versioned like APIs; consumers tolerate unknown fields
  • Decoupling: the agent emits invoice.created, 3 consumers act

SSE + WebSockets at Scale

  • SSE: one direction, reconnects with Last-Event-ID, works through proxies; the default for streaming text
  • WebSockets: bidirectional, stateful; sticky sessions or a pub/sub backplane across instances
  • Serverless and long connections: duration limits, streaming responses, where a persistent server is still required

Serverless

  • Good at: request-response and short jobs with bursty load
  • Bad at: long-running agents, large memory, warm connections
  • Limits to read before designing: cold starts, concurrency, execution time, payload size
  • Mixing: serverless for web and API, containers for workers and WebSockets, a workflow engine for durable runs

Build

Features shipped today

  • WorkBot: SSE for chat and WebSockets for voice across 2 instances; per-tenant queues with fairness and dead letters; domain events with 2 consumers

Rebuild your processing layer on explicit execution models:

  1. Classify every operation in the monorepo as request, stream, job or workflow; write it down
  2. Jobs: a queue (BullMQ on Redis, or a managed queue) with bounded concurrency per tenant, exponential backoff, a dead-letter queue and a small UI to inspect it
  3. Events: the agent emits domain events to a topic; 2 consumers (notifications, analytics) subscribe; event schemas are versioned
  4. Streams: SSE for chat with reconnection; WebSockets for the voice path behind a pub/sub backplane, so 2 instances can run
  5. Load test: 500 queued jobs with one misbehaving tenant; fairness holds, the dead-letter queue fills with the right failures, nothing is lost

Walkthrough

Put every part of WorkBot on the execution model it needs. The recurring mistake is an operation filed under the wrong kind. Today each operation is written down as one of the four, the documents API stops reading documents inside a request, jobs share the workers fairly between senders and leave what still fails for a person, domain events reach their consumers from Postgres, a chat reply survives a reload, and live voice calls run over WebSockets on a server of their own, across two instances. A 500-job load test with one misbehaving sender checks the queue.

The four kinds of work
Requests and streams in serverless functions, jobs and workflows on Inngest, WebSockets in a container, and Postgres under all of them, including the realtime servers' NOTIFY backplane.

Part 1

Classify every operation

Study first

  • The four kinds: a request answers in a few seconds while the caller waits; a stream sends as it goes, server-sent events one way and WebSockets both ways; a job runs off the request for seconds to minutes and is retried; a workflow runs for minutes to days, durable, waiting on people and events (Day 18).
  • The question for each operation is how long it takes, whether someone is waiting, and what happens when it fails halfway.

Build

  1. docs/execution.md: every operation in a table, the kind, where it runs, and why. Pages and writes, webhooks and voice turns are requests; the chat is a stream; reading a document, answering a text message, delivering events and the crons are jobs; document approval is a workflow.
  2. Read the table for what is filed wrong. POST /api/documents read each document with two model calls and retries inside the request, 5 to 15 s: it now stores the document, hands it to the workflow and answers 202 with an address to follow. Investigating a request runs an agent for minutes inside a server action: still a request today, named as the next to move.

Done when

  • Each operation has a row, and the ones in the wrong place are either moved or named.

Part 2

Jobs: a queue with fairness and dead letters

Study first

  • Queue mechanics: producers, consumers, acknowledgement, and at-least-once delivery, so every effect must be safe to repeat.
  • Backpressure: bounded concurrency per worker, per tenant and per provider. Fairness: one noisy sender must not take every slot.
  • Retries with exponential backoff and jitter; dead-letter queues and who looks at them.
  • Options: a Postgres-backed queue at small scale, Redis with BullMQ, or a managed queue. WorkBot already runs Inngest for its workflows.

Build

  1. Caps on each Inngest function, counted in steps running at once: two documents per sender and ten in all; one reply per phone number, so a person's messages are answered in order, mostly (a retry can let a later one through first), and twenty in all.
  2. failFast in src/inngest/fail-fast.ts: an error the provider marks as not retryable, such as a malformed PDF, ends the run at once; anything else is retried three times with backoff.
  3. A run that gives up is kept in dead_letters, with its event, its owner and the error. /admin/jobs lists them by source and owner, and an admin sends one back once, after fixing or accepting the cause.
  4. The documents API keeps text and page addresses in the inbox beside files, under the sender's id, and answers 202 with /api/documents/<id> to follow.

Done when

  • A failure no retry can fix is in the dead letters after one attempt, a sent-back letter runs again once, and the API answers in milliseconds.
Jobs
Dead letters by source and owner, each with Send back for an admin, and each event consumer's position.
Technical notes
  • A second queue engine, Redis and BullMQ, would mean a second system to run and watch, so jobs and workflows share Inngest, whose functions take a concurrency key, retries and a failure handler.
  • Sending back a text message's run needs a new event id: the engine drops a repeat of an event's id for a day, and the original id was the provider's message id.

Part 3

Load-test the queue

Study first

  • What a load test of a queue must show: fairness under a noisy neighbour, the right failures in the dead letters, and nothing lost.

Build

  1. pnpm load: a noisy integration sends 300 documents first, 100 of them PDFs the provider rejects; 4 others send 50 each after it. Every document is read by the small model on the running engine.
  2. Count, per sender: sent, read, failed, dead letters, and the time from sending to done.

Done when

  • The others are not stuck behind the backlog, the dead letters are exactly the broken documents, and every document is accounted for.
500 jobs
The others finished within 97 s while the noisy sender's backlog ran to 385 s; the 100 dead letters are its 100 broken PDFs; nothing lost.
Technical notes
  • Every good document ended read and waiting for a reviewer, a field under the confidence bar, which the approval workflow intends. Waiting for a person counts as done for the queue.
  • Once the others finished, the noisy sender was the only one with work and its cap of 2 left 8 of the 10 slots unused for nearly 5 minutes. With a cap of 5 the batch finished in 152 s instead of 385 s, at about 40 s more for the others. WorkBot keeps 2: people uploading a few documents matter more than a bulk run. Lending idle slots, or a lower priority for bulk senders, would get both.
  • The run tests the caps on this workload, not fairness in general: Inngest schedules within a key first in, first out, on a best-effort basis.
  • The two 500-document runs cost $0.27 in model calls. A queue test does not need real work, but this one exercises the documents' real path too.

Part 4

Domain events

Study first

  • Topics and subscribers: one event, several consumers, each acting on its own; ordering per key.
  • The outbox pattern: an event written in the same transaction as the change it describes is never lost and never invented.
  • Versioning: an event schema is an API. Consumers ignore fields they do not know; a change that would break one is a new version.

Build

  1. createRequest writes the request and its request.created event in one SQL statement.
  2. src/lib/events.ts: consumers, notifications and analytics today, each reading the events after its own position in event_offsets, in order. The notification is made once per event id, so a repeat delivery is harmless.
  3. Readers by topic and version. A consumer that fails one event three times dead-letters it and moves past it, so one bad event cannot hold up the rest.
  4. An Inngest function delivers every minute and on demand, one run at a time.

Done when

  • A request filed anywhere notifies its team once, however often its event is read, and an event of an unknown version waits in the dead letters while the ones after it go through.
Technical notes
  • Analytics accepts a rare repeat after a crash between its call and its position moving; a notification, which a person reads, does not.
  • Inngest can fan one event out to several functions, but sending to it cannot happen inside the database statement that files the request. A row written with the change, read by consumers, has no gap between the two.
  • The code review found a gap in the first version, which read events by id. Ids are handed out before commit, so two requests filed at once can commit out of order, and a reader that has passed id 2 never goes back for id 1. Each event now keeps the transaction that wrote it (pg_current_xact_id()); a consumer reads only events from transactions older than every one still running (pg_snapshot_xmin), in transaction order. On Postgres, with one event held uncommitted, nothing was delivered until it committed, and then both in order.

Part 5

A chat reply that survives a reload

Study first

  • Server-sent events: one direction, through proxies; a reconnecting client can say the last event it saw (Last-Event-ID), if the server numbers its events.
  • Serverless and streams: a function can stream while its invocation runs, but a reconnect is a new request, so resuming needs state saved somewhere.

Build

  1. record in src/lib/resumable.ts: a copy of each reply's stream kept in parts, about a tenth of a second each.
  2. GET /api/chat/<id>/stream: the reply under way, replayed from its start and followed to its end; 204 when there is none.
  3. useChat({ resume: true }) asks for it on load. The browser keeps the conversation as soon as a message is sent, so a reload mid-reply knows which one to resume.

Done when

  • A reply cut off mid-way arrives whole after the reconnect, and the address answers 204 once it is done.
Resumed
A reply cut off after 211 characters, replayed from its start and followed to its end 1.5 s later.
Technical notes
  • The AI SDK's resumable streams use the resumable-stream package, built on Redis. Postgres holds the parts here and the reader polls every 200 ms for at most a minute, which fits inside a function's limit.
  • The replay starts from the beginning, not from a Last-Event-ID, which fits a reload: the new page has nothing. Continuing a live page's dropped connection from where it was would need the events numbered and a cursor.
  • The first version could not resume a new conversation: the browser kept a conversation only when its reply finished, so a reload mid-reply opened a fresh chat.

Part 6

WebSockets across two instances

Study first

  • WebSockets: two-way and stateful. Vercel Functions support them, generally available since October 2026, but a connection is pinned to one instance and closes at the function's maximum duration, 300 s by default; a call can run longer, and the backplane's listener needs a process that stays up.
  • Several instances: a socket stays on the instance that accepted it, so a message for it must reach that instance. A pub/sub backplane does that; this one sends every frame to every instance, and each delivers to the sockets it holds.

Build

  1. src/lib/realtime.ts and pnpm realtime: a WebSocket server. A voice call joins a room named by an id only its page holds and publishes each turn; staff watch every call; an agent can say something into one.
  2. The backplane is Postgres: each instance listens on a channel named for its tenant and publishes every frame with NOTIFY.
  3. The voice line publishes its turns and shows what the team says; /admin/calls follows every call.

Done when

  • A caller on one instance and a supervisor on another hear each other, and a viewer can watch but not speak.
A note from the team
The caller's voice line, connected to the first instance, showing a supervisor's note sent from the second.
Calls
The Calls page following the call: the caller, the assistant, and the team's note.
Two instances
A supervisor on 3302 receives the caller's turn from 3301, and the caller's page shows the supervisor's note.
Technical notes
  • NOTIFY keeps nothing: a supervisor who connects late sees calls from then on, and an instance that drops its listener misses what was sent meanwhile. A payload is at most 8,000 bytes. If missed frames matter, store them and send only their ids through NOTIFY.
  • Starting the two instances found a race. Each process applies the schema on its first query, and create table if not exists is not safe against itself: three processes started at once on an empty database, and two of the three failed. The schema is now applied under an advisory lock.

Part 7

Serverless, and what it cannot do

Study first

  • The limits to read before designing: execution time, request size, cold starts, concurrency, and no long-lived connections.

Build

  1. Write them down for this app in docs/execution.md: a function's request body is capped at 4.5 MB on Vercel (its limit at the time of writing), so large recordings must go to storage directly; an investigation can run past a function's 300 s default and shows no progress meanwhile, which is why it should be a job; live calls outlast a function's connection, so they run in the container.

Done when

  • Anyone deploying WorkBot can say which part runs where, and why.

Resources