Day 33

Process lifecycle: queues, pub/sub, SSE, WebSockets and serverless

You ship
A processing layer with a fair queue, dead letters, versioned events and streams across 2 instances, load tested with 500 jobs.

What you'll build today

After Day 32, WorkBot answers on four channels, and its work comes in different shapes. Some operations answer while a person waits, some stream, some run in the background, and some wait days for a reviewer. The recurring mistake is an operation filed under the wrong kind, such as slow model calls made inside a request.

Today you sort every operation into four kinds of work, a request, a stream, a job and a workflow, and give each kind what it needs. Jobs get a queue that shares the workers fairly between senders and keeps what still fails for a person. Domain events, records of what happened such as a request being filed, reach the code that acts on them through Postgres. A chat reply survives a reload, and live voice calls run over WebSockets on a server of their own, across two instances.

Along the way you learn what serverless functions cannot do, and how to design around it. By the end, a load test of 500 jobs with one misbehaving sender shows that the queue stays fair and loses nothing.

The four kinds of work
Requests and streams run in serverless functions, and jobs and workflows on Inngest. WebSockets run in a container. Postgres sits under all of them, including the NOTIFY backplane that links the realtime servers.

By the end of the day you can

  • Choose the execution model for each part of an agent system: request, stream, job or workflow
  • Build queue-backed and pub/sub-backed processing with backpressure, retries and dead letters
  • Know the limits of serverless runtimes and design around them

Topics

4 Kinds of Work

  • Request-response: under a few seconds, synchronous, cacheable
  • Streams: server-sent events (SSE) for token streams from server to browser (Days 6 and 8), and WebSockets for two-way audio and shared state (Day 16)
  • Jobs: seconds to minutes, retried, run by workers off a queue
  • Workflows: minutes to days, durable, with several steps that wait on people and events (Day 18)
  • Mapping: place every operation of an agent in one of the 4. Most products need all of them

Queues

  • Mechanics: producers, consumers, visibility timeouts and acknowledgements. Delivery is at least once, so handlers must be idempotent, safe to run twice
  • Backpressure: bounded concurrency per worker, per tenant and per provider rate limit, and what happens when the queue grows
  • Retries: exponential backoff with jitter. Dead-letter queues hold what keeps failing, and someone must look at them
  • Fairness: priorities across tenants, so one noisy tenant cannot starve the rest
  • Options: Postgres-backed queues at small scale, Redis-backed (BullMQ), or managed. Managed queues include Amazon Simple Queue Service (SQS), QStash and Google Pub/Sub

Pub/Sub + Events

  • Topics and subscribers: one event fans out to several consumers, with ordering guaranteed per key
  • Event schemas are versioned like APIs, and consumers tolerate unknown fields
  • Decoupling: the agent emits invoice.created, and 3 consumers act on it

SSE + WebSockets at Scale

  • SSE: one direction, reconnects with Last-Event-ID, and works through proxies. It is the default for streaming text
  • WebSockets: two-way and stateful. Across instances they need sticky sessions or a pub/sub backplane
  • Serverless and long connections: duration limits, streaming responses, and where a persistent server is still required

Serverless

  • Good at: request-response and short jobs with bursty load
  • Bad at: long-running agents, large memory, warm connections
  • Limits to read before designing: cold starts, concurrency, execution time, payload size
  • Mixing: serverless for web and API, containers for workers and WebSockets, a workflow engine for durable runs

Build

Features shipped today

  • WorkBot: Runs SSE for chat and WebSockets for voice across 2 instances, queues with a fair share per sender and dead letters, and domain events with 2 consumers

Rebuild your processing layer on explicit execution models:

  1. Classify every operation in the monorepo as request, stream, job or workflow, and write it down
  2. Jobs: a queue with bounded concurrency per sender, exponential backoff, a dead-letter queue and a small UI to inspect it. Use BullMQ on Redis, a managed queue, or the Inngest engine WorkBot already runs
  3. Events: WorkBot emits domain events to a topic, and 2 consumers (notifications, analytics) subscribe. Version the event schemas
  4. Streams: SSE for chat with reconnection, and WebSockets for the voice path behind a pub/sub backplane, so 2 instances can run
  5. Load test: 500 queued jobs with one misbehaving sender. Fairness holds, the dead-letter queue fills with the right failures, and nothing is lost

Step by step

Classify every operation

Before you change anything, find out what kind of work each operation is. The table you write here guides every later part of the day.

Study first

  • There are four kinds of work. A request answers in a few seconds while the caller waits, and a stream sends as it goes, with server-sent events one way and WebSockets both ways. A job runs outside the request for seconds to minutes, and is retried. A workflow runs for minutes to days, survives restarts, and waits on people and events (Day 18).
  • For each operation, ask how long it takes, whether someone is waiting, and what happens when it fails halfway.
  • The asynchronous request-reply pattern is how a request hands its work to a job. The endpoint answers 202 with an address to follow, and that address reports the work's state until it is done.

Build

  1. Write docs/execution.md, with every operation in a table that gives its kind, where it runs, and why. Pages and writes, webhooks and voice turns are requests, and the chat is a stream. Reading a document, answering a text message, delivering events and the crons are jobs, and document approval is a workflow.
  2. Read the table for what is filed wrong. Since Day 17, POST /api/documents has read each document inside the request, with two model calls and retries, which took 5 to 15 s in the reference run. Move the reading out, so the endpoint stores the document, hands it to the workflow, and answers 202 with an address to follow.
  3. Look at investigating a request, the job of the investigation agent from Day 11, which runs for minutes inside a server action. Leave it as a request today, and name it as the next to move.

Done when

  • Each operation has a row, and each one in the wrong place is either moved or named.

A fair queue with dead letters

The documents API now hands its reading to a job, and jobs share a queue. That queue must stay fair when one sender floods it, and it must keep the work that keeps failing.

Study first

  • A queue has producers, which add messages, and consumers, which take them and acknowledge them. Delivery is at least once, so every effect must be safe to repeat.
  • Backpressure means limiting work so that a backlog waits instead of overloading the system. Bound concurrency per worker, per tenant and per provider, where a tenant is how WorkBot separates one customer's data and settings from another's. Fairness means one noisy sender must not take every slot.
  • Retry with exponential backoff and jitter. Amazon's article on timeouts, retries and backoff explains both, and why retries at every layer multiply the load. A dead-letter queue holds the work that keeps failing, and someone must look at it.
  • For the queue itself, a Postgres-backed queue works at small scale, with consumers claiming rows through SKIP LOCKED, as the locking clause of SELECT describes. There is also Redis with BullMQ, or a managed queue. WorkBot already runs Inngest for its workflows.

Build

  1. Cap each Inngest function, counting the steps running at once, with Inngest's concurrency keys. A sender is whoever sent the work, by their user id. Let documents run two at a time per sender and ten in all.
  2. Let replies run one at a time per phone number and twenty in all. That answers a person's messages in order, mostly, since a retry can let a later one through first.
  3. Write failFast in src/inngest/fail-fast.ts. An error the provider marks as not retryable, such as a malformed PDF, ends the run at once, while anything else is retried three times, with backoff.
  4. Keep a run that gives up in dead_letters, with its event, its owner and the error, from the function's failure handler. /admin/jobs lists them by source and owner, and an admin can send one back once, after fixing or accepting the cause.
  5. Make the documents API keep text and page addresses (URLs) in the inbox beside files, under the sender's id. It answers 202 with /api/documents/<id> to follow.

Done when

  • A failure that no retry can fix is in the dead letters after one attempt, and a sent-back letter runs again once.
  • The API answers in milliseconds.
Jobs
The Jobs page lists dead letters by source and owner, each with Send back for an admin, and each event consumer's position.
Technical notes
  • Jobs and workflows share Inngest, whose functions take a concurrency key, retries and a failure handler. A second queue engine, Redis and BullMQ, would mean a second system to run and watch.
  • Sending back a text message's run needs a new event id. The engine drops a repeat of an event's id for a day, and the original id was the provider's message id.

Load-test the queue

The caps make fairness a claim, and a load test turns it into a measurement. This part floods the queue on purpose with one noisy sender.

Study first

  • A load test of a queue must show fairness under a noisy neighbor, the right failures in the dead letters, and nothing lost.
  • Amazon's article on queue backlogs covers isolating a noisy customer, alarming on dead letters, and measuring a backlog by the age of its messages.

Build

  1. Write pnpm load. A noisy integration sends 300 documents first, and 100 of them are PDFs the provider rejects. 4 other senders send 50 each after it, and the small model reads every document on the running engine.
  2. Count, per sender, what was sent, read and failed, the dead letters, and the time from sending to done.

Done when

  • The other senders are not stuck behind the backlog, and the dead letters are exactly the broken documents.
  • Every document is accounted for.
500 jobs
In the reference run, the others finished within 97 s while the noisy sender's backlog ran to 385 s. The 100 dead letters are its 100 broken PDFs, and nothing was lost.
Technical notes
  • In the reference run, every good document ended read and waiting for a reviewer, because a field fell under the confidence bar, as the approval workflow intends. For the queue, waiting for a person counts as done.
  • Once the others finished, the noisy sender was the only one with work, and its cap of 2 left 8 of the 10 slots unused for nearly 5 minutes. With a cap of 5, the batch finished in 152 s instead of 385 s, at about 40 s more for the others.
  • The reference run keeps 2, because people uploading a few documents matter more than a bulk run. Lending idle slots, or a lower priority for bulk senders, would get both.
  • The run tests the caps on this workload, not fairness in general, because Inngest schedules within a key first in, first out, on a best-effort basis.
  • In the reference run, the two 500-document runs cost $0.27 in model calls. A queue test does not need real work, but this one exercises the documents' real path too.

Domain events on Postgres

Some changes matter to more than one part of WorkBot. A new request needs a notification and a line in analytics. Domain events let each consumer act on its own, without the code that files the request knowing about them.

Study first

  • With topics and subscribers, one event goes to several consumers and each acts on its own. Ordering holds per key.
  • The outbox pattern writes an event in the same transaction as the change it describes, so the event is never lost and never invented. Read its notes on duplicates and order, because an event can be delivered twice.
  • Postgres can write both rows in one statement, with a data-modifying statement in WITH that passes the new request's id on through RETURNING.
  • Version events like APIs, because an event schema is an API. Consumers ignore fields they do not know, and a change that would break one is a new version.

Build

  1. Make createRequest, which files a request (Day 5), write the request and its request.created event in one SQL statement.
  2. Write src/lib/events.ts, with two consumers today, notifications and analytics. Each reads the events after its own position in event_offsets, in order. The notification is made once per event id, so a repeat delivery is harmless.
  3. Add readers by topic and version. A consumer that fails one event three times dead-letters it and moves past it, so one bad event cannot hold up the rest.
  4. Add an Inngest function that delivers the events every minute and on demand, one run at a time.

Done when

  • A request filed anywhere notifies its team once, however often its event is read.
  • An event of an unknown version waits in the dead letters while the ones after it go through.
Technical notes
  • Analytics accepts a rare repeat after a crash between its call and the move of its position, while a notification, which a person reads, does not.
  • Inngest can fan one event out to several functions, but sending to Inngest cannot happen inside the database statement that files the request. A row written with the change, and read by consumers, leaves no gap between the two.
  • Do not read events by id. Ids are handed out before commit, so two requests filed at once can commit out of order. A reader that has passed id 2 never goes back for id 1.
  • Keep on each event the transaction that wrote it (pg_current_xact_id()). Let a consumer read only events from transactions older than every one still running (pg_snapshot_xmin, one of Postgres's system information functions), in transaction order.
  • In the reference run's test on Postgres, one event was held uncommitted. Nothing was delivered until it committed, and then both arrived in order.

Chat replies that survive a reload

The chat is a stream, and a stream breaks when the page reloads in the middle of a reply. Keeping a copy of the stream lets the new page pick the reply up again.

Study first

  • Server-sent events go one way and pass through proxies. A reconnecting browser can send the last event it saw (Last-Event-ID), if the server numbers its events. Mozilla's guide to server-sent events shows the stream's format, and the HTML standard's section on that header says when the browser sends it.
  • A serverless function can stream while its invocation runs, but a reconnect is a new request, so resuming needs state saved somewhere.

Build

  1. Write record in src/lib/resumable.ts. It keeps a copy of each reply's stream in parts, about a tenth of a second each.
  2. Add GET /api/chat/<id>/stream, which replays the reply under way from its start and follows it to its end. It answers 204 when there is none.
  3. Set useChat({ resume: true }), as in the AI SDK's guide to resuming streams, so the page asks for the reply on load. Keep the conversation in the browser as soon as a message is sent, so a reload mid-reply knows which one to resume.

Done when

  • A reply cut off midway arrives whole after the reconnect, and the address answers 204 once the reply is done.
Resumed
In the reference run, a reply cut off after 211 characters was replayed from its start and followed to its end 1.5 s later.
Technical notes
  • The AI SDK's resumable streams use the resumable-stream package, built on Redis. WorkBot keeps the parts in Postgres instead, and the reader polls every 200 ms for at most a minute, which fits inside a function's limit.
  • The replay starts from the beginning, not from a Last-Event-ID, which fits a reload, since the new page has nothing. Continuing a live page's dropped connection from where it was would need the events numbered and a cursor.
  • Keep the conversation in the browser when the message is sent, not when the reply finishes. A browser that keeps it only at the end opens a fresh chat on a reload mid-reply, so a new conversation cannot resume.

WebSockets and serverless limits

Live voice calls need two-way connections that last, and that is where serverless functions stop fitting. This part runs the calls on a server of their own, then writes down the limits that decided it.

Study first

  • WebSockets are two-way and stateful. Vercel Functions support them, generally available since October 2026, but a connection is pinned to one instance and closes at the function's maximum duration, 300 s by default. A call can run longer, and the backplane's listener needs a process that stays up. Vercel's page on WebSockets in functions describes both limits and how a page should reconnect.
  • With several instances, a socket stays on the instance that accepted it, so a message for it must reach that instance. A pub/sub backplane, a channel that every instance listens on, does that. WorkBot's backplane sends every frame to every instance, and each instance delivers to the sockets it holds.
  • Read the limits of serverless before designing: execution time, request size, cold starts, concurrency, and no long-lived connections. Vercel lists its own on its page of function limits.

Build

  1. Write src/lib/realtime.ts and pnpm realtime, a WebSocket server. A voice call joins a room named by an id that only its page holds, and publishes each turn. Staff can watch every call, and a staff member with the agent role can say something into one.
  2. Use Postgres as the backplane, with its LISTEN and NOTIFY commands. Each instance listens on a channel named for its tenant, and publishes every frame with NOTIFY.
  3. Make the Day 16 voice line publish its turns and show what the team says. Add /admin/calls, which follows every call.
  4. Write the serverless limits for this app down in docs/execution.md. On Vercel, a function's request body is capped at 4.5 MB, its limit at the time of writing, so large recordings must go to storage directly.
  5. Note there that an investigation can run past a function's 300 s default and shows no progress meanwhile, which is why it should be a job. Live calls outlast a function's connection, so they run in the container.

Done when

  • A caller on one instance and a supervisor on another hear each other, and a viewer can watch but not speak.
  • Anyone deploying WorkBot can say which part runs where, and why.
A note from the team
The caller's voice line, connected to the first instance, shows a supervisor's note sent from the second.
Calls
The Calls page follows the call, with the caller, the assistant and the team's note.
Two instances
A supervisor on the instance at port 3302 receives the caller's turn from port 3301, and the caller's page shows the supervisor's note.
Technical notes
  • NOTIFY keeps nothing, so a supervisor who connects late sees calls from then on, and an instance that drops its listener misses what was sent meanwhile. A payload is at most 8,000 bytes. If missed frames matter, store them and send only their ids through NOTIFY.
  • Apply the schema under an advisory lock, a lock that Postgres holds on the application's behalf. Each process applies the schema on its first query, and create table if not exists is not safe against itself.
  • In the reference run, three processes started at once on an empty database, and without the lock two of the three failed.

Resources