A browser voice assistant and a timestamped video analysis tool.
What you'll build today
After Day 15, WorkBot can chat, file requests from what people write, and investigate a request with its research analyst. So far, everything people give it is text. Today you add 2 new ways in, speech and video.
First you build a voice line, a page where a caller describes a problem out loud and WorkBot asks a question or two before it files a request. Along the way you learn to model a conversation as a small set of states, and to stop speaking the moment the caller cuts in. You use the browser's own speech services, behind interfaces that a hosted speech provider could fill instead.
Then you teach WorkBot to read video. A vision model reads still images, so you cut a recording into frames, have the model describe them, and turn the descriptions into a timeline.
By the end of the day, a caller can file a request by talking to WorkBot. A recording attached to a request becomes a timeline that you can ask questions about.
By the end of the day you can
Add speech-to-text and text-to-speech to a TypeScript app
Build a real-time voice assistant that handles interruptions
Read and analyze video with a vision model
Topics
Speech-to-Text (STT)
Whisper API from OpenAI: high accuracy, batch processing, more than 50 languages
Deepgram: real-time streaming speech-to-text over a WebSocket, custom vocabularies, and speaker diarization, which labels who spoke when
AssemblyAI: real-time streaming and asynchronous processing, with summarization and entity detection
Choosing: Deepgram for real-time streaming, Whisper for batch work and accuracy, AssemblyAI for its wider feature set
Audio input: the browser's MediaRecorder API, the audio formats each provider requires, and sample rates
State of the art: impressive, but in most cases not yet reliable enough for production
Build
Features shipped today
WorkBot: A support voice line that handles interruptions and files the caller's request, and video recordings read frame by frame into a timeline on the request
Part 1: Voice AI Assistant
Build a voice assistant in the browser:
Speech-to-text from the browser's speech recognition, behind a Listener interface. Deepgram's WebSocket, fed by MediaRecorder, can fill the same interface
The next line from the small model, at most 300 characters, spoken whole rather than streamed
Text-to-speech from the browser's speech synthesis, behind a Speaker interface. Cartesia or ElevenLabs streaming can fill the same interface
Interruption handling: the caller can cut in at any time, and WorkBot stops speaking
Visual feedback: a waveform and a status that shows when WorkBot is listening or speaking
A typed conversation when the browser has no speech services
A filed request at the end of the call, through Day 12's intake workflow
Part 2: Video Analysis Tool
Build a video analysis pipeline:
Take a video file, from the command line or as an upload on a request's page
Extract frames with ffmpeg: 1 every 2 seconds by default, or 1 per second. Scene-change detection is the other way to pick frames
Send the frames to a vision model in batches, and keep a description of each frame
Write a timestamped summary: "0:00-0:15: Person enters room and sits at desk. 0:15-0:30: Opens laptop and begins typing..."
Answer follow-up questions about the video from the frame descriptions ("What was the person wearing?")
Step by step
1
The call as a state machine
A voice call is harder to get right than a chat, because the caller and WorkBot can both speak at once. So start with the rules of the conversation alone, before any microphone or speaker is involved. Model the call as 4 states and 5 events in a pure reducer, a function that takes the current state and an event and returns the next state. Because it touches no browser objects, you can test it on its own.
Study first
The pipeline in today's topics runs from the microphone to speech-to-text, the model, text-to-speech and the speaker. A production voice line streams at every stage and starts speaking before the model has finished. This build keeps each reply short, 300 characters at most, and speaks it whole.
Interruption and turn-taking. The caller may speak while WorkBot is speaking, and WorkBot must then stop and listen.
React's guide to extracting state logic into a reducer, which moves every change of state into one pure function of the state and an action. Its comparison with useState explains why a pure reducer can be tested on its own, apart from any component.
Build
1
Create src/lib/voice-model.ts and define Voice as { state, turns, interim }, with the states idle, listening, thinking and speaking. Define the events start, stop, heard, reply and spoken, where a heard event carries the text and whether it is final.
2
Write the reducer next(voice, event). A final transcript while listening adds a caller turn and moves to thinking, a reply adds a WorkBot turn and moves to speaking, and spoken returns to listening.
3
Put the interruption rule in one branch, so that heard while speaking moves to listening with the words as the interim text. Ignore everything heard while idle or thinking.
4
Write 3 tests, for a full turn, the interruption, and the events the reducer ignores.
Done when
pnpm test src/lib/voice-model passes 3 tests, and the interruption test starts in speaking and ends in listening with interim set.
Technical notes
The reducer holds no browser objects, so the component is the only place that touches the microphone, the speech recognizer and the speech synthesizer.
Ignoring words heard while thinking is a choice. The model is already answering what the caller said, and a final transcript at that moment would otherwise start a second request in parallel.
interim holds the caller's words so far, shown in italics, and the reducer clears it when a final transcript arrives or the call stops.
2
Speech behind 2 interfaces
With the rules in place, the call needs ears and a voice. The browser already has speech recognition and speech synthesis, so you can build the whole line without a speech provider's account. Wrap each one in an interface of 2 methods, a Listener and a Speaker, so that a hosted provider can replace either one without touching the rest.
Study first
The Web Speech API. Learn SpeechRecognition, prefixed webkit in Chrome, with its continuous and interimResults options, and speechSynthesis with SpeechSynthesisUtterance. Chrome ends a continuous recognition on its own after a while and fires end, so restart it until the call stops.
Where the hosted services fit. The browser's MediaRecorder API captures the microphone as chunks of audio, and Deepgram takes the chunks over a WebSocket and streams transcripts back, so it fills the Listener interface. Cartesia and ElevenLabs stream audio back, so they fill the Speaker interface, while the Web Audio API queues their chunks for playback.
Voice activity and interruption on a hosted line. A voice activity detector such as silero-vad says when the caller starts and stops speaking, and so can the provider's endpointing, its own detection of where speech ends. On an interruption, the line flushes the queued audio. In this build, the recognizer's interim results are the activity signal, and speechSynthesis.cancel() is the flush.
The getUserMedia and AnalyserNode browser APIs, which give you the microphone's samples for a waveform.
Build
1
Create src/components/voice-line.tsx and write webSpeechListener(), which returns { start(on), stop() }. The callback on gets the latest result's text and whether it is final, and recognition restarts on end until the call stops.
2
Write webSpeechSpeaker(), which returns { speak(text), stop() }. speak resolves on the utterance's end or error event.
3
Hold the reducer in the component. When the state enters thinking, post the turns to /api/voice, and when it enters speaking, speak the last line. When the caller cuts in, the state leaves speaking early, which runs the effect's cleanup and cancels the speech.
4
Create src/lib/voice.ts with voiceTurn(turns, model), which asks the small model for the next line, say, and whether it knows enough, done. When done is true, pass the caller's words through Day 12's intake to file the request. The last line names the team and the title the request was filed under.
5
Add POST /api/voice, which takes { turns }, and build the /voice page with a status pill, a waveform and the transcript as bubbles. Add a card that links to the filed request, and a text field that feeds the same reducer when the browser has no speech services.
6
Test voiceTurn with a scripted model, a test model that returns set replies, first a question and then a filing. Test the route too, and render the component in a test.
7
Add a live test, pnpm test:live src/lib/voice.live.test.ts, with the Anthropic key in .env.local. It files a caller's report on the small model, chatModel("small") from Day 6's model list, which was Haiku in the reference run.
Done when
You open /voice in Chrome and click Start the call, and the page shows Listening and a moving waveform.
You say what happened and WorkBot asks a question. When you cut in while it speaks, it stops.
After at most 2 questions, the card shows the filed request and the call ends.
pnpm test:live src/lib/voice.live.test.ts passes, filing the caller's report as an open request with priority high and a title that names the printer.
In another browser, or with the microphone refused, you can have the same conversation by typing.
A call on the small model, with the caller's turns, WorkBot's questions, and the request filed with its team at the end.Technical notes
The line ends the call itself after it speaks the filing line, because another turn would run intake again and file a second request.
speechSynthesis.cancel() fires error on the cut-off utterance, and that resolves speak. The effect's current flag stops that resolution from dispatching spoken into a state that has moved on.
Chrome does not always fire the utterance's end event, and in the reference run the line stayed in Speaking in 2 of 4 headless runs. Bound the wait with a timer sized to the text, which resolves speak when end never comes. The reference run made this fix at the start of Day 17.
The route's reply schema caps say at 300 characters. The prompt forbids lists and markdown, and allows at most 2 questions in a call.
In the reference run, headless Chrome driven by Playwright had both services. Synthesis fired start and end with no audible voice and recognition started without error, so the page ran for screenshots. Headless Chrome could not hear a WAV audio file played as a fake microphone, though, and the transcripts came back garbled.
3
A recording, frame by frame
The second half of the day turns from sound to pictures. A vision model reads still frames, not videos, so you turn a recording into frames first. With ffmpeg you take 1 frame every 2 seconds, then the model describes a batch of frames per call, and a second call writes the timeline.
Study first
The fps filter of ffmpeg and its image2pipe output, which writes frames as PNGs to standard output with no files on disk. Every PNG starts with the same 8-byte signature, so you can cut a stream of whole PNGs into frames at each signature.
Several images in one model call. The Vercel AI SDK sends each image as a file part, labeled by the text part before it.
Other ways to get frames. Use select='gt(scene,0.3)' in place of fps to take a frame at each scene change, or --every 1 for 1 frame per second. In the browser, a canvas drawImage takes a frame from a <video> or a webcam. After the frames, the pipeline is the same.
Video generation with Runway, Luma or Kling is an overview topic only, and nothing in WorkBot generates video.
Build
1
Create src/lib/video.ts and write extractFrames(file, { every, limit }). It spawns ffmpeg with fps and a scale to at most 1024 pixels wide, then cuts the output at each PNG signature into { at, png } frames. Write duration(file) too, which asks ffprobe for the length.
2
Write describeFrames(frames, model, { batch }), which sends 8 frames per call, each after a text part with its time, and returns { at, text } per frame through Output.object.
3
Write summarize(scenes, model), which writes one line per span of time and merges consecutive frames that show the same thing. Then write askRecording(scenes, question, model), which answers from the frame notes and gives the time.
4
Add the command pnpm video <file> [--every n] [--ask "<question>"].
5
Write the tests. Extract frames from a 5-second test pattern that ffmpeg generates inside the test, and check that a 1600-pixel-wide recording comes out at 1024. Batch descriptions 3 at a time in the test, against 8 in production, and check that the first call has 3 file parts. Check the timeline and an answer from scripted text.
6
Add a live test, pnpm test:live src/lib/video.live.test.ts, which needs Chrome, ffmpeg and the key. It draws 3 labeled scenes with Chrome and stitches them with ffmpeg. Then it checks that the small model reads each label, writes a timeline from 0:00, and answers which scene said what.
Done when
pnpm video walk.mp4 --ask "What did the second scene say?" prints the frame count, a timeline with time spans, and an answer naming the scene and its time.
Technical notes
fps=1/2 gives the first frame at 0 seconds and one every 2 seconds after. -frames:v caps a long recording at 90 frames, which is 12 model calls at 8 per batch.
The scale bounds what a frame costs in memory and in the request, whatever the recording's resolution, and a test checks that a 1600-pixel-wide recording comes out at 1024.
WorkBot answers questions from the frame notes, not the frames, which takes 1 text call instead of another pass over the images.
The ffmpeg build in the reference run had no drawtext filter. So the live test draws its labels in Chrome and uses ffmpeg only to stitch the images, with -framerate 1/3 showing each image for 3 seconds.
WorkBot keeps no transcript of a recording, because the audio track needs a hosted speech-to-text provider and this build does not have one. The recording's spoken words stay out of scope until you configure one.
4
Recordings on the request
A command proves the pipeline, but people attach recordings to their work, so this part puts the pipeline on the request page. A recording attached to a request becomes a timeline on the page, with a question box under it. The frame notes stay in the database, so WorkBot answers later questions without reading the video again.
Study first
Uploads through a Server Action. The file arrives in FormData, and the default body limit is 1 MB, which serverActions.bodySizeLimit raises, as the Server Actions configuration page explains.
useActionState shows a form's result in place, and useFormStatus lets a button show that the submission is running. useFormStatus reports on the form the button is rendered inside.
Build
1
Add a recordings table with the request, name, seconds, scenes as jsonb, summary and time. Then write src/lib/recordings.ts with saveRecording, listRecordings and getRecording.
2
Add attachRecording(id, formData) to the request actions. It writes the file to a temporary directory, extracts and describes the frames, reads the duration and writes the timeline. Then it saves the row, tracks a recording_attached analytics event with the request id and the seconds, and calls refresh(), while a finally block removes the directory.
3
Add askAboutRecording(recordingId, state, formData), which answers the question from the stored scenes and returns { question, answer } for useActionState.
4
Write src/components/pending-button.tsx, which shows a spinner and a busy label while the form runs, for both Day 15's Investigate button and the upload. Write recording-upload.tsx, a hidden file input and that button, so that choosing a file submits the form. Then write recording-question.tsx, the question box.
5
Add a Recordings section to the request page, listing each recording with its length, when WorkBot read it, its timeline and the question box. Set bodySizeLimit: "50mb" in next.config.ts.
Done when
On a request's page, you click Attach a recording and choose a short video. The page shows "Reading the recording…" until the timeline appears, with the length in m:ss.
A question under the timeline gets an answer with a time.
A screen recording of WorkBot attached to a request, with the timeline the small model wrote from its frames and a question answered from them.Technical notes
Bind the scenes column as the array itself, as Day 15 does for the reports table's findings. Both postgres.js and PGlite, the embedded Postgres from Day 5, serialize a value bound to a jsonb column, so a string would be encoded twice.
A 2-minute recording is 60 frames and 8 model calls on the small model, and the action waits for all of them within the page's maxDuration of 300 seconds.
ffmpeg reads the file from the temporary path, which the action then removes, so WorkBot stores only the frame notes and the timeline.
Call requestSubmit() from the file input's change event. It submits the form through its action, so the pending state and the refresh work as for any form.
