Day 10

Context and memory

You ship
WorkBot's web chat with context management and memory, deployed.

What you'll build today

After Day 9, WorkBot streams chat with tool calls and generates pages from a description, but it forgets the person between conversations. Someone who said last week that they work in finance and want short answers has to say it again. A long conversation has a different problem, because each reply sends the whole history to the model, and the history keeps growing.

Today you fix both. After each reply, a small model reads the exchange and extracts lasting facts about the person. These are their preferences, the people and systems they work with, and the cases they are on. The next conversation's prompt carries those facts, and a settings page lets the person forget any of them. You also give the history a token budget, a limit on how much of it goes with each reply, and summarize the older messages you cut.

By the end of the day, WorkBot remembers the person across conversations and still answers well when a conversation runs long.

By the end of the day you can

  • Keep a conversation's context within the model's token limits
  • Implement long-term memory for AI applications
  • Choose the right memory architecture and storage for each kind of memory

Topics

Context Window Management

  • Token counting in practice: read the usage field, and estimate with tiktoken
  • The context budget: divide it between the system prompt, memory, conversation history and the response
  • Truncation strategies:
    • Drop the oldest messages: simple, but it loses important early context
    • Keep the system prompt, the first message and the most recent N turns: this preserves the original intent
    • Summarize and truncate: compress older messages into a summary, and keep recent ones word for word
  • Dynamic context: include more or less history depending on the budget left
  • When to do what Claude Code's /compact does, from your own code

Conversation Thread Memory

  • Sliding window: keep the last N messages and drop the rest
  • Sliding window with summarization: summarize messages before you drop them
  • Hybrid approaches: always keep the system prompt and the user's first message, summarize the middle, and keep recent messages word for word
  • Thread branching: managing context when a conversation forks

Long-Term Memory

  • User preference memory: store likes, dislikes, settings and personal facts extracted from conversations
  • Fact extraction: use a model to pull structured facts from conversations ("user prefers dark mode", "user works at company X")
  • Conversation summaries: store a summary at the end of each conversation for future reference
  • Memory retrieval: when and how to put stored memories into the prompt of a new conversation
  • Memory decay: some memories matter more than others. Score them by recency, frequency and importance

Memory Architectures

  • Episodic memory: specific past interactions ("last Tuesday we discussed X")
  • Semantic memory: general facts and preferences ("user is a backend engineer who prefers Postgres")
  • Procedural memory: learned patterns ("user always wants code examples in TypeScript")

Storage Patterns

  • Redis: fast session memory with TTL (time to live) expiry, good for short-term context
  • Postgres: structured user profiles, conversation history and stored facts
  • Vector store: semantic memory search. Embed memories, and retrieve the relevant ones by similarity
  • Combining stores: Redis for the session, Postgres for facts and a vector store for semantic retrieval

Build

Features shipped today

  • WorkBot: Memory of each visitor's entities, cases and preferences, context management, and a settings page to view and manage memories. Deployed

Build WorkBot's full-stack chat app, combining everything from the week:

  1. Chat app with streaming (the Day 8 foundation)
  2. Tool calls with status indicators in the interface (Days 7 and 8)
  3. Generative UI through json-render: the model renders React components (Day 9)
  4. Context management: truncate long conversations, and summarize what you cut
  5. Memory system:
    • Extract the user's facts from conversations and store them (Postgres)
    • Retrieve the relevant memories at the start of each conversation
    • Let the user view and manage their stored memories (a settings page)
  6. Deploy to Vercel with the full stack working

Step by step

Know who is asking

Memory belongs to a person, so WorkBot first needs to know who is asking. On Day 5, src/proxy.ts gave every browser an anonymous visitor cookie, and for now that cookie is the person's identity, read in one function. Sign-in arrives on Day 30, and with it per-tenant scoping, which keeps each customer's data apart.

Study first

  • Read about cookies from next/headers, which is request-time. It works anywhere in a route handler, or in a Server Function, which runs on the server when the page calls it. On a page under Cache Components, the rendering model from Day 1, it must run inside a Suspense boundary.

Build

  1. Write src/lib/visitor.ts, where visitorId() returns the cookie's value, or undefined for an API request without the cookie.
  2. Change src/lib/analytics.tsx to read the id through visitorId(), instead of from the cookie jar.

Done when

  • pnpm test passes.
  • A curl to /api/chat without the cookie still gets an answer.
Technical notes
  • A caller without an id gets no memory, so WorkBot reads nothing for it and keeps nothing. A shared "anonymous" bucket would mix strangers' facts.

Extract and store memories

With an identity in place, WorkBot can keep facts about the person. A small model reads each exchange after the reply and returns any lasting facts as structured data, and the next conversation's prompt carries what is stored.

Study first

  • Read the AI SDK's guide to generating structured data. generateText with Output.object({ schema }) returns a parsed, typed object.
  • Learn what is worth keeping. Keep preferences (how the person wants answers), entities (their team, systems, vendors and people) and cases (the requests they work on). Do not keep what the assistant said, or what is already known.
  • Read about Next's after, which runs work once the response has been sent.

Build

  1. Add a memories table as one more statement in the schema list from Day 5. Give it visitor text, kind text, text text and created_at timestamptz, with unique (visitor, text), so the same sentence twice is one memory, whatever its kind.
  2. Write src/lib/memory.ts with listMemories, forget, forgetAll and remember, which inserts with on conflict do nothing from Postgres's INSERT. Add memoryNote(memories), which writes the note for the system prompt.
  3. Write extractMemories(model, exchange, known). It calls generateText with Output.object and the schema { memories: [{ kind, text }] }, with at most 3 entries.
  4. Pass the exchange ({ user, assistant }) as the prompt and the known facts in the instructions, and ask for each fact as one sentence about "the person". Run it on chatModel("small"), the provider's small model from the Day 6 list.
  5. In the route, read the visitor's memories before the reply and put memoryNote in the instructions.
  6. Then extract after the reply. toUIMessageStream calls its onEnd with the assistant's finished message, and there after() extracts facts from the question and the reply and stores what comes back.
  7. Write tests for the table's behavior on the in-memory database, the note, extraction against a mock model, and the route round trip with the small model mocked.
  8. Write live tests on the small model as smoke tests, quick checks that the pieces work at all. Extraction returns at least one fact about the person, and the route stores at least one memory after a real reply.

Done when

  • Tell the chat your team, and that you want short answers. The reply in the next conversation is short.
A second conversation
In a new conversation after the first, the prompt carries what WorkBot learned, and the reply stays short, as the person asked.
Technical notes
  • Extraction runs in after(), once the reply has gone out, so it adds nothing to the reply's latency. The next conversation sees the new facts, and the current one does not.
  • The known facts go into the extraction prompt, so the model returns what is new instead of what it has read before, and the unique constraint catches the rest.
  • chatModel("small") picks the provider's small model for extraction, while the reply still uses the large one.

View and forget memories

People should be able to see what WorkBot keeps about them and remove it. A settings page lists the memories and forgets them through Server Functions, so every change runs on the server, where the visitor can be checked.

Study first

Build

  1. Add /settings, which lists the memories grouped by kind, with a Forget button per memory and one for all.
  2. Make the buttons call Server Functions, marked "use server", that check the visitor.

Done when

  • /settings lists the facts, with a way to forget each one.
The settings page
The settings page shows what WorkBot remembers, grouped by kind. The small model extracted these from the first conversation, and the person can forget each line.
Technical notes
  • The settings actions parse the id as a UUID and scope every write by visitor, so a request cannot forget someone else's memory.

Fit long conversations to a budget

Memory carries facts between conversations, and this last part keeps a single conversation within limits. When a long history exceeds the budget, WorkBot fits it to the budget before each reply and summarizes the part it drops.

Study first

  • Read Anthropic's page on context windows. A model's context window is the most it can read in one call.
  • Learn token estimation. English text runs about 4 characters per token, a rough estimate that needs no tokenizer. usage on each response gives the exact count after the fact, and tiktoken counts exactly up front for OpenAI models.
  • Learn what to keep. The first user message holds the intent and the most recent messages hold the thread, so the middle can become a summary.

Build

  1. Write src/lib/context.ts, where sizeOf(message) counts every part, tool inputs and results included.
  2. Add fitContext(messages, budget), which keeps the first message and as many recent ones as fit, and hands back the rest.
  3. Add summarize(model, dropped), which asks the small model for a summary under 150 words, and compact, which combines the two.
  4. In the route, compact the validated messages with the small model, and send the kept ones. When there is a summary, add "Earlier in this conversation: …" to the instructions.
  5. Write tests. A history within budget comes back untouched, and over budget the first message and the latest stay. A large tool result counts, and compact summarizes only what was dropped.
  6. Add a live test that summarizes 3 real messages.

Done when

  • A conversation pushed past the budget, with a few page reads, still gets an answer, and the reply refers to something said at the start.
Technical notes
  • The budget is 24,000 tokens by default, which is under the reply model's context window, so the summary controls cost before it is ever a limit.
  • The summarizer's own input is bounded too. A transcript over 24,000 tokens keeps its newest part and says the rest was omitted. Tool results appear in it, cut to 500 characters each, so a fact a tool returned survives the cut.
  • Fitting walks back from the newest message, so the cut falls between turns and never splits a tool call from its result.
  • The summary is the model's account, not the messages, and the instructions say so. The person's own words from the start of the conversation stay as the first message.

Resources