Day 17

Structured output and data extraction

You ship
An extraction engine with 4 schemas, measured on 18 documents.

What you'll build today

After Day 16, people can reach WorkBot by typing, by talking and with a recording. A lot of work still arrives as documents, though: invoices, emails, letters and forms, often as a PDF or an image. Today WorkBot learns to read them, so nobody has to retype what a document says.

You build an extraction engine that turns a document into a typed record. You describe the record with a Zod schema, the model fills it in, and the engine checks the result. When the result does not fit, the engine sends the error back so the model can correct it. The model also says how sure it is of each field, so a person needs to check only the doubtful ones.

Then you measure the engine on 18 labeled documents, so you know how often it is right before you rely on it. At the end of the day, a document given to WorkBot becomes a request for the right team, with its fields read and the doubtful ones marked.

By the end of the day you can

  • Extract structured data from unstructured inputs, reliably
  • Validate the model's output with Zod schemas, and retry when it fails
  • Build an extraction pipeline fit for production

Topics

Structured Output from Models

  • The challenge: models generate free-form text, but applications need structured data
  • Approaches:
    • JSON mode: ask for JSON in the prompt, then parse the response
    • Tool use for structured output: define a tool whose input schema is your output schema, and the model "calls" it with the structured data
    • XML tags: ask the model to wrap its output in tags, then parse them with a regular expression or an XML parser
  • With Claude, the most reliable approaches constrain the output to your exact Zod schema: a tool whose input schema it is, which the model fills in, or Anthropic's structured-outputs API
  • OpenAI structured outputs: a JSON Schema with strict: true. The same Zod schema serves both providers

Zod Schema Validation

  • Defining output schemas with Zod: objects, arrays, enums, unions, optional fields
  • Validation: parsing the model's output through Zod to get typed results
  • Error handling: what to do when validation fails
  • Retry with feedback: send the validation error back to the model, and ask it to fix the output
  • Progressive extraction: extract the simple fields first, then use them as context for the complex fields

Extraction Patterns

  • Entity extraction: pull people, companies, dates and amounts out of text
  • Classification: sort documents, emails and support tickets into predefined types
  • Document parsing: extract structured data from invoices, receipts, contracts and resumes
  • Multi-page extraction: process a long document page by page, then merge the results
  • Table extraction: turn tables in PDFs and images into structured data

Production Extraction Pipelines

  • Pipeline stages: receive the input, pre-process it, extract, validate, store. Pre-processing cleans the input, and runs optical character recognition (OCR) when needed
  • Input types: an email as text, a PDF parsed with pdf-parse, a web page fetched and cleaned, an image read by a vision model
  • Confidence scoring: how sure the model is of each field, with the low-confidence fields flagged for a person to review
  • Batch processing: many documents at once, in parallel, within rate limits
  • Quality monitoring: tracking extraction accuracy over time, and catching regressions

Build

Features shipped today

  • WorkBot: A document intake for invoices, emails, job postings, and letters and forms, read from text, a web page, a PDF or an image and filed as requests

Build a data extraction pipeline:

  1. Define 4 extraction schemas (Zod):
    • Job posting: title, company, location, salary range, requirements[], responsibilities[], and the workplace (remote, hybrid or onsite)
    • Invoice: vendor, number, dates, currency, line items[], subtotal, tax, total, payment terms
    • Email: sender, recipients[], subject, intent (question, request, information or complaint), action items[], urgency (low, medium or high)
    • Letter or filled form: sender, recipient, date, subject, summary, fields[], and what it requests
  2. Build the extraction engine:
    • Takes any input (URL, PDF file, raw text, image) and an extraction schema
    • Prepares each input by its type: fetches a URL and keeps the page's text, and sends a PDF or an image whole, since the model reads them
    • Extracts structured data from one Zod schema through the AI SDK: Anthropic's structured-outputs API or a tool, and OpenAI's strict structured outputs
    • Validates with Zod, and on a failure retries up to 3 times with the error sent back
    • Returns typed, validated data with a confidence score per field, and the fields to review
  3. Measure it on 18 labeled documents:
    • 5 job postings in different layouts
    • 5 sample invoices in different formats
    • 5 emails with different intents
    • 3 letters and forms
    • Measure extraction accuracy: the kinds classified right and the fields extracted right
  4. File each document as a request: classify it, extract it with its kind's schema, and file it for the right team. Keep the record on the request, with the doubtful fields marked

Step by step

One schema, both providers

Everything today rests on one idea, that the model fills in a schema you give it instead of writing free text. On Anthropic, that is tool use or the structured-outputs API, and on OpenAI it is a strict JSON Schema. With the Vercel AI SDK, the same Zod schema serves both, so you write each schema once.

Study first

  • The 4 ways to get structure. You can ask for JSON in the prompt and parse it, or ask for XML tags and parse them, for a model with no constrained mode. You can also define a tool whose input schema is the output, or use a provider's structured-output mode. The last 2 constrain the output to the schema's shape. What a schema cannot express, such as a z.refine rule or a date's format, can still come back wrong, and validation and the retry catch that.
  • Output.object in the AI SDK, from its guide to generating structured data. generateText({ output: Output.object({ schema }) }) picks the provider's mode and validates with Zod. When the output does not fit, the call rejects with NoObjectGeneratedError (see its reference page), which carries the raw text and the cause.
  • The model is chatModel("small"), which was Haiku in the reference run. It reads PDFs and images itself, so the pipeline needs no optical character recognition (OCR) step and no pdf-parse. Those belong to a model without document input, or to a text-only pipeline where a page's text is enough and cheaper.
  • Zod for documents. Use nullable() for what a document may not state, describe() on every field, and enums for the fixed vocabularies.

Build

  1. Create src/lib/document-model.ts with 4 schemas. Invoice has vendor, number, dates, currency, lines, subtotal, tax, total and terms, and Email has from, to, subject, intent, actions and urgency. Letter, for letters and filled forms, has from, to, date, subject, summary, fields and requested. JobPosting has title, company, location, salary range, requirements, responsibilities and workplace. Export them as schemas, keyed by kind.
  2. Describe the enums in words the model can apply. For example, an email is a complaint when something went wrong, even if it asks for a fix.

Done when

  • pnpm test src/lib/documents passes the toRequest cases for 3 kinds. You can run it once you write toRequest in the last part.
  • pnpm extract content/eval/documents/email-1.txt --as email prints an Email record whose intent is complaint. You can run it once you add the command in the next part and write email-1.txt, the damaged-delivery email, in the third.
Technical notes
  • In the installed @ai-sdk/anthropic, structuredOutputMode defaults to auto, which uses the structured-outputs API on the models that support it and a JSON tool on the rest. The OpenAI provider sends the JSON Schema with strict: true, and your code does not change between them.
  • Ask for dates as YYYY-MM-DD, and keep them as strings in the schema. A schema constraint cannot parse "October 1, 2026", but the model can.
  • Define each intent sharply. In the first live test of the reference run, the model read a damaged-delivery email as a request because it asked for replacements. The definition of a complaint in step 2 fixes that.

Validate, retry and score

A schema shapes the output, but it cannot promise that the content fits, so the engine checks every record. The model returns the record and a confidence for each top-level field. When the record does not fit the schema, the engine retries with the output and the validation error, up to 3 times.

Study first

  • Retry with feedback. The model can act on a validation error, but a bare retry gives it nothing to act on. Keep the conversation, the output and then the error, so the second attempt is a correction rather than a fresh guess. Zod's error lists each issue with its path and message. The AI SDK's message prompts show how an assistant message hands the model its own earlier output.
  • The model's confidence is a self-report, useful for triage but not a probability. Set a threshold and route what falls under it to a person. This build uses 0.7, a starting point to tune against the eval, a scored test of WorkBot's output that you build on labeled documents in the third part.

Build

  1. Create src/lib/extract.ts and define Source as { kind: "text" }, { kind: "url" } or { kind: "file", data, mediaType }. A file has one of 4 media types, application/pdf, image/png, image/jpeg or image/webp.
  2. Write content(source), which turns a source into message parts. It reads a page through fetchPage from Day 14 and textOf from Day 7, both in src/lib/web.ts. They allow public addresses only, check redirects and cap a page at 512 KB. A file goes in as a file part.
  3. Write fromPath(path), which reads .pdf, .png, .jpg, .jpeg and .webp as files, .html through textOf, and anything else as text.
  4. Write extract(source, schema, model, { attempts, threshold }). Wrap the schema as { data, confidence: [{ field, score }] }, and call generateText with Output.object. On NoObjectGeneratedError, append 2 messages before the next attempt: the raw text as an assistant message, then the user message That did not match the schema: <reason>. Return the corrected record only. Both providers take that shape.
  5. Return data, and confidence as a record keyed by field, built from the array. Add review, the fields under the threshold, and attempts.
  6. Add pnpm extract <file or url> [--as invoice|email|letter|job], which prints the record, the confidences and the fields to check. Without --as, it first classifies the document with classify, which you write in the last part.
  7. Test with a scripted model, a test model that returns set replies. A first output with a string where a number belongs goes back with the reason, and the engine accepts the second. 3 misses throw, and a page's text and a PDF's bytes arrive as the right parts.
  8. Add a live test, pnpm test:live src/lib/extract.live.test.ts, which renders an invoice as a PDF and a form as an image with Chrome. It expects the invoice's vendor, number, due date and total, and the visitor's name among the form's fields. For the damaged-delivery email, it expects complaint with high urgency.

Done when

  • pnpm test src/lib/extract passes, and the retry test shows exactly 2 model calls with an assistant message between them.
  • pnpm extract content/eval/documents/email-1.txt prints email (classified), an Email record, and a confidence per field.
Technical notes
  • The model returns confidence as an array of { field, score }, not as a record keyed by field. In JSON Schema, a record with open keys needs additionalProperties, which the object reference describes. OpenAI's strict mode requires additionalProperties: false on every object, so it can express the array but not the record.
  • The engine cuts the text of a page or a document at 20,000 characters. A PDF or an image goes to the model whole and is billed by the token, so a long PDF costs more than its text would. Past the model's page limit, split the PDF by page, extract each part, and merge the records by key.
  • The reason sent back is the cause's message, which is Zod's issues for a validation miss and the parser's message for malformed JSON. The raw text goes back as the assistant's own message, so the model corrects it rather than starting again.
  • Progressive extraction, in the fuller form the topics describe, is a second call. It puts the simple fields from the first pass into the prompt as context for the hard ones. The last part's classify, then extract, is its simplest form.

Measure on 18 documents

The engine works on one document at a time, but you need a number that says how well it works across many. Run classification and extraction on 18 labeled documents in 4 kinds, and print the fields it got right for each document and in total.

Study first

  • Field-level scoring. A string is right when the expected text appears in the extracted one, ignoring case and spacing. A list is right when every expected item is in it, a record when every expected field is, and a number when it is equal. Score the fields the document states plainly, and leave out fields where more than one value is defensible.
  • The grading section of Anthropic's guide to building evaluations. Code-based grading, by exact match or string match, is the fastest and most reliable, and field-level scoring is that kind.
  • Why the sources are HTML and text, not PDFs. Chrome renders an invoice as a PDF and a form as an image at run time. So the repository holds no binary files, and the eval still exercises the PDF and image paths.

Build

  1. Write the documents in content/eval/documents. Write 5 invoices, in table and prose layouts, with 3 currencies, one paid receipt and one bilingual invoice. Write 5 emails, which are a complaint, an FYI (for your information), a question, a past-due reminder and a leave request.
  2. Write 5 job postings, with hourly and yearly pay, and remote, hybrid and on-site work. Then write 3 letters and forms. That makes the curriculum's 5 of each of its 3 kinds, and 3 of the fourth.
  3. Write expected.json with the kind of each document and the fields to check.
  4. Write cli/extract-eval.ts. For each document, it renders HTML with Chrome, as a PDF for an invoice and as an image otherwise. It classifies the document, extracts it with the expected kind's schema, compares field by field with a same function and prints a Markdown row. At the end, it prints the kinds right and the fields right.
  5. Run pnpm extract:eval.

Done when

  • The table has 18 rows and ends with the kinds right and the fields right out of the fields scored, with the percentage. In the reference run, the eval got 18 of 18 kinds and 87 of 90 fields (97%). The misses were a shortened job title and a free-text list of requests.
The extraction eval in the terminal
18 documents on the small model, with the kind each was classified as, the fields right, and the ones wrong.
Technical notes
  • Treat a wrong field in the table as a question, not a verdict, because the expected value may be the one to fix. In the reference run it was, for the intent of the damaged-delivery email. The scoring rule may also be too strict, as the reference run's first scorer was: its exact string match failed a sender's name inside a longer line.
  • The eval costs 36 model calls on the small model, 2 per document, with a PDF or an image in 8 of them. The documents run one after another. A batch would run them in groups with Promise.all, and the SDK's maxRetries would ride out a rate limit.
  • Keep the eval in CI with a threshold, and add every corrected document to it. The number then tells you when a prompt or model change made extraction worse.
  • The job postings are 5 documents in different layouts, not pages from 5 live job sites, because live job sites change and an eval needs sources that stay the same.

Documents file their own requests

With the engine measured, the last part puts it to work inside WorkBot. Treat each document that arrives as a request: classify it, extract it with the schema of its kind, and file it for the right team. Keep the extracted record on the request, with the low-confidence fields marked for review.

Study first

  • Progressive extraction. A cheap first pass names the kind of document, and the kind decides the schema for the full pass. It is the routing workflow in Anthropic's "Building effective agents", which classifies an input and sends it to a specialized follow-up task.
  • Human review in the product. A person checks the fields below the threshold, and WorkBot saves the rest as extracted.

Build

  1. Create src/lib/documents.ts and write classify(source, model), which uses a one-field schema that describes the 4 kinds.
  2. Write toRequest(document), which maps each kind to a title, a team and a priority. Invoices go to finance, complaints and questions to support, and the rest to operations, with an email's urgency as the priority.
  3. Write ingest(source, name, model), which classifies, extracts, creates the request and saves the record.
  4. Add a documents table with the request, type, name, time, and data, confidence and review as jsonb. Write saveDocument and listDocuments.
  5. Add POST /api/documents, which takes JSON { text } or { url }, or a file under document as form data with one of the 4 media types. A URL must be a public http or https address, checked with isPublic in src/lib/web.ts. The route answers 201 with the request and the document, or 400 with the validation error. Set maxDuration = 120 on the route.
  6. Turn Day 16's recording-upload.tsx into a general FileUpload button, and use it for a File it from a document button on the new-request page. Its input is <input type="file" name="document" accept=".pdf,image/png,image/jpeg,image/webp">, submitted by requestSubmit on change.
  7. Write a Server Action submitDocument(formData), which reads formData.get("document") as a File, ingests it and redirects to the request. Set export const maxDuration = 120 on the page.
  8. Add a Documents section to the request page, where DocumentFields shows the record field by field with a Check badge on the fields to review.
  9. Test toRequest for 3 kinds, and ingest with a scripted model from classification to the row. Test the route with text and with bad input, and render FileUpload and DocumentFields in tests.

Done when

  • You print content/eval/documents/invoice-1.html to PDF from Chrome. On the new-request page, you click File it from a document and choose the PDF. You land on a request for the finance team titled "Pay Acme Supplies Co. invoice INV-4471, USD 1,551.35".
  • The invoice's fields show under Documents, with any doubtful field marked Check.
A request filed from an invoice
An invoice PDF read into a request for finance, with the record as WorkBot read it, field by field, and the fields to check marked.
Technical notes
  • toRequest takes a discriminated union { type, data }, so each branch sees its own record's type. NewRequest.parse keeps the title within 120 characters and the body within 5,000.
  • The route tells the 2 shapes apart with one check. A multipart body means a file, and the route parses anything else as JSON through DocumentBody.
  • A file comes to the Server Action as a File in FormData. Check its media type against the engine's list before you read a byte. The page's maxDuration of 120 seconds covers the 2 model calls.
  • Bind the data column, a jsonb column, as the object itself, as on Days 15 and 16.

Resources