LLMs speak prose, but software needs data. Structured outputs make the model return machine-readable JSON that conforms to a schema you define, so its answers drop straight into your code, APIs, and databases without brittle string parsing or fingers-crossed regex.
Structured outputs are model responses that follow a machine-readable shape you specify in advance - typically JSON matching a schema - instead of free-form prose. You hand the model a contract ("return an object with these fields and these types"), and the response comes back as data your program can parse, validate, and act on directly.
You describe the target as a JSON Schema or a typed model (Pydantic, zod). This is the contract the output must satisfy.
The model is steered to emit only tokens that keep the output valid against the schema, so it cannot drift into prose.
The result is parsed into a typed object and validated. Anything that slips through is caught before it reaches your code.
| Approach | What it does | Guarantee |
|---|---|---|
| Prompt-and-hope | Ask nicely for JSON in the prompt; parse whatever comes back | None - best effort only |
| JSON mode | Provider flag forcing syntactically valid JSON, but any shape | Valid JSON, not your schema |
| Schema-constrained | Decoding restricted to a supplied JSON Schema at token level | Valid JSON and your schema |
| Grammar-based | A formal grammar (GBNF, regex) masks illegal tokens each step | Any formal shape you can express |
| Typed models | Pydantic / zod define, coerce, and validate the result in code | Runtime type safety + retries |
Prose is great for humans and terrible for machines. The moment an LLM output feeds another system - a database, an API call, a UI component - you need a predictable shape, not a paragraph you have to reverse-engineer.
Downstream systems consume fields, not sentences. A guaranteed schema means you can index, store, and route the output without fragile text extraction.
No more regex against "roughly JSON-ish" text. Constrained decoding removes the whole class of "the model added a friendly sentence before the braces" bugs.
Function and tool calling is structured output under the hood - arguments must match the tool's parameter schema exactly or the call fails.
A schema is a typed interface. Producers and consumers agree on it, so teams can evolve each side independently and catch drift early.
Same idea, four ways to picture it, so it clicks whoever you are.
It's a multiple-choice answer sheet instead of an essay. You cannot scribble in the margins; you fill the exact boxes, so the grading machine can read your answers instantly.
It's a typed return value instead of a raw string. The function signature promises a shape, and the type checker refuses anything that does not match before it reaches your logic.
Think of a standardized intake form. Every applicant supplies name, date, and amount in the same slots, so processing is uniform and nothing gets misfiled.
Like filling in a strict paper form with labeled boxes: one letter per square, date in DD/MM/YYYY. The boxes force the shape, so whoever reads it later never has to guess.
Structured output has two moving parts: constraint during generation, which shapes what the model can emit, and validation after generation, which verifies and coerces the result into a typed object your code can trust.
At each step the decoder computes which tokens would keep the output valid against the schema, masks the rest to zero probability, and samples only from what remains. Invalid JSON becomes impossible to produce.
The raw text is parsed and checked against a typed model. Missing fields, wrong types, or extra keys are rejected, and a failed check can trigger an automatic reask instead of a crash.
flowchart LR
subgraph Define["📐 Define"]
SC["🧾 JSON Schema / typed model"]
end
subgraph Generate["🔒 Constrained generation"]
PR["🧩 Prompt + schema"] --> CD["⛓️ Constrained decode"]
CD --> RAW["📄 Raw JSON tokens"]
end
subgraph Consume["✅ Validate & use"]
VAL["🔍 Validate"] --> OBJ["📦 Typed object"]
OBJ --> SVC["🚀 Downstream service"]
end
SC --> PR
RAW --> VAL
from pydantic import BaseModel, Field
from openai import OpenAI
class Invoice(BaseModel):
vendor: str
invoice_number: str
total_amount: float = Field(ge=0)
currency: str = Field(pattern=r"^[A-Z]{3}$")
due_date: str # ISO 8601
client = OpenAI()
# The SDK sends Invoice's JSON Schema and constrains decoding to it.
resp = client.chat.completions.parse(
model="gpt-model",
messages=[
{"role": "system", "content": "Extract invoice fields from the text."},
{"role": "user", "content": document_text},
],
response_format=Invoice, # schema-constrained output
)
invoice: Invoice = resp.choices[0].message.parsed
save_to_ledger(invoice) # already a validated, typed objectThe response_format is the whole trick: the schema derived from Invoice both steers generation and validates the result, so parsed is a real typed object, not a string you still have to trust.
Three views: token-by-token constrained decoding, the validate-then-reask loop, and the end-to-end path from schema to a typed object handed downstream.
sequenceDiagram
autonumber
participant App as 🖥️ App
participant Dec as ⛓️ Decoder
participant SM as 📐 Schema mask
participant LM as 🧠 Language model
App->>Dec: generate(prompt, schema)
Dec->>SM: compile schema to token constraints
SM-->>Dec: valid next-token set
loop for each token
Dec->>LM: next-token logits
LM-->>Dec: raw probabilities
Dec->>SM: which tokens keep JSON valid
SM-->>Dec: allowed token mask
Dec->>Dec: mask illegal tokens then sample
end
Dec-->>App: guaranteed schema-valid JSON
sequenceDiagram
autonumber
participant App as 🖥️ App
participant LLM as 🧠 LLM
participant Val as 🔍 Validator
App->>LLM: prompt + schema
LLM-->>App: candidate JSON
App->>Val: validate(candidate, schema)
alt valid
Val-->>App: typed object
App-->>App: proceed
else invalid
Val-->>App: errors (wrong type, missing field)
loop until valid or max retries
App->>LLM: reask with errors + prior output
LLM-->>App: corrected JSON
App->>Val: validate again
end
Note over App,Val: give up gracefully after max retries
end
sequenceDiagram
autonumber
participant Dev as 👩💻 Schema author
participant App as 🖥️ App
participant LLM as 🧠 LLM
participant Svc as 🚀 Downstream service
Dev->>App: define typed model (Pydantic / zod)
App->>App: derive JSON Schema from model
App->>LLM: prompt + schema (constrained)
LLM-->>App: schema-valid JSON
App->>App: parse into typed object
App->>Svc: call API with typed payload
Svc-->>App: 200 OK, record created
App-->>Dev: structured result logged
From an unstructured request to a validated, typed object flowing into your systems - the whole journey in order.
Write the desired output as a typed model or JSON Schema: fields, types, enums, required keys, and constraints like ranges and patterns.
Turn the typed model (Pydantic, zod) into a JSON Schema the provider understands. This is the contract both sides agree on.
Combine your instructions with the schema. Describe each field's meaning so the model fills the right value, not just the right type.
Pass the schema as the response format so the provider switches on schema-constrained decoding rather than free-form text.
The decoder compiles the schema into a state machine that knows, at every position, which tokens keep the JSON valid.
At each step the model's logits are masked to the allowed set, so it can only emit tokens that continue a valid document.
Optionally stream tokens as they arrive, parsing the growing JSON so the UI can render fields the moment they complete.
Once complete, parse the JSON and load it into the typed model, coercing values where the schema allows it.
Run semantic validation the schema can't express - cross-field logic, referential checks, sane ranges - beyond mere type correctness.
If validation fails, send the errors and prior output back for correction, looping up to a retry cap before falling back gracefully.
Pass the validated object to the API, database, or UI. Logs of failures and reasks feed evals to tighten the schema over time.
Valid JSON is not the same as correct data. These are the failure modes that bite in production.
Hitting the token limit mid-object yields JSON that starts valid and ends abruptly. Set a generous max, detect unterminated output, and reask rather than ship half a record.
Without constraint the model returns "42" for a number or "true" for a boolean. Typed validation with coercion catches and fixes these before they poison downstream math.
The model invents keys you never defined or fills required fields with confident guesses. Forbid extra properties and treat "unknown" as a first-class allowed value.
Deeply nested, sprawling schemas raise error rates and latency. Flatten where you can and split huge extractions into smaller, focused calls.
A schema guarantees shape, not truth. A perfectly typed invoice can still carry the wrong total. Pair structural checks with semantic evals and spot audits.
Retrying forever on an impossible constraint burns tokens and stalls requests. Always cap retries and define a clean fallback for genuine failures.
Track structural validity and semantic correctness separately, so you know whether to fix the schema or the prompt.
| Metric | What it tells you | Good sign |
|---|---|---|
| Schema validity rate | Fraction of outputs that parse and match the schema first try | High: constraints are working |
| Field-level accuracy | Are extracted values actually correct against ground truth? | High: right data, not just right shape |
| Reask rate | How often a retry is needed to reach a valid output | Low: fewer wasted round-trips |
| Truncation rate | Share of responses cut off before completion | Low: token budget is sufficient |
| Hallucinated-field rate | How often invented or wrongly-filled fields appear | Low: model stays inside the contract |
| Latency & cost / call | Extra time and tokens from constraints and reasks | Within your SLA and budget |
Three representative patterns showing structured outputs in production-style use.
A finance team needs vendor, amount, currency, and due date pulled from thousands of PDF invoices into their ledger.
An assistant must translate natural-language requests into precise calls against internal APIs - search, booking, refunds.
An onboarding flow collects applicant details from free-form chat and must populate a strict backend form.
Structured outputs are moving from a bolt-on parsing trick into a first-class, guaranteed capability of the model runtime.
Grammar and schema enforcement built into the serving stack, so valid output is the default rather than an SDK wrapper around retries.
Standardized partial parsing that yields typed, incrementally-complete objects, so UIs render fields the instant they resolve.
Fuller JSON Schema coverage - recursion, unions, conditionals, formats - closing the gap between what you can express and what decoding can enforce.
Typed outputs as the connective tissue of agents: retrieval fills fields, tools consume them, and each hop stays machine-checkable.
Model-graded and rule-based checks that verify meaning, not just shape, catching the "valid but wrong" outputs schemas miss.
Faster constraint compilation and token masking, shrinking the latency cost so structured output is free to turn on everywhere.