Structured output is constrained decoding: before the sampler picks, a schema-compiled state machine masks every token that would break your JSON, making invalid shape mechanically impossible. Three tiers of guarantee, why tool calls are the same machinery, and the quality tradeoff nobody warns you about.
Every post in this series has circled the same awkward fact: a language model is a function that emits text, one token at a time. But the code that calls it doesn't want prose. It wants a record — an object with named fields and correct types that the next function can consume without a regex and a prayer. Structured output is how you close that gap. And the way it actually works is more interesting, and more mechanical, than "we trained the model to speak JSON."
Here's the thesis for this post: structured output is constrained decoding, and constrained decoding is a filter on the sampler from the sampling post. The model still produces a probability distribution over the next token. But before the sampler picks, a constraint engine looks at your schema, figures out which tokens would keep the output valid, and sets the probability of every illegal token to zero. The model isn't "trying" to produce JSON. It is mechanically incapable of producing anything else.
That single idea reorganizes everything else. Let's build up to it.
Ask a model to "respond in JSON" in the prompt and you'll get JSON most of the time. The failures are the problem. The model wraps its answer in Markdown code fences. It prepends "Sure! Here's the JSON you asked for:". It drifts a field name from user_id to userId. It invents a field you never asked for. It returns "2" where you expected the number 2, or "two" where you expected "2". As models have gotten better these syntactic slips have become rarer — but not zero — and the schema-level failures (wrong names, wrong types, missing required fields, hallucinated keys) are stubborn.
This matters because of a distinction the rest of the post depends on: syntactic validity and schema conformance are different guarantees. "This parses as JSON" is like "this compiles." "This matches my schema" is like "this typechecks." You can have the first without the second, and it will hurt you at 3am when a downstream worker expects a field that quietly went missing.
Vendors sell you three different points on a reliability curve, and confusing them is the most common structured-output mistake.
Tier 1 — Prompt-only. You ask nicely in the prompt. Easiest to write, no guarantee at all. The model can still add prose, wrap in fences, rename fields, invent keys. In production this needs a validate-and-retry loop: parse, catch the error, re-prompt, hope. Sometimes fine for a prototype; a liability for a pipeline.
Tier 2 — JSON mode. OpenAI's original response_format: {type: "json_object"} (introduced at their 2023 DevDay) guarantees the output parses as JSON. That's it. It does not guarantee your schema — required fields can still be absent, types can still be wrong, extra keys can still appear. OpenAI's own docs now treat json_object as legacy and steer you to the schema-based mode. Anthropic and Google have equivalent "give me JSON" modes. Useful, but "it parses" is a weak promise.
Tier 3 — Strict / constrained decoding. OpenAI's response_format: {type: "json_schema", strict: true}, Anthropic's strict tool use and JSON outputs, Google's responseSchema, and the open-source engines (Outlines, XGrammar, llguidance, LM Format Enforcer, llama.cpp's GBNF) all do the same thing: they compile your schema into a state machine and mask the model's logits at every step so it cannot emit a token that breaks the schema. OpenAI reported that gpt-4o with Structured Outputs scored a perfect 100% on their internal complex-schema-following eval, versus under 40% for the older gpt-4-0613 without it. The guarantee is structural and absolute: on supported models, invalid shape is impossible.
The mental model that sticks: JSON mode is "the model tries to output JSON, you verify." Strict mode is "the decoder physically cannot emit tokens that violate your schema."
This is the mechanical heart of the post, and it's pure software-engineering plumbing — no math required.
Start with the schema, regex, or grammar. Compile it into an automaton. For a regular language (a regex, an enum, a fixed date format) that's a finite-state machine (FSM). For a context-free language like JSON — which needs to track arbitrarily nested, balanced braces and brackets — you need a pushdown automaton (PDA), which is an FSM plus a stack to remember how deep you are. This is the exact same distinction from a compilers course: regular languages get finite automata, nested languages need a stack.
Now generate. At each step the model does its normal forward pass and produces logits over the whole vocabulary — tens of thousands of tokens. Before sampling, the constraint engine asks the automaton one question: given the current parse state, which tokens are legal next? Then it sets the logits of every illegal token to negative infinity. After the softmax, those tokens have probability zero. The sampler from the sampling post then runs its normal pipeline — temperature reshaping, top-p/top-k truncation, the actual draw — but only over the surviving legal tokens. Because the mask is reapplied at every single step, an invalid sequence can never be constructed. The randomness still lives in the sampler; the constraint just deletes the illegal options first.
A tiny worked example. Suppose the schema says the output is an object and we're at the very start. The automaton says: the only legal first character is {. Every token that doesn't begin an opening brace gets masked to −∞. Next the automaton expects either whitespace or a " to start a key name — so only those tokens survive. Once we're inside a string value, the mask opens up to let the model write freely until it's time to close the quote, at which point the automaton demands a , or a }. The model's content choices are still its own; its structural choices are on rails.
Step through it above and watch where the mask bites: the model's top pick at the start is the natural-language opener "Sure", but only { is legal; the key is pinned to the schema's one property; the value must be a string, so a number or true is masked; and once the single field is filled, a comma — which would begin an illegal second field — is masked, leaving only }.
If you've used lex/yacc, this is a familiar shape: the grammar is your specification, the compiled automaton is your parser, and the logit mask is the enforcer that rejects illegal tokens at the door. The constrained decoder is a parser generator pointed backwards — instead of validating a string after the fact, it prevents the invalid string from ever being emitted.
The obvious worry is cost. Doesn't checking every token against a grammar slow generation to a crawl? Early implementations did have real overhead, especially for context-free grammars, because the PDA's state depends on the stack and can't be fully precomputed. The modern engines solved most of it. Outlines pioneered precomputing an index over the vocabulary per FSM state so that the per-step lookup is roughly O(1). XGrammar (Dong et al., MLSys 2025) splits the vocabulary into context-independent tokens it can check ahead of time and context-dependent tokens it interprets at runtime, and overlaps the grammar work with GPU execution — the authors report up to 100x speedup over prior approaches and near-zero end-to-end overhead. Microsoft's llguidance computes a token mask in around 50 microseconds; OpenAI shipped llguidance in its Structured Outputs backend on May 20, 2025, to widen JSON Schema coverage and speed. The takeaway: on a well-built stack, the throughput cost of constrained decoding is small. The cost you should actually worry about is a different one — coming up shortly.
Here's the part most people miss: the most common structured output in production isn't the response_format feature at all — it's tool/function calling. When you define a tool with a JSON Schema for its arguments, and the model decides to call it, the arguments come back as JSON that matches that schema. Under the hood, the provider runs the exact same constrained-decoding machinery against the tool's argument schema. The tool's schema is the grammar.
This is why tool calls are so much more reliable than "output JSON please" — when you use a vendor's tool API, you're getting Tier 3 automatically, whether you realized it or not. Anthropic's docs are explicit that strict tool use "compiles tool input_schema definitions into grammars using the same pipeline as structured outputs," so a booking tool that declares passengers: int will get 2, never "2" or "two". OpenAI's strict: true on function definitions does the same; Google's Gemini function calling enforces the argument schema too. When we give tool use its own full treatment in the Extending section, this is the foundation it stands on.
Everything above makes constrained decoding sound like a free win. It isn't, and the reason is subtle enough that it's worth representing the genuine debate in the field honestly rather than picking a side.
The provocative finding came from Tam et al. (2024), "Let Me Speak Freely?" (EMNLP Industry Track). They compared models answering in free-form natural language versus being forced into structured formats, and their abstract states plainly: "we observe a significant decline in LLMs' reasoning abilities under format restrictions... stricter format constraints generally lead to greater performance degradation in reasoning tasks." The task-level numbers are dramatic in places — on GSM8K grade-school math, they report Claude 3 Haiku dropping from about 86.5% in free text to about 23% under a schema-constrained JSON format, and GPT-3.5 Turbo falling from roughly 77% to 49%. Crucially, they showed this wasn't a parsing artifact: on the Last Letter task with Llama 3 8B the parsing error rate was 0.148% yet the performance gap was 38%. Their diagnosis of the mechanism is the important part — when the schema forces the model to emit the answer field before it has written any reasoning, you've amputated its chain of thought. In one striking case, 100% of GPT-3.5's JSON-mode responses put the answer key before the reason key, turning chain-of-thought into blurt-the-answer.
But the same paper contains a nuance that got lost in the headline: format restriction is task-dependent. Tam et al. found that on classification tasks, constraining the output often helped — on the DDXPlus medical dataset, Gemini 1.5 Flash jumped from about 42% to 60% with JSON mode enabled — because narrowing the answer space reduces selection errors. Their actual conclusion is that "format restrictions, particularly constrained decoding (JSON-mode), can hinder reasoning abilities while enhance classification task accuracy."
The pointed rebuttal came from Will Kurt at .txt (Dottxt), "Say What You Mean" (November 2024). Re-running the same tasks, .txt found structured generation matched or beat free text across the board (GSM8K 0.77→0.78, Last Letter 0.73→0.77, Shuffle Objects 0.41→0.44). Their critique: Tam et al. used different prompts for the structured and unstructured conditions, so it wasn't apples-to-apples; the structured prompts didn't give the model room to reason; and the paper conflated true schema-constrained generation with OpenAI-style "JSON-mode." Their analogy: you can "prove" Rust is slower than Python by writing terrible Rust.
So who's right? The honest synthesis, which later work supports, is: the danger isn't structure, it's premature structure. Forcing the answer to be the first token is what hurts. If the schema lets the model reason first, the gap disappears. This is why the dominant practical pattern is to put a free-text reasoning (or analysis, or think_step_by_step) field first in the schema, before the fields that hold the final answer. Because generation is left-to-right, the model literally thinks in prose into that field, and the constrained fields that follow are conditioned on that reasoning. OpenAI's own launch post recommends exactly this: "It can be useful to give the model a separate field for chain of thought to improve the final quality of the response." The alternative is a two-step pipeline: one unconstrained call to reason, a second constrained call to extract. Either way, the framing for engineers is clean: premature structuring is refactoring before you understand the code. Give the model room to think, then lock the shape.
The streaming post covered streaming as SSE deltas and flagged that tool-call arguments arrive as JSON fragments that must be reassembled. Structured output has the same property. When you both constrain and stream, the deltas are partial JSON — {"loca, then tion": "San Fra, and so on — which is not valid JSON at any intermediate moment. You have two choices. Buffer everything and parse once at the end (simple, but you lose the streaming UX). Or parse incrementally with a tolerant parser that can handle a truncated document.
The SDKs increasingly do this for you. Anthropic's streaming emits input_json_delta events carrying partial_json fragments, and its SDK re-parses the accumulated buffer in partial mode on every delta (the Python SDK uses jiter's partial mode; the docs suggest Pydantic's partial parsing or the SDK helpers). OpenAI's SDK offers .stream() with a response_format that yields incrementally-parsed objects, and .parse() for the buffer-and-parse-once path. Libraries like partial-json-parser exist for when you're rolling your own. The mechanical point from the streaming post still holds: partial parsing is a state-machine problem, and you should let a library own that state machine.
The developer experience that has become dominant in Python codebases hides all of this. You define a Pydantic model — an ordinary typed class — and hand it to the SDK. OpenAI's client.chat.completions.parse() (and the newer responses.parse()) accepts the Pydantic model, generates the JSON Schema behind the scenes, sets strict: true, applies constrained decoding, and hands you back a typed, validated object. Anthropic and Google support the same Pydantic/Zod pattern, and the Instructor library (built on Pydantic) offers it uniformly across many providers with automatic validation-and-retry on top. TypeScript gets the identical ergonomics via Zod schemas.
"Pydantic in, Pydantic out." You define your output type once, and you get that type back. The FSMs, the pushdown automata, the logit masks — all invisible. That invisibility is exactly why understanding the mechanism matters: when something goes wrong, you need to know what the machine underneath is and isn't promising you.
Constrained decoding guarantees shape, not truth. This is the single most important caveat and it connects straight back to why models hallucinate. The JSON will be valid and match your schema; the values inside it can still be wrong. A field typed as an ISO date will be a valid ISO date — but it can be the wrong date. An enum will be one of your allowed values — but the model, boxed into your options, may pick the least-bad one when none fit, collapsing to a "safe" default. Constrained decoding does not fix hallucination; it just ensures the hallucination is well-formed.
Recent benchmarks make this vivid. The Structured Output Benchmark (arXiv:2604.25359) reports best value accuracy of 83.0% on text, 67.2% on images, and 23.7% on audio, "while JSON Pass remains substantially higher than Value Accuracy across all three source domains... schema compliance alone is an insufficient measure of structured output quality." And correctness can actually fall when you switch strict decoding on for a hard task: ExtractBench (arXiv:2602.12247) found GPT-5's field-level pass rate on credit agreements dropped from 86.9% to 70.0% under structured-output mode, "suggesting that the constrained decoding overhead — maintaining a valid grammar state across thousands of tokens — may compete with the model's capacity to attend to" the document content. Shape is solved. Correctness is still your job — which is what evals (a later post) are for.
None of this machinery is JSON-specific. Because the enforcer is just "mask tokens the automaton rejects," you can constrain to anything you can describe formally. Outlines lets you constrain to a bare regex — outlines.generate.regex(model, r"\(\d{3}\) \d{3}-\d{4}") forces a US phone-number format, digit by digit, with the format guaranteed. You can constrain to a context-free grammar (XGrammar, or llama.cpp's GBNF) to force SQL that parses, or a custom DSL, or valid UUIDs and IPv4 addresses. The same mechanism extends to YAML, XML, and CSV. JSON is just the most common target, not the boundary.
reasoning field first, then the structured fields — or split into reason-then-extract calls. Don't make the answer the first token.And the misconceptions worth killing: the model wasn't "trained to output JSON" — post-training nudged it, but the guarantee comes from the sampler mask, not the weights. JSON mode does not guarantee your schema. Constrained decoding is not free of quality risk on reasoning tasks. And no, even a frontier model produces invalid JSON at some nonzero rate without constraints — rare, but expensive when it happens in production, which is the whole reason this feature exists.
That closes the Interface section. Over these posts we've covered both halves of talking to a frozen function: how you write to it — the context window as the one shared budget, and roles as a learned convention over a single token stream — and how tokens come back out — sampling and temperature reshaping the distribution, streaming as a delivery pattern, and now structured output as a filter on that same sampler. Every one of these is a fact about the interface, not the model's insides. The model has been a black box that takes tokens and returns a distribution.
The next section, Representations, opens the box. We'll go below the API surface into embeddings — how tokens become vectors with meaning — attention, and why the transformer architecture won. The behavior we've been describing from the outside is about to get an inside.
Research papers
Vendor documentation
json_schema strict:true, json_object legacy status, SDK .parse() / .stream() helpers, schema restrictions like additionalProperties:false and all-fields-required).input_json_delta / partial_json).responseMimeType: application/json, responseSchema / responseJsonSchema).Libraries & engines
Commentary