Streaming: publishing, not generating

If you've used ChatGPT or Claude, you've watched the answer type itself out — a word, a few more words, a sentence unspooling in real time. It's tempting to read that as the model "thinking out loud," composing as it goes. It isn't. By the time you see the first word, the model has done nothing special; it's doing exactly what the inference post described — autoregressive decoding, one token at a time. Streaming is not a different way of generating. It's a different way of delivering.

That distinction is the whole post. Back in the inference post we established that generation is inherently sequential: the model runs prefill over your prompt, then enters the decode loop, producing one token, appending it, feeding the sequence back in, producing the next. That loop runs the same whether or not you stream. The only question is what the server does with each token the sampler picks (from the sampling post: the sampler is the separate step that turns the model's probability distribution into an actual token). It has two choices:

• Buffer: hold every decoded token until generation finishes, then return one big JSON object.
• Stream: flush each token to the client the instant it's produced.

Same tokens, same order, same total time. From the model's perspective streaming is essentially free — it was already emitting tokens one at a time; you're just forwarding them instead of hoarding them. From your perspective as the engineer, streaming changes almost everything: the transport, how you parse, how you handle errors, how you cancel, how you measure latency and cost, and what the UX feels like. A colleague of mine likes to put it this way: autoregressive decoding was always streaming internally — the server was just holding it for you.

Mechanically, what streaming is

Non-streaming: you POST a request, the server runs prefill and the full decode loop, serializes the complete choices (or content) array, and sends one HTTP response with a Content-Length. You wait for all of it.

Streaming: you send the same request with stream: true (OpenAI/Gemini) or "stream": true (Anthropic). The server keeps the HTTP response open and writes each new token — or small group of tokens — as a separate event, then closes the stream when decoding stops. The generation is byte-for-byte identical. What changed is framing: instead of one buffered payload, you get a sequence of small deltas.

That's the entire mechanical story. Everything else is the engineering that a sequence-of-deltas forces on you.

The transport: Server-Sent Events

Almost every major text LLM API streams over Server-Sent Events (SSE) — OpenAI, Anthropic, and Google Gemini all use it, and OpenAI-compatible self-hosted servers like vLLM do too. SSE is worth knowing precisely because it's so unglamorous: it's a plain HTTP response with a specific content type and a dead-simple text format.

The server responds with Content-Type: text/event-stream. The body is a stream of UTF-8 text. SSE is an old idea that turned out to be perfect for LLM output: it was first conceptualized around 2004, folded into HTML5 by Ian Hickson, published as a standalone W3C Working Draft in October 2009, and is now maintained in the WHATWG HTML Living Standard (§9.2). The format is deliberately trivial: each message is a block of lines like data: ..., and messages are separated by a pair of newlines (a blank line). A line beginning with a colon is a comment, often used as a keep-alive. An optional event: line names the event type. That's essentially the whole spec.

A minimal OpenAI-style stream looks like this:

data: {"choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"content":" world"},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Each event carries a delta; you append delta.content as it arrives; a final finish_reason tells you why generation stopped; and the literal sentinel data: [DONE] closes the OpenAI stream. Anthropic instead ends with a typed message_stop event; Gemini terminates by closing the stream.

Underneath, at the HTTP layer, this rides on HTTP/1.1 chunked transfer encoding (RFC 9112 §7.1), which lets a server start sending a body before it knows the total length — exactly the situation here. On HTTP/2 and HTTP/3, chunked encoding is forbidden; those protocols stream natively via DATA frames, and the SSE layer works the same on top.

Two framing notes worth internalizing. First, SSE is one-way — server to client only. That's fine for chat completion, where the client already sent the whole prompt up front. Second, the browser EventSource API only does GET, and these APIs are POST, so in practice you (or your SDK) parse the SSE stream manually rather than using EventSource directly.

When it's not SSE. WebSockets and WebRTC show up for realtime voice and multimodal — OpenAI's Realtime API uses WebRTC for browser audio and WebSockets for server-to-server, because those need bidirectional, low-latency audio, not one-way text. And a few text servers don't use SSE at all: Ollama streams newline-delimited JSON (NDJSON) by default, not text/event-stream. So: check the docs before assuming SSE.

What's in a chunk, and how vendors differ

Every streamed chunk carries a delta — new text (already detokenized; you get characters, not token IDs), sometimes with metadata like a finish reason or usage counters. The shapes diverge, and the differences matter when you write a parser.

OpenAI (Chat Completions) is the flat one: choices[0].delta.content holds the new text. The first delta carries role: "assistant"; subsequent ones carry content; the last carries a finish_reason.

Anthropic (Messages) is a structured, typed event stream. A response is: message_start → then, for each content block, content_block_start → one or more content_block_delta → content_block_stop → then one or more message_delta (top-level changes like stop_reason and cumulative usage) → message_stop. Interspersed ping events may appear. Crucially, a content block can be text, tool_use, or thinking — the block type tells you what channel you're reading. Text arrives as text_delta; extended-thinking content arrives as thinking_delta (and a signature_delta just before the thinking block closes).

Google Gemini uses streamGenerateContent, and there's a real gotcha: you must append ?alt=sse to the URL. Without it, Vertex/Gemini returns one big JSON array instead of an SSE stream — a bug that regularly bites people wiring up gateways. With alt=sse, each event is a GenerateContentResponse and the text lives at the verbose candidates[0].content.parts[0].text.

The lesson: "just parse the chunk" means something different per vendor, and abstracting over all three (which is what libraries like LiteLLM try to do) is where a lot of subtle streaming bugs live.

TTFT vs ITL: the latency structure streaming makes visible

Here's the most useful thing streaming gives you as an engineer — not speed, but visibility. Because tokens arrive over time, you can suddenly see that "latency" was never one number. It's two.

Time-to-first-token (TTFT) is the time from sending the request to the first streamed chunk. It's dominated by network + queueing + prefill — the model processing your whole prompt before it can emit anything. So TTFT scales with prompt length. This is a callback to the inference post: TTFT is a prefill problem.

Inter-token latency (ITL), sometimes called time-per-output-token (TPOT), is the average gap between successive chunks. It's dominated by the decode step and scales mainly with model size and how loaded/batched the server is (and weakly with context length via KV-cache size). ITL is a decode problem.

The total wall-clock time is:

total ≈ TTFT + (ITL × output_tokens)

Streaming does not reduce this total. What it does is change which number the user feels. Buffered, the user waits the whole total staring at a spinner. Streamed, the user's perceived latency collapses to roughly TTFT, because they start reading as soon as text appears.

A worked example. Say TTFT is 500 ms and ITL is 50 ms (i.e., 20 tokens/sec), generating a 200-token answer:

total = 0.5 s + (0.05 s × 200) = 0.5 + 10 = 10.5 s

Buffered, that's 10.5 seconds of blank screen. Streamed, the first words land at half a second and the rest scrolls in while the user reads. Same 10.5 seconds of compute — completely different experience. And it lands nicely against human reading speed: Brysbaert's 2019 meta-analysis (190 studies, 17,887 participants, Journal of Memory and Language) puts average adult silent reading at 238 words per minute for non-fiction (with most adults falling in a 175–300 wpm range) — very roughly 4–5 tokens/second — while chat-scale models emit tens to hundreds of tokens/second, so a streamed response comfortably outpaces the reader.

For real 2026 numbers, the independent benchmarker Artificial Analysis (median/P50 over rolling 72-hour windows, first-party APIs) is the usual reference. For non-reasoning, chat-scale models, median TTFT clusters in roughly the 0.3–1.5 s range and output speed runs from tens to a few hundred tokens/second — for example, on the Artificial Analysis leaderboard (accessed July 7, 2026) GPT-5.5 in non-reasoning mode sits around 1.0 s TTFT and ~77 tokens/sec, Google's Gemini 3.5 Flash (minimal) around 0.88 s and ~191 tokens/sec, with the lowest-latency production models (e.g. Cohere's North Mini Code and Command A+) near 0.3–0.4 s TTFT. One big caveat: for reasoning models, the same benchmark measures time to the first reasoning token, which can be tens of seconds — Artificial Analysis lists reasoning-mode SKUs like GPT-5.5 (xhigh) at ~100 s and Claude Sonnet 5 (max) at ~188 s TTFT. For those, the number users actually feel is "time to first answer token" after the thinking phase — a distinction worth keeping straight, and one the Latency and UX post will treat properly. (These are 72-hour rolling medians and drift day to day, so treat any single figure as indicative, not fixed.)

The engineering realities streaming introduces

Turning "call and wait" into "consume a stream" spawns a family of concerns that buffered responses never had. This is the part that actually costs you time.

Parsing partial content. For plain chat text this is trivial — append each delta to a buffer and render. Structured output (JSON, XML) is where it hurts: a JSON object is split across many chunks, so any given chunk is usually invalid JSON on its own. You either wait for completion, use a tolerant/incremental parser, or maintain a partial-parse state machine. (The mechanics of constraining and parsing structured output are the next Interface post; here just note that "JSON-parse each chunk" does not work.)

Tool calls stream too — as fragments. When the model calls a function, the arguments arrive in pieces. OpenAI sends tool_calls deltas where only the first carries the id and name; subsequent deltas set id: null and are keyed by index, and you concatenate function.arguments fragments per index. So you'll literally see {"lo then cation then ":" then Boston then "} across chunks, and you must reassemble by index before executing anything. Anthropic frames the same idea as a tool_use content block emitting partial-JSON deltas, closed by content_block_stop. A common real bug: dropping every delta whose id is null (because only the first has one) makes the whole tool call appear to "freeze" and then dump at the end. Full tool mechanics belong to the function-calling post.

Reasoning tokens stream on a separate channel. Claude's extended thinking streams as thinking_delta events inside a dedicated thinking content block; whether reasoning is exposed at all depends on the model and API version. Treat thinking output as a distinct stream, not as part of the answer text.

Errors can hit mid-stream — after you've already shown text. This is the nasty one. Once the SSE stream has started, the HTTP status is already committed at 200 OK. If the model server falls over, the error can't come back as an HTTP 4xx/5xx — it arrives inside the stream. Anthropic sends a typed error event (e.g. overloaded_error, which would have been a 529 in a non-streaming call); a well-known Anthropic SDK issue is that mid-stream errors get surfaced with status_code=200 because the underlying HTTP response was 200, breaking naive retry logic that switches on status code. OpenAI-style streams may instead just terminate abruptly. Either way: treat any in-stream error event as terminal, and design for "I've already rendered half an answer and now it's broken."

Cancellation actually works with streaming. Because you hold a live connection, aborting it (e.g. an AbortController) can tell the model server to stop decoding — which means you stop paying for further tokens. But semantics vary by provider: per OpenRouter's documentation, providers such as OpenAI, Azure, Anthropic, DeepSeek, Together, and xAI stop model processing and billing immediately on abort, while AWS Bedrock, Google/Google AI Studio, Groq, Mistral, and Perplexity keep generating upstream and can bill the full completion even after you disconnect. If a hard cost ceiling matters, set max_tokens as a backstop and check your provider's cancellation behavior rather than assuming. Cancellation by connection close is one of streaming's real wins over buffered calls, where there's nothing to cancel.

Backpressure and slow consumers. Mostly TCP handles this for you — if the client reads slowly, the OS flow-control slows the sender. It matters mainly for resource-constrained or on-device clients, where a slow reader can stall or (in some SDKs) even drop the tail of a response.

Retries get awkward. With buffered calls you just re-issue on failure. With streaming, you may have already delivered half the output. Your options: retry from scratch (loses/duplicates the partial), keep the partial with an "incomplete" marker, or fall back to a non-streaming call. There's no clean "resume from token N" in mainstream APIs.

Metering and usage counters. Streaming delivers text, not token counts, so usage often arrives only at the end. OpenAI historically didn't return usage in streaming mode at all until it added stream_options: {include_usage: true}, which appends a final chunk with the token counts (and, notably, that chunk may not arrive if the stream is interrupted or cancelled). Anthropic puts cumulative usage on message_delta / message_stop events. So: don't count on token accounting mid-stream; capture the final usage event, and handle the case where it never comes.

Proxy buffering — the classic gotcha. This one has burned nearly everyone. Reverse proxies and CDNs (nginx, some load balancers, corporate proxies) buffer responses by default — they accumulate your carefully-flushed chunks and hand them to the client in one lump, silently defeating streaming while everything looks fine locally. The canonical nginx fix is to send the X-Accel-Buffering: no response header (or set proxy_buffering off), which is exactly why you'll see that header set in production streaming code. If streaming works from your laptop but arrives all-at-once through your infra, suspect a buffering intermediary first.

What streaming enables — and what it doesn't change

The payoff is almost entirely about perceived responsiveness. A TTFT in the few-hundred-milliseconds range feels essentially instant; the user reads while the model writes; if the answer is heading the wrong way, they can cancel early instead of waiting for a wall of text; long generations get natural progress indication; and downstream pipelines can consume the stream — text-to-speech that starts speaking mid-generation, IDE completions that update live, or a UI that shows "calling weather_api…" the moment a tool_use block starts streaming.

Equally important is what streaming leaves untouched. It does not reduce total generation time. It does not change total cost — tokens are tokens whether or not you stream them. It does not change output quality, the sampling behavior (from the sampling post), or the context-window budget. And per token it is not more expensive. Streaming is purely a delivery choice layered on top of an unchanged generation process.

Where this goes next

Streaming is the default in modern LLM apps for one blunt reason: TTFT is typically well under a second to a few seconds, total generation can run many seconds, and streaming converts "wait ten seconds for a wall of text" into "start reading in half a second." It's a delivery optimization with an enormous UX return and effectively zero cost to the model.

Two of the threads opened here get their own posts. The next Interface post, on structured output, picks up the partial-JSON problem head-on: how to constrain a model to valid JSON and how to parse it while it streams in fragments. And in the Building section, Latency and UX will do the full TTFT/ITL treatment — perception thresholds, latency budgets, how prompt caching specifically attacks TTFT (cached prefixes let the server skip reprocessing, which is a prefill/TTFT win, not a decode/ITL one; Anthropic and AWS document up to ~85% latency reduction for long, fully-cached prompts), how batching trades latency for throughput, and how TTS pipelines ride the stream. For now, the one idea to keep: streaming is publishing, not generating. The model was always producing one token at a time. Streaming just stops the server from making you wait for the last one.

Further reading

Specifications

• WHATWG HTML Living Standard, §9.2 — Server-sent events.
• MDN — Using server-sent events and the EventSource API.
• IETF RFC 9112 (HTTP/1.1), §7.1 — chunked transfer coding.
• MDN — Transfer-Encoding header.

Vendor streaming docs

• OpenAI — Streaming API responses and Function calling (streaming tool-call deltas).
• Anthropic — Streaming messages (Messages API event types), Errors (mid-stream overloaded_error), and Prompt caching (the TTFT effect).
• Google — Generating content (streamGenerateContent?alt=sse).
• OpenAI — Realtime API via WebRTC / WebSocket.
• Ollama — Streaming (NDJSON).

Inference servers & performance

• vLLM — Metrics (TTFT, inter-token latency, prefill/decode).
• Red Hat Developer — 5 steps to triage vLLM performance.
• Artificial Analysis — performance methodology and model leaderboard.

Explainers & war stories

• Simon Willison — How streaming LLM APIs work.
• Julia Evans — Server-sent events: a simple way to stream events from a server (the nginx X-Accel-Buffering bug).
• OpenRouter — Cancelling a streaming request & billing.

Reading rate

• Brysbaert, M. (2019). How many words do we read per minute? A review and meta-analysis of reading rate, Journal of Memory and Language (190 studies, 17,887 participants; 238 wpm silent non-fiction).