Inference vs. training

In the neural network post we framed these systems as Software 2.0: dataset + architecture is the source, training is the compiler, the weights are the binary. In the last post we walked through what training actually does — pretraining bakes in capability, post-training shapes behavior — and ended with a fact that's easy to say and harder to internalize: once training is done, the weights are frozen.

This post is about what that means for the other 99.999% of a model's lifetime: inference. Every chat reply, every code completion, every API call is inference. And inference is a fundamentally different kind of computation from training, even though they share a single building block — the forward pass — that we already met in the first post.

The shortest way to say it: training writes the weights; inference reads them. Training is the build step. Inference is the program executing. They are as different as gcc is from ./a.out.

That distinction is the spine of this post. We'll spend most of our time on the read-path, because that's where users actually live — what happens, mechanically, when you send a prompt to an LLM — and then we'll line up the five contrasts that matter for engineering decisions.

The forward pass, alone

Recall the forward pass: tokens go in, they get embedded into vectors, those vectors flow through stacks of matrix multiplications and nonlinearities (attention layers and MLPs, in a transformer), and at the very end you get a vector of scores — one per token in the vocabulary — that becomes a probability distribution over "what comes next."

During training, this forward pass is just step one of three. Step two is the backward pass (backprop), which computes how every weight should nudge to reduce the loss. Step three is the optimizer step, which actually nudges them. Repeat ~10¹³ times.

During inference, only step one runs. No loss. No gradient. No optimizer. The weights are read out of memory, used to transform inputs into outputs, and put back unchanged. If you've ever shipped a compiled binary and watched it run, you already have the right mental model: the executable on disk is not modified by being run. The weights are not modified by being inferenced.

This is the single most important fact in this post, because every misconception about LLMs — "the model is learning from me," "the model gets smarter as I chat," "the model remembers our last conversation" — collapses against it. The weights are frozen. The model is a pure function of (weights, input tokens, sampling settings). Same inputs in, same distribution out.

One token at a time: autoregressive decoding

Here's where inference gets interesting, and where most engineers get a little surprised the first time they look closely.

Generating "Hello, world!" from an LLM is not one forward pass that emits a sentence. It's many forward passes that each emit a single token. The loop looks like this:

1. Feed the prompt into the model.
2. The forward pass produces a probability distribution over the next token.
3. Pick one (greedy = highest probability; otherwise sample according to temperature and top-p).
4. Append that token to the sequence.
5. Feed the new, one-token-longer sequence back into the model.
6. Repeat until you hit a stop token or a max length.

This is autoregressive decoding, and it's a structural property of how these models were trained — to predict the next token given all the previous ones — not an implementation choice. There is no "generate a paragraph" primitive. There is only "generate one more token, given everything so far," called in a loop.

Two consequences fall out immediately. First, generation is inherently sequential: the n-th token can't be computed until the (n−1)-th has been picked and appended, because it's part of the input. This is also why every LLM UI streams tokens — there's nothing to display until the next loop iteration finishes, so you might as well show it. Second, latency is shaped like prompt-length-plus-(per-token-cost × output-length), not like a single fixed-cost API call. Long answers are slow because they are literally many forward passes.

Two phases: prefill and decode

If you watch a careful explanation of LLM serving — NVIDIA's Mastering LLM Techniques: Inference Optimization is a good one — you'll see the inference loop split into two phases, because they behave very differently on real hardware.

Prefill is the first forward pass: the model ingests the entire prompt at once and produces the very first output token. Because the whole prompt is known up front, the model can process all of its tokens in parallel through every layer — it's one big matrix multiply per layer, the kind of dense compute GPUs love. NVIDIA's blog describes the prefill phase as "process[ing] input tokens in a highly parallelized manner" and characterizes it as compute-bound.

Decode is everything after that: one new token per forward pass, appended to the sequence, fed back in. Each decode step has very little parallel work — essentially one new vector going through each layer — so the GPU spends most of its time waiting for weights to stream in from memory rather than actually multiplying. As NVIDIA puts it: "The speed at which the data (weights, keys, values, activations) is transferred to the GPU from memory dominates the latency, not how fast the computation actually happens. In other words, this is a memory-bound operation."

That asymmetry — prefill is compute-bound, decode is memory-bandwidth-bound — is why prompt length and response length contribute to latency in different ways, and why "time to first token" and "tokens per second after first token" are reported as separate numbers. We'll come back to those in the later Latency and UX post; for now, just hold the shape.

The KV cache: memoization for transformers

Look at the autoregressive loop again. Step 5 says "feed the new, one-token-longer sequence back into the model." Naïvely, that means: at every step, the model attends across the entire sequence so far, re-doing all the work it did last step plus a little more. For a response of length n, that's quadratic work — and worse, you're recomputing things you already computed.

You don't have to. Inside each transformer layer, attention works by projecting every token into three vectors — a query, a key, and a value. The query for the current token gets compared against the keys of all prior tokens to decide where to look; the matching values get blended into the output. Crucially, the keys and values of past tokens don't change when a new token arrives. They depend only on the past tokens themselves and the (frozen) weights.

So you cache them. The KV cache is exactly that: a per-layer, per-token store of the key and value tensors from every prior step, kept in GPU memory across the decode loop. With the cache in place, each decode step only has to compute K and V for the one new token, append them to the cache, and run attention against the whole cached prefix. HuggingFace's text-generation-inference docs put it cleanly: "all the attention keys and values generated for previous tokens are stored in GPU memory for reuse." It's a memoization trick — the same one you'd reach for any time you had a sequential computation with an invariant prefix.

The KV cache is what makes long-context, real-time chat tractable. It is also why long context isn't free. The cache scales linearly with sequence length, number of layers, hidden size, and batch size, and it lives on the same expensive HBM as the weights themselves. The vLLM team's PagedAttention paper (Kwon et al., SOSP '23) opens with the warning that "the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically," and their accompanying blog reports that "existing systems waste 60% – 80% of memory due to fragmentation and over-reservation." KV-cache memory is one of the central engineering problems of modern LLM serving.

Five contrasts that change everything

Now that we've walked the read-path, the differences from the write-path become sharp. Here are the five contrasts I'd want any engineer who works near an LLM to internalize.

1. Compute shape. Training = forward + backward + optimizer step, in a tight loop. Inference = forward only. The backward pass alone costs roughly twice what the forward pass does, which is why Kaplan et al.'s 2020 scaling-laws paper writes — in §2.1 — that you should account for "the backwards pass (approximately twice the compute as the forwards pass)" and arrives at the famous heuristic of about 6N FLOPs per training token (where N is parameter count). The corollary is the canonical inference number: roughly 2N FLOPs per inference token, the forward pass alone. Same model, same matmuls, three times as much arithmetic per token at training time. Plus the optimizer step. Plus the activation storage backprop needs.

2. Frequency and cost shape. Training is a one-time (per model) enormous batch job. Inference is a small per-request cost paid forever — once per chat message, once per API call, millions to billions of times a day. The arithmetic compounds. AWS's SageMaker documentation states that "for production ML applications, inference accounts for up to 90% of total compute costs." NVIDIA's Jensen Huang, on the BG2 Podcast (Episode 17, October 2024), put the long-term framing more starkly: "This will make AI inference 1 billion times larger than it is today." Per call cheap; in aggregate, not cheap at all. (We'll spend a whole post on the economics of this in Caching, batching, and cost management.)

3. Memory and state. Training is a memory pig. It has to hold, simultaneously: the weights, the gradients (one per weight), the optimizer states (Adam-style optimizers keep two extra fp32 tensors per parameter — first and second moments), and the activations from the forward pass (because backprop needs them to compute the gradients). The ZeRO paper (Rajbhandari et al., 2020) does the accounting concretely in §3.1: for a model with Ψ parameters trained in mixed precision with Adam, it needs 2Ψ + 2Ψ + KΨ = 16Ψ bytes of memory. For 1.5B-parameter GPT-2 that is ~24 GB for the model state alone, versus only 3 GB to hold the weights in fp16. Inference has to hold only the weights (~2 bytes per parameter in fp16/bf16, so about 14 GB for a 7B model per NVIDIA's inference-optimization blog) plus the KV cache. Roughly an order of magnitude difference, sometimes more.

4. Hardware and latency profile. Training is a throughput-bound, latency-tolerant batch job. Nobody cares whether a particular training step finishes in 1.2 seconds or 1.5 seconds; what matters is tokens-processed-per-day across thousands of GPUs and that the cluster doesn't crash for a month. Inference is the opposite: interactive, latency-sensitive, with hard SLOs measured in time-to-first-token and inter-token latency. The two workloads have such different shapes that they often run on different hardware: H100s and massive training clusters for the build job; everything from those same H100s to AWS Inferentia, Groq LPUs, and consumer Apple Silicon for serving. Same model, very different boxes.

5. Determinism. With frozen weights, the forward pass is deterministic: given the same input tokens and the same numerics, you get the same logits out. Sampling is what introduces randomness at inference — temperature, top-p, top-k — and that randomness lives in the sampler, not in the model. Set temperature to 0 (greedy decoding) and, to a first approximation, you get the same output every time. (There are subtler reasons that doesn't quite hold in practice — batch effects, non-deterministic GPU kernels, floating-point reduction order — and we'll do a dedicated post on those. The big point for now: the model isn't "deciding differently" on different calls. It's the same function each time.)

What this means for the misconceptions

With training and inference firmly separated, the consequences from the last post become almost mechanical.

"The model is learning from our conversation." It can't be. Inference is forward-only; nothing writes back to weights. What's actually happening is that the model is conditioning on your conversation: the entire chat history lives in the context window, gets re-fed into the model at every step (efficiently, thanks to the KV cache), and shapes the probability distribution over the next token. That's often called "in-context learning," and the name is misleading — there's no learning in the gradient-descent sense, just very flexible pattern completion on the tokens you've given it.

"Knowledge cutoffs are a bug." They're a structural consequence of the write-path/read-path split. Whatever was in the training corpus is in the weights; nothing after is. Retrieval-augmented generation patches around this by stuffing fresh tokens into the context window — i.e., by feeding new information through the read-path at inference time. We'll do a dedicated post on cutoffs.

"Bigger context is free." Every extra token of context costs prefill FLOPs and adds to the KV cache. Context is not free; it's just often worth it.

What's next

Inference is the substrate beneath every real interaction users have with an LLM, and now we've named its moving parts: autoregressive decoding, prefill, decode, the KV cache, the five contrasts with training. That's enough scaffolding to ask harder questions about behavior.

Next in the series: Why models hallucinate. With frozen weights and a forward-only read-path, hallucination isn't a glitch — it's what a next-token predictor does when the right next token isn't strongly supported by the weights or the context. That post leans heavily on everything we just built.

Further out, in the Building section, we'll come back to the engineering consequences of this post: Caching, batching, and cost management (why the KV cache and continuous batching shape every serving stack), Latency and UX (why streaming exists and how time-to-first-token works), and Determinism (why "frozen weights" doesn't quite mean "byte-for-byte reproducible," and what you can rely on instead).

For now, hold the spine: training writes the weights, inference reads them. Everything else falls out of that.

---

Further reading

• Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. — Source for the C ≈ 6N FLOPs-per-training-token heuristic and the "backward pass ≈ 2× forward pass" accounting (§2.1). The ~2N-per-inference-token corollary follows directly.
• Rajbhandari, S. et al. (2020). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. — The clearest breakdown of why training memory is ~16 bytes per parameter (weights + gradients + fp32 master copy + Adam moments) versus the weights alone.
• Kwon, W. et al. (2023). Efficient Memory Management for LLM Serving with PagedAttention. — The vLLM paper. Why the KV cache is the central memory-management problem of modern serving, solved with an OS-paging analogy.
• vLLM team (2023). vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention. — Accessible companion to the paper; source of the "60–80% of memory wasted" figure.
• NVIDIA. Mastering LLM Techniques: Inference Optimization. — The prefill-vs-decode framing and the "decode is memory-bound" claim, with concrete per-model memory math. Start here for serving.
• AWS. Reduce ML inference costs on Amazon SageMaker. — Source for "inference accounts for up to 90% of total compute costs."
• Alammar, J. The Illustrated GPT-2. — The best visual explainer of autoregressive decoding specifically. Pair with The Illustrated Transformer for the forward pass.
• HuggingFace. KV cache strategies. — Practical, code-level treatment of the cache if you want to see it in the API rather than the abstract.
• Karpathy, A. (2023). Intro to Large Language Models. — The "parameters file + run.c" mental model for inference, for a general audience.