Temperature and sampling: the dice live outside the model

In the inference post I made a claim that I deliberately left half-finished: at every step, a language model doesn't output a token. It outputs a probability distribution over every token in its vocabulary. Then something picks one. I named that something — the sampler — and I named its most famous knob — temperature — and then I waved my hands and said "later." This is later.

Here's the thesis, and it's the single most useful thing to internalize about generation: temperature and sampling are not in the model. The model is a pure function. You feed it a sequence of tokens, it runs one forward pass, and it produces the same vector of numbers every time. That vector is deterministic. The randomness you experience when you chat with an LLM — the reason the same prompt gives you different answers — is added afterward, by a separate step that takes the model's distribution and chooses a token from it.

If you're a software engineer, this is a clean separation of concerns, and it's worth drawing the boundary sharply: the model is the deterministic part, the sampler is the random part. Almost every misconception about temperature dissolves once you put the knob on the correct side of that line.

The pipeline: distribution in, token out

Let's trace one step. The forward pass ends by producing a logit for every token in the vocabulary — one raw, unbounded score per token. Modern vocabularies are large; Llama 3, for example, uses a tokenizer with a vocabulary of 128,256 tokens (up from 32,000 in Llama 2), so that's a vector of 128,256 numbers. Higher logit means "more plausible here," but logits aren't probabilities yet — they don't sum to anything in particular.

To turn them into a probability distribution, you apply softmax: exponentiate every logit and normalize so the whole thing is positive and sums to 1. That's the entire math story of this post, and you already met it in the inference post. Softmax over logits gives you a proper distribution: token A might get 0.6, token B 0.25, token C 0.1, and a long tail of tens of thousands of tokens splitting the last sliver.

Now — and only now — the sampler runs. Its job is to pick one token from that distribution. The simplest picker is greedy decoding: just take the highest-probability token (the argmax). Greedy is fully deterministic. But most chat models don't run greedy by default; they sample — they roll a weighted die where each token's chance of coming up equals its probability. Token A wins 60% of the time, B 25%, and so on.

So the loop is: forward pass → logits → softmax → distribution → sample one token → append it → do it all again. Steps 1–4 are the model and are deterministic. Step 5 is the sampler and is where the dice live. Temperature, top-k, top-p — every knob in this post — operates between the distribution and the die. None of them touch the model's weights or what it "knows."

What temperature actually does

Temperature is a single scaling factor you apply to the logits before softmax. The whole operation is softmax(logits / T). That's it. Dividing the logits by a number and then running the same softmax as always.

The intuition is what matters. Dividing every logit by the same T doesn't change which token is on top, but it changes the contrast between tokens:

• T < 1 sharpens the distribution. Gaps between logits get magnified, so the model's favorite tokens dominate even more. At T = 0.2, a token that was at 60% might climb well above 90%.
• T = 1 leaves the distribution exactly as the model produced it — the "native" shape that training optimized for next-token prediction.
• T > 1 flattens the distribution. The contrast shrinks, so lower-probability tokens get a bigger share of the mass. More surprising tokens become reachable.
• T = 0 is a special case. You can't literally divide by zero, so implementations treat T = 0 as "skip the dice, take the argmax" — i.e., greedy decoding. Deterministic in principle.
• T → ∞ drives every token toward equal probability — a uniform distribution, which is pure noise.

A useful mental model: temperature is a dial that continuously reshapes the curve. Low and the curve becomes a spike on the model's top pick; high and it spreads out into the tail. It does not add information or take any away — it only redistributes confidence.

Top-k, top-p, and min-p: truncation, not reshaping

Temperature reshapes the whole distribution. The next family of knobs does something categorically different: it truncates the distribution — it throws tokens out of consideration entirely before the dice roll. If temperature is a dial, these are gates.

• Top-k keeps only the k highest-probability tokens and discards the rest. k = 40 means "only ever consider the 40 most likely next tokens." Simple, but rigid: 40 might be too many when the model is confident and too few when it's genuinely uncertain.
• Top-p (nucleus sampling) is the adaptive fix, introduced by Holtzman et al. in "The Curious Case of Neural Text Degeneration" (2019). Instead of a fixed count, you keep the smallest set of tokens whose cumulative probability exceeds p — e.g., p = 0.9 keeps just enough top tokens to cover 90% of the mass and cuts the rest. When the model is confident, that might be two tokens; when it's unsure, it might be fifty. Top-p is the most common truncation method in modern deployments.
• Min-p is a newer variant (Nguyen et al., 2024) that sets the cutoff relative to the top token: keep any token whose probability is at least p times the most likely token's probability. min_p = 0.1 with a top token at 50% keeps everything above 5%. It adapts to how peaked the distribution is. (Worth noting: a 2025 reanalysis by Schaeffer et al., "Min-p, Max Exaggeration," disputed the original paper's quality claims, so treat min-p as promising-but-contested rather than a clear winner.)

The crucial point: these cut the tail, then the survivors are renormalized back to sum to 1, and then sampling happens. They change which tokens are eligible; temperature changes how the eligible ones are weighted. Two distinct operations on the distribution.

And they interact. Temperature and top-p are applied together in the same pipeline, and the order and combination matter. Crank temperature up to flatten the curve, and a low top-p will immediately clip most of the new long tail you just created — they fight. As one sampling guide puts it, at low temperature with top-p 0.9 the nucleus typically contains only a handful of tokens because the sharpened distribution concentrates mass quickly, whereas a high temperature flattens the distribution so the same top-p 0.9 admits a far larger nucleus — potentially hundreds of tokens. This is exactly why most vendors tell you to tune one or the other, not both — more on that below.

Other knobs, briefly

A few more levers live in the same post-model layer:

• Repetition / frequency / presence penalties nudge the logits of tokens you've already used downward, to discourage loops. OpenAI's frequency_penalty scales with how often a token has appeared; presence_penalty is a one-off hit for any token that has appeared at all. Both work by subtracting from logits before sampling.
• Logit bias lets you add or subtract a fixed amount from specific tokens' logits — to forbid a word, or force one. At the extreme it's a ban or a guarantee.
• Beam search keeps several candidate sequences alive at once and maximizes total sequence probability. It's standard in machine translation, but it's mostly not used for chat-style LLMs: maximizing probability produces bland, repetitive text — the exact "degeneration" Holtzman's paper was about. (Name it, move on.)
• Speculative decoding (Leviathan et al., 2023) is a speedup: a small "draft" model proposes several tokens and the big model verifies them in parallel. Critically, it's designed to leave the output distribution unchanged — it's a latency trick, not a sampling strategy.

The practical surprises

This is the part worth the price of admission.

"Temperature = creativity" is misleading. Temperature changes the shape of the distribution, not the model's capability. It is a shape knob, not a quality knob. Higher temperature doesn't make the model smarter or more imaginative; it makes it less predictable, which for some tasks reads as "creative" and for others reads as "wrong." A bad answer at temperature 0 is still a bad answer at temperature 1.5 — just phrased with more flair. If the output is wrong in kind, no slider fixes it; fix the prompt.

There is no universal sweet spot. The right temperature is task-dependent. The patterns that hold up across vendors and practice:

• Code, SQL, JSON, structured extraction: 0 or very low.
• Factual QA, summarization, math: 0 to ~0.3.
• General chat and instruction-following: ~0.5–0.7.
• Brainstorming and creative writing: ~0.7–1.0.
• Above roughly 1.2, most modern instruct models visibly degrade — at OpenAI's max of 2.0 you mostly get gibberish.

"Temperature 0" is mostly deterministic but not byte-perfect. Greedy decoding always takes the argmax, so in principle the same input gives the same output. In production it usually doesn't, and the reason is worth understanding. The classic explanation — "floating-point math isn't associative, so (a + b) + c ≠ a + (b + c), and GPUs add things in unpredictable orders" — is partly right but misses the real culprit. The September 2025 Thinking Machines Lab post "Defeating Nondeterminism in LLM Inference," by Horace He and collaborators, pins it on batch invariance failure: production servers batch your request with other users' requests, the batch size changes with traffic, and several common GPU kernels produce slightly different numerical results depending on batch composition. Different numbers, occasionally a different argmax, occasionally a different token. In their experiment — sampling Qwen3-235B-A22B-Instruct-2507 a thousand times at temperature 0 on the prompt "Tell me about Richard Feynman" — they report: "we generate 80 unique completions, with the most common of these occuring 78 times… the completions are actually identical for the first 102 tokens! The first instance of diverging completions occurs at the 103rd token." (992 of the completions then said "Queens, New York"; 8 said "New York City.") Their batch-invariant kernels made all 1,000 bitwise identical — at a modest performance cost. Anthropic says the same thing more plainly in its docs: even at temperature 0, results "will not be fully deterministic." This is the bridge to the Determinism post; here, just know that temperature 0 buys you near-determinism, not a cryptographic guarantee.

Vendor advice is genuinely inconsistent, so read the docs for the model you're using. OpenAI: temperature ranges 0–2, default 1, and the reference says "We generally recommend altering this or top_p but not both." Anthropic: temperature ranges 0–1, default 1, "use temperature closer to 0.0 for analytical / multiple choice, and closer to 1.0 for creative and generative tasks," and on the newest Claude models you literally cannot set both temperature and top_p in one call. Google Gemini: temperature ranges 0–2, default 1 — but Google now tells you to leave Gemini 3 at the default 1.0, warning that lowering it "may lead to unexpected behavior, such as looping or degraded performance, particularly in complex mathematical or reasoning tasks." Same number, different meaning, different advice, per vendor.

Reasoning models change the rules. This is an evolving area, so represent it carefully. OpenAI's o-series and GPT-5 reasoning models reject custom sampling values outright — send a temperature and you get a 400 error ("Only the default (1) value is supported"), because those models fix the sampling parameters internally. Anthropic's extended thinking mode is "not compatible with temperature or top_k modifications." And DeepSeek's open-weights R1 family goes the other way: its model card says "Set the temperature within the range of 0.5-0.7 (0.6 is recommended) to prevent endless repetitions or incoherent outputs" — i.e., greedy decoding at T = 0 is actively discouraged. So "set it to 0 for determinism" is wrong advice for some reasoning models. There is no single rule; check the model card.

One more myth to kill: "cranking temperature lets the model explore more reasoning paths." It explores more token paths. Sometimes a different token opens a genuinely different line of reasoning; often it just swaps a synonym. If you actually want to explore reasoning paths, the technique is self-consistency (Wang et al., 2022): sample several full chains at moderate temperature and take the majority answer. That's a deliberate ensemble, not a side effect of a hot sampler.

Practical recommendations

• Start with the defaults. Only change temperature when you have a specific reason.
• Don't raise temperature to "make the model smarter." There's no such effect.
• For deterministic-feeling output, use temperature 0 — but don't bet your business on byte-identical reproducibility.
• Tune one of temperature or top-p, not both. They occupy the same job and fight when set against each other.
• When iterating on a prompt, develop at low temperature so you can tell prompt changes apart from sampling noise.
• To gauge reliability, sample multiple times at moderate temperature and look at the spread of outputs — that distribution tells you how stable your task really is.
• For reasoning models, ignore generic advice and use the model card's numbers (or accept that the knob is locked).

What's next

Everything in the rest of this Interface section sits directly on top of this one sampling step. Streaming, the next post, is about when you see the tokens the sampler picks — delivering them as they're chosen rather than all at once. Structured output, after that, is about constraining which tokens the sampler is even allowed to pick — JSON mode, grammars, constrained decoding all reach into this same gate. And the Determinism post in the Building section will go deep on why "temperature 0" still drifts, picking up the batch-invariance thread we only touched here.

The model hands you a distribution. Everything interesting about controlling an LLM's output is a decision about how to pick from it.

Further reading

Primary papers

• Holtzman, Buys, Du, Forbes, Choi. The Curious Case of Neural Text Degeneration — arXiv:1904.09751 (2019; ICLR 2020). Nucleus / top-p sampling.
• Nguyen, Baker, Neo, Roush, Kirsch, Shwartz-Ziv. Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs — arXiv:2407.01082 (2024; ICLR 2025).
• Schaeffer, Kazdan, Denisov-Blanch. Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models — arXiv:2506.13681 (2025).
• Leviathan, Kalman, Matias. Fast Inference from Transformers via Speculative Decoding — ICML 2023; arXiv:2211.17192.
• Wang, Wei, Schuurmans, Le, Chi, Narang, Chowdhery, Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models — arXiv:2203.11171 (2022).

Vendor documentation

• OpenAI — API reference (temperature default 1, range 0–2; "altering this or top_p but not both") and the reasoning guide (sampling params fixed for reasoning models).
• Anthropic — Messages API (temperature 0–1, default 1; "will not be fully deterministic") and extended thinking (incompatible with temperature / top_k).
• Google — Gemini 3 developer guide (default temperature 1.0; the looping warning).
• DeepSeek — DeepSeek-R1 model card (temperature 0.5–0.7, 0.6 recommended).
• Meta — Introducing Meta Llama 3 (the 128,256-token vocabulary).

Blogs & explainers

• Thinking Machines Lab (Horace He et al.). Defeating Nondeterminism in LLM Inference (Sept 2025), and Simon Willison's notes on it.
• Maxime Labonne (Hugging Face). Decoding Strategies in Large Language Models.