What training actually means

In the last post, we ended on a frame borrowed from Andrej Karpathy: a neural network is a piece of Software 2.0. The dataset and architecture are the source code. The weights are the binary. And the training process is the compiler that turns one into the other.

That frame is useful, but it hides a detail that matters a lot once you start using these models: there isn't one compiler pass. Modern LLMs are not produced by a single training run. They're produced by a pipeline of distinct stages, each with its own data, its own objective, and its own purpose. The thing you talk to in a chat window is the output of at least three of those stages stacked on top of each other.

This post is the map of that pipeline. We'll walk it end to end — pretraining first, then post-training — at the level of "what is each stage doing and why does it exist." We won't derive any math. Names like PPO, DPO, and Constitutional AI will show up, but they're signposts for future posts, not topics for this one.

The big picture: training is a build pipeline

A useful starting point: stop thinking of "training" as a single verb, and start thinking of it as a build pipeline.

At the top of the pipeline you have a freshly initialized network — billions of weights set to small random numbers. It can do nothing. Run a forward pass on it and you get noise.

At the bottom of the pipeline you have a chat assistant: something that takes a question, follows the instruction, and responds in a recognizable voice with reasonable judgment about what to say and what not to say.

Between those two endpoints, the industry has converged on roughly this shape:

1. Pretraining. Self-supervised learning on a vast text corpus. Produces a base model.
2. Supervised fine-tuning (SFT). Train on curated (prompt, ideal response) pairs. Teaches the model the assistant format and basic instruction-following.
3. Preference / alignment tuning. Use human preference data — pairs of model responses where a human picked the better one — to nudge the model toward responses people actually like. The canonical version is RLHF; modern variants include DPO and Constitutional AI.

Stages 2 and 3 together are usually called post-training.

The proportions matter enormously, and they're almost the opposite of what most people guess. In his "State of GPT" talk at Microsoft Build 2023, Karpathy put it bluntly: pretraining "is 99% of the training compute time and also flops." OpenAI made the same point in its InstructGPT announcement, noting that their fine-tuning step "uses less than 2% of the compute and data relative to model pretraining." Almost all the money, almost all the time, and almost all the hardware goes into stage 1. And yet stage 1 doesn't get you the product. The thin sliver at the end of the pipeline is what makes the model feel like an assistant.

Hold onto that asymmetry. It's the most important fact in this post.

Stage 1: Pretraining — predicting the next token at scale

Pretraining is conceptually almost embarrassingly simple. You take a huge pile of text — most of the public internet, books, code, Wikipedia, and so on — and you train the network on exactly one task: given some text so far, predict the next token. A token is a short chunk of text, roughly half a word on average; for now, just think "next word."

That's it. That's the whole objective. There is no labeling step. The data labels itself: for every span of text in the corpus, the "correct answer" is just the next token that actually appears. This is what people mean by self-supervised learning. There are no humans in the loop here, no annotation budget, no judgment calls per example. You just need a lot of text and a lot of compute.

The forward pass, loss, and gradient descent loop from the last post do all the work. The model makes a prediction; the loss measures how wrong it was; backprop pushes the weights in the direction that would have been less wrong. Repeat a few trillion times.

The trick — and it's still a slightly miraculous trick — is that this trivial objective, applied to enough text at enough scale, produces a model that has implicitly learned grammar, world facts, code, several languages, basic arithmetic, the structure of arguments, and a great deal of reasoning ability. None of that was a training target. It's all incidental to "predict the next token," because to predict the next token well across a sufficiently diverse corpus, you eventually have to model the things the text is about.

The scale here is hard to overstate. The original GPT-3 paper (Brown et al., 2020) reports that "all models were trained for a total of 300 billion tokens." By 2024, Meta's Llama 3.1 405B was pretrained on over 15 trillion tokens — a 50× jump in four years — using, in Meta's words, "over 16 thousand H100 GPUs." The official model card lists 30.84 million H100-hours for the 405B model alone, out of 39.3 million across the full Llama 3.1 collection. As for cost: asked at an April 2023 MIT event whether GPT-4 cost $100 million to train, Sam Altman said "it's more than that"; Epoch AI's amortized estimate of the GPT-4 training run lands at $40 million (Cottier et al., 2024). A pretraining run is a months-long, multi-thousand-GPU batch job whose checkpoint files are measured in hundreds of gigabytes.

What you get at the end of all that is a base model. It is enormously knowledgeable. It can complete a paragraph in the style of any author it's seen, write working code, translate between languages, and follow patterns. But there's something off about it as a product: it doesn't answer questions. It continues text. If you type "What is the capital of France?" into a raw base model, you are roughly as likely to get back a list of more trivia questions as you are to get "Paris." It saw plenty of quiz lists during training. From its perspective, your "question" is just a prefix, and a sensible continuation is more quiz.

This is the thing post-training fixes.

Why the base model isn't the assistant

The mismatch between pretraining objective and product behavior is the entire reason post-training exists. Pretraining optimizes one thing: average next-token accuracy across the corpus. That objective is agnostic about what the model should do when you show up with a request. It has no notion of "user." It has no notion of "helpful." It has no notion of "don't help me build a bioweapon."

You can squint at a base model and see a lot of latent capability. But to get a coherent assistant — one that follows instructions, stays on topic, and behaves within some envelope of acceptable behavior — you need a second compiler pass on top of the pretraining binary. And that pass is doing something fundamentally different. It isn't trying to give the model more knowledge. It's trying to shape which of the model's existing capabilities are activated by which inputs.

Stage 2a: Supervised fine-tuning (SFT)

The first piece of post-training is supervised fine-tuning, often called instruction tuning. Compared to pretraining, this stage looks small. The data is curated, often hand-written. You're not slurping the internet; you're paying skilled labelers (or generating with another model and then filtering) to produce a few tens of thousands to a few million examples of the form:

Prompt: Explain bubble sort to a junior engineer in three sentences.

Response: [a good, well-formatted answer]

You then run the same training machinery as pretraining — same loss, same backprop — but on this new, much smaller dataset. The model adjusts to do two things at once. First, it learns the format of being an assistant: there's a user turn, there's an assistant turn, the assistant addresses the user's request, the assistant doesn't ramble. Second, it learns to follow instructions: when the input is shaped like a request, generate a response, not more requests.

You can think of SFT as showing the model 10,000 worked examples of the job and saying: "this is what good looks like here." It is not, primarily, a knowledge transfer step. The most striking evidence here is Zhou et al.'s LIMA paper (NeurIPS 2023), which fine-tuned a 65B LLaMA model on only 1,000 carefully curated examples — 750 from Stack Exchange and wikiHow, 250 hand-written — and reported that the result outperformed RLHF-trained DaVinci003 in human preference evals. A follow-up, Lin et al.'s URIAL paper (ICLR 2024), pushed even further: with as few as three carefully designed in-context examples and a system prompt, they got non-trivial instruction-following out of a base model with zero weight updates. The technical term for the underlying intuition is the superficial alignment hypothesis: SFT mostly teaches the model a style and a format for emitting capabilities it already has.

That's also why fine-tuning is a surprisingly poor tool for adding knowledge. Gekhman et al. (2024), in a controlled study at Google, found that when you fine-tune on new factual content the model didn't see during pretraining, the model learns it slowly and its hallucination rate goes up roughly linearly with how much novel-fact content you push in. The blunt summary: pretraining is where knowledge enters the model. Fine-tuning is where behavior is shaped.

Stage 2b: Preference tuning — RLHF and its cousins

SFT gets you an assistant-shaped artifact, but it's still missing something. SFT only knows about responses a human bothered to write down as ideal. It doesn't know how to choose between two plausible responses, or how to push toward responses that humans actually prefer over the merely "good enough."

That's where preference tuning comes in. The canonical version is RLHF — reinforcement learning from human feedback — formalized for language models by OpenAI's InstructGPT paper (Ouyang et al., 2022), with intellectual roots going back to Christiano et al.'s 2017 work on deep RL from human preferences, and Stiennon et al.'s 2020 summarization paper that bridged the two worlds.

The canonical InstructGPT pipeline runs in three steps on top of the pretrained model:

1. SFT. As described above.
2. Reward model training. You take pairs of model responses to the same prompt, show them to human labelers, and ask which is better. From tens of thousands to a few hundred thousand of these comparisons, you train a separate neural network — the reward model — to predict the human-preferred response. It's essentially a learned proxy for human judgment over outputs.
3. RL against the reward model. You then fine-tune the SFT model using reinforcement learning — typically PPO, proximal policy optimization — to produce responses the reward model scores highly. To keep the model from drifting into weird, reward-hacking corners of output space, the objective includes a KL penalty that pulls it back toward the SFT model when it strays too far.

The two things to take away from this without doing any of the math:

• The reward model is a stand-in for "what humans like." It's much cheaper to query than to ask a human every time, and once trained, you can run RL against it for as many steps as you can afford.
• The KL penalty is the leash. Without it, RL would happily exploit the reward model's blind spots and produce something that scores high but reads as nonsense.

RLHF works, but it is finicky. PPO is famously unstable, the reward model can be gamed, and human labelers disagree with each other and themselves. So newer techniques have emerged that achieve similar goals with less machinery. Direct Preference Optimization (DPO), introduced by Rafailov et al. in 2023, mathematically rearranges the RLHF problem so that you can skip the explicit reward model and the RL loop entirely — you just do a clever supervised loss directly on the preference pairs. Constitutional AI (CAI), introduced by Anthropic in late 2022, replaces a lot of the human labeling with model-generated labels guided by a short list of written principles — a "constitution" — making the alignment step faster to iterate on. Both of these deserve their own posts, and they'll get them. For now, just file them as "same goal as RLHF, simpler operationally."

The end product of preference tuning is the model you actually interact with. It's helpful by default. It refuses certain requests. It has a tone. It hedges where it should. None of that was in the base model. It was sculpted in this last sliver of the pipeline.

The mental model: capability vs. behavior

If you take one thing from this post, take this:

OpenAI made this exact point about InstructGPT: the fine-tuning step "'unlocks' capabilities that GPT-3 already had, but were difficult to elicit through prompt engineering alone." The capability was always there. The behavior wasn't.

This split is a robust mental model. It survives every new post-training technique people invent. DPO, CAI, RLAIF, the various reasoning-RL recipes used to train models like o1 and DeepSeek-R1 — they're all variations on "shape behavior on top of a capable base model." The base model is the substrate. Post-training is the polish.

What this means in practice

A few practical consequences fall out of this picture, and they tend to surprise people:

The model doesn't learn from your chats. Once training is done, the weights are frozen. Whatever the model picks up "about you" in a conversation lives in the context window — the tokens currently being processed — and disappears when the conversation ends. The model you chat with today is the same set of weights you chatted with yesterday. (This is also why every model has a knowledge cutoff: the binary was built at a specific date, and it hasn't read anything published since. We'll come back to cutoffs in a later post.)

Fine-tuning is mostly not how you add knowledge. The instinct for engineers coming from databases is that "fine-tuning on our docs" should make the model know our docs. It largely doesn't, and the Gekhman et al. paper above is one piece of evidence why: fine-tuning on facts the base model never saw mostly teaches it to make confident things up. If you want the model to "know" your docs at inference time, retrieval-augmented generation (putting the relevant docs into the context window) is almost always the better tool. Fine-tuning is for behavior, format, and style.

Bigger isn't the only thing that matters. A lot of the gains between Llama 2 and Llama 3 came from more and better data, not bigger models — Meta trained Llama 3 8B on roughly 75× the Chinchilla-optimal token count (15T tokens versus the ~200B their own scaling laws would have predicted) and kept seeing gains long past the point where the curves "should" have flattened. Post-training quality matters even more. A small, well-post-trained model can feel dramatically better than a larger one with sloppy alignment.

The "base model" you can sometimes download is not the product. When Meta releases Llama 3.1 405B, they release both a base and an instruction-tuned variant. The base model is the raw pretraining output; almost nobody uses it directly. The instruction-tuned variant is what feels like a product. The gap between the two is the entire substance of post-training.

What's next in the series

This post is deliberately a map, not a tour. Each of the stages we walked through deserves its own dive, and they're coming:

• Inside pretraining — what the dataset actually looks like, what tokenization is doing, why data quality dominates, what scaling laws say, and why the bill is so high.
• Post-training and RLHF — the three-step InstructGPT pipeline in detail, the role of the reward model and the KL penalty, why PPO is hard, and how the field is moving.
• Constitutional AI and the alignment stack — Anthropic's principle-based approach, RLAIF, and where these fit into modern training recipes.
• Reasoning models and RL on verifiable rewards — the newer twist where the reward signal isn't a human preference but a correctness check, and what changes when you can grade the model automatically.

For now, the headline is the one we started with: training is a pipeline, not a single step. Pretraining gives the model what it knows. Post-training gives it how it behaves. The compute lives almost entirely in the first half; the personality lives almost entirely in the second.

---

Further reading

• Karpathy, A. (2023). "State of GPT," Microsoft Build. — The single best high-level diagram of the training pipeline, and the source for the "99% of the compute is pretraining" line. Start here.
• Ouyang et al. (2022). Training Language Models to Follow Instructions with Human Feedback (InstructGPT). — The canonical SFT → reward model → PPO pipeline. The paper that turned RLHF into the industry default.
• OpenAI. "Aligning Language Models to Follow Instructions." — The accessible companion to the InstructGPT paper; source of the "unlocks capabilities GPT-3 already had" framing.
• Christiano et al. (2017). Deep Reinforcement Learning from Human Preferences. — The founding RLHF idea, done on Atari and MuJoCo before anyone applied it to language.
• Zhou et al. (2023). LIMA: Less Is More for Alignment. — 1,000 curated examples rival full RLHF. The strongest evidence for the superficial alignment hypothesis.
• Gekhman et al. (2024). Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? — Controlled evidence that fine-tuning on novel facts raises hallucination rates. The reason "fine-tune on our docs" usually disappoints.
• Bai et al. (2022). Constitutional AI: Harmlessness from AI Feedback. — Anthropic's principle-based alternative to all-human preference labeling (RLAIF). Defer the mechanics; grab the intuition.
• Rafailov et al. (2023). Direct Preference Optimization. — Rearranges RLHF to drop the explicit reward model and the RL loop. The technique quietly eating PPO's lunch.
• Meta AI (2024). The Llama 3 Herd of Models. — The most transparent frontier-scale training report we have: data mix, token counts, GPU-hours, and post-training recipe in one place.
• Lambert, N. et al. "Illustrating Reinforcement Learning from Human Feedback (RLHF)." — The clearest illustrated walkthrough of the RLHF loop if the InstructGPT paper is too dense.