AI Basics

What a neural network actually is

A neural network is a function with billions of tunable weights fit from examples by an optimizer — not a brain simulation. A ground-up explainer for software engineers on how they actually work.

What training actually means

Training an LLM isn't one step — it's a pipeline. Pretraining (99% of the compute) gives the model its knowledge; post-training (SFT, RLHF) shapes how it behaves. A ground-up map for engineers.

Inference vs. training

Once training ends, the weights freeze. Inference is the read-path — a forward pass run one token at a time, forever. Autoregressive decoding, prefill vs. decode, the KV cache, and the five ways inference differs from training.

Why models hallucinate

Hallucination isn't a bug grafted onto a truthful system — it's the default behavior of a next-token predictor. The same operation produces every correct answer and every confident fabrication. Why it happens, and why grading rewards it.

Knowledge cutoffs: the model thinks it's still last year

A knowledge cutoff isn't a date on a sticker — it's the trailing edge of a distribution, and the model can't reliably see its own boundary. Why frozen weights make every chat a conversation with an archive, and what actually moves the line.

Tokens, Tokenizers, and How Claude Counts Them

How subword tokenization algorithms work under the hood, why they shape context window economics and model accuracy, and what Claude's proprietary tokenizer changes mean for your API budget.

Context windows: the only door into a frozen model

A frozen model has exactly one write surface — the context window. Why it's bounded, why its usable length is far shorter than the number on the model card, and why deciding what to put in it is the core skill of working with LLMs.

Roles are a convention, not an architecture

System, user, and assistant are special tokens in one flat sequence that the model was post-trained to honor — not privileged channels. Why the instruction hierarchy is a learned preference, and why that's exactly why prompt injection works.

Temperature and sampling: the dice live outside the model

The model emits a fixed probability distribution; everything that feels random happens in the sampler on top of it. What temperature actually does, how top-k/top-p/min-p differ, and why 'temperature 0' isn't truly deterministic.

Streaming: publishing, not generating

Streaming doesn't change how a model generates—only how the server delivers what it was already producing one token at a time. What SSE actually is, why TTFT and ITL are different problems, and the engineering a stream forces on you.

Structured output: making a text generator return typed data

Structured output is constrained decoding: before the sampler picks, a schema-compiled state machine masks every token that would break your JSON, making invalid shape mechanically impossible. Three tiers of guarantee, why tool calls are the same machinery, and the quality tradeoff nobody warns you about.

Embeddings and vector space: turning meaning into geometry

An embedding turns meaning into geometry — text mapped to a vector so that similar meanings land close together. The two very different artifacts both called 'embeddings,' how sentence models are actually trained, and what vectors can and can't tell you.

Attention, conceptually: a soft lookup over every token

Attention is a soft, weighted, differentiable lookup — every token asks which others matter and reads a weighted blend of their values. Q/K/V as a soft hashmap, why multi-head exists, the causal mask, and why the whole thing costs N².

Why transformers won: the right shape for the compute

Transformers didn't win because attention beats recurrence — they won because attention is parallelizable at training time, which made them GPU-native and unlocked the scaling laws. Why the moat is the stack, not the architecture, and who's actually challenging it.

Prompt engineering: the tricks died, the discipline didn't

Tipping, threatening, and 'let's think step by step' mostly failed replication or got absorbed into the models. What survived is less glamorous and more useful: prompting is steering a frozen distribution, and the only thing that turns it into a discipline is measuring.

RAG: the hard part was never the model

Retrieval-augmented generation is one hard problem and one trivial one wearing the same name. The generation half is a prompt with text pasted into it. Everything that actually breaks is a search problem that predates LLMs by thirty years — which is why so many teams staff it wrong.