Attention is a soft, weighted, differentiable lookup — every token asks which others matter and reads a weighted blend of their values. Q/K/V as a soft hashmap, why multi-head exists, the causal mask, and why the whole thing costs N².
In the last post, we established that once you're past the first-layer token lookup, everything inside a transformer is vectors flowing through layers — and that a dot product between two vectors measures how similar they are. This post is about the single operation that puts those vectors to work: attention. If embeddings are the model's vocabulary of meaning, attention is the grammar — the mechanism by which each token in your prompt reaches out to every other token and asks, "which of you matters to me right now, and how much?"
Here's the thesis, up front: attention is a soft, weighted, differentiable lookup. Every position in the sequence pulls in a weighted blend of information from all the other positions, where the weights are computed from the same dot-product-over-vectors machinery you already know. That's it. Everything else — queries, keys, values, multi-head, causal masks, the KV cache — is bookkeeping around that one idea.
Before transformers, the dominant sequence models were recurrent neural networks — RNNs, and their better-behaved cousins the LSTM and GRU. These process a sequence one token at a time, carrying a running summary called a hidden state along a chain. Token 1 updates the state, hands it to token 2, which updates it and hands it to token 3, and so on.
The problem is the chain itself. If the pronoun on line 40 refers to a noun on line 2, the information about that noun has to survive dozens of sequential updates without being overwritten. In practice it often didn't; this is the vanishing-gradient / long-range-dependency problem that plagued recurrent models. It's also inherently sequential — you can't compute step 40 until you've computed step 39 — which is terrible for GPUs that want to do thousands of things at once.
Attention was originally invented within this recurrent world, not as a replacement for it. Bahdanau, Cho, and Bengio introduced it in 2014 for machine translation: instead of forcing the decoder to rely on a single fixed-length summary vector, they let it "soft-search" over all the encoder's hidden states and pull in the relevant ones for each output word. The 2017 paper "Attention Is All You Need" made the radical move of throwing away the recurrence entirely and keeping only the attention. Hence the title.
At every layer, every token asks: which other tokens matter for me right now? — and pulls in a weighted combination of their information. Attention lets every position look directly at every other position, with no chain to travel down. The noun on line 2 is exactly as reachable as the word right next door.
Concretely, consider the sentence "The cat sat on the mat because it was tired." When the model processes "it," it needs to figure out what "it" refers to. Attention is the mechanism that lets the representation at "it" look back at "cat" and "mat," score both, and mix in more from "cat" — because agents of being-tired tend to be animate. The model isn't following a rule; it's computing similarity scores and taking a weighted average.
Here's the part worth internalizing. For each token position, the model computes three vectors, each a learned linear projection (a matrix multiply) of the token's current representation:
If you're a software engineer, the cleanest mental model is a hashmap lookup. You look something up with a query. The keys advertise what's stored. The values are what you get back. In an ordinary hashmap the match is exact — one key wins, you get one value. In attention the match is soft: you compute the similarity between your query and every key, turn those similarities into weights, and return a weighted blend of all the values. Every entry contributes something; the relevant ones just contribute more.
That's the whole conceptual leap. Attention is a hashmap where every key matches a little, and you get back a weighted average instead of a single hit.
Here's the actual computation — this is "scaled dot-product attention," the core operation from the 2017 paper. For a given query token:
Three operations you already know: dot product for similarity, softmax for weights, weighted sum for aggregation. Do this for every query token at once — and because each token's query is independent, you can — and the whole thing collapses into a few big matrix multiplications, which is exactly what GPUs are built for. This parallelism is the quiet reason attention won, and it's the subject of the next post.
One attention operation can only express one notion of relevance at a time. But "which tokens matter to me" has many simultaneous answers — grammatical subject, coreference, topic, tense. So instead of running attention once over the full-width vectors, transformers split Q, K, and V into several smaller heads, run attention independently in each, then concatenate the results and project them back.
The database analogy: multi-head attention is like running several parallel indexes over the same data, each keyed on a different aspect. The original 2017 transformer used 8 heads of dimension 64 each. Modern models use varied counts and split the roles apart: Llama 3 70B, per Meta's "The Llama 3 Herd of Models" (Dubey et al., 2024), uses 64 attention heads of dimension 128, but only 8 key/value heads under grouped-query attention (GQA) — a trick where several query heads share one K/V head "to improve inference speed and to reduce the size of key-value caches during decoding." Different heads demonstrably specialize: some track syntactic dependencies, some track position, some track long-range topic.
A misconception worth killing: multi-head attention is not multiple models. It's the same model and the same tokens, with Q/K/V projected into several lower-dimensional subspaces so different relationships can be attended to in parallel.
In self-attention, Q, K, and V all come from the same sequence — every token attends to the tokens around it. This is what decoder-only LLMs (GPT, Claude, Llama, Gemini) use throughout. In cross-attention, the queries come from one sequence and the keys and values from another — for example, in the original encoder-decoder transformer, the decoder's queries attend to the encoder's output. Modern chat models are decoder-only and rely on self-attention; cross-attention is mostly relevant if you're working with encoder-decoder architectures like T5. It's worth naming so the term isn't a mystery when you meet it.
There's a catch specific to autoregressive language models. When the model is learning to predict the next token, a token at position N must not be allowed to attend to tokens at positions after N. You can't let the model peek at the future it's supposed to predict — during training that would leak the answer.
The implementation is delightfully mundane. Before applying softmax, you take the attention score matrix and set every "future" cell to −∞. After softmax, −∞ becomes exactly zero weight. It's a triangular boolean matrix: every position can attend to itself and everything before it, nothing after. This is the causal mask, and it's the one place in attention where a weight is genuinely zero rather than just small.
Note what this is and isn't. It's a training-and-generation necessity so autoregressive prediction stays honest. It is not a security feature, despite the name sounding locked-down.
Now the engineering payoff. During generation, the model produces one token at a time (the autoregressive decoding from the inference post). Naively, each new token would recompute attention over the entire sequence from scratch. But notice: when you add a new token, its query is new, but the keys and values for all the previous tokens haven't changed — they were computed on earlier forward passes and are still valid.
So we cache them. This is the KV cache: each new token computes exactly one new Q, K, and V, appends its K and V to the cache, and attends against all the stored keys and values. That's why generating token number 5,000 doesn't cost 5,000× what generating token number one did.
But there's a hard limit lurking, and it's the callback to the context-windows post. Every query attends to every key, so the attention scores form an N×N matrix, where N is the sequence length. Both compute and memory scale with N². Double your context and you quadruple the attention cost. This is the nested-loop-over-token-pairs cost, and there's no way around the pairwise interaction without changing the mechanism itself.
This is also where the phases split. Processing the initial prompt — prefill — is the O(N²) step, because every prompt token attends to every other. That's why time-to-first-token grows with prompt length. Generation after that — decode — is roughly linear per token thanks to the KV cache. The quadratic cost is why long context is genuinely hard, why advertised context windows outrun effective ones, and why there's a whole industry chasing cheaper attention: FlashAttention (an exact but IO-aware implementation that made long context practical without approximating anything), sliding-window and sparse attention (Longformer), linear attention, and state-space models like Mamba that drop the quadratic term entirely. We'll dig into that competition next post; for now, just know the N² is the thing everyone is fighting.
Attention weights are one of the few semi-interpretable signals inside a transformer — you can literally visualize which tokens attend to which and sometimes recognize what a head is doing. This has produced real insights. Anthropic's work on induction heads (Olsson, Elhage et al., 2022) identified attention heads that implement a copy-and-continue pattern — completing sequences like [A][B] … [A] → [B] — and put forward "a hypothesis that induction heads might constitute the mechanism for the majority of all in-context learning in large transformer models." Separately, Xiao et al. (2023) studying streaming inference found attention sinks: they observed that "keeping the KV of initial tokens will largely recover the performance of window attention," because a disproportionate share of attention lands on the first few tokens "as a 'sink' even if they are not semantically important." Remarkably, keeping just four initial tokens as sinks was enough to stabilize models over sequences of up to four million tokens.
That finding has since crossed from curiosity into production design. A 2025 study by Barbero et al., "Why do LLMs attend to the first token?", argues the sink is functional — it's how models "avoid over-mixing," keeping representations from collapsing into mush over long contexts. And OpenAI's gpt-oss models, released in August 2025, bake it directly into the architecture: per the official model card, "each attention head has a learned bias in the denominator of the softmax, similar to off-by-one attention and attention sinks, which enables the attention mechanism to pay no attention to any tokens." The sink went from bug to feature to shipped hyperparameter.
Here's where honesty is required. There was a genuine and instructive academic fight over whether attention weights explain anything. A 2019 paper titled "Attention is not Explanation" (Jain & Wallace) argued that attention weights don't reliably tell you why a model made a decision — you can often find different weights that produce the same output. A rebuttal, "Attention is not not Explanation" (Wiegreffe & Pinter), argued the picture is more nuanced and depends on how you define explanation. The honest summary: attention weights are suggestive, not a complete explanation of what the model is doing. Treat visualizations as a useful lens, not ground truth — full mechanistic interpretability is still an open problem.
Here's a subtle and important property: attention as described is permutation-invariant. Shuffle the tokens and the attention outputs come out the same, just reordered. The operation genuinely doesn't know that "dog bites man" differs from "man bites dog" — the sequence is a bag of vectors until we say otherwise.
So transformers explicitly inject positional information. Early approaches added fixed sinusoidal patterns or learned position vectors to the token embeddings. Modern models overwhelmingly use RoPE (rotary position embeddings, Su et al., 2021), which rotates the query and key vectors by an amount that depends on their position, so relative distance falls out of the dot product directly. You don't need the internals here — just the fact that position is a bolt-on, and that the range over which positions were trained is one more thing that caps effective context length.
A few clarifications that save you from common traps:
This post was about what attention does: a soft, weighted lookup where every token blends in information from every other token, built entirely out of dot products and softmax over vectors. The next post — Why transformers won — is about why this specific mechanism replaced recurrence and took over: the parallelism, the scaling behavior, the historical arc from the 2017 paper to today, and the serious modern challengers like Mamba that are trying to beat the quadratic cost. And when we get to the Building section, the N² you just met will show up again wearing a business suit, in the posts on latency, time-to-first-token, and cost — because the shape of the attention matrix is, in the end, the shape of your bill.
Core mechanism
Positional information
KV cache & efficient implementation
Sub-quadratic alternatives
Interpretability