Transformers didn't win because attention beats recurrence — they won because attention is parallelizable at training time, which made them GPU-native and unlocked the scaling laws. Why the moat is the stack, not the architecture, and who's actually challenging it.
If you've followed this series, you now know what attention does: a soft, differentiable lookup where every token asks a question and reads a weighted blend of answers from the tokens it's allowed to see. That was the last post. This post asks a different question — not "how does attention work?" but "why did the transformer, the architecture built around attention, take over the entire field and refuse to let go?"
Here's the short answer, and it surprises people: transformers did not win because attention is smarter than the recurrence it replaced. They won because attention was parallelizable at training time. That one property made transformers a natural fit for the hardware we already had, which made it possible to train them on enormous datasets, which unlocked a set of predictable relationships — the scaling laws — that turned "spend more compute" into a rational business plan. Everything else compounded from there.
Let me build that case.
For roughly 2014–2017, sequence modeling meant recurrent neural networks: RNNs, and their better-behaved cousins the LSTM and GRU. A recurrent network reads a sequence one step at a time, carrying a hidden state forward: to compute what it knows at position t, it must first have computed position t−1.
Attention itself was not new in 2017. Bahdanau, Cho, and Bengio introduced it in 2014 as a bolt-on to a recurrent translation model — a way to let the decoder look back at all the encoder's hidden states instead of cramming a whole sentence into one fixed-length vector. Attention predates the transformer by three years. There were also convolutional approaches to sequences (ByteNet, WaveNet) that were more parallel-friendly than RNNs. So the pieces were lying around.
The problem was structural: RNN training was inherently sequential. You couldn't get past a certain scale, because the architecture fought the hardware.
Vaswani et al.'s "Attention Is All You Need" (NeurIPS 2017) made the radical move of throwing out recurrence entirely. No hidden state passed step to step — just attention, stacked in layers, with feed-forward networks in between. The original was an encoder-decoder built for machine translation.
The results were state-of-the-art, but the part worth internalizing is why the authors were excited. The abstract emphasizes that the models are "superior in quality while being more parallelizable and requiring significantly less time to train." The big model hit 28.4 BLEU on English-German and 41.8 BLEU on English-French after training for 3.5 days on eight NVIDIA P100 GPUs — a small fraction of the training cost of the best prior models. The headline wasn't only accuracy. It was accuracy per unit of training time.
This is the part to actually understand, because it's the whole argument.
Consider training on a sequence of length N. In an RNN, computing the loss requires N sequential steps, because hidden state t depends on hidden state t−1. Even on a machine with thousands of cores sitting idle, you wait: the dependency chain is linear in sequence length. The paper says it plainly — this "inherently sequential nature precludes parallelization within training examples."
Now the transformer. With causal masking and teacher forcing, you feed in the whole ground-truth sequence at once and ask the model to predict the next token at every position simultaneously. Position 5 predicting token 6 doesn't wait for position 4's prediction — it just needs the actual tokens 1–5, which you already have because it's the training text. So the loss for all N positions is computed in one forward pass, and that forward pass is a stack of big matrix multiplications: Q·Kᵀ, softmax, ·V, then the MLPs.
The transformer's own comparison table makes it crisp: a self-attention layer needs a constant number of sequential operations, O(1), while a recurrent layer needs O(n). Concretely: training on a 4,096-token sequence in an RNN means 4,096 sequential dependent steps; in a transformer, all 4,096 positions' losses land in a single parallel pass.
That's the win. Transformers won at training time, not (only) at inference time. (At inference they still generate one token at a time — that's the autoregressive decode loop and KV cache from earlier posts. The asymmetry is the point: parallel to train, sequential to serve.)
Here's where hardware enters. GPUs are, fundamentally, machines for dense matrix multiplication done thousands of lanes at a time. A transformer's forward pass is almost entirely matrix multiplication. The fit is almost suspicious.
And it got tighter. In 2017 — the same year as the transformer paper — NVIDIA shipped its Volta architecture with the first tensor cores: hardware units built specifically to do small matrix multiply-accumulate operations fast, in mixed precision. NVIDIA reported the V100's tensor cores delivered up to 12× higher peak training throughput than the prior-generation Pascal. The industry then spent years co-designing hardware around matmul-heavy workloads, along with the software stack (CUDA, cuBLAS, cuDNN), and eventually FlashAttention (Dao et al., 2022), which made attention dramatically faster and more memory-efficient by being smart about GPU memory movement — without approximating the math. FlashAttention is exact; it just stops shuttling the giant attention matrix in and out of slow memory.
Put those together and you get a flywheel. The architecture suited the hardware; the hardware evolved to suit the architecture; the software matured around both. A new architecture doesn't just need to be better on paper — it needs to be better on this hardware, with this toolchain. That's a high bar.
The parallelism made big training runs possible. The scaling laws made them rational.
In 2020, Kaplan et al. ("Scaling Laws for Neural Language Models," arXiv:2001.08361) showed that a transformer's loss falls as a smooth power law in three things: model size, dataset size, and compute — across more than seven orders of magnitude. This is the quietly revolutionary bit. It meant you could train small models, fit the curve, and predict what a much larger model would achieve before spending the money. Scale stopped being a gamble and became a forecast. This is what turned "spend more compute" into a defensible R&D strategy.
In 2022, Hoffmann et al.'s Chinchilla paper (arXiv:2203.15556) corrected the recipe. Kaplan-era practice built models that were too big and fed them too little data. Chinchilla showed that for a fixed compute budget, parameters and tokens should scale together — roughly 20 tokens per parameter — and demonstrated it by training a 70B model (Chinchilla) that beat the 280B Gopher at equal compute. The illustration that lands: GPT-3 had 175B parameters trained on ~300B tokens (under 2 tokens per parameter). Chinchilla-optimal for that size would have been ~3.5 trillion tokens — more than ten times the data.
A caveat worth stating honestly: scaling laws are empirical regularities, not laws of physics. They've held remarkably well, but they're fits to data, and even Chinchilla's exact coefficients have been the subject of replication debates (Besiroglu et al., 2024).
And there's the emergence debate, which you should represent fairly. Wei et al. (2022) catalogued "emergent abilities" — capabilities that seem to appear suddenly at a certain scale. Schaeffer, Miranda, and Koyejo (2023, "Are Emergent Abilities a Mirage?", a NeurIPS 2023 outstanding paper) pushed back: many of those sharp jumps are artifacts of the metric. Score a task all-or-nothing and you get a cliff; measure it continuously and the underlying capability grows smoothly and predictably. The honest read: the underlying loss improves smoothly; whether that looks like a sudden new ability depends heavily on how you measure.
By around 2020, transformers had a lead. What turned a lead into a moat is that everything else compounded around them:
This is the framing that matters for a software engineer: the transformer moat is not the architecture, it's the stack. Beating attention on a whiteboard is not the same as beating the accumulated investment of an entire industry.
To displace transformers, a challenger needs roughly three things at once: better quality per FLOP at the top of scale (not just on small models); scaling behavior at least as good (ideally steeper); and compatibility with existing hardware and pipelines, or its own hardware allies. So far nothing has done all three convincingly. Here's the honest state of the competition in mid-2026.
Mamba and state-space models (SSMs). Gu and Dao's Mamba (2023, arXiv:2312.00752) is the most serious architectural challenger: a selective state-space model that runs in linear time in sequence length and keeps a constant-size state during generation — no quadratic attention, no growing KV cache. Mamba-2 (2024) refined it. SSMs are genuinely competitive at small-to-medium scale. But they have a known weakness: a fixed-size state means they're worse at copying and retrieving from context — exactly the in-context recall that few-shot prompting and retrieval lean on (Jelassi et al., "Repeat After Me," 2024, which shows this is inherent to any fixed-memory model, not a Mamba bug). Pure SSMs have not decisively beaten transformers at the frontier.
Hybrids — the consensus play. This is where the action actually is. Interleave a few attention layers (for precise recall) with many SSM layers (for cheap bulk sequential processing) and you get most of both. AI21's Jamba (2024) was the first production-grade hybrid at scale, combining Transformer + Mamba + MoE at roughly a 1:7 attention-to-Mamba ratio, which yields an 8× smaller KV cache than a vanilla transformer at long context. NVIDIA's Nemotron-H (2025) replaces most self-attention layers with Mamba-2 and reports comparable accuracy with up to 3× faster inference; NVIDIA has since carried the hybrid Mamba-Transformer-MoE pattern into its flagship Nemotron 3 family. Hybrids are promising and increasingly common — but note what they are: attention is still in the recipe, doing the one job only it does well.
Mixture of Experts (MoE) — horizontal scaling, not a rival. This is the most misunderstood item on the list. MoE is not an alternative to attention. It's a change to the other half of the transformer block — the MLPs. Instead of one big feed-forward network firing on every token, you have many "expert" networks and a router that sends each token to just a couple of them. You get the knowledge capacity of a huge model while only paying to run a slice of it per token. Think of it as horizontal scaling for models: add more experts the way you'd add more database shards. Mixtral 8x7B (2023) has ~46.7B total parameters but activates ~12.9B per token. DeepSeek V3 (2024) has 671B total, 37B active. Llama 4 Maverick (2025) has 400B total, 17B active across 128 experts. The frame: MoE lets you scale within the transformer framework, not beyond it.
RWKV, RetNet, and friends. These are recurrence-inspired architectures engineered to train in parallel (like a transformer) and serve cheaply (like an RNN). RWKV scaled a dense RNN-style model to 14B parameters; RetNet targets the same "parallel training, low-cost inference, strong performance" trifecta. Genuinely interesting research, not dominant.
Diffusion for text. Borrowing from image generation, diffusion LLMs generate by iteratively refining a whole block of tokens in parallel rather than strictly left-to-right. LLaDA (2025) trained one from scratch at 8B parameters; Inception Labs' Mercury Coder reported ~1,109 tokens/sec on H100 GPUs — several times faster than comparable autoregressive models — and its 2026 successor Mercury 2 pushed reasoning-capable diffusion to roughly 1,000 tokens/sec. It's a real research direction with striking speed properties, but not yet a serious contender for mainstream chat LLMs.
Three things worth keeping in view so you don't overlearn the lesson.
"Won" ≠ "best possible." Transformers are a local maximum that the whole field converged on and then invested in until the investment itself became the advantage. That's not the same as being theoretically optimal. The pieces (attention, dot products, residual connections, layer norm) mostly existed before 2017; the win was the assembly plus the scale, not a single Eureka.
The scaling story can hit walls. Chinchilla says to feed models ~20 tokens per parameter, but Villalobos et al. project that the stock of high-quality public human text gets fully used around a median year of 2028 (80% confidence interval 2026–2032). The compute-optimal token count for a frontier model may exceed the quality data that exists. This is a live 2026 concern, and part of why synthetic data and data efficiency are hot topics. (In practice, labs already "overtrain" small models far past 20 tokens/param — Llama-2-7B saw ~290× the Chinchilla ratio — because cheaper inference over a model's lifetime is often worth burning extra training compute.)
Capability is increasingly bought elsewhere. More and more gains now come not from a bigger base transformer but from how you use it: retrieval, tool use, and inference-time compute (chain-of-thought and reasoning models that spend more tokens thinking). The architecture is increasingly the substrate, and the leverage is moving up the stack.
That's a wrap on how these models represent the world. The embeddings post said meaning lives as geometry in vector space. The attention post said attention is a soft, weighted lookup over those vectors. This post said the transformer — attention plus MLPs plus residuals plus a training recipe plus a co-evolved hardware-and-software stack — took over because it was the right shape for the compute, and then compounded that head start into a moat nobody has crossed.
Next we start the Extending section: leaving "what the model is internally" and moving back out to "how you get useful work out of it," beginning with prompt engineering as an actual discipline. The internals were the foundation. Now we build on top.
Foundations
Scaling laws & emergence
Hardware & systems
Challengers