What Changed Since 2017: RMSNorm, SwiGLU, RoPE, KV Cache & GQA

Open the "Attention Is All You Need" paper from 2017 and the core loop is already recognizable. Embed tokens, add sinusoidal positions, run multi-head attention with softmax(QK^T / sqrt(d_k)), pass through a ReLU feed-forward network, wrap each sublayer in LayerNorm, repeat. That skeleton still runs every frontier model today.

But a 2025 model — Llama, Mistral, Gemma, DeepSeek — swaps four components inside that skeleton. None of them changes what a transformer is. Every one of them changes how stably it trains or how cheaply it generates. The shift over eight years was not a new idea about attention; it was a relentless focus on training stability and inference efficiency.

Here are the four swaps: LayerNorm became RMSNorm (and Post-Norm became Pre-Norm), the ReLU FFN became SwiGLU, sinusoidal positions became RoPE, and Multi-Head Attention became Grouped-Query Attention riding on a KV cache. Learn these four and you can read the config of any modern open-weight model.

RMSNorm: LayerNorm, trimmed#

LayerNorm does two things to a hidden vector: it subtracts the mean (centering), then divides by the standard deviation (scaling), then applies a learned gain γ and bias β. RMSNorm keeps only the scaling — and it scales by the root-mean-square instead of the standard deviation:

Two things are gone: the mean subtraction and the bias term. RMSNorm's bet is that centering does nothing for stability in practice, so the extra reductions and the bias are wasted compute. The denominator is now defined around zero (RMS) rather than relative to the mean (standard deviation). With eps = 1e-6, that is the entire mathematical difference.

The other half of the swap is placement. The 2017 block used Post-Norm — normalize after the residual add. Modern blocks use Pre-Norm — normalize the input before attention and the FFN, leaving the residual stream a clean, un-normalized highway. Pre-Norm is what lets you stack 80+ layers without the gradients exploding.

SwiGLU: a gated feed-forward network#

The classic FFN is ReLU(x·W_1)·W_2 — one projection up, a hard cutoff at zero, one projection back down. SwiGLU replaces it with a gated unit that uses two separate learned projections of the same input:

Read that carefully, because the common mistake is to apply the activation to the whole thing. There are two independent matrices. W_gate and W_value both project the same input h. SiLU is applied only to the gate path. Then the two paths are multiplied elementwise, and a third matrix W_out projects back down.

SiLU (also called Swish) is SiLU(x) = x · sigmoid(x). Unlike ReLU's hard zero, it is smooth and non-monotonic: small negative pre-activations leak through instead of dying, with a minimum of about -0.278 at x ≈ -1.278. The elementwise product is the real trick — for each output dimension the network learns, via the gate, how much of the value projection to let through. That is a far richer control than a fixed on/off cutoff, which is why SwiGLU consistently improves quality per parameter.

Walkthrough: normalizing one vector two ways#

Take a single hidden vector x = [2.4, -1.2, 0.8, 3.1] and run both normalizers, with γ = 1 and eps = 1e-6. This is exactly what the interactive steps through, one operation at a time.

The outputs differ because LayerNorm re-centers around zero first — notice its output has mean ≈ 0, while RMSNorm's does not. RMSNorm only guarantees unit RMS. In a trained network the learned γ absorbs most of what the mean subtraction would have done, so removing it costs almost nothing while cutting two reduction passes and a parameter vector per layer. Across 80 layers and billions of tokens, "almost nothing" times a lot is real throughput.

RoPE: encoding position as rotation#

Sinusoidal and learned position embeddings add a position signal to the token embedding. RoPE instead rotates the query and key vectors by an angle proportional to position. Split each Q/K vector into pairs of dimensions; pair i rotates at frequency:

The magic is what happens in the attention dot product. When you rotate the query at position posA and the key at position posB, each pair's contribution to the score collapses to cos((posB - posA) · theta_i) — it depends only on the relative distance, never on the absolute positions. Attention becomes translation-invariant for free.

Different pairs spin at different rates, which spreads position across scales. With d = 8 (four pairs) and base = 10000:

For a gap of 3 tokens (posA = 2, posB = 5), the high-frequency pair 0 contributes cos(3·1.0) ≈ -0.99, while the low-frequency pair 3 contributes cos(3·0.001) ≈ 1.0. The slow dials barely move over short gaps, which is precisely why RoPE extrapolates to long context — you can stretch the low frequencies without scrambling the high-frequency, local ones.

The KV cache, the memory wall, and GQA#

Generation is autoregressive: to produce token 1001 the model attends to the Keys and Values of tokens 1 through 1000. Recomputing all of those every step would be quadratic waste. So we compute each token's K and V once and cache them. That turns per-step attention from O(n²) into O(n) — the single optimization that makes real-time generation possible.

The cache is not free. Its size is exact:

The 2 is Keys plus Values. Plug in Llama-70B-class numbers — layers = 80, head_dim = 128, fp16 so dtype_bytes = 2 — with kv_heads = 8 and an 8K context:

• Per token, all layers: 2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 0.31 MB
• Full 8,192-token context: 0.31 MB × 8,192 ≈ 2.5 GB sitting resident in VRAM the entire generation.

Now scale the context to 128K and the cache alone approaches 40 GB, competing with the model weights for the same 80 GB GPU. That is the memory wall: the term that explodes is seq_len, and the knob you control is kv_heads.

GQA is that knob. Multi-Head Attention gives every query head its own K/V (in Llama-70B, 64 heads). Multi-Query Attention shares a single K/V across all query heads. Grouped-Query Attention sits between them: partition the query heads into G groups, and all heads in a group share one Key/Value set.

Crucially, every query head is retained — each still computes its own attention pattern. Only the cached K/V are shared. That is why the quality drop is tiny while the savings are large: the 2.5 GB figure above is already Llama-70B's GQA number, computed with kv_heads = 8. If the same model ran full Multi-Head Attention instead — one KV set per query head, so kv_heads = 64 — the identical formula gives roughly 20 GB at the same 8K context, 8× more. Grouping those 64 query heads into 8 KV groups is precisely what pulls the cache back down from that 20 GB to the ~2.5 GB we computed, with almost no measurable loss in quality. MHA, GQA, and MQA are the same wiring at G = 64, G = 8, and G = 1.

What to remember#

• RMSNorm = LayerNorm minus mean-subtraction and bias, scaling by RMS instead of std. Pre-Norm placement keeps deep stacks stable.
• SwiGLU = two projections of h, SiLU on the gate path only, multiplied elementwise. Not SiLU(x)·x on a scalar.
• RoPE rotates Q/K by position so scores depend on relative distance; multi-frequency dials give both local precision and long-range reach.
• KV cache turns generation from O(n²) to O(n) but scales linearly with context length — the memory wall.
• GQA shrinks the cache by query_heads / kv_groups while keeping every query head, trading a sliver of quality for multiples of memory.

Keep going#

These four swaps are the architecture layer. The rest of the LLM lifecycle builds on top:

• Train Your First Language Model: From Raw Text to Generation — how these blocks stack into a trainable model.
• Supervised Fine-Tuning: How a Base Model Becomes an Assistant — turning a next-token predictor into a chat model.
• LLM Fine-Tuning, Part 1: When to Fine-Tune & How LoRA Works — adapting a pretrained model cheaply.
• RLHF vs DPO: How Models Learn Human Preferences — the alignment step that follows.

You can toggle each of these upgrades on and off and watch a 2017 block morph into a 2024 one in the interactive deep dive — explore it interactively on YeetCode.

FAQ#

What is the difference between RMSNorm and LayerNorm?#

They diverge in how much of a vector's statistics they lean on. LayerNorm first shifts the vector to zero mean, then rescales by its own standard deviation, and finishes with a per-channel weight and offset. RMSNorm skips both the shift and the offset — it divides purely by the vector's root-mean-square and applies a single learned weight. Losing the centering step means one less statistic to compute at every layer, and in practice accuracy holds up because that surviving weight quietly compensates for whatever the mean-removal would have contributed. Most implementations pair the swap with a placement change too: normalize the input feeding each sublayer instead of the output leaving it, which is the part that keeps gradients tame when you stack dozens of layers.

Why do modern LLMs use RoPE instead of learned position embeddings?#

Learned and sinusoidal embeddings inject position by adding a vector to each token; RoPE takes a different route, spinning every query and key through an angle set by where the token sits, with the angles governed by theta_i = base^(-2i/d). When two of those rotated vectors meet in the attention dot product, the two spins combine so the resulting score reflects only how far apart the tokens are, never their absolute slots — slide the whole sequence over and nothing changes. The frequencies also span a wide range of scales, and the slowest ones barely turn across a short gap, so you can stretch them to reach context lengths the model never trained on. That graceful stretch is exactly why long-context systems standardized on it over absolute or learned schemes.

What is the KV cache and why does it matter?#

Each decoding step emits one token, and that token has to look back over the key and value vectors of everything produced before it. Instead of rebuilding those vectors on every step, the model saves them the first time they are computed and reads them back afterward, which drops the per-step cost from quadratic to linear in sequence length. The trade is resident memory: the store grows in lockstep with context length — 2 × layers × kv_heads × seq_len × head_dim × dtype_bytes — and once the context is long enough it can swell to rival or overtake the weights themselves. That ceiling on how long a session a single GPU can carry is what people mean by the memory wall.

How does Grouped-Query Attention save memory without hurting quality?#

GQA bundles the query heads into a handful of groups and gives each group one shared set of keys and values. Since cache size tracks the number of KV groups rather than the full head count, collapsing 64 heads into 8 groups shrinks the store eightfold — a factor of query_heads / kv_groups. Accuracy barely shifts because no query head loses its own voice: each one still produces a distinct attention distribution, and all they hold in common is the underlying K/V they read from. That lands GQA squarely between vanilla Multi-Head Attention, where every head owns its keys and values, and the more aggressive Multi-Query Attention, where all heads fold onto a single shared set.