Transformer Block Anatomy

One pre-norm decoder block: the residual stream flows through RMSNorm, multi-head attention (QKV heads), a residual add, another RMSNorm, a SwiGLU MLP, and a second residual add.

The transformer block is the core unit of modern frontier LLMs. Most frontier models are still recognizably stacks of decoder-only transformer blocks, with many engineering changes around normalization, positional encoding, attention variants, MLP design, routing, and systems.

Primary source: Attention Is All You Need.

The Decoder-Only Shape

A causal LLM takes a sequence of token ids:

x_1, x_2, ..., x_T

It maps them to embeddings, processes them through many transformer blocks, and predicts the next-token distribution at each position:

P(x_{t+1} | x_1, ..., x_t)

A modern pre-norm decoder block usually looks like:

h = h + Attention(Norm(h))
h = h + MLP(Norm(h))

This means each block adds updates into the residual stream.

The Residual Stream

The residual stream is the model's working memory at each token position. Attention and MLP sublayers do not replace the representation; they write updates into it.

Why this matters:

Information can flow through many layers without being overwritten.
Features can be incrementally composed.
Interpretability work often studies what is written into the residual stream.
Pre-norm makes optimization more stable at depth.

Attention Sublayer

Self-attention is the token-mixing part.

At each position, the model computes:

Query: what this position is looking for.
Key: what each position offers as an address.
Value: what each position contributes if attended to.

Core equation:

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

Causal masking prevents a token from attending to future tokens.

Related note: Attention Mechanics and KV Cache.

MLP Sublayer

The MLP is applied independently at each token position. It does not directly move information across positions. Its job is feature transformation and nonlinear computation.

Modern LLM MLPs often use gated activations such as SwiGLU:

MLP(x) = W_out( SiLU(W_gate x) * W_up x )

Why MLPs matter:

They hold a large share of model parameters.
They are a major target for Mixture of Experts Architectures.
They can be interpreted as feature detectors and feature writers.

Normalization

Modern LLMs often use RMSNorm instead of LayerNorm.

Why normalization matters:

Stabilizes training.
Controls activation scale.
Helps deep residual networks optimize.

Pre-norm is common:

x + Sublayer(Norm(x))

Post-norm was used in the original Transformer:

Norm(x + Sublayer(x))

Positional Information

Attention itself is permutation-invariant unless position is added. LLMs use schemes such as:

Learned position embeddings.
Sinusoidal embeddings.
Rotary position embeddings, or RoPE.
ALiBi-style attention biases.

RoPE is especially common in modern decoder LLMs because it injects relative position structure into query/key space.

Primary source: RoFormer: Enhanced Transformer with Rotary Position Embedding.

Modern Frontier Block Ingredients

Common ingredients in contemporary LLM blocks:

Pre-norm.
RMSNorm.
RoPE.
SwiGLU or gated MLPs.
Multi-head attention, often with GQA or MQA for inference efficiency.
Larger MLP hidden dimension than attention dimension.
Sometimes sparse expert MLPs via Mixture of Experts Architectures.

What To Be Able To Derive

You should be able to write shapes for:

Token ids: [batch, seq]
Embeddings: [batch, seq, d_model]
Q/K/V: [batch, heads, seq, d_head]
Attention weights: [batch, heads, seq, seq]
MLP hidden: [batch, seq, d_ff]
Logits: [batch, seq, vocab]

Research Questions

Which architectural changes still matter at frontier scale?
How much of capability comes from attention versus MLP capacity?
Why do pre-norm and residual scaling choices change trainability?
When does a new block design survive scale-up?