Glossary Frontier LLM Architectures
Active Parameters
The number of parameters used for a token's forward pass. In MoE models, total parameters can be much larger than active parameters.
Activation Checkpointing
Training memory technique that saves only selected activations and recomputes others during backpropagation. It trades extra compute for lower memory.
Attention Head
One projection subspace in multi-head attention. Each head has its own query/key/value projections unless using MQA or GQA.
Causal Mask
Mask that prevents a position from attending to future tokens.
Chinchilla Scaling
Compute-optimal training result arguing that many large LMs were undertrained and should use more tokens for a given compute budget.
Cross-Entropy Loss
The standard next-token prediction loss. It penalizes the model when it assigns low probability to the true next token.
Dense Model
A model where all parameters in a layer are activated for every token, unlike sparse MoE models.
DPO
Direct Preference Optimization. A preference-tuning method that optimizes directly from preferred/rejected response pairs.
Expert Parallelism
Distributed training/serving strategy where different devices host different MoE experts.
Expert
In MoE, usually an MLP subnetwork selected by a router for a subset of tokens.
FlashAttention
An IO-aware exact attention algorithm that avoids materializing the full attention matrix in slow GPU memory.
GQA
Grouped-Query Attention. Multiple query heads share fewer key/value heads, reducing KV-cache memory compared with full multi-head attention.
In-Context Learning
The ability of a model to adapt behavior from examples in the prompt without weight updates.
MFU
Model FLOPs utilization. A measure of how much of the theoretical hardware compute is being used for model computation.
KV Cache
Stored keys and values from previous tokens during autoregressive generation.
LoRA
Low-Rank Adaptation. A parameter-efficient fine-tuning method that trains small low-rank update matrices instead of all weights. Primary source: LoRA.
MLP
Position-wise feed-forward network inside a transformer block. Often the largest parameter component and the target of MoE replacement.
MQA
Multi-Query Attention. Many query heads share one key/value head to reduce decoding memory and bandwidth.
Pre-Norm
Transformer layout where normalization happens before attention or MLP sublayers.
Perplexity
Exponentiated cross-entropy loss. Lower perplexity means the model assigns higher probability to the held-out text.
Prefill
Inference phase where the prompt is processed and the KV cache is created.
Residual Stream
The main hidden representation carried through transformer layers. Sublayers write updates into it through residual connections.
RLHF
Reinforcement Learning from Human Feedback. Typically trains a reward model from preference data and optimizes the policy against it with a constraint to remain close to the original model.
Reward Model
A model trained to score outputs according to preference data. Used in RLHF pipelines.
RoPE
Rotary Position Embedding. Encodes position by rotating query/key representations in a way that supports relative position structure.
Router
The network component in MoE that assigns tokens to experts.
SFT
Supervised Fine-Tuning. Instruction tuning by imitation on prompt-response examples.
Sparse Model
A model where only a subset of parameters activates per token. MoE is the main LLM example.
SwiGLU
Gated activation commonly used in modern transformer MLPs.
Temperature
Sampling parameter that controls sharpness of the output distribution. Lower temperature is more deterministic; higher temperature is more random.
Tensor Parallelism
Distributed strategy that splits large matrix operations across devices.
Token
A discrete unit produced by the tokenizer. Tokens may be words, subwords, bytes, punctuation, or code fragments.
Top-k Routing
MoE routing method where each token is sent to the k highest-scoring experts.
Top-p Sampling
Sampling method that keeps the smallest set of tokens whose cumulative probability reaches threshold p, then samples from that set.
Verifier
A model or program that checks candidate answers. Verifiers are important for math, code, and reasoning training/evaluation.