Frontier Model Systems and Inference
Frontier architecture is constrained by hardware. A design that looks elegant on paper may lose if it is slow, memory-heavy, communication-heavy, or hard to serve.
Training Versus Inference
Training bottlenecks:
- FLOPs.
- Activation memory.
- Optimizer state.
- Distributed communication.
- Data loading.
- Checkpointing.
- Stability at scale.
Inference bottlenecks:
- Prefill cost for long prompts.
- Decode latency token by token.
- KV-cache memory.
- Batch scheduling.
- Quantization quality.
- Hardware utilization.
Prefill And Decode
Prefill
The model processes the prompt and fills the KV cache. Long prompts make prefill expensive because attention sees the prompt sequence.
Decode
The model generates one token at a time using the KV cache. Decode is often memory-bandwidth-bound because it repeatedly reads model weights and KV cache for small increments of work.
Parallelism
Frontier models use combinations of:
- Data parallelism.
- Tensor parallelism.
- Pipeline parallelism.
- Sequence/context parallelism.
- Expert parallelism for MoE.
The hard part is not naming these. The hard part is knowing which communication pattern dominates.
Quantization
Quantization reduces precision to lower memory and improve throughput.
Common targets:
- Weights.
- Activations.
- KV cache.
Tradeoffs:
- Lower memory.
- Faster serving.
- Possible quality loss.
- Hardware/kernel dependency.
Speculative Decoding
Speculative decoding uses a cheaper draft model to propose tokens, then the larger model verifies them.
Goal:
- Reduce latency while preserving the larger model's output distribution.
Research question:
- How do draft quality, acceptance rate, and serving complexity trade off?
MoE Serving
MoE adds serving complications:
- Tokens route to different experts.
- Expert load can be uneven.
- Expert parallelism creates communication.
- Small-batch interactive serving can underutilize experts.
This links directly to Mixture of Experts Architectures.
Long Context Serving
Long context stresses:
- KV-cache memory.
- Prefill latency.
- Attention kernels.
- Request scheduling.
- Cache eviction and reuse.
Long context is not just "increase max sequence length." It is an infrastructure problem.
Research Questions
- Which architecture improves quality per dollar, not just quality per FLOP?
- How should models be designed for decoding latency specifically?
- Can KV-cache compression preserve reasoning?
- When does quantization damage tool use, code, or math?
- How should serving systems schedule MoE and long-context workloads?