Evaluation Benchmarks and Scientific Method
Evaluation is how LLM research knows whether it made progress. Bad evals create fake progress. Good evals reveal capability, failure modes, and scaling behavior.
Primary sources:
Why Evaluation Is Hard
LLMs are broad systems. A single score can hide:
- Memorization.
- Data contamination.
- Prompt sensitivity.
- Format sensitivity.
- Overfitting to public benchmarks.
- Model verbosity.
- Judge bias.
- Cost and latency regressions.
- Safety regressions.
Types Of Evaluation
Loss Evaluation
Measures predictive performance on held-out text.
Pros:
- Smooth.
- Scales predictably.
- Useful during pretraining.
Cons:
- Does not directly measure instruction following, reasoning, tool use, safety, or user value.
Multiple-Choice Benchmarks
Examples: MMLU-style tasks.
Pros:
- Easy to run.
- Comparable.
- Covers many domains.
Cons:
- Guessing and calibration issues.
- Contamination risk.
- Can reward test-taking artifacts.
Generative Reasoning Benchmarks
Examples: GSM8K, MATH, code tasks.
Pros:
- Tests multi-step outputs.
- Closer to real use.
Cons:
- Prompt-sensitive.
- Harder to grade.
- Chain-of-thought can be misleading.
Human Or Model Preference Evals
Pros:
- Captures style and usefulness.
- Useful for chat behavior.
Cons:
- Judge model bias.
- Human preference inconsistency.
- Can reward sycophancy or verbosity.
Systems Evals
Measure:
- Latency.
- Throughput.
- Memory.
- Cost per token.
- Time to first token.
- Max batch size.
These are essential for Frontier Model Systems and Inference.
Contamination
Benchmark contamination happens when eval examples appear in training data.
Research defense:
- Deduplicate against benchmarks.
- Use private evals.
- Use generated variants.
- Test robustness to paraphrase.
- Track suspiciously high performance.
Prompt Sensitivity
A model's score can change with:
- Few-shot examples.
- System prompt.
- Answer format.
- Chain-of-thought instruction.
- Temperature.
- Tool access.
Always log prompts.
Capability Versus Elicitation
If a model fails, there are multiple explanations:
- It lacks the capability.
- It has the capability but the prompt fails to elicit it.
- It knows the answer but refuses.
- It reasons correctly but outputs invalid format.
- It is derailed by context or sampling.
This connects to Post Training Alignment and Reasoning.
Scientific Method For LLM Research
For any claim:
- Define the bottleneck.
- Choose a baseline.
- Isolate one variable.
- Run enough seeds or sizes to avoid noise.
- Evaluate across domains.
- Include systems cost.
- Check contamination.
- Inspect failures.
- State what would falsify the claim.
A Good Eval Suite For Architecture Work
Use a mix:
- Validation loss by domain.
- Short-context language modeling.
- Code generation.
- Math reasoning.
- Long-context retrieval and synthesis.
- Instruction following.
- Safety/refusal behavior.
- Latency and memory.
- Robustness to prompt variants.
Research Questions
- Which evals best predict user value?
- How do we measure reasoning without rewarding fake rationales?
- How do we evaluate agentic tool use reproducibly?
- Can private evals stay private long enough to matter?
- How should model cost be included in benchmark reporting?