Data Tokenization and Pretraining Objective
Architecture is only half the story. Frontier LLM behavior is strongly shaped by data, tokenization, objective, filtering, mixture design, and training curriculum.
Next-Token Prediction
Most decoder-only LLMs are pretrained with causal language modeling:
maximize log P(x_t | x_1, ..., x_{t-1})
Or equivalently, minimize cross-entropy loss over next-token predictions.
Why this works:
- Predicting the next token requires modeling syntax, facts, reasoning patterns, styles, formats, code, and latent world structure.
- A broad web-scale dataset turns this simple objective into broad representation learning.
Why it is not enough:
- The base model imitates data distribution rather than user intent.
- It may continue harmful, low-quality, or inconsistent patterns.
- It may know how to solve a task but not reliably choose to.
This motivates Post Training Alignment and Reasoning.
Tokenization
Tokenization maps text into discrete ids. Common approaches use subword tokens such as BPE or unigram tokenization.
Tokenization affects:
- Sequence length.
- Multilingual fairness.
- Code representation.
- Numeric reasoning.
- Rare-word handling.
- Context-window efficiency.
A bad tokenizer can make some languages or domains more expensive by requiring more tokens for the same content.
Dataset Mixture
Frontier pretraining data is a mixture:
- Web text.
- Books.
- Academic text.
- Code.
- Math.
- Q&A.
- Multilingual data.
- Synthetic data.
- Long documents.
- Domain-specific corpora.
Mixture design decides what the model gets good at.
Data Quality
Important operations:
- Deduplication.
- Filtering low-quality or toxic content.
- Removing benchmark contamination.
- Language identification.
- Code license filtering.
- PII reduction.
- Domain balancing.
- Synthetic-data validation.
Data curation can beat naive scale.
Curriculum And Stages
Training can happen in stages:
- Pretraining on broad data.
- Continued pretraining or midtraining on higher-quality/domain-specific data.
- Long-context adaptation.
- Instruction tuning.
- Preference optimization.
- Reasoning RL or specialized post-training.
Each stage changes behavior and can trade off capabilities.
Evaluation Risk: Contamination
If benchmark examples appear in training data, evaluation can overestimate capability.
Good research asks:
- Was deduplication performed?
- Is the benchmark public and likely contaminated?
- Are there private or held-out evals?
- Does the model generalize to variants?
Research Questions
- How should synthetic data be mixed without causing model collapse or overfitting?
- Which data improves reasoning rather than memorization?
- How much code/math data should a general model see?
- How should long documents be sampled for long-context ability?
- Can we design tokenizers that improve reasoning or multilingual efficiency?