Post Training Alignment and Reasoning

Pretraining creates broad capability. Post-training shapes how that capability is elicited: instruction following, helpfulness, refusal behavior, tool use, reasoning style, and preference alignment.

Primary sources:

• InstructGPT: Training language models to follow instructions with human feedback
• Training a Helpful and Harmless Assistant with RLHF
• Constitutional AI
• Direct Preference Optimization
• DeepSeek-R1

The Post-Training Stack

Typical stages:

1. Base model pretraining.
2. Supervised fine-tuning, or SFT, on instruction-response data.
3. Preference learning, often RLHF or DPO-style methods.
4. Safety tuning.
5. Reasoning-specialized training or reinforcement learning.
6. Evaluation, red-teaming, and deployment-specific tuning.

Supervised Fine-Tuning

SFT teaches the model the interaction format:

• Follow instructions.
• Answer concisely or in desired style.
• Use chat formatting.
• Demonstrate task solutions.

SFT is imitation learning. It can improve behavior but may not optimize preference tradeoffs deeply.

RLHF

RLHF usually means:

1. Collect human preference comparisons.
2. Train a reward model.
3. Optimize the policy model to maximize reward while staying near the SFT model.

Why the KL penalty matters:

• Without it, the policy can exploit reward-model weaknesses.
• The model may become weird or overoptimized.

DPO

Direct Preference Optimization reframes preference learning so the model can be optimized directly from preferred/rejected pairs without separately training and optimizing a reward model in the same way as PPO-style RLHF.

Why it matters:

• Simpler pipeline.
• Often more stable and cheaper than RLHF.
• Strong baseline for preference tuning.

Constitutional AI

Constitutional AI uses AI feedback guided by written principles to reduce dependence on human preference labels for every example.

Key idea:

• The model critiques and revises outputs according to a constitution.
• Preference models can be trained from AI-generated comparisons.

Reasoning Post-Training

Reasoning-focused systems often use:

• Math/code data.
• Chain-of-thought style demonstrations.
• Verifier or outcome rewards.
• Process supervision.
• Reinforcement learning on tasks with checkable answers.
• Test-time compute policies.

DeepSeek-R1 is a public example of reasoning behavior strengthened through reinforcement-learning-centered training.

Architecture Versus Post-Training

Important distinction:

• Architecture controls what computations are easy or efficient.
• Pretraining learns broad capabilities.
• Post-training controls which capabilities are elicited and how behavior is shaped.

Do not attribute every behavior change to architecture. Often it is data or post-training.

Failure Modes

• Reward hacking.
• Over-refusal.
• Sycophancy.
• Verbosity or style collapse.
• Hidden benchmark overfitting.
• Reasoning traces that sound plausible but are wrong.
• Capability suppression rather than true safety.

Research Questions

• How do we train reliable reasoning without teaching models to fake reasoning traces?
• What is the best way to allocate compute between pretraining and RL?
• Can preference methods preserve calibration?
• How should post-training work for MoE models?
• How do we evaluate alignment without conflating politeness with truthfulness?

Related

• Data Tokenization and Pretraining Objective
• Scaling Laws and Compute Optimal Training
• Open Research Questions Frontier LLM Architectures
• Researcher Skill Stack for Frontier Labs