Post-training

Understanding Its Purpose First

Post-training further adjusts the behavior of a base model to make it more suitable for following instructions, answering questions, or completing specific tasks.

Understanding with an Example

Being able to continue writing a textbook does not mean it knows you want a "40-minute lesson plan." Demonstrating correct answers can teach format and task behavior; preference feedback can also help compare which responses better match requirements.

Going a Step Deeper

Supervised fine-tuning, preference optimization, and reinforcement learning are all possible components. A specific training scheme may combine several of these, or use only some of them. Post-training is a collective term for stages and activities; the roles of SFT, DPO, PPO, and GRPO should be understood separately.

What It Does Not Imply

Post-training does not mean being superior to the original model in all capabilities, nor does it automatically enable integration with external tools. The diagram only illustrates that further adaptation is possible; it does not prescribe the same training sequence for every company.

What to Explore Next

• Supervised Fine-Tuning (SFT)
• Reinforcement Learning from Human Feedback (RLHF)
• Direct Preference Optimization (DPO)
• Group Relative Policy Optimization (GRPO)
• Low-Rank Adaptation (LoRA)

References

• Training language models to follow instructions with human feedback: InstructGPT example: demonstration data, human ranking, post-training through feedback.
• Direct Preference Optimization: Direct objective on preference pairs; the standard form does not require a separately trained reward model or PPO online loop.
• DeepSeekMath · GRPO: Section 4.1: candidate groups for the same question, relative rewards, no need for a separate value model; outcome supervision and process supervision.

Data verified on 2026-09-09; original papers are cited to explain mechanisms, and examples in the text are for instructional purposes.