Lambert's Lecture 10 explains why SFT forgets, RL generalizes

A quiet argument runs through the tenth lecture of Nathan Lambert's RLHF & Post-Training Book course: the order in which you fine-tune a model, supervised first and then reinforcement, is a consequence of how the two loss functions push probability mass around, not just tradition.

The lecture, titled 'Regularization in RL, Why RL Generalizes, and why SFT Forgets,' pairs with the book's regularization chapter and, per the course page, covers 'the KL penalty, implicit regularization, and why RL forgets less.' The framing is a direction-of-KL argument. SFT trains against a forward KL, which the chapter describes as mode-covering: it 'tries to cover both modes of the target distribution, which pushes the policy to stretch and redistribute probability mass from the old mode, disrupting its shape and causing forgetting.' RL uses reverse KL, which is mode-seeking, and 'can shift a new mode it samples from toward the target without touching the old mode at all, leaving prior knowledge intact.'

Alongside that implicit story, the lecture covers the explicit tool practitioners actually configure: a KL penalty against a frozen reference policy, which the chapter calls 'the most popular variant, used in most RLHF implementations as of 2026.' The reward is offset by a lambda-weighted KL between the current policy and the reference, discouraging drift into nonsensical output. Cited work from Chu et al. (2025) is summarized in the chapter as 'RL learns transferable rules, while SFT memorizes the training data and collapses under distributional shift,' and Shenfeld et al. (2026) is summarized as showing on-policy methods are 'inherently biased toward solutions that remain closer to the original policy in KL divergence.'

Why this matters for anyone shipping post-trained models: it gives a first-principles reason to expect regressions when you SFT on top of an RL-tuned checkpoint, and a reason to keep data collection on-policy when preserving the base model's breadth matters.

The honest caveat is that YouTube blocked a direct fetch of the video, so this reads the chapter and course-page description the lecture tracks against, not the spoken talk. Numeric ablations, live demos, and any updated framing Lambert gives on camera are not captured here. For teams designing post-training pipelines, the testable takeaway is the ordering assumption: if reverse KL really is what protects prior knowledge, SFT patches applied on top of RL-tuned models deserve the same forgetting audits you would run on any base-model swap.