SFT, RL, and On-Policy Distillation Viewed Through the Lens of Distribution

Viewed as a sequence distribution, a language model's post-training is the process of realigning that distribution toward a new target. The key question is how directly and how a given target distribution is defined.

SFT pulls the model directly toward a fixed external data distribution. cross-entropy is effectively close to minimizing forward KL, and the tokens presented by the data matter more than the starting model. This makes it strong for format conversion or cold-start, but catastrophic forgetting easily occurs the further it moves from the original distribution.

RL moves in the direction of increasing expected reward, evaluated on rollouts the model itself samples. Here, the reference is not an external ground-truth distribution but the current policy, and this direction is relatively clear in RLVR, where rewards are verifiable. For this reason, an explicit KL penalty is sometimes set weakly, or the trust-region constraint is relaxed, as in GRPO instead of PPO.

On-Policy Distillation (OPD) sits between SFT and RL. The student generates samples on-policy, but the update is done via reverse KL to match the teacher distribution. In a variant called OPSD, the teacher and student are the same model, but the teacher is given a reference solution prefix, and in per-token KL analysis, style tokens like wait, alright showed larger values than math tokens like power, exponent, logarithm. This made per-token clipping necessary to prevent excessive updates. Because of this signal, OPSD is closer in character to RLVR than to RLHF.

In the Minimal Code Editing experiment, a buggy corrupted function was given and only the bug was to be fixed. After training on one corruption set, generalization was assessed by evaluating on a different corruption type, and general code generation ability was checked with LiveCodeBench v6. The RL teacher achieved a higher final reward on its own training and generalized better, but the OPD student came out nearly the same regardless of which teacher was used. Even when the SFT teacher already showed forgetting in general code generation, the OPD student did not inherit much of that loss.

• Teacher: SFT Pass@1 0.775, Norm. Levenshtein 0.450, Added CC 0.450, LiveCodeBench v6 0.286
• Teacher: RL Pass@1 0.792, Norm. Levenshtein 0.063, Added CC 0.206, LiveCodeBench v6 0.320
• OPD student: SFT teacher 0.800 / 0.059 / 0.206 / 0.297
• OPD student: RL teacher 0.787 / 0.055 / 0.228 / 0.314

These results suggest that on-policy sampling itself may matter more than the teacher distribution. The author considers the view that explains why RL forgets less solely in terms of forward KL vs reverse KL to be insufficient, and finds the following two explanations more persuasive.

• Data-dependent regularization: SFT applies uniform pressure to all tokens, whereas RL relatively reduces the update when reward variance within a group is large.
• Sparse parameter updates: RL produces sparse full-rank updates that mainly touch a small subnetwork, and reducing the number of updated parameters causes performance to collapse faster than with SFT, showing that RL updates are more essential.

Meanwhile, the outcome reward in RLVR has the limitation of providing only O(1) bits per episode, while OPD provides a unique signal per token, but at the cost of greater noise and bias.