The Gradient Is a Covariance: Why Post-Training Updates Are Sparse

Reinforcement learning, on-policy distillation, and supervised fine-tuning on near-policy data change only 5–30% of a model's parameters. While reweighting existing samples explains why updates are small, it does not explain why they are sparse; a dense update of small magnitude is still dense. The key distinction is that the expected gradient is a covariance and the Fisher information is a variance, which imposes a strict upper bound on the update magnitude.

The Covariance Structure

On the sampler's own samples, the score (the derivative of the log-probability) has a zero mean. Consequently, the expected gradient $G_i = E[A \cdot score_i]$ becomes the covariance $Cov(A, score_i)$, and the Fisher $F_i$ becomes the variance $Var(score_i)$. This leads to the inequality $|G_i| \le std(A) \cdot \sqrt{F_i}$. A large Fisher is necessary for a large gradient, but not sufficient; the reward signal must also align with the parameter's sensitivity. For instance, the top 1% of parameters by Fisher magnitude hold 47.40% of the top 1% by gradient magnitude, compared to 1% at random.

Token-Level Sensitivity

At the token level, the softmax operation subtracts a mean sensitivity, meaning the score reflects how much a parameter moves the drawn token's logit relative to the average candidate. The token Fisher is the variance of these logit sensitivities across likely candidates. A parameter has a small token Fisher if it moves all likely candidates equally, either because the model is confident or because the candidates react similarly. Across a full response, cross-terms between positions average to zero, so the total Fisher is the sum of token-level Fishers.

Score Centering for Off-Policy Data

This framework extends to off-policy settings through score centering. If samples are drawn from a different distribution $\mu$ (e.g., a stale model or inference engine), the score must be centered under $\mu$ rather than the current policy $p$. This requires knowing the sampler's full next-token distribution. Without centering, the score does not average to zero, breaking the covariance structure. With centering, the gradient remains a covariance, and the Fisher remains a variance, preserving the theoretical bounds even when the data is not on-policy.