bf16 Fine-Tuning Freezes 95% of Weights, Masking Optimizer Instability

Fine-tuning with bf16 weights and torch.optim.AdamW without an fp32 master copy causes weight updates to freeze for large-magnitude weights, effectively acting as a sparse fine-tuner.
In a rerun of the Quiet-STaR recipe on Mistral-7B, 97-99.7% of proposed updates rounded to zero in bf16, while correct fp32 runs at the same learning rate diverged, with loss peaking significantly higher than the stable bf16 run.
The bf16 rounding error hides instability by anchoring the model to pretrained weights; a correct fp32 optimizer at a lower learning rate (3e-7) matches bf16 performance at higher rates (1e-6).
The phenomenon explains why sparse update methods like PaFi and NanoAdam perform well, as the bf16 bug accidentally implements magnitude-based sparsity.

Mechanism and Impact

bf16 has 7 mantissa bits, creating gaps 65,536 times wider than fp32; updates smaller than half the gap round back to the original value.
At learning rate η, only weights with magnitude less than ~512·η can update; for lr 1e-6, weights above ~4.88e-4 are frozen.
AdamW's weight decay and second moment (v) also freeze in bf16, as their relative changes round away.

Diagnostic: Run a single step at full learning rate; if <10% of weights change, the optimizer is acting as a sparse fine-tuner.
Preferred fix: Use fp32 master weights; alternative: Use stochastic rounding (e.g., torchao's bf16_stochastic_round=True) to average updates over time.
Reproduction advice: Divide learning rates by 3 to 10 when moving bf16 recipes to correct fp32 stacks like DeepSpeed or FSDP.