What We Learned This Week (20 min read)

Preventing distillation is harder than it seems. Even if you hide the chain of thought, there are workarounds, and agentic workflows involving locally run tool use and file edits are hard to fully obscure. In fact, a product company could effectively distill a frontier model by training its own model using the "gold diff" — the final output a user actually wanted — as an RL target.

Two main reasons are highlighted for why pretraining fails. One is designs that break causality. In expert routing, token routing suffers from severe imbalance, and expert choice causes token placements to influence each other, using future information and thereby creating a train-serve mismatch. Token dropping can also disturb causality in a way where later tokens affect the processing of earlier tokens.

The other is the accumulation of bias. Variance can be averaged out, but bias keeps accumulating. For example, in FP16 collectives, adding a small gradient to a large accumulated value can cause significant distortion due to precision issues, and bugs of this kind are extremely hard to find in large-scale training.

Training parallelism is described as a staged solution to get past these problems. The basic approach is data parallel, followed by FSDP. With FSDP, each GPU holds only a portion of the parameters and performs an all-gather per layer, while gradients are handled via reduce-scatter instead of all-reduce, lowering communication volume to around params × 3. However, as the number of GPUs increases, compute time decreases but comms time does not, creating a crossover point, at which pipeline parallelism needs to be added.

But pipeline parallelism comes with its own cost. Bubbles that occur at the front and back of a batch reduce GPU utilization, and when residual or attention structures become entangled across stages, it even slows down research velocity. So the key isn't more parallelism, but discipline — rigorously controlling where causality is broken and where bias accumulates.