Apple Research Improves Discrete Flow Matching Trajectories with Energy Compass… 32% Perplexity Reduction vs. Teacher Model with 8-Step Inference

Existing Discrete Flow Matching requires hundreds of forward passes to convert noise tokens into language, and during distillation, the cause of student model performance degradation was often attributed to insufficient capacity. However, the researchers point out that the Trajectory itself is the bottleneck. This is because existing trajectories repeatedly perform random jumps without quality assessment at intermediate time points, leading to the accumulation of initial errors.

The researchers addressed this issue through TS-DFM (Trajectory-Shaped Discrete Flow Matching). During training only, a lightweight energy compass is used to evaluate candidate continuity and guide the trajectory to select the most consistent path. This method does not affect inference costs and operates only during the training stage.

Experimental results on a 170M parameter language model showed that the student model applying TS-DFM recorded 32% lower perplexity than the 1,024-step teacher model with 8-step inference, which corresponds to a 128x improvement in inference speed. It also demonstrated superior performance compared to existing discrete generation-based methods that increased data volume by 6x or model size by 5x.