Scaling Categorical Flow Maps

Apple researchers scaled Categorical Flow Maps (CFM), previously evaluated only on models with fewer than 1 billion parameters, to a 1.7B parameter model. After training the base Flow model on 2.1T tokens, they distilled it to generate diverse, high-quality text with just 4 inference steps.

CFM connects Gaussian distributions and discrete data distributions encoded as one-hot vectors via Flow Matching. This enables the use of fast sampling and generation result adjustment capabilities of continuous generative models in language modeling, while maintaining the entropy of generated tokens close to real data levels.

The researchers also derived a likelihood lower bound for CFM in semi-discrete settings. They used this to evaluate the model on standard language model benchmarks, finding performance comparable to discrete Diffusion models.

They analyzed issues arising during large-scale training and provided the following practical guidelines:

• Adjusting training loss weights
• Optimizing time scheduling
• Addressing key challenges in large-scale CFM training

These results demonstrate that CFM can be applied to large language models and fast text generation beyond small-scale experiments.