Meta Releases FLAT, a Variable-Length Continuous Token Model Integrating Image and Text

Meta has released FLAT, a multimodal model that integrates understanding and generation by mapping images and text into a single flexible-length continuous token sequence. The model generates up to 256 continuous 64-dimensional register tokens for both modalities via a single VLM encoder.

Core Mechanism and Performance

FLAT uses a nested dropout technique to organize information in a coarse-to-fine manner. During inference, selecting the prefix length (K) allows balancing between computational cost and detail representation. The first token handles global semantics, while longer prefixes restore components and fine-grained attributes. Training objectives include cross-modal alignment, image synthesis, and captioning performed via joint training.

Key performance metrics are as follows:

• GenEval: 83.1
• COCO CIDEr: 138.6 (BLEU-4: 40.5)
• Retrieval (COCO R@5): I2T 86.8 / T2I 75.8
• ImageNet linear probe: 81.8

Scalability and Flexibility

FLAT's most notable feature is performance stability across changes in representation width (K). On COCO, performance changes by less than 1 point as K increases from 1 to 256, remaining nearly constant, and it shows competitive performance even at K=1. Conversely, for generation tasks, compositional details such as count, spatial relationships, and attributes are progressively resolved as K increases, without requiring model switching.

Additionally, FLAT continuous tokens significantly reduce the modality gap. A single FLAT token halves the CLIP image-text centroid distance, while the full sequence reduces it to approximately 1/4. Thanks to this shared decodable space, interpretable semantic transitions via vector operations and semantic compositions like "lighthouse" and "full moon" are possible.