Betting on Early Fusion

Falcon Perception is a 0.6B-parameter early-fusion Transformer that processes image patches and text as a single sequence.

The core is a hybrid attention mask. Image tokens attend to each other bidirectionally, while text and task tokens are generated causally, attending to preceding tokens and the entire visual prefix. This allows a single backbone to implement both the visual encoder and the autoregressive predictor together.

The output interface is composed as Chain-of-Perception.

• <coord>: first determines the center position of the target.
• <size>: then predicts the size.
• <seg>: finally generates a segmentation embedding, recovering the mask via dot product with upsampled image features.

The coordinate and size heads use Fourier features to handle continuous values precisely, and segmentation is handled lightly without a separate complex query/Hungarian matching process.

For evaluation, PBench was also proposed. This benchmark divides samples by ability into categories such as attributes, OCR-guided, spatial, relations, dense, allowing weaknesses to be viewed as a profile rather than a single score.

Training was conducted at a scale of 54M images, 195M positive expressions, 488M hard negatives, using multi-teacher distillation to bring together the strengths of DINOv3 and SigLIP2. This was followed by 3-stage training through in-context listing, task alignment, and long-context finetuning, refining dense scenes and presence calibration.

As a result, it recorded 68.0 Macro-F1 on SA-Co, surpassing SAM 3's 62.3, and on PBench the gap widened significantly especially in segments requiring compositional reasoning such as OCR, spatial, relational, dense. Falcon OCR, also released in the same post, is a 0.3B model that also emphasizes high throughput and OCR performance.