NeoMME Releases Single Encoder

Architecture and Features

NeoMME is a multimodal encoder with 260M and 800M parameters that processes text tokens and raw image patches with a single bidirectional Transformer, without using a pretrained vision tower or causal language model (LM). The entire model was trained from scratch with a masked discrete-diffusion objective function.

• Native multimodal input: Text is input as decomposed token embeddings, and images are divided into 32x32 patches and projected through a small MLP into the same Transformer encoder.
• Dynamic image resolution: Maintains the aspect ratio and size of images, adjusting the number of tokens based on information density.
• Long context: Supports a context length of 16,384 tokens, applying a modern encoder stack that alternates between sliding window attention and global attention.

Performance and Efficiency

NeoMME-Retriever, fine-tuned for visual document retrieval using ColPali's page image approach, returns dense and late-interaction embeddings in a single forward pass.

• Processing speed: On an NVIDIA L40S GPU, with 2048x2048 image inputs, the 260M model encodes approximately 51 pages per second, showing about 2x the throughput of ColModernVBERT.
• Storage optimization: Through hierarchical token pooling and asymmetric quantization, the storage space for the late-interaction index was reduced from approximately 1.5MB per page to 6kB, a 255x reduction, while maintaining over 95% of the performance at nDCG@10.
• Benchmarks: Located on the Pareto frontier based on nDCG@10 and model size on ViDoRe v3.

Deployment

Model checkpoints are released under the Apache 2.0 license and are available on Hugging Face Transformers.