Elastic Cores for ViT Released

Dense self-attention in ViT gets expensive as N² with more tokens, but this method lowers the computation to 2NC + C² using core-periphery block-sparse attention.

It trains with nested dropout so the number of core tokens can be adjusted at test time, allowing inference cost to be elastically matched accordingly.

• It showed competitive dense features and classification accuracy compared to DINOv3.
• Performance was stable across the 256~1024 resolution range.
• Attention maps in early layers were isotropic, becoming semantically aligned in deeper layers.
• Reducing the number of core tokens made attention wider and more diffuse, while increasing it made attention smaller and more concentrated.

The paper was published on arXiv, and the code is still in progress on GitHub.