New Sparse Attention Architecture DWARF Released

DWARF-55M-Base, the first model based on the new architecture DWARF, has been released. This model uses DSQG (Dynamic Sparse Query-Gather) layers, which have a sparse structure in almost every layer, as its backbone, and includes only a single Full Causal Attention layer to coordinate the entire context.

Instead of scanning all past tokens, DSQG reduces computation by sampling adjacent and long-range token offsets within a fixed range. This design is intended to make KV cache bandwidth and attention computation converge to nearly O(1), regardless of context length.

Key features are as follows:

• Efficient context extension: Although trained on 2,048 tokens, testing showed reliable retrieval performance up to 6,144 tokens, up to 3x the training length.
• HISA path support: As an experimental feature, the HISA (Hybrid Sparse Attention) path can be used instead of Full Attention to achieve competitive performance in a purely sparse manner.
• Training data: The model was trained using 10B tokens from the Dolma3 Mix 150B dataset.