Subquadratic Breaks the Context Window Limit, Unveils 12 Million Token Window

Subquadratic has released a model based on Subquadratic Selective Attention (SSA) with a 12 million token context window. The company offers this window via API, and aims for a 50 million token window by Q4.

The industry has already entered the era of million-token windows, but models that actually make good use of long context remain rare. On MRCR v2, GPT-5.5 managed only 74.0%, and Anthropic's Claude Opus 4.7 only 32.2%, which is why workarounds like RAG, agentic decomposition, and hybrid model architectures have been widely used.

The core claim of SSA is that it selects important positions based on the content of queries and keys, while keeping the selection itself linear. Subquadratic stated it is 7.2x faster at 128K and 52.2x faster at 1 million tokens compared to dense attention, and achieved 92.1% on a 12 million token needle-in-a-haystack test and 83 on MRCR v2, surpassing GPT-5.5's 74.0%.

Key benchmarks are as follows.

• RULER 128K: scored 97.1, exceeding Anthropic's Opus 4.6 (94.8).
• SWE-Bench Verified: 82.4%, narrowly ahead of Opus 4.6's 81.4% and Google's Gemini 3.1 Pro's 80.6%.
• Each model was run only once, and the paper itself acknowledges that SWE-Bench results are heavily influenced by the evaluation environment.

Previous approaches have also shown limitations. Fixed-pattern sparse attention like Longformer is strong on nearby information but misses distant dependencies, while Mamba, Mamba-2, RWKV, and RetNet suffer from lossy state compression. In an 8B-scale study by Nvidia, pure Mamba-2 lagged behind transformers on MMLU and phonebook lookup, and the gap only narrowed when attention was reintroduced.

The recent Native Sparse Attention received the ACL 2025 best paper award, and its successor, DeepSeek Sparse Attention (DSA), was incorporated into DeepSeek V3.2-Exp. However, even though the actual attention in DSA is sparse, the indexer that selects keys must compare all queries and keys, making the selection stage quadratic again. Hybrids like Jamba, Kimi Linear, Qwen3-Next, and Nemotron v3 also supplement retrieval with some dense attention, but ultimately fail to eliminate the remaining O(n²) cost.

The current product is offered in beta as an API, the SubQ Code CLI agent, and the SubQ Search deep research tool. Both products run on neoclouds rather than major hyperscalers, and while the company does not release weights, it plans to offer post-training tools for enterprises. The company was previously called Aldea, and after pivoting from a speech model, has raised $29 million to date at a $500 million valuation. Magic.dev raised over $500 million in August 2024 by touting a 100 million token model and 1000x efficiency, but as of early 2026, little external evidence of LTM-2-mini adoption has been found. In this category, the gap between demos and real-world adoption remains wide.