The Rise of Hybrid LLMs

Transformer is still the standard architecture, but the quadratic complexity of self-attention and the growing KV-cache cause costs to spike sharply at long context lengths. Fixed context limits also make it easy to miss global information, requiring workarounds like chunking and retrieval.

Mamba, a selective state-space model (SSM) proposed by Albert Gu and Tri Dao in December 2023, demonstrated linear-time inference and a 5x throughput improvement without attention. Through input-dependent parameters and content-aware computation, it showed strong performance in domains like language, audio, and genomics, becoming the starting point for hybrid LLMs.

Since then, the industry has moved quickly toward mixing Mamba and Transformer. Jamba mixed attention and Mamba at a 1:7 ratio and added MoE to support a 256K token context, and Jamba 1.5 scaled this up to 398B total / 94B active parameters. MambaVision validated hybrid design in vision, while Codestral Mamba and Mamba-Llama did so in code and conversational models.

In 2025, this trend became even clearer. Falcon Mamba, a pure Mamba model, showed consistent memory usage and throughput in long-form reasoning, while Nemotron-H, Bamba, Hunyuan TurboS, and Phi-4-mini-flash-reasoning combined Mamba2 with attention, MoE, GMU, and other elements to pursue faster inference and long-context handling. The key point is that hybrid and SSM approaches are moving beyond the experimental stage to become the new default for deployable foundation models.