AI BriefingZyphra Releases ZAYA1-8B
Key point
Zyphra has released ZAYA1-8B, an 8B MoE model trained on AMD Instinct MI300.
Details
ZAYA1-8B has been released as a 760M active / 8.4B total MoE language model. Pretraining, midtraining, and SFT were all conducted on an AMD Instinct MI300-based stack, using a jointly built cluster with IBM featuring 1,024 MI300x nodes and AMD Pensando Pollara interconnect.
On benchmarks, it showed strong performance in math, coding, and reasoning. The model card presents AIME'26 89.1, HMMT'26 71.6, LiveCodeBench-v6 65.8, GPQA-Diamond 71.0, and MMLU-Pro 74.2, with comparisons made to similarly-sized Qwen3/Qwen3.5 and Gemma 4 series models as well as larger open-weight models.
The core architecture is as follows.
- CCA (Compressed Convolutional Attention)
- MLP router for expert selection
- learned residual scaling that controls residual norm across depth
On top of this, the Markovian RSA test-time compute method was added to further boost performance, and the model was released under the Apache-2.0 license on Hugging Face and Zyphra Cloud. Its small total parameter count also suggests potential for on-device deployment.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.