AI Briefing#mixture-of-experts
The latest AI and developer news about #mixture-of-experts, with the original source and a short summary.
The latest AI and developer news about #mixture-of-experts, with the original source and a short summary.
Developed with Fraunhofer using NVIDIA GH200 GPUs, the Mixture-of-Experts model targets all 24 official EU languages.
The team optimized FSDP and Expert Parallelism on 384 GH200 GPUs to reach 1,300 tokens/s per GPU, overcoming inter-node bandwidth bottlenecks.
The method uses second-order interactions to maintain performance during aggressive expert removal, outperforming first-order baselines on Qwen3.5 and GLM-4.5 models.
The open-source family of four models uses non-autoregressive decoding to achieve 67.52 on Decision Index 0.4.1 and rank top 3 on OpenRouter.
Thinking Machines AI introduces Inkling 2 Small, a 594B MoE model with 17B active parameters, initially available with closed weights on Tinker, with full Apache 2.0 open weights planned for next week.
The handbook covers Megatron-LM configurations and system insights derived from 800k GH200 GPU-hours of training on the Alps cluster.
The 2.8T parameter Mixture-of-Experts model features a 1M token context window and achieves 65.7 on Terminal-Bench 4.0.
The 1.6T parameter model achieves 88.4 on Terminal Bench v2.1 and 91.9 on BrowseComp, outperforming LongCat-2.0 and approaching frontier models like Claude-Opus-5.
The study uses staggered synchronization and compressed communication to train a 32B Sparse MoE model on two geographically separated clusters, achieving 0.2525 bits/byte on general coding evals compared to 0.2549 for standard training.
The 1T parameter model features a 1M token context window and ranks 5th on the Artificial Analysis Intelligence index for open-weight models.
The new architecture uses a Mixture-of-Experts model and fused GPU kernels to deliver near-ranking-level performance at 300x fewer FLOPs than the 2024 system.
The model features a 35B-parameter mixture-of-experts backbone with approximately 3B active parameters per token and a 128K context window.
The framework uses MoE expansion and RL-based distillation to consolidate short-term context into long-term parameters, achieving 79.2% accuracy on AIME 24/25.
The method compresses the first loop visit and reallocates saved computation to increase hidden size across 1.3B-7B parameter models.
Cross-layer shared Meta Experts reduce parameter redundancy and improve downstream task accuracy compared to hierarchical Local Experts.
The model features a hybrid-linear MoE architecture supporting 256K token contexts and is available on Hugging Face and ModelScope.
The system uses a lightweight quantized draft model to predict expert usage, reducing PCIe I/O bottlenecks on commodity hardware.
Built on Qwen3.6-35B-A3B, the model features agentic orchestration and verification for long-horizon tasks.
The compact Mixture-of-Experts model supports speech recognition, audio reasoning, and optional synthesized speech responses in Chinese and English.
The open-weights MoE model achieves a score of 45, significantly above the median of 19 for similar-sized models, while offering 230.4 tokens/sec output speed.
The open-weight reasoning model features 30B parameters with 3B active per token and a 1M token context window.
The new quantized models achieve near-BF16 performance at 3.50 bpw, while the expert-pruned Coder build reduces the resident working set to 29.6 GB.
Vercel AI Gateway supports the open-source MoE model Hy4 Preview with 770B parameters.
Qwen4Exp maintains inference speed by offloading 25% of weights to SSD using N-grams.