AI Briefing#moe
The latest AI and developer news about #moe, with the original source and a short summary.
The latest AI and developer news about #moe, with the original source and a short summary.
On a Mac Mini M4 Pro 64GB, the 14B dense model hit 16.2 t/s, while Qwen3.6 35B NVFP4 reached 69.8 t/s.
Qwen has released official Sparse Autoencoders for Qwen 3.5 and Qwen3.
Qwen has released Qwen-Scope, an SAE-based interpretability tool built on Qwen3 and Qwen3.5.
An analysis of DeepSeek-V4's architectural innovations, infrastructure optimizations, and the trial-and-error process during training.
Poolside has unveiled the Laguna XS.2 and Laguna M.1 models, optimized for coding agent performance.
Poolside released Laguna XS.2, a 33B A3B MoE model, under Apache 2.
This is a research paper that directly compares and analyzes the performance and efficiency of MoE and Dense architectures.
The llama.cpp CUDA MMQ kernel improved speed in some ranges using fastdiv and dynamic tiling.
Using OpenMythos as a case study, this piece analyzes the 'sheepwave' phenomenon where architectures overheat without technical verification, and examines the actual implementation level.
A PR supporting Cohere MoE models has been submitted to vLLM.
DeepSeek has released the V4 model family, which supports a 1M token context.
A DeepSeek-V4 preview has been released, supporting a 1.6T MoE and 1M context.
Growing a 32-expert MoE to 64 experts mid-training cut GPU hours by about 32%.
Tencent has unveiled Hy3-preview, an MoE model with 21B active parameters out of a total of 295B parameters.
A merged model combining Qwen3.6-35B-A3B with Qwopus and Holo3, along with benchmarks, has been released.
In a comparison of the new 3.6-27B models, the MoE model is rapidly catching up to the Dense model, particularly in coding performance.
Shared a setup for running Qwen3.6-35B-A3B with llama-server on an RTX 4060 8GB VRAM, along with a thinking budget pitfall.
MoE models achieve higher inference acceleration when Speculative Decoding techniques are applied.
In a 4x RTX 3090 test, MoE Qwen models violated strict bash rules more often than a dense 27B.
Boosted Qwen3.6-35B-A3B speed by 54% on an RTX 5070 Ti using --n-cpu-moe.
A 35-domain LoRA routing model based on Qwen3.5-35B-A3B has been released.
In an apples-to-apples FP8 comparison, the advantage between Ironwood and B200 reversed depending on the model.
On dual RTX 5060 Ti GPUs, Qwen 3.6-35B-A3B hit 21.7 tok/s at 90K context.