AI Briefing#inference-optimization
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
The open-source vllm-xtu-moe project achieves up to ~20× faster prefill than CPU-only paths for medium contexts by streaming expert weights from CPU to GPU.
A Jev-compatible open-weight model that returns probability distributions in a single forward pass to address LLM hallucination and latency issues.
CLM-v0.1-8B, based on Qwen3-8B, released with candidate action selection speed up to 13x faster than Jev
A router trained on RouteLLM data achieved 0.84 AUC but dropped to 0.838 when labels were shuffled, indicating it learned task identity rather than item difficulty.
The new 280M-parameter draft model accelerates LFM2.5-VL-3B inference by up to 3.13x on device and 2.66x on H100 GPUs.
CLM-8B achieves SOTA on agent coding benchmarks by separately encoding and caching states and actions.
Meta has introduced Muse Realtime Avatar, a system that generates synchronized, expressive video from speech tokens with subsecond latency, outperforming Runway and HeyGen in user preference tests.
The C2C framework has been released, directly connecting KV-caches between LLMs without text generation to reduce latency and improve accuracy.
Video Rebirth has open-sourced 'HyperFlow,' a data-free self-distillation LoRA that reduces MiniMax-H3 inference steps from 49 to 8.
The GLM-5.3-based Infra Agent built GLM-5.3-Flash inference infrastructure on a cluster of over 100,000 Chinese accelerators in less than two weeks, boosting end-to-end serving performance by approximately 3x.
Pinterest has released its 3-Tower Engagement Co-Train model, achieving a 1% CTR lift and a P99 latency of 30ms.
A study featured on the Apple Research blog proposes TS-DFM to improve the trajectory quality of discrete flow matching, reducing perplexity by 32% compared to a 1,024-step teacher model with 8-step inference and improving speed by 128x.
Researchers from the National University of Singapore and others released OpenWAM, a modular stack for comparing design variables of robot World-Action Models.
DeepSeek released the 552B MoE model V4.1-Flash, replacing V4-Pro and optimizing KV cache to 890 bytes per token.
Pinterest built a VLM serving stack using NVIDIA Dynamo and Blackwell GPUs, improving TTFT by 85x.
NVIDIA Dynamo has unveiled a serving architecture that separates vision encoders to reduce the TTFT of multimodal LLMs by up to 93%.
VDN-H3 applied hybrid attention to MiniMax H3 to increase video generation speed by up to 74x, but usage is prohibited in South Korea due to license restrictions.
vLLM released detailed benchmark results comparing the throughput and optimal settings of five speculative decoding methods, including Native MTP, EAGLE-3, and DFlash, on AMD MI300X/MI355X GPUs.
The 'Declarative Attention' technique has been released, allowing language models to declare areas requiring attention during their reasoning process, reducing computational costs by up to 52%.
A technique combining Diffusion draft generation and AR verification with the GLM-OCR model has been released, increasing decoding speed by 3.85x.
The Apple Silicon-specific inference engine Lily achieved a 35% improvement in decoding speed and high hardware utilization compared to MLX-LM.
A new llama.cpp branch featuring Metal and n-gram SSD offload optimizations has been released, improving Qwen-3.8-Flash-Next inference performance on Apple Silicon.
Qwen4Exp maintains inference speed by offloading 25% of weights to SSD using N-grams.
The QAH technique is released, recovering performance higher than the original model after structural compression and 4-bit quantization.