AI Briefing#inference-optimization
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
llama.cpp now supports NVIDIA FP4 quantization and Multi-Token Prediction (MTP) features.
Native MTP support for Qwen3.6 MLX models has been added to the distributed inference tool exo.
NVIDIA has released a Diffusion language model family that delivers inference speeds up to 6x faster than existing AR models.
The BeeLlama v0.2.0 update improves inference speed for Qwen 3.6 and Gemma 4 models by up to 4.9x.
The BeeLlama v0.2.0 update significantly boosts inference speed for Qwen and Gemma models through DFlash optimization.
The VRAM leak issue that occurred when using MTP models in the latest llama.cpp build has been resolved.
An update has been proposed in llama.cpp to improve MTP performance by moving the draft path to backend sampling.
Forge, a guardrail framework that improves tool-calling reliability of local LLMs, has been released.
The UI rendering API 'M1', which reduces the cost and latency of UI generation for AI agents, has launched.
DwarfStar 4 enables LLM steering experiments that manipulate the internal activations of the DeepSeek-V4-Flash model.
cyankiwi has released an AWQ 4-bit update that jointly optimizes scales and quantization ranges to reduce error.
Hugging Face presented up to a 24% speed improvement by asynchronizing continuous batching.
llama.cpp is undergoing updates to resolve conflicts between MTP and multimodal features and support their integration.
BeeLlama.cpp, a new fork that maximizes the performance of llama.cpp, has been released.
z-lab has released the DFlash drafter model and paper for Gemma 4.
Meta improved recommendation system inference efficiency by eliminating user embedding duplication with IKBO.
A method for optimizing Qwen3.6 inference performance by grafting an MTP layer onto an Unsloth quantized model is introduced.
vLLM Factory, an open-source tool that efficiently serves encoder models such as GLiNER and ColBERT via a vLLM plugin, has been released.
MTP support in llama.cpp can boost inference speed for the Qwen 3.6 27B model by 2.5x.
Google excluded Gemma 4's MTP feature from the public release, then began supporting it via an auxiliary model after the community discovered it.
Supports real-time conversation and high-quality TTS simultaneously through OpenAI API compatibility and a process isolation architecture.
FastDMS has been released, optimizing the KV-cache compression technology DMS to achieve less memory usage and faster speed than vLLM.
APEX, a mixed-precision quantization strategy for MoE models, has expanded to over 30 model types and launched a new ultra-compressed tier.
Applying Speculative Decoding to NeMo-RL boosted RL rollouts by 1.8x.