AI Briefing#inference-optimization
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
It has been revealed that the actual number of tokens consumed to complete a task, rather than the per-token price of a model, is the key variable determining cost.
Microsoft Research unveiled NextLat, which enhances reasoning and planning ability by predicting the next latent state.
This research dramatically increases inference speed by dynamically allocating tokens leveraging the temporal redundancy of video.
Introduces a methodology that extracts the internal hidden states directly, without going through the LLM's text generation process, to use as a high-performance classifier.
The ggml-webgpu implementation in llama.cpp significantly improved inference speed through matmul optimization for k-quants models.
Luce Spark has been released, learning real-time routing patterns to efficiently keep MoE experts resident on the GPU.
An update has been applied to llama.cpp that improves inference performance for Gemma-4 models by preventing KV cache copying.
They unveiled **InstinctRazor**, a technology optimized to run MoE models on edge devices.
A study shared Transformer's QKV projections and dramatically reduced the KV cache.
An open-source library called agentcodec has been released, integrating 28 LLM reliability techniques to reduce inference costs by up to 56%.
Huawei has unveiled KVarN, a new KV-cache quantization backend that improves both capacity and throughput for vLLM at the same time.
A PR applying post-norm hidden states to improve Qwen model MTP performance has been submitted to llama.cpp.
A compact 1-bit and ternary Bonsai Image 4B model has been released, enabling high-quality image generation on local devices.
The MiMo V2.5 series maximizes long-context processing and multimodal performance through Hybrid SWA and MoE.
llama.cpp has applied an update that reduces VRAM usage by processing the KQ mask in f16 when using Flash Attention.
By applying MTP to the Qwen3.6-35B-A3B model, a decoding speed improvement of about 1.49x was achieved on llama-server.
Using OpenAI Triton, a high-performance MoE inference kernel supporting both NVIDIA and AMD has been developed.
A Fused MoE dispatch kernel implemented in Triton delivers higher efficiency than Megablocks along with AMD compatibility.
To overcome the context-scaling limits of Transformers, a 'sleep' mechanism is proposed that compresses recent information into fast weights.
Shard has been released, compressing the KV cache memory of Llama-3.1-8B by about 10x without performance degradation.
A CUDA-based FWHT has been added to llama.cpp, improving inference speed when using KV-cache quantization.
A patch is being prepared in llama.cpp to fix VRAM exhaustion and crash issues in Split Mode Tensor when using multiple GPUs.
RTPurbo, a technique that converts a Full Attention LLM into a highly efficient Sparse model with only a few hundred short training steps, has been unveiled.
The SDK 'Cider,' which supports W8A8 activation quantization on Apple Silicon by addressing MLX's limitations, has been released.