AI Briefing#inference-optimization
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
This piece examines the latest techniques for optimizing inference efficiency and resource usage in LLMs, centered on research presented at ACL 2024.
This piece analyzes the latest efficiency research aimed at solving the computational cost and inference speed issues that come with the growing scale of LLMs.
It's a caching proxy tool that saves GPU resources by preventing redundant computation for embedding models like Ollama.
llama.cpp has rolled out an update adding Flash Attention support on Intel SYCL and improving Qwen model inference speed.
Meta has unveiled IKBO technology that eliminates the redundant data duplication overhead occurring during recommendation model inference.
Unsloth has released NVFP4 quantized models for Qwen3.6 that boost inference speed by up to 2.5x on Blackwell GPUs.
The patch fixes unnecessary regex recompilation occurring when using sm-tensor in llama.cpp, improving decode performance.
Flex-Forcing flexibly switches how video diffusion models generate by chunking temporal and denoising steps.
How to maximize performance by optimizing GLM-5.2 with NVFP4 and TP=4 settings in an NVIDIA B200 environment
NVIDIA has released Nemotron-3 Puzzle-75B, an MoE-based model that dramatically improves inference efficiency.
This article introduces engineering optimization techniques that leverage the Hybrid SWA architecture of the MiMo-V2.5 model to reduce KVCache storage space and compute to about 1/7 the level.
This introduces a technique that uses a small LLM to remove 68% of RAG context while maintaining 96% recall.
An ultra-lightweight 51M-parameter routing model that selects the appropriate model based on user requests has been released.
A technique is proposed that maximizes the efficiency of Diffusion-based Speculative Decoding by predicting the optimal block size according to the input data.
ELDR routing, which accounts for expert locality in MoE models, cuts decode latency by up to 13.9%.
Manticore Search has introduced an ONNX Runtime backend, boosting embedding inference speed by an average of 14x compared to before.
Cerebrium announced that it reduced AI model cold start times by over 80% using CPU and GPU memory snapshotting technology in a gVisor-based runtime.
openPangu-2.0-Flash, a 92B MoE model based on Ascend, has been released.
vLLM Semantic Router has unveiled a micro-agent runtime that makes multiple models collaborate at the model API layer.
JetBrains has unveiled the Mellum2 model, a 12B-2.5A scale model optimized for fast inference.
DeepSeek released DeepSpec, a tool for training and evaluating models for Speculative Decoding.
A router that distributes queries between local and cloud LLMs by analyzing prompt structure without making model calls.
DeepSeek has open-sourced an optimization technique that maximizes inference efficiency, boosting generation speed by up to 85%.
NVIDIA has unveiled Nemotron-TwoTower-30B-A3B, a diffusion-based language model that boosts generation speed by 2.42x.