AI Briefing#llm-inference
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The latest AI and developer news about #llm-inference, with the original source and a short summary.
Nvidia proposed a system for Windows PCs with 128GB of unified memory for running local AI models.
Domino, a technique that separates modeling from drafting to maximize the efficiency of Speculative Decoding, has been unveiled.
A feature has been proposed for llama.cpp that dynamically changes the KV cache quantization level without reloading the model.
DeepLearning.AI has released a course on optimizing efficient LLM inference using vLLM.
This introduces Speculative KV coding, a technique that uses a predictor model to losslessly compress an LLM's KV cache by up to 4x.
**OCTOPUS**, which maximizes transformer KV Cache compression performance by combining octahedral parameterization with joint quantization, has been released.
A Thinking mode toggle that lets users control a model's reasoning process has been added to the llama.cpp UI.
A Pull Request has been submitted to llama.cpp for support of Stepfun AI's Step-3.7-Flash model.
A Pull Request has been proposed in llama.cpp that improves the way logits space is reserved to reduce VRAM usage.
Support for the EXAONE 4.5 model has been added to the llama.cpp open-source project.
The llama.cpp project has released a unified `llama` binary and an official website (llama.app).
This describes a technique that optimizes single-request decoding speed, a core metric for AI agents, achieving 3,000 tokens per second on standard GPUs.
A monokernel technology for LLM inference has been unveiled, achieving up to 3,300 tokens/s per request on AMD MI300X.
In llama.cpp, an optimization was made to reduce VRAM usage by using an f16 mask during Flash Attention.
A major update for AMD/ROCm environments has been released in llama.cpp version B9387.
Zai cut GLM-5.1 inference infrastructure costs by 33% and boosted throughput by 15% by introducing the ZCube architecture.
LG AI Research leveraged vLLM metrics to repurpose idle GPUs for research jobs, maximizing infrastructure efficiency.
TogetherAI has open-sourced OSCAR, a new KV cache quantization technique that improves inference efficiency.
A 2-bit quantization technique has been released that compresses KV cache memory usage by about 7x while minimizing performance degradation.
A checkpoint feature has been proposed to solve the problem of reprocessing the entire prompt whenever the context changes in llama.cpp.
hipEngine, an open-source engine that maximizes inference performance for Qwen 3.6 models on AMD RDNA3 GPU environments, has been released.
A problem where CUDA acceleration is limited when using asymmetric KV cache quantization in llama.cpp, along with solutions, is being discussed.
A study on Mix-Quant has been published, improving efficiency by applying aggressive quantization to the prefill stage while keeping the decoding stage at high precision
l9gpu, an open-source GPU observability tool that can track GPU usage by workload, has been released.