AI Briefing#llm-inference
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The latest AI and developer news about #llm-inference, with the original source and a short summary.
ik_llama.cpp has officially merged MTP support for Qwen3.8-Flash-Next, resulting in approximately 2x faster inference speeds on RTX 5090 environments.
A technique combining Diffusion draft generation and AR verification with the GLM-OCR model has been released, increasing decoding speed by 3.85x.
A comparison of nine AI coding harnesses using the same LLM revealed up to a 17x difference in task success rates and cost efficiency.
WebLLM, which runs LLMs in the browser via WebGPU without a server, has been released with OpenAI API compatibility.
NVIDIA has released Model Optimizer, which integrates six optimization techniques including quantization and pruning to support checkpoint generation for inference engines such as vLLM and TensorRT-LLM.
Introduces techniques for managing the latency and throughput tradeoffs in LLM inference and expanding the efficient frontier through quantization, speculative decoding, and more.
Alexia Jolicoeur-Martineau's research team published a paper combining sliding window attention and attention sinks to replace linear attention.
vLLM v0.28.0 adds large-scale performance optimizations and inference capabilities for the Kimi-K3 and DeepSeek V4 models.
vLLM v0.28.0 introduces major performance optimizations and new inference features for Kimi-K3 and DeepSeek V4.
vLLM resolved the non-deterministic output issue for Qwen3.8-Flash-Next at temperature 0 by fixing the sparse-attention kernel.
IST-DASLab released a Qwen3.8-27B GGUF model with significantly improved performance over previous versions by applying GSQ and RCO technologies.
Numerous cases have been discovered where GGUF filenames differ from the actual saved bit counts due to quantization logic issues in llama.cpp.
Tontaube has released TontaubeV1, a 2.9B parameter open-source TTS model specialized for long-form generation and low-latency inference.
Ollama's KV cache quantization settings are applied globally, which can cause performance degradation for specific models.
Samsung Electronics unveiled an LPDDR5X package featuring Processing-in-Memory (PIM) technology at Hot Chips 2026, improving Llama 3.1 inference speed by 3x.
Ollama's default settings for context length, parallel requests, and resident model limits significantly increase memory usage, causing GPU offloading failures.
A checkpoint quantizing Qwen3.8-27B to NVFP4 using the QUASAR QAT algorithm has been released, maintaining BF16-level performance.
LayerStoRm, an open-source engine that runs large MoE LLMs on consumer GPUs using PCIe streaming, has been released.
HBF is discussed as a potential alternative to address DRAM capacity issues in LLM inference.
The addition of ConvRot quantization to llama-cpp-turboquant brings the quality of Q6 quantized models close to Q8 levels.
Version 0.2.0 of the local LLM inference tool Llama.cpp has been officially released.
PagedAttention applies virtual memory concepts to significantly reduce KV cache memory waste during LLM inference.
Applying DSpark speculative decoding to LFM2.5 models improves GPU and on-device inference speed by up to 3.2x.
Ollama 0.32.15 introduces a metadata cache to reduce per-request processing overhead.