AI Briefing#llm-inference
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The latest AI and developer news about #llm-inference, with the original source and a short summary.
In Tesla P40 benchmarks, the saw-family KV quantization was the most stable.
PFlash cut 128K prefill time by 10x compared to llama.cpp on an RTX 3090.
Local inference speeds for Qwen3.6 and Gemma 4 on an RTX 4070S 12GB were shared.
QuietBox 2 is equipped with a Ryzen 7 9700X and 2 Blackhole cards, for a total of 128GB VRAM.
In an RTX 5090 Llama.cpp experiment, MiniMax-M2.7 processed at 22 tps even at 100k context but only generated at 3~4 tps.
On 4x 5060 Ti, vLLM recorded pp 1444.9 tok/s and tg 47.4 tok/s.
DeepInfra has been added to Hugging Face's Inference Providers.
eLLM, a CPU-based LLM inference framework, has been released, delivering higher performance than GPUs on long-context tasks.
Qwen3.6-27B ran at 85–85–100 t/s on a 24GB RTX 5090 Laptop GPU.
Reproduction code and benchmarks for Cartridges and STILL on a single GPU have been released.
Compressed the KV cache on Qwen 3.6's 1M context, reducing it from 10.7GB to 6.9GB.
Ran Qwen3.6 35B 4bit for multi-user use via vLLM Docker on 2x RTX 3090.
In an apples-to-apples FP8 comparison, the advantage between Ironwood and B200 reversed depending on the model.
On an M2 Ultra 64GB, Qwen3.5-35B-A3B Q8_K_XL achieved 63 t/s in generation.
DFlash doesn't speed up as much as expected with MoE, quantization, concurrent use, and long context.
In Gemma 4 26B a4b, multi-step think/tool loops break off midway.
In the latest oMLX commit, DFlash optimization significantly boosted tg TPS for small contexts.
Shared real-world benchmarks and issues from running Qwen3.5-27B on the Arc B70.
Running MiniMax M2.7 AWQ on 2 GX10s to complete a local agentic coding stack.
Global partners including Samsung SDS announced commercial products for FuriosaAI's RNGD.
With its latest SDK, RNGD boosted the number of users per 15kW rack to up to 2x that of the RTX Pro 6000.
Aurora is an open-source RL-based framework that continuously improves an LLM's speculative decoding performance by learning from real-time inference data.
Differential Transformer V2 has been released, improving inference efficiency by expanding query heads while keeping the number of KV heads unchanged.
A router mode has been added to the llama.cpp server that allows multiple models to be dynamically loaded and switched without restarting.