AI Briefing#llm-inference
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The open-source Kubernetes-native stack supports vLLM and SGLang across heterogeneous hardware, claiming up to 3x higher output throughput and 2x faster TTFT.
The new Hopper kernel and distributed extension improve throughput by 1.80x to 3.71x over Megatron-Core baselines at context-parallel degrees 4-32.
The method uses soft memory tokens to help frozen LLMs approximate full-history continuations, reducing tool errors on agent benchmarks.
The framework reuses target model hidden states to avoid maintaining historical KV caches, outperforming DSpark by up to 70%.
The device features the Ryzen AI Max+ 395 CPU and 128GB unified memory, offering a $4k alternative to Nvidia's $7k DGX Spark.
The update introduces native FP8 support and optimizes Metal kernels, reducing driver memory peaks from 270.135 GiB to 8.108 GiB for video decoding.
While performance is similar under default settings, the optimal engine varies depending on output length and prefix reuse.
The collaboration delivers up to 54% cost reduction for beam search workloads on Qwen3 1.7B models.
Generation speed improved 4.6x compared to llama.cpp, enabling 125B MoE model execution with 12GB VRAM
Cache misses caused by harness modifications like timestamp changes or tool list updates increase energy consumption by up to 2.5x compared to fully cached requests.
New NVFP4 and GGUF checkpoints use quantization-aware distillation to achieve lower KL divergence than reference 4-bit models from NVIDIA, Google, and community quantizers.
The open-source runtime achieved a 77.73% reduction in total provider tokens but saw a drop in verified task completions from 38 to 34.
The experimental C-based Janas-LLM achieves 24 tokens/s generation for the 48.4GB Qwen3-Next-80B-A3B on a 32GB RAM laptop, significantly outperforming llama.cpp.
Architecture and performance data released for serving 79 model rows, from text to robot actions, within a single runtime
A study of over 700 firms shows AI agents boost code generation by 30% but increase code review time by 49%, resulting in no significant change in overall software output.
The new DLoop method reduces target-model forward passes by performing multiple confident drafting stages before a single verification.
Speculative decoding accelerates LLM inference by increasing total FLOPs to utilize idle compute resources, shifting the workload from memory-bound to compute-bound at low batch sizes.
The openTPU project presents an open-source AI accelerator developed by AI agents, capable of running models like LFM2.5-230M and Qwen3.5 on a Xilinx Kintex-7 FPGA with bit-exact simulation results.
The new DTOC framework allows agents to dynamically hide or unhide tool outputs, improving solve rates and reducing token costs on DeepSWE tasks.
Debian announced a self-service LLM inference portal on September 23, 2026, with recent updates adding the scaleway/qwen3.8-27b model and improved documentation.
The new module supports PCIe cascading and runs on embedded Linux and Android systems.
A security benchmark found that default Ray and vLLM configurations on Kubernetes expose critical control-plane ports to cross-namespace attacks, which standard scanners fail to detect.
CoWA and MALA achieve up to 8.6x attention-operator speedups at 128K tokens while maintaining comparable model capabilities to FullAttn.
The 2B parameter Jeff model achieves an 83.1% score on a five-benchmark panel, slightly exceeding the published 83.0% score of the larger Jev model.