AI Briefing#inference
The latest AI and developer news about #inference, with the original source and a short summary.
The latest AI and developer news about #inference, with the original source and a short summary.
The open-source Kev series of decision models based on Qwen3.5 has been released, offering TypeSafe API compatibility and detailed performance metrics.
The open-source library tool-prune has been released, reducing prompt tokens by 92% through pre-filtering of tool schemas.
A study reveals that the Intelligence per Watt (IPW) efficiency of local LLMs has improved 5.3x over three years, demonstrating the potential to offload central infrastructure demand for a significant subset of queries.
An open-weight model, PCCG-Qwen3-4B, was released to control generation halting using discontinuity experiments across 11 LLMs and the Jacobian lens.
HP has started taking orders for the ZGX Fury, an enterprise AI inference workstation featuring the NVIDIA GB300 Grace Blackwell Ultra superchip and 748GB of unified memory.
Hugging Face has released 207 optimized WebGPU kernels and a loader library to enhance in-browser AI inference performance.
Artificial Analysis, in collaboration with Liquid AI, released a benchmark measuring the intelligence and inference speed of small LLMs under 8GB on the iPhone 17 Pro.
The 120-million-parameter open-source TTS model Sopro V2 delivers SOTA-level voice cloning performance and enables fast inference even on CPUs.
ik_llama.cpp has added support for DSpark models and Vulkan quantization features.
Applied Compute has released AC2, an AI model training and deployment platform previously used by its internal research team.
kimodo.cpp, a C++/GGML implementation of the NVIDIA Kimodo motion generation model without PyTorch, has been released, simplifying local inference environments.
FreeToken proposes a system for efficiently serving frontier-grade MoE models on hardware comparable to gaming PCs.
NVIDIA Vera Rubin NVL72 achieves 30x higher throughput and 35x lower token costs compared to previous generations in AI agent workloads.
Shortening LLM output reduces costs by up to 3x, whereas shortening input prompts actually increases costs.
Optimized Qwen3-TTS to achieve sub-50ms latency and 10 RPS throughput on a single H100.
OpenRouter has released 'Ox Alpha', a reasoning model developed by an anonymous third-party developer, as a free API.
Escha Labs released a 2-bit quantized model of Qwen3.8-27B, demonstrating performance equivalent to FP8.
SyzygyResearch has released the 7GB Mach-1-Additive-35B model and a dedicated llama.cpp fork.
Cerebras launched the CS-4, featuring three wafers and delivering up to 30x faster inference performance compared to GPUs.
LMSys has released optimization strategies to maximize DeepSeek-V4-Pro serving performance on H20 GPUs.
Inco AI released DFlash 2, a parallel inference acceleration technology that improves throughput by 16–25%.
AI inference chip developer Etched raised $700 million, more than doubling its valuation in just one month.
vLLM's non-standard KV cache structure is reported to hinder inference interoperability across heterogeneous hardware such as Cerebras.
Qwen3.6-35B is faster in speed but has longer inference times, resulting in slower final response times.