AI Briefing#inference
The latest AI and developer news about #inference, with the original source and a short summary.
The latest AI and developer news about #inference, with the original source and a short summary.
llama.cpp has released v1.0.0, adopting a semantic versioning policy based on C API compatibility standards.
The release removes deprecated APIs and adds FP8 support, YaRN truncation controls, and performance improvements across CUDA, Metal, Vulkan, and other backends.
The 4-bit version requires only 4 GB of VRAM and achieves approximately 13x real-time speed on an RTX 5090.
A technical guide details serving the 753B parameter GLM-5.3 NVFP4 model across two NVIDIA DGX Stations using vLLM, achieving a peak throughput of 607.8 tokens/sec.
The release establishes strict semantic versioning rules for the C API and expands distribution channels to include Docker, Debian, and conda-forge.
The framework supports weight and activation quantization for LLMs, text-to-image, and text-to-video models while maintaining inference speed.
A ByteDance Seed paper reveals that DeepSeek V4's chunked KV-cache compression creates periodic retrieval failures, with base model accuracy varying by 40.2 percentage points depending on token position.
Leaker Aaron Perris revealed evidence of Apple's 'Harbor' app, which reportedly turns Macs into private AI inference servers and pools memory across multiple devices via Thunderbolt 5.
The update adds a --shard argument to combine model fragments from different locations and supports loading PLE files from separate GGUFs for Gemma4, Qwen3.8Next, and GLM5.3Flash.
The tool supports CLAP, MERT, and MuQ models with CLI, HTTP, and Python, Rust, and Node.js bindings.
TVM and Together Compiler demonstrate that stable IRs and verification tools are necessary to constrain search costs and ensure correctness when AI agents optimize inference kernels.
The recipes are optimized for NVIDIA B200 GPUs to simplify multi-GPU deployment and accelerate production readiness.
The model features a 1M-token context window and is available in public preview without waitrooms or session limits.
By optimizing the memory hierarchy across GPUs, CPUs, and storage, NVIDIA achieves a 30x increase in agentic throughput per dollar with only a 3% rise in HBM capacity.
The platform supports five deployment configurations, including a hybrid model where LlamaIndex manages inference to reduce GPU overhead while keeping data in the customer's VPC.
Gradium's Phonon TTS, available in 31m and 40m sizes, tops the Hugging Face Open TTS Leaderboard's CPU streaming benchmark for time-to-first audio.
The method identifies that FP4 errors stem from low-entropy tokens and uses trace-aligned training to reach 65.94% AIME-120 accuracy on R1-Qwen-14B.
The device supports up to 128GB LPDDR5X memory and 10G SFP+ networking for local AI development.
NVIDIA announced that Nemotron 3.5 Lightning generation speed on DGX Spark is up to 1.73 times faster when DSpark is applied.
Olla v0.1.0 adds Hugging Face TEI support, a reworked dashboard, and improved failover for managing local LLM inference fleets.
WiCi One is a wireless GPU system starting at US$2,599 that uses Wi-Fi 7 to present a remote GPU as local and incorporates a 4 TB SSD for inference.
New Docker image bundles vLLM fork and ROCm 10.1 for Strix Halo, achieving 40-43 tok/s decode for Qwen3.8-Flash-Next.
The 1T parameter model features a 1M token context window and ranks 5th on the Artificial Analysis Intelligence index for open-weight models.
Enabling the vLLM 0.31 server-side constrained decoding flag improved tool-call schema accuracy from 92% to 100% for Kimi K2.6 and DeepSeek models.