AI Briefing#inference-optimization
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
In hybrid model serving, it increased prefill TPM by up to 3.14x and reduced p90 TTFT by up to 88%.
A 164-line patch adds a missing Arm NEON kernel for llama.cpp's tiled K-quant matmul, achieving up to 6.5x faster prompt processing on Neoverse N1 without affecting generation speed.
The open-source plugin uses coding agents to find the fastest inference path for specific hardware configurations.
The model achieves 80+ tokens/s decoding and 29GB peak memory usage on compatible laptops, outperforming several cloud models on coding and reasoning benchmarks.
Open-source KQuant packs for Qwen models on Hugging Face achieve up to 3.7x lower KL divergence than public 4-bit baselines, with 6-15% faster inference on M5 Pro for 4B, 9B, and 27B models.
The MIT-licensed tool analyzes KV-cache events from vLLM and SGLang to pinpoint why requests miss the prompt cache, achieving 32/32 agreement with Anthropic's diagnostics.
NVIDIA VFlash reduces latency by up to 5.1x for robot and autonomous driving trajectory prediction.
The method uses second-order interactions to maintain performance during aggressive expert removal, outperforming first-order baselines on Qwen3.5 and GLM-4.5 models.
A linear program optimizes the weight edit found by GPT-6 Astra on Qwen3.5-0.8B, achieving comparable math performance with 8x fewer changed weights but slightly lower instruction-following scores.
Removing unnecessary draft-token reservations allows larger prefill chunks and more efficient CUDA graph execution, raising output throughput from 1,373 to 3,390 tok/s.
The method achieves a 27.8% increase in mean acceptance length over DFlash on Qwen3-8B at temperature zero.
The open-source engine reduces batch-8 decode latency to 24.2 ms by fusing layers into single NKI kernels to minimize HBM traffic and compiler overhead.
The team combined mixed-precision quantization, DFlash2 speculative decoding, and custom CUDA kernels to reduce latency and improve throughput on a single GPU.
The open-source Cosmos Framework supports training and simulation for autonomous vehicles, robotics, and surgical workflows, while NIM enables sub-second live stream alerts.
Recent commits resolve Hadamard sign tensor bottlenecks, allowing full GPU offload and fast CPU prompt ingestion for PQ2_0 models.
The combined approach matches the quality of a 70-billion-parameter teacher model while running 10 to 20 times faster and at less than one-third the cost.
The open-source plugin targets gfx1151 hardware to optimize MoE GEMMs and attention layers, reducing 256k prompt prefill time from over two hours to four minutes.
The system uses high-fidelity production replay and parallel sub-agents to diagnose bottlenecks and test speculative decoding, quantization, and scheduling changes.
The optimization achieves 5.7% to 6.1% gains on RTX 5090 for Qwen3 and Qwen3.6 models.
The new adapters reduce image generation from 40 to 6 steps and remove the separate prompt-upsampling phase, saving 41 to 80 seconds per image.
The system prioritizes full-attention KV cache admission to reduce SSD wear without sacrificing cache hit rates.
VHA uses linear operations to halve physical query heads, achieving 4x KV cache reduction and +0.70% quality improvement over MLA on a 30B MoE model.
PACE uses periodic rounding and channel-aligned scaling to maintain accuracy at four bits, matching 8-bit performance on models up to 35B parameters.
Combining block-level Flow generation with AR verification increased throughput by 2.54x.