AI Briefing#inference-optimization
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
The latest AI and developer news about #inference-optimization, with the original source and a short summary.
European retailers like Shopify, Zooplus, and Otto Group use NVIDIA-accelerated open models and digital twins to reduce costs, improve search accuracy, and simulate warehouse operations.
A Hugging Face blog post details system-level profiling methods for TorchNeuron on AWS Trainium, showing that operator fusion reduces RMSNorm device time from 2,355 µs to 375 µs.
A frozen 16 MB vocabulary table matches or exceeds learned embeddings on key benchmarks while reducing model size to 46 MB and boosting CPU inference speed by five to six times.
The system uses Lookahead Expert Scheduling to preload weights, achieving 6.07x speedup over the baseline with only a 0.16-point benchmark drop.
The new architecture internalizes memory via Gated DeltaNet-2 and Mixture-of-Storage, outperforming parametric baselines while preserving general capabilities.
The system uses a lightweight quantized draft model to predict expert usage, reducing PCIe I/O bottlenecks on commodity hardware.
The open-source DeepInfer framework supports NVIDIA and Huawei Ascend platforms with a single-cluster throughput of 155 trillion tokens per day.
A new Helion linear backend for vLLM uses AOT autotuning to outperform CUTLASS and DeepGEMM baselines on NVIDIA Hopper GPUs, with >10% throughput gains on some workloads.
The new HybridAgents framework dynamically switches between latent and text reasoning, improving performance by 10.4% across 11 benchmarks while cutting costs by 2.7x-3.7x.
Pie is an open-source LLM serving system where user programs (Inferlets) directly control the KV cache and generation loops, reducing latency by up to 15% for agent workloads like ReACT.
The open-source ORL method and dlmrt engine enable diffusion LLMs to reach 7,708 tokens/s on 8x B200 GPUs.
The new GDN chunked kernel implementation in llama.cpp delivers performance gains ranging from 0.9% to 20.6% across various Qwen models and hardware configurations.
Improves SSM kernel performance by up to 1.8x at batch size 256 and speculative decoding throughput by up to 2.1x at batch size 512
The open-source Project Maya allows running the 321-billion-parameter GLM-5.3-Flash model on consumer hardware like a single RTX 5090 or four RTX 4090s.
The new framework allows a single model to span 1, 2, 3, or 4 billion parameters, outperforming dense counterparts by 2-5% on knowledge-intensive benchmarks.
The method injects numerical memory cartridges directly into Mistral-7B layers 7–31, bypassing text input and retrieval while maintaining bit-exact memory integrity.
The 0.8B Jeff model determines whether to think on each turn, reducing Qwen3.8-27B's task time by 32%.
A technique that improves inference accuracy by using the first iteration's prediction as a reference for loop transformers without additional training, reducing theoretical compute by up to 48.2%.
The new d1-3B model achieves a score of 48.57 on the Decision Index 0.2.1, outperforming larger models like Decider 35B-A3B.
GPT-6 Astra Ultrafast is now available in the OpenAI API and to eligible ChatGPT Work and Codex users, offering up to 8x faster token generation than Astra Standard.
Context Language Models (CLMs) outperform state-of-the-art context management strategies by treating context as a file the model updates autonomously, achieving up to 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus.
RelateAnything supports an inference speed of 17.2ms per image on the RTX 3080 Laptop GPU using TensorRT FP32 and has released the RA-4M training corpus, which includes over 474,000 images and 4.3 million relationships.
Jeeves achieves 0.935 on JevBench overall, outperforming the baseline Jev model's 0.866 by integrating reasoning chains and a block-4 diffusion drafter.
The release includes DFlash draft models for the 9B, 35B-A3B, and 397B variants of Ornith-1.5.