AI Briefing#inference
The latest AI and developer news about #inference, with the original source and a short summary.
The latest AI and developer news about #inference, with the original source and a short summary.
Ramp cut AI costs by over 25% and reduced error rates using dynamic LLM routing based on online learning.
A price trend analysis of Claude models based on Anthropic's official data: the top-tier model has remained stable at $11→$10 over two years, Sonnet's price has stayed fixed, and Haiku has seen a 4x price increase.
MTP-based speculative decoding and support for Hy3, Tencent's 299B MoE model, have been added to llama.cpp.
This paper proposes a new optimization methodology to solve the 'Doom Loop,' a repetitive generation error in LLMs.
With Hugging Face Jobs infrastructure, you can build an OpenAI-compatible vLLM endpoint with just a single command.
Some claim that AI GPUs only last around 3 years, but real-world cases suggest they are likely to remain in use for much longer.
Apple has released MLX LM Server, which supports Continuous Batching and distributed inference.
Xiaomi has unveiled MiMo-V2.5-Pro-UltraSpeed, which generates 1,000 tokens per second by applying FP4 quantization and DFlash technology.
The UK is accelerating its AI technological independence through NVIDIA's infrastructure and the government's sovereign AI fund.
Groq, which licensed its technology to NVIDIA, has attracted $650 million in investment on the strength of its data center infrastructure.
A native HIP W4A16 kernel that optimizes performance on AMD GPUs has been added to vLLM.
FuriosaAI is partnering with Broadcom to develop a 3rd-generation inference platform optimized for the agentic AI era.
Tokenizer support for the MiniCPM5 model has been added to the llama.cpp open-source project.
A bug in llama.cpp causes the KV cache of the Qwen3.6-35B-A3B model to be dropped, significantly degrading inference speed.
This analyzes various AI hardware technologies and corporate strategies aimed at solving the memory bandwidth problem, the core bottleneck in LLM inference.
Llama.cpp has added PDL support for Blackwell GPUs, improving token generation speed by 4-10%
OWC has announced a Thunderbolt 5 device that enables local execution of large AI models on Windows and Linux PCs.
TabPFN has released a foundation model for tabular data along with extension features.
Qwen3.7-Max showed top-tier performance on coding, reasoning, and agent benchmarks.
Anthropic expands its partnership with SpaceX to increase GB200 capacity at Colossus 2 throughout June.
ByteShape released Qwen 3.6 35B GGUF NTP and MTP quantizations along with benchmarks.
NVIDIA has unveiled Nemotron-Labs-Diffusion, which supports AR, diffusion, and self-speculation.
Bug fixes related to MTP and speculative decoding improvements have been merged into llama.cpp.
Maximum activation values across 27 open LLM checkpoints varied significantly depending on architecture and training stage.