AI Briefing#model-serving
The latest AI and developer news about #model-serving, with the original source and a short summary.
The latest AI and developer news about #model-serving, with the original source and a short summary.
InternLM has released Intern S2 Mobius, a derivative model of Qwen3.5-35B.
MiniMax has released the weights for its 33B H3 model, which jointly generates video and stereo audio, but Korea is excluded from the license.
AirLLM has revealed a memory-saving method that runs a 70B model on a 4GB GPU.
Toss Bank standardized collaboration criteria between DS and MLE into an interface.
WASTE streams activated expert weights from NVMe, enabling Kimi K3 to run larger than RAM.
The llama.cpp team has unveiled a Mac app and simplified model serving feature.
LocalAI has revealed the performance and cost of its own C/C++ inference engines, which reduce Python dependencies.
WASTE ran the 2.78T Kimi K3 via disk streaming on a 64GB MacBook Pro.
NVIDIA ModelExpress dramatically reduces model weight loading times through direct GPU-to-GPU data transfer.
Stripe is considering acquiring OpenRouter, an AI model routing startup, hinting at possible expansion into AI infrastructure.
NVIDIA has unveiled ModelExpress, which dramatically speeds up model loading by leveraging GPU-to-GPU RDMA.
The Petals project uses a BitTorrent-style approach to let consumer GPUs run huge LLMs like Llama 3.1 405B in a distributed manner.
pi version 0.81.0 has added integrated support for llama.cpp (llama-server router).
ExLlamaV3 v1.0.0 has been released, offering improved inference speed and support for a variety of models.
Weave Router has unveiled a single-endpoint proxy that routes prompts to the optimal model within 50ms, cutting costs by 40-70%.
vLLM's Transformers modeling backend has been optimized to deliver throughput on par with or faster than existing custom implementations.
Cerebrium announced that it reduced AI model cold start times by over 80% using CPU and GPU memory snapshotting technology in a gVisor-based runtime.
vLLM Semantic Router has unveiled a micro-agent runtime that makes multiple models collaborate at the model API layer.
DeepLearning.AI has released a course on optimizing efficient LLM inference using vLLM.
This describes a technique that optimizes single-request decoding speed, a core metric for AI agents, achieving 3,000 tokens per second on standard GPUs.
Forge is an open-source guardrail layer that improves the tool-calling stability of local LLMs.
LM Studio added support for MTP speculative decoding in 0.4.14 Beta.
Bug fixes related to MTP and speculative decoding improvements have been merged into llama.cpp.
The Claude Code performance degradation was not due to the model but to changes in execution settings.