AI Briefing#llm-inference
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The latest AI and developer news about #llm-inference, with the original source and a short summary.
AirLLM has been updated to enable Kimi K3 (2.8T) to run on 4GB VRAM via sparse MoE model streaming.
A new technique leveraging parallel ANE and GPU execution on Apple Silicon improves LLM Prefill speed by up to 50%.
llama.cpp has released version v0.1.0, adopting semantic versioning instead of its previous build number scheme.
The Qwen 3.8 2.4T model achieved a throughput of 288,000 tokens per second in an NVIDIA GB300 NVL72 environment.
bitsandbytes developer Tim Dettmers has previewed a new quantization methodology, demonstrating high inference performance.
Unsloth has launched the Desktop beta, integrating local model execution, training, and agent capabilities.
Qwen has released the Qwen3.8 open model with a total size of 2.4T and 95B active parameters.
WARP has implemented local inference for the 2.78-trillion-parameter Kimi K3 on a 64GB MacBook.
llama.cpp inference speed on Apple Silicon macOS VMs was increased up to 16x using a Metal compatibility layer.
inclusionAI has released an 8B MoE model with 1.3B active parameters.
Unsloth has released the massive GGUF quantized files for Kimi K3.
ARBITRAGE predicts model superiority to reduce mathematical reasoning latency by up to 2x.
Diffusion language models enable parallel generation but showed limitations in long contexts and batch processing.
DiffusionGemma adapts Gemma 4 into a discrete diffusion model, achieving 1,500 TPS on H100.
Quantization, KV cache compression, and inference chips reduce the cost and latency of agentic AI.
InclusionAI released a 124B MoE model with 5.1B active parameters.
AirLLM has revealed a memory-saving method that runs a 70B model on a 4GB GPU.
WASTE streams activated expert weights from NVMe, enabling Kimi K3 to run larger than RAM.
BTB predicts repeated-prefix bursts and proactively replicates KV caches into GPU HBM.
Manifest deprecated its LLM router based on the experience of 7,000 cloud users.
WASTE ran the 2.78T Kimi K3 via disk streaming on a 64GB MacBook Pro.
The Neutrino-1 model family has been released, applying a ternary-style format from the training stage to maximize efficiency.
A 2-bit Qwen3.6-35B-A3B model and a dedicated runtime have been released, dramatically reducing size while maintaining FP8-level performance.
AI Gateway has launched a unified Fast Mode in beta that provides lower latency and higher throughput on a per-model basis.