AI Briefing#local-inference
The latest AI and developer news about #local-inference, with the original source and a short summary.
The latest AI and developer news about #local-inference, with the original source and a short summary.
The model achieves 80+ tokens/s decoding and 29GB peak memory usage on compatible laptops, outperforming several cloud models on coding and reasoning benchmarks.
MACaQ quantizes a 122B-parameter model in 64 minutes using 32.5 GB of device memory, supporting text, image, video, and audio models across llama.cpp, vLLM, Diffusers, and PyTorch.
The model operates as a true speech-to-speech system with inference speeds faster than real-time and runs locally on MacBook Pro M4 Max or higher.
The update makes all of Cohere's latest model architectures compatible with llama.cpp and allows fine-tuning of Cohere Transcribe in Unsloth.
The new OpenUI Lang DSL reduces token usage by ~67% compared to JSON-based approaches and supports streaming component rendering.
Hugging Face Transformers now directly supports the GGUF format from llama.cpp, simplifying local inference on Apple Silicon.
FreeToken has released an edge-native serving engine that enables interactive-speed execution of frontier MoE models on consumer hardware.
Bad Theory Labs has released BTL-3, a 27B open-weight agentic coding model that runs from a single 8.39GB file, under the Apache-2.0 license.
Neuphonic has open-sourced NeuTTS-2E, a 125M-parameter on-device TTS model capable of controlling 7 emotions.
InstinctRazor technology has been unveiled, running a 122B MoE model even in an 8GB VRAM environment by placing experts on the CPU.
The next-generation computer use agent series Holo3.1, delivering strong performance in local and mobile environments, has been released.
Ultra-lightweight 1-bit and Ternary Bonsai Image 4B models have been released, enabling high-quality image generation on local devices.
DwarfStar 4, a Metal-based local inference engine, has been released supporting DeepSeek V4 Flash.
ds4, a dedicated local inference engine for running DeepSeek V4 Flash on Metal, has been released.
On MBP M3 Max benchmarks, the gap between three Qwen3.6 variants and gpt-oss:120b was revealed.
Comparing 10 image models on an M1 Max 64GB revealed differences in non-English cultural accuracy.
The RTX 6000 Pro was significantly faster than the 4080 Super at Qwen3.6 27B inference.
DFlash for Qwen3.5-35B-A3B was confirmed running on an RTX 2080 SUPER 8GB.
AMD has announced 'Agent Computers' and related hardware that support continuous local execution of AI agents.
llama.cpp now supports GGUF conversion for NVIDIA Nemotron Nano 3 Omni.
Canonical is rolling out Ubuntu AI features in stages, centered on local inference and Snap security.
Qwen3.6-27B was run up to a 100K context on an A5000 16GB using IQ4_XS.