AI Briefing#local-llm
The latest AI and developer news about #local-llm, with the original source and a short summary.
The latest AI and developer news about #local-llm, with the original source and a short summary.
In a MacBook Pro M5 Max test, Gemma 4 31B outperformed in Pac-Man generation quality.
On a single RTX 3090, Qwen3.6-27B achieved a 218K context and stable tool calling.
On an RTX 5080 16GB, Qwen3.6-35B-A3B hit 30 t/s at 128K and 89 t/s for new tokens.
In an 18GB M3 Pro bench, Gemma4:e4b ranked first with 72%, while Qwen3.5:9b scored 68%.
Local inference speeds for Qwen3.6 and Gemma 4 on an RTX 4070S 12GB were shared.
Creation OS improved accuracy by filtering local LLM uncertainty through a σ-gate.
On dual RTX 5060 Ti 16GB GPUs, Qwen3.6 27B ran at about 60 tok/s.
On an M2 MacBook Pro with 32GB RAM, Qwen 3.6 27B's processing speed plummeted to 4t/s at a 52,000-token context.
Thoth, a local-first AI agent emphasizing personal data sovereignty, has been released.
Benchmark results were released for 17 local LLMs and 5 agent harnesses.
A hackathon was held where participants use OpenAI Codex to optimize Mac Metal kernels.
Torrix, a self-hosted observability tool that lets you monitor requests, responses, latency, and more for local LLMs, has been launched.
Hipfire, a new inference engine optimized across AMD GPUs, along with the mq4 quantization method, has been released.
Experimental inference support for DeepSeek v4 Flash has been released in llama.cpp.
Qwen3.6-27B-INT4 achieved 256k context and 105~108 tps on a single RTX 5090.
On an M5 Pro 64GB, Qwen 3.6 35B A3B was faster and more accurate than the 27B.
Qwen3.6-27B ran at 85–85–100 t/s on a 24GB RTX 5090 Laptop GPU.
On a 3070 8GB setup, Qwen3.6-35B ran faster with larger quants.
Max-and-Omnis released GGUF files for Nemotron-3-Super-64B-A12B-Math-REAP.
On a combination of m720q i7, 64GB RAM, and T1000 8GB, Qwen 3.6 27B recorded 20 TPS.
A setup connecting llama.cpp with VSCode to run Gemma-4-26B-A4B-IT-Q8_0 has been shared, along with its performance results.
Connecting Qwen3.6 to Claude Code on a dual 3090 setup cut the estimated 8-hour cost from $142 to under $4.
Gemma4 and Qwen 3.5/3.6 were compared on RTX 5090 using the same document-writing task.