AI Briefing#local-llm
The latest AI and developer news about #local-llm, with the original source and a short summary.
The latest AI and developer news about #local-llm, with the original source and a short summary.
Unsloth updated Qwen3.8-27B GGUFs with Dynamic V3.1 for 5-10% accuracy gains and introduced new 1-bit quantization types for the 2.4T parameter model.
The update reduces system prompts from ~8k to ~1k characters and uses plain text markers instead of JSON to ensure reliable performance on smaller local models like Gemma 12B.
Precision Numerics releases Boole-32B-A2.5B-Preview, a 1.8-bit MoE model that is 9x smaller than BF16 and 2x faster than INT4.
The free Windows tool uses a shared queue to stop CUDA OOM crashes when multiple apps access the GPU simultaneously.
The open-source plugin uses coding agents to find the fastest inference path for specific hardware configurations.
The release includes Mini, Micro, and Atom variants, with the 34.31M parameter Mini model supporting reference voice tuning and 48 kHz enhancement on CPU and CUDA.
New 2-bit to 6-bit quantizations achieve higher accuracy on agentic and coding benchmarks using 256K and 128K context windows.
Reducing the 30B model size to 10GB and recovering 89.4% of benchmark scores makes it possible to run on a 24GB MacBook.
The tool uses a custom VMM and WebAssembly components to enforce strict network and filesystem policies, supporting Linux, macOS, and Windows.
The open-source framework separates cognitive planning from execution using pre-vetted software bundles to improve reliability and reduce token usage.
New quantization files improve accuracy by 10-15% on agentic benchmarks and introduce a new engram switching process for multiple precisions.
uzu, a Rust inference engine for Apple Silicon, recorded a speed of 142.3 tok/s on the M5 Max.
Prospero is a new open-source coding harness written in C++ that uses approximately two orders of magnitude fewer resources than web-based alternatives while supporting headless deployment and desktop control.
The free Python package ports models like Cloudflare's Clef and Perplexity's pplx-decider to MLX, achieving sub-20ms latency on small encoders on an M5 Pro and 96.4% accuracy on TREC after fine-tuning.
NVIDIA has released Nemotron 3.5 Bolt E4B, a small model for interactive agents that operates on 4GB VRAM.
The 816B parameter model scores 39, slightly below the 180B Qwen3.8-Flash-Next which achieves a score of 40.
The 1.0 release adds support for OpenAI, Anthropic, and Gemini APIs alongside local models like Gemma 4 E2B and Qwen3.8 27B.
The Apache 2.0 licensed tool uses custom Metal kernels and speculative decoding to outperform standard MLX engines on local coding models.
The new version eliminates repetition loops found in the preview, achieves 0/50 loops on SWE-bench tasks, and introduces an open-source engine enabling 262K context within a 16 GB memory budget.
LLMxRay is a new open-source tool that provides detailed observability for local Ollama models, including prompt caching analysis, token-level streaming visualization, and protocol comparison.
Generation speed improved 4.6x compared to llama.cpp, enabling 125B MoE model execution with 12GB VRAM
The 2B parameter model achieves up to 500 tokens per second locally in the browser on an M4 Max using custom WebGPU kernels.
The release includes quantized versions of the 32B, 7B, 3.7B, and 0.9B parameter models.
Runs on laptops with 2.45B active parameters, outperforming Qwen3.6 in tool-calling performance