AI Briefing#inference
The latest AI and developer news about #inference, with the original source and a short summary.
The latest AI and developer news about #inference, with the original source and a short summary.
The 1.58-bit MoE model achieves 172 tok/s decode and 7000+ tok/s prefill speeds in both Stock llama.cpp and the PrismML fork.
The deployment uses 4-bit quantization and pipeline parallelism to keep model weights static on-chip while streaming activations between wafers.
The new Space verifies if popular Hugging Face models run unmodified on Trainium2, providing execution scripts and a public dataset of results.
The 10B-parameter model uses roughly 2-4x fewer reasoning tokens than leading peers while maintaining comparable accuracy.
The chip was announced at Hot Chips in late August 2026 and achieved tapeout in 9 months using AI-accelerated design processes.
The pre-trained foundation model requires no feature engineering or table flattening, using up to 10,000 historical examples for single-pass inference on databases with over 500 billion rows.
The experimental proof of concept burns the model into silicon, achieving speeds 213x faster than an RTX 5070.
NVIDIA showcases its local AI hardware lineup at GTC Berlin, while Recursive Superintelligence releases the Recursive Flint model optimized for RTX Spark.
The model supports a 1,048,576-token context and offers controllable thinking budgets to balance reasoning quality and latency.
The release supports NVIDIA SM100 and Huawei Ascend 950 hardware, featuring optimized kernels and asynchronous scheduling for offline inference.
The 1.0 release adds support for OpenAI, Anthropic, and Gemini APIs alongside local models like Gemma 4 E2B and Qwen3.8 27B.
The new vllm-torchtpu plugin allows standard PyTorch models to run on TPU v7x with no code changes, using custom JAX/Pallas kernels for attention and quantization.
The Apache 2.0 licensed tool uses custom Metal kernels and speculative decoding to outperform standard MLX engines on local coding models.
The update introduces a playground tab for testing decision models like Kev and OpenJev with probability bars.
The new open-weight model features only 2B active parameters, achieving up to 1.7x faster inference than comparable MoEs on-device while matching larger models on key agentic benchmarks.
Moondream 3.1 is now available through Prime Inference, powered by the Photon engine which delivers over 60 inferences per second on a single H100.
The new tool ranks GPU kernels by execution time with readable names and supports remote monitoring via SSH or host flags.
FLE Studio v0.1.0, an MIT-licensed local chat interface for llama.cpp and the FastLocalExperts engine, offers live inference metrics, model management, and launch profiles for macOS, Linux, and Windows.
The Apache-2.0 licensed model achieves a composite score of 72.5 on JevBench, surpassing Jev 1.13.
Achieves 0.743 accuracy by performing decisions with a single forward pass of a 27B model
Cognition benchmarks show Vera Rubin NVL72 delivers up to 4.8x higher token throughput for SWE-2 inference compared to GB200 NVL72.
Mica v0.1 4B is an Apache-2.0 licensed model that outputs calibrated probabilities for agent decisions, achieving 67.0 accuracy on held-out English sets while maintaining a small local footprint.
The Qwen3-based CLM supports all of Jev's features and achieved 87.6% performance on Terminal-Bench.
NVIDIA has released AIPerf, an LLM inference benchmarking tool that eliminates client-side bottlenecks.