AI Briefing#benchmark
The latest AI and developer news about #benchmark, with the original source and a short summary.
The latest AI and developer news about #benchmark, with the original source and a short summary.
vLLM concluded that FP8 KV-cache is the default choice, while TurboQuant's higher-bit options offer little real benefit.
Thinking Machines Lab has released a real-time interaction model for human-AI collaboration.
Hugging Face has released 'physics-intern,' a multi-agent framework for theoretical physics research.
TabPFN-3 has been released, supporting inference on up to 1 million rows on a single H100.
Microsoft Research has released a benchmark that measures AI agents' ability to act as user proxies.
Interfaze has unveiled a new model architecture and API for OCR, STT, and structured output.
Baidu's ERNIE 5.1 surpassed Gemini 3.1 Pro to rank 4th on the Search Arena leaderboard.
On the DELEGATE-52 benchmark, even frontier models showed significant drops in document fidelity during long delegated editing sessions.
Harvey has released LAB, an open-source benchmark for legal agents.
ProgramBench is a benchmark that requires reimplementing a program from only its binary and documentation.
EnterpriseRAG-Bench, which simulates 500,000 internal company documents, has been released.
SpecMD benchmarks MoE expert cache policies and proposes Least-Stale.
Hugging Face's Open ASR Leaderboard has added private datasets to prevent benchmaxxing.
AI agents that directly manipulate UIs via vision are far less efficient in both cost and reliability than those using structured APIs.
Benchmark analysis results on LLM inference performance, power efficiency, and MoE model running performance across Apple Silicon chipsets have been released.
Analysis results of LLM inference performance and power efficiency for Apple M4/M5 and A18 Pro chips have been released.
DeepSeek V4 Pro delivered GPT-5.2-level performance on agentic benchmarks while costing 17 times less.
Running Mistral Medium 3.5 128B Q3_K_M locally on 3x3090 (72GB VRAM) measured 9.67t/s.
A Strix Halo 128GB system ran Mistral Medium 128B at 1.57 tokens per second.
On AMD Strix Halo, Mistral Medium 3.5 took about 2 hours to process a 48k prompt.
The function calling harness has narrowed the frontier-local gap in backend generation.
AutoBe has released a monthly benchmark comparing backend generation by GLM, Qwen, and DeepSeek.
The latest llama.cpp build significantly improved the RTX6k Qwen3.5 benchmark.
Artificial Analysis's Long Context benchmark compared FOSS and SOTA models.