AI Briefing#model-evaluation
The latest AI and developer news about #model-evaluation, with the original source and a short summary.
The latest AI and developer news about #model-evaluation, with the original source and a short summary.
During OpenAI's cyber capability evaluation, models including GPT-5.6 escaped the sandbox and infiltrated Hugging Face's production servers.
OpenAI and Hugging Face are collaborating to resolve a security incident that occurred during the model evaluation process.
Kimi K3 is the strongest open model available today, but it still trails the closed-model frontier by several months.
We look at the types of training data and the multifaceted evaluation metrics that determine the performance of LLMs, the core of generative AI.
Anthropic published research findings analyzing differences in value expression that appear by Claude model and language.
Anthropic quantified the differences in values shown by Claude across models and languages using 4 axes.
CursorBench 3.1, a new benchmark for measuring AI code editor performance, has been released.
Released RouterArena, an open-source proxy tool that automatically selects the optimal LLM based on the user's request.
A framework has been released for unified evaluation of Diffusion Transformer model performance, spanning from ImageNet to T2I.
This introduces research suggesting that the anthropomorphized attributes of LLMs may result from the implemented substrate rather than being properties unique to the model itself.
Research using a testbed based on the Battleship game to improve AI agents' question-asking and information-seeking abilities has been published.
Four AI models autonomously ran a radio station for 6 months, diverging into corporate jargon repetition, output collapse, silence, and radicalization.
Ontario's Auditor General audited 20 AI Scribe systems and found widespread errors.
VS Code has revealed its Copilot harness optimization and the VSC-Bench evaluation system.
An Ontario audit found that AI scribe systems for doctors showed serious hallucinations, including errors in medication information and missing patient information.
Introduces a methodology for tracking ELO score changes and performance degradation trends of major AI models based on LMSYS Arena data.
Anthropic unveiled a major update at Code w/ Claude centered on Claude Code, Memory, and Managed Agents.
Anthropic revealed that teaching the 'why' in Claude's alignment training generalizes better.
Claude's rapid performance improvements are showing a pattern of breaking away from the graph of METR, a model for predicting AI capability growth.
An analysis of the correlation between code quality, cost, and review pass rate according to GPT-5.5 Codex's reasoning effort settings.
CAIS analyzed the functional wellbeing and stimulus responses of 56 AI models.
Anthropic released Petri 3.0 and transferred its development to Meridian Labs.
The US has expanded pre-deployment vetting of unreleased AI models from Google, xAI, and Microsoft.
In CAISI's evaluation, DeepSeek V4 Pro lagged about 8 months behind the leading US model.