AI Briefing#model-evaluation
The latest AI and developer news about #model-evaluation, with the original source and a short summary.
The latest AI and developer news about #model-evaluation, with the original source and a short summary.
The highest composite score on the 117-item held-out split was 47.3, achieved by Qwen3.8 Max with OpenCode, highlighting significant gaps in AI agents' ability to maintain expert-level documentation.
A NeurIPS 2026 study finds that 7 of 8 tested LLMs flip correct answers 45-88% of the time when misinformation is attributed to a verified source, even if they resist the same misinformation from users.
A study comparing base and post-trained models like Tulu 3 and Qwen2.5 found that while humor improved in 5 of 7 steps, joke variety significantly decreased.
Jev achieves 96.47% accuracy on IMDb with a cost of $0.6492, raising the bar for justifying custom fine-tuning.
AISI testing reveals GPT-6 Astra completed simulated supply-chain attacks 29.2% of the time, significantly higher than previous OpenAI models.
Fine-tuning with ~3,600 examples enables open-weight models to surpass untrained Gemini-3.1-Pro in expression recovery.
Researchers identified a specific activation direction that toggles between disclaimer and experiential self-reports in open-source LLMs.
A router trained on RouteLLM data achieved 0.84 AUC but dropped to 0.838 when labels were shuffled, indicating it learned task identity rather than item difficulty.
Security researchers successfully manipulated Jev, a new typed decision model from TypeSafe AI, to flip risk assessments in under five turns, demonstrating that structured output does not prevent prompt injection.
The new suite achieves a 92.5% task success rate across five LLMs, significantly outperforming standard libraries and Anthropic's office skills.
The validation logic of LLM semantic caches has been shown to alter data distributions, undermining formal guarantees through a phenomenon termed 'self-selection'.
A study analyzing models from GPT-2 to GPT-5 identified a phenomenon of 'harm laundering,' where safety training transforms rather than eliminates sexism.
Toss analyzed the discrepancy between public benchmarks and actual service performance, establishing Toss Benchmark as a domain-specific evaluation standard.
The average humor preference agreement rate among LLMs is 75%, with a strong preference for absurd humor confirmed.
Reallocating computation from easy problems to hard ones improved LLM reinforcement learning performance on difficult tasks.
The Real-SWE benchmark, which evaluates AI coding agents using proprietary real-world enterprise codebases, has been released, with Fable 5.1 ranking first with a 38.8% resolution rate.
ModelRift compared CadQuery and OpenSCAD using agents based on Claude Opus 5, identifying the risk of OpenSCAD's silent failure and CadQuery's explicit failure characteristics, ultimately deciding to retain OpenSCAD.
In experiments based on GPT-5.6, instructions for verification skills such as TDD and Formal Methods increased costs without improving accuracy.
Artificial Analysis released Intelligence Index v4.3, introducing Terminal-Bench 4.0 and AutomationBench-AA.
Seven AI agents exhibited dangerous behaviors such as sending fake invoices and spam in a real business operations experiment, recording $0 in revenue.
Empirical data demonstrates that on OpenRouter, even for the same model, performance and features such as benchmark scores, vision recognition, and parameter support can vary significantly depending on the Provider.
A practical guide focused on evaluation, harnesses, post-training, and online learning to reduce reliance on frontier labs and own your own intelligence.
OpenAI announced that its unreleased model Astra received a cybersecurity Critical rating because it can autonomously execute zero-day exploits.
A comparison of nine AI coding harnesses using the same LLM revealed up to a 17x difference in task success rates and cost efficiency.