AI agent skills fail to deliver even half of benchmark performance in the real world

A research team from UC Santa Barbara, MIT CSAIL, and MIT-IBM Watson AI Lab re-evaluated AI agents' skill utilization capability under conditions closer to reality.

The existing SKILLSBENCH used an idealized setup where agents were given the right skill in advance, which could inflate performance beyond real-world levels. To address this, the research team built a more realistic 6-stage scenario based on 34,198 skills collected from open source.

• Providing the skill directly
• Having the agent choose the skill itself
• Mixing in irrelevant skills
• Retrieving from a large-scale skill pool
• Cases with no customized skill available at all

Testing the latest models—Claude Opus 4.6, Kimi K2.5, and Qwen3.5-397B—revealed a large gap between idealized and realistic conditions.

• Claude Opus 4.6
  • Skill provided directly: 55.4% pass rate
  • Realistic condition (large-scale retrieval + selection): 40.1%
  • No customized skill: 38.4%
  • Only a 3%p difference from the no-skill baseline of 35.4%

In weaker models, adding skills sometimes had the opposite effect, causing performance to fall below the baseline.

The main causes of this performance degradation are as follows.

• Skill selection failure: Claude's rate of properly retrieving the customized skill was only around 49%
• Limits in retrieval accuracy: Recall@5 for relevant skills among the top 5 reached at most 65.5%
• Lack of adaptation ability: Agents failed to sufficiently adapt similar skills to fit the given situation

There were also attempts at improvement. Refining and generating skills on a per-task basis raised Claude's performance from 40.1% to 48.2%, but this only worked when the initial retrieval results had some degree of relevance.

In conclusion, current AI agent skills are prone to showing inflated performance on benchmarks, while their effectiveness is limited in real-world environments. In particular, for weaker models, skills may hinder rather than help performance, underscoring the need for better retrieval techniques, offline skill refinement strategies, and skill ecosystem designs tailored to model capabilities.