Study Evaluates How AI Agents Perform Tasks

Tessl researchers built a framework that evaluates AI agent performance by measuring not just whether a task is solved (Success rate), but how accurately the agent follows the user's intended workflow, coding conventions, and preferences.

Based on approximately 500 real-world skills and 1,000 generated coding tasks, they evaluated 19 agent and model configurations, with the following results:

• Claude Code and Anthropic models: Showed the strongest overall performance, but the key differentiator was not simply completing tasks, but adherence to workflows based on the configured skill.
• Correlation with model performance: When given an appropriate skill, lower-cost models approached flagship model performance in terms of instruction following.

This study suggests that when adopting AI agents in production environments, the ability to follow user-defined rules may be a more important metric than simple benchmark scores.