Agent Teams Increase Costs 1.8–5.1x with Limited Score Gains on Vibe Code Bench

A case study on Vibe Code Bench compared single-agent versus multi-agent team performance for GPT-6 Sol and Claude Opus 5.5 across 50 full-stack web app tasks. The study found that agent teams increased costs by 1.8 to 5.1 times compared to single agents, with statistically significant score improvements observed only for GPT-6 Sol in medium reasoning settings.

Cost and Performance Metrics

The experiment measured average cost per app and score percentage, revealing distinct trade-offs between models and reasoning efforts:

• GPT-6 Sol (medium): Single agent cost $1.22 with a score of 77.6%, while the team cost $3.07 and scored 84.9%. The team achieved a +7.3 point improvement, which was statistically significant (p=0.005).
• GPT-6 Sol (max): Single agent cost $4.82 with a score of 89.0%, while the team cost $8.54 and scored 90.4%. The +1.4 point gain was not statistically significant.
• Claude Opus 5.5 (medium): Single agent cost $4.08 with a score of 91.5%, while the team cost $9.10 and scored 91.2%. The team showed a -0.3 point change, which was not significant.
• Claude Opus 5.5 (max): Single agent cost $23.77 with a score of 89.8%, while the team cost $122.00 and scored 93.2%. The +3.4 point gain was not statistically significant.

Runtime and Cost Drivers

The primary driver of increased team costs was cached input tokens resulting from context re-transmission to each subagent. For Opus max teams, the median usage was 224 million cached tokens per app, compared to 55 million for single agents.

Runtime behavior varied by model:

• GPT-6 Sol: Teams were faster due to parallel processing. In medium settings, teams took 11.9 minutes versus 12.4 minutes for single agents. In max settings, teams took 27.6 minutes versus 38.4 minutes.
• Claude Opus 5.5: Teams were slower, operating in sequential waves where each wave waited for the slowest member. In max settings, teams took 174.1 minutes, approximately 2.3 times longer than the 74.7 minutes for single agents.

Agent Team Structures

The lead agents employed different delegation strategies:

• GPT-6 Sol Lead: Divided tasks based on architecture (DB, API, UI) and spawned subagents immediately (within ~1 minute). The lead performed testing via a headless browser in 46 out of 50 runs, identifying integration bugs like schema mismatches.
• Claude Opus 5.5 Lead: Created a CONTRACT.md file (defining schema, API routes, file ownership) before starting work in 40–42 out of 50 runs. Subagents were spawned in sequential waves (foundation -> features -> tests). In max effort, the number of subagents increased to 6.8 per app, and test subagents were mandatory.

Impact of Reasoning Effort vs. Team Structure

Comparing high reasoning effort single agents against medium effort teams revealed that increasing reasoning effort was often more efficient:

• GPT-6 Sol: A max effort single agent (89.0%, $4.82) outperformed a medium effort team (84.9%, $3.07), though the difference was not statistically clear (p=0.08). Max effort provided a larger score boost (+11.4 points) than team structure (+7.3 points).
• Claude Opus 5.5: A max effort single agent (89.8%) scored lower than a medium effort single agent (91.5%). While the max effort team achieved the highest score (93.2%), it cost 30 times more than the medium effort single agent, with no significant score difference (p=0.36).

Methodology and Limitations

The study used OpenAI Agents API for GPT-6 Sol and Claude Code CLI headless mode for Claude Opus 5.5. Evaluation relied on the Vibe Code Bench standard UI test suite. Key limitations include:

• Single Run Constraint: Each setting was run once, so confidence intervals reflect variation across apps, not across repeated runs.
• Delegation Prompt: Team runs included delegation instructions, meaning the comparison was effectively 'subagents + delegation prompt' vs 'none'.
• Harness Differences: Different harnesses were used for each model, mixing model effects with harness effects.