Stripe benchmark: Coding agents score 29.4% on strict pass rate despite 78.1% average criteria success

·2026.11.19 09:00

Key point

Stripe's 69-scenario benchmark reveals that while agents achieve 78.1% average criteria success, their strict pass rate drops to 29.4% due to final detail failures.

Details

  • Stripe's new benchmark of 69 real-world integration scenarios shows coding agents achieve a 78.1% average criteria success rate but only a 29.4% strict pass rate, indicating they produce almost correct code that fails on final MUST criteria.
  • The Stripe agent plugin improved scores for all 14 model/harness pairs by an average of 3 to 8 percentage points, with the largest gains seen in Gemini 3.8 Flash (+7.9%) and Minimax M3.1 Flash (+7.4%).
  • Plugin usage reduced web searches by 96% (from 1.26 to 0.05 per eval) and page fetches by 63%, while decreasing outdated parameter references from 7% to 1.3%.
  • Performance varies significantly by business model; agents strong in Payments or Subscriptions did not necessarily lead in Connect, where the plugin provided larger gains.

Methodology and Scope

  • Benchmark covers 69 scenarios (19 Connect, 24 Payments, 26 Subscriptions) across 14 model/harness pairs, including Claude Code, Codex, and OpenCode harnesses.
  • A secondary agent (GPT 6.1 Sol) simulates a developer with latent needs to steer the coding agent, while GPT 6.1 Sol also serves as the judge using RFC 2119-based rubrics.
  • Each case has a 45-minute timeout; errors are excluded from scores, and the benchmark does not measure runtime execution, product recommendation quality, or long-horizon maintenance.

Availability

  • The Stripe benchmark and Forge eval harness are published as open source on GitHub (stripe/forge).
  • Results will be updated periodically with new models and test cases.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.