Can AI agents build real Stripe integrations? Stripe verified it with a benchmark

This effort started from the problem awareness that while LLMs are good at implementing functions or refactoring at the file level, actual software engineering—long-term planning, maintaining state, and recovering from failures—is a different matter entirely. Noting that payment integration requires 100% accuracy, Stripe wanted to see whether models could go beyond simple code generation to perform verification and testing as well.

To this end, Stripe built the Stripe integration benchmark, composed of 11 environments. Each environment includes code, a database, and scripts like a real repository, and also provides a test Stripe API key. Grading was mainly done via deterministic tests, judging submissions through API calls, automated UI tests, and verification of Stripe objects. The execution harness is based on goose, with an MCP server attached that gives the model a terminal, a browser, and Stripe-specific search tools.

The evaluation tasks were divided into three categories.

• Backend-only tasks: server-side work such as Stripe API version changes, data migrations, and backend API updates
• Full-stack tasks: integration tasks involving both server and client, requiring browser use for final submission
• Gym problem sets: simulation problem sets that dig deep into specific Stripe features like Checkout and subscriptions

The results were stronger than expected. The authors expected some level of performance on backend-only tasks, but in practice the models also navigated UIs, debugged live issues, and even handled behavior that was poorly documented. In particular, Claude Opus 4.5 scored an average of 92% on full-stack API integration tasks, and OpenAI GPT-5.2 scored an average of 73% on gym problem sets. The best-performing runs went up to an average of 63 turns.

There were also impressive cases. In the card element to checkout session task, an agent converted the legacy Card Element UI to Checkout, then even carried out an actual test purchase in the browser. The payment method wasn't specified, but in one run the agent completed the purchase using Link, the digital wallet built by Stripe.

In the Checkout gym task, agents had to look at 20 pre-built Checkout UIs and work backward to figure out the API calls needed behind them. The agent first had to check the product and quantity in the UI, find the product ID via the Products API, identify settings such as shipping amount, custom fields, and whether tax collection was enabled, and then translate these into Checkout Session API parameters. In the evaluation, agents got over 80% of the parameters for this task right, and for more complex UIs, they even clicked through additional dropdown options to check and reflect them.

However, the limitations were also clear. Some agents failed to properly interpret ambiguous situations—for example, entering nonexistent Stripe data, receiving a 400 response, and still concluding the task was done. There were also cases where, during browser tasks, agents lost input focus or got stuck inside a frame, and even though a simple refresh or re-click could have recovered the situation, the agent concluded on its own that it had failed.

The core message is clear: real integration isn't just about writing a good piece of code—it also involves verification, state management, browser interaction, and error recovery. By open-sourcing this benchmark, Stripe hopes it will become a testbed for repeatedly measuring how much interventions like agent prompts, skills, and browser tools actually improve real-world performance.