How Ora Benchmarks Major AI Agents on Vercel

Ora is a benchmarking platform that instructs AI agents to perform product sign-ups, integrations, and payments on real websites, and analyzes the causes of failure. Co-founder Assaf Elovic founded Ora after determining that it is not enough for agents to simply navigate the web; it is necessary to measure whether they can actually use products.

Ora simultaneously tests major agents, including Claude Code, ChatGPT, Gemini, Hermes, OpenClaw, and Vercel's agent framework eve. Since each agent (harness) requires unique infrastructure and execution environments, Ora operates independent runtimes for each and tracks every step. This allows customers to clearly identify at which step an agent failed.

The entire system is built on Vercel, with the frontend, backend, and agent runtimes sharing the same deployment path, logs, and authentication system. When Vercel launched eve, Ora conducted benchmarks under the same conditions as other agents.

Comparing eve and Claude Code, eve showed the following results:

• 7% fewer steps to reach the goal
• 2x higher rate of completing tasks directly on customer sites without switching to web search
• 9% higher discovery rate of valid endpoints that can actually be called

After fixing prompt caching issues discovered during the benchmarking process, the total cost of eve decreased by approximately 15%.