AutoBe Benchmark Narrows the Gap With Local Models

AutoBe is a backend generation benchmark that produces requirements analysis, ERD, OpenAPI spec, E2E tests, NestJS implementation, and a type-safe SDK from a single natural language request.

Each stage fills a predefined AST via function calling rather than free-form code. Scoring also uses only 100-point static analysis, so the same artifact receives the same score no matter who runs it.

On the April leaderboard, 13 models clustered within about 5 points of each other. GLM 5(82.8) took first place, followed closely by Claude Sonnet 4.6(82.2) and Qwen 3.5-27B(81.8). Some local models passed enterprise-scale backends with 100% compile success.

There are two key takeaways.

• The structured harness, along with retries and diagnostics, significantly reduced complex type errors.
• In backend generation, harness design may influence performance differences more than model reputation.

The cost is also substantial. At frontier pricing, a full run costs about $1,000-$1,500 per model, and the next round plans to narrow the field to models priced at $0.25/M input or below, or ones that can run on a 64GB unified-memory laptop.

However, since only 4 reference projects were used, there remains a possibility that models good at following procedural function-calling were favored.