How Databricks Built Its Own Coding AI Benchmark and What It Found

Databricks engineers built a proprietary coding benchmark to evaluate model performance, based on coding tasks performed on real multi-million-line codebases.

The key findings are as follows:

• Model Performance Tiering: Models fall into three Capability Tiers based on performance and cost. Top-tier models are strong across all problems but come at high cost, while mid-tier models are far cheaper and highly efficient for common tasks.
• The Rise of Open Models: Open models such as GLM 5.2 have entered the top performance tier, showing quality comparable to Opus 4.8 on certain tasks while costing less.
• The Pitfall of Cost Estimation: Predicting overall task cost based solely on per-token pricing is risky. Since inference efficiency varies by model, evaluation should be based on the actual task-level cost required to complete the work.
• The Importance of the Harness: How a model is invoked (the harness) significantly affects both cost and quality. Experiments showed that a simpler approach like Pi performed best for certain workloads.