Parsimony: Open-Source Benchmark Measures Code Bloat in AI Models

Measuring Code Efficiency

Existing benchmarks like SWE-bench focus on solve rates and cost but ignore code bloat, which increases long-term maintenance burdens. Parsimony addresses this by measuring net coding units—syntax elements such as statements, calls, comparisons, names, and literals—rather than lines of code. This prevents rewarding code cramming and accounts for deletions by calculating net units = added − removed.

Benchmark Results

The primary dataset evaluates 26 models on 34 DeepSWE Python tasks (136 attempts per model). Rankings are determined by the mean net units added on solved tasks, with lower scores indicating more concise code. Key findings include:

• Gemini 3.1 Pro Preview (high) ranked first with 1,479.5 net units added (8.1% solve rate).
• Claude Fable 5 (xhigh) followed with 1,557.5 net units (74.3% solve rate).
• Claude Opus 5 (max) added 2,183.4 net units (74.3% solve rate), showing higher verbosity despite high performance.
• GPT-5.6 Sol (max) added 1,964.2 net units (74.3% solve rate).

Methodology and Limitations

Parsimony applies patches to task code and parses changed files without running tests, relying on DeepSWE's published pass/fail results. The benchmark supports multiple languages and datasets, including DeepSWE (Python, JavaScript, TypeScript, Go), SWE-bench Verified, and SWEeper-Bench (PHP).

• Correlation: Models that write more code tend to solve more tasks (Spearman correlation 0.33–0.60).
• Agent Influence: All models use the same agent (mini-SWE-agent), but agent behavior still impacts code volume.
• Scope: Smaller code size does not guarantee better readability or design.