Quantifying Infrastructure Noise in Agentic Coding Evaluations

Agentic coding benchmarks like SWE-bench and Terminal-Bench show very fine performance differences between models, so infrastructure configuration alone can distort results. Experiments showed that on Terminal-Bench 2.0, score differences of up to 6%p occurred depending on how resources were configured.

Agentic coding evaluation encompasses the entire runtime environment in which a model writes code and runs tests. Therefore, resource limits (CPU, RAM) go beyond being a simple container environment and act as a core element of the problem-solving process; with different resource budgets, models are effectively taking different tests.

Experiments in a Kubernetes environment confirmed the following issues when resource quotas were strictly limited:

• Increased infrastructure error rate: When memory spikes occur, containers are forcibly terminated (OOM-kill), raising the task failure rate.
• Distorted performance: Scores are measured lower due to infrastructure instability, regardless of the model's actual coding ability.

As resource headroom was increased, performance changes appeared in two stages:

• Below 3x: The infrastructure error rate decreases, but there is no significant change in the model's actual success rate. This is the stage where environment stability improves.
• Above 3x: The success rate rises sharply. As large-scale dependency installation or memory-intensive test execution becomes possible, the benchmark ends up measuring "resource availability" rather than the model's actual capability.