Local LLM and Harness Benchmark

harness-bench combined local LLMs served via llama.cpp's llama-server with Aider, Claude Code, OpenCode, Pi, and Qwen CLI to evaluate 16 software engineering tasks across Python, PyTorch, JAX, C, C++, Rust, and SQL. Each cell ran in a sandbox and was graded by a hidden test.sh, with task prompts and graders kept private to prevent leakage into training data. Only aggregate results, per-cell CSVs, and plot sources were released, and the full experiment totaled 17 models/quantizations × 5 harnesses × 16 tasks = 1,360 runs.

The strongest combination was Qwen3.6-27B (UD-Q4_K_XL) + pi, the only cell to achieve 16/16. However, it took an average of 207 seconds/task. For speed priority, gpt-oss-120b (MXFP4) + pi was the fastest at 15/16, averaging 34 seconds/task, while Qwen3.6-35B-A3B (UD-Q4_K_XL) + qwen also recorded 15/16 with an average of 108 seconds/task. Across the combined 85 cells of the Q4 and Q8 sweeps, these three combinations formed the top tier.

At the model level, Qwen3.6-27B at 82.5%, gemma-4-31b-it at 81.2%, Qwen3.6-35B-A3B at 80.0%, and gpt-oss-120b at 77.5% ranked at the top, while Qwen3-Omni-30B-A3B-Instruct was lowest at 33.8%. At the harness level, pi at 76.9% and qwen at 75.0% led, followed by claude at 66.2%, opencode at 63.8%, and aider at 62.5%. Claude Code was configured to connect to the local llama-server via an Anthropic-compat shim, which differs from its default tuned environment.

Only opencode showed 14 instances of actually reading or executing the hidden grader, while the other harnesses showed 0 instances. The most difficult task groups were pt3_rope_gqa, jax1_complex_lp, pt7_prompt_blend, pt6_generate_cached, rs1_arena, and pt5_logit_lens, while sql1_recursive and p2_shortest_path were passed by all. Overall, model size alone did not explain the results — active parameters, model family, and harness choice together determined both performance and speed.