AI Briefingvc-bench: AI coding agents achieve 100% pass rate on version control tasks
·2026.11.07 23:58
Key point
A new benchmark evaluates Codex and Claude on git operations, finding a 359/360 pass rate across the full batch, with GitButler showing the lowest latency and command count in the primary comparison.
Details
- vc-bench is a new benchmark that measures AI coding agents' ability to perform complex version control operations. The results history for the 2026-07-20 batch indicates a total of 359/360 passes across all runs, although the primary comparison table shows 60/60 passes for each tool-agent combination.
- The benchmark compares three version control tools: standard git, Jujutsu, and GitButler. In the primary comparison, GitButler completed tasks significantly faster (34.0s average) and with fewer commands (4.5 average) than git (92.2s, 20.4 commands) and Jujutsu (110s, 21.0 commands).
- The test suite includes six scenarios: selective commit, multi-amend, split commit, reorder commits, squash commits, and update dirty branch, each requiring precise manipulation of commit history and working trees.
- A deterministic grader verifies the final Git state rather than the command sequence, ensuring that different valid approaches are scored equally; the benchmark is designed to compare tools while holding the AI agent constant.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Thinking Machines AI introduces Inkling 2 Small, planning Apache 2.0 open weights release next week