VS Code Unveils VSC-Bench

The VS Code team revealed how it refines the harness, which matters more than the model itself, in the Copilot agent.

The harness handles context assembly before a request, tool exposure for what the model can call, and tool execution that processes results. System prompts, tool sets, and loop behavior are also tuned differently per model, and a new model isn't simply added but is subject to re-validation before release.

Public benchmarks are used as a baseline, but the team believes internal evaluation is necessary due to contamination concerns with frontier models and coverage limits for real editor workflows.

• SWE-bench Verified carries potential contamination risk, and
• Terminal-Bench is strongly single-task in nature, differing from actual VS Code development flows.

As an alternative, VS Code built an offline evaluation set called VSC-Bench.

• Scope: custom agent modes, extension workflows, MCP/tool use, terminal/browser interaction, multi-turn conversations
• Task families: multi-language coding tasks including TypeScript, Python, C++, etc.
• Metrics: solution correctness, agent effort, token efficiency, latency
• The example chart compared 40 runs across 8 configurations, where xhigh used more tokens than high while yielding a slightly lower number of solutions.

Harness-related PRs are tagged with the ~requires-eval-assessment label to trigger automated evaluation, with a workflow that posts results back to the PR. After release, the team continues measuring via A/B testing, aggregate usage signals, and weekly reporting, and stated that VS Code will keep shipping harness improvements alongside model updates going forward.