Vercel Adds Support for Harbor Evaluation Framework with Parallel Execution on Firecracker microVMs

Vercel now supports running Harbor, an open-source evaluation harness, on Vercel Sandbox. Harbor serves as the underlying technology for Terminal-Bench and includes various benchmarks such as SWE-bench, tau3-bench, and OSWorld.

Users can add the --env vercel option to the harbor run command to execute each trial in an isolated Firecracker microVM. This enables large-scale parallel processing beyond the limitations of local machines. Network policies are enforced by the sandbox firewall outside the VM, and the credential injection feature connects secrets only to outbound requests without passing them into the sandbox.

Integrating with AI Gateway allows access to hundreds of models from multiple providers using a single API key. By simply changing the --model flag, users can easily switch to different models for the same benchmark. This feature is supported in Harbor 0.22.0 and later versions.