NVIDIA Releases Open-Source LLM Router 'Switchyard'… Up to 30% Cost Savings Compared to Opus

NVIDIA has released Switchyard, an open-source project that automatically routes LLM calls to the model with the best cost-performance balance. This tool allows users to continue using OpenAI/Anthropic APIs without modifying agent code, intelligently selecting between capable (high-performance) and efficient (low-cost) models based on request complexity.

Key Features and Architecture

• Automatic Routing: The Algorithm selects a model category (e.g., efficient), and the Driver maps it to the actual ModelId for invocation.
• Various Integration Paths: Available as a NeMo Relay plugin (requires Rust build) or via library embedding (PyPI or source build). It can also run as a standalone proxy, providing standard APIs such as /v1/chat/completions at localhost:4000.
• Coding Agent Integration: Changing ANTHROPIC_BASE_URL or OPENAI_BASE_URL to the Switchyard server address immediately applies it to existing tools like Codex CLI.

Performance Benchmarks (Terminal-Bench 2.1)

Based on NVIDIA's internal endpoint testing, using routing algorithms resulted in significant cost savings compared to fixed models.

• Baseline (Claude Opus 4.8): Accuracy 76.0%, Cost $98.06
• Escalation Routing: Accuracy 75.7%, Cost $85.00 (13.3% cost savings)
• Stage Routing: Accuracy 72.7%, Cost $68.19 (30.5% cost savings)
• Capability Routing: Accuracy 71.2%, Cost $79.32 (19.1% cost savings)

Among standalone models, Kimi K2.6 (55.8%, $76.28) and GLM 5.2 (52.4%, $16.47) were used for comparison, but composite strategies via routing proved to better manage the trade-off between performance and cost.