The Pitfalls of OpenRouter Routing: Performance and Feature Disparities Across Providers for the Same Model

Based on the case of Olly (an iMessage AI assistant) using open-source models via OpenRouter, this analysis examines performance disparities and caveats across routed Providers. Although model weights are identical, actual behavior differs greatly depending on the hosting company's GPU, precision, and optimization methods.

Performance and Feature Disparities by Provider

• Benchmark Gaps: For DeepSeek V4 Flash, some Providers (such as DigitalOcean) saw TAU-Bench scores drop by more than 20 points compared to first-party offerings. For the GLM-5.3 model, Provider rankings were completely reversed.
• Vision Recognition Errors: Qwen3.5 122B exhibited errors in recognizing colors and text on DeepInfra, while MiniMax M3 ignored image inputs on Venice and Together, responding with 'no image provided'.
• Ignored Parameters: The reasoning.effort setting was ignored by some Providers (such as digitalocean and venice), having no impact on the number of reasoning tokens.

Quantization and Response Quality

• fp4 vs fp8: Contrary to the common belief that fp4 performs worse than fp8, fp4 hosts sometimes ranked in the middle or top of fp8 groups, indicating that precision is not a good indicator of quality. Filtering should be based on leaderboard scores rather than hard filtering.
• Hollow Completions: Cases occur where HTTP 200 OK is returned, but content, reasoning, and usage are all null. Approximately 20% of DeepSeek traffic on StreamLake experienced this phenomenon, necessitating logic to treat it as a failure and retry.

Caveats for Production Operations

• IP-Based Restrictions: Venice and Novita worked normally in Mac environments but frequently returned 429 Too Many Requests errors from production infrastructure IPs. Benchmarks must be conducted in the actual deployment environment.
• Risks of Provider Pinning: Allowing only three trusted Providers and disabling fallbacks can lead to total service downtime due to temporary outages or model unavailability at those Providers. Flexible routing and retry strategies are essential.