Kimi Vendor Verifier – Verifying Accuracy of Inference Providers

Alongside Kimi K2.6, Kimi has open-sourced Kimi Vendor Verifier (KVV). The goal is to verify that open-source models run identically and accurately everywhere.

Kimi investigated benchmark anomalies following K2 Thinking, and revealed that a significant portion stemmed from misuse of decoding parameters. To prevent this, in Thinking mode it enforces Temperature=1.0 and TopP=0.95, and has added safeguards at the API level to verify that thinking content is returned correctly.

Afterward, it discovered more subtle anomalies as well. Results for a specific LiveBenchmark evaluation differed significantly between third-party API and official API, and Kimi explained that after testing multiple infrastructure providers, such discrepancies were widespread. Kimi views this as a trust issue for the open-source ecosystem, and emphasized the need to distinguish between "model performance defects" and "implementation deviations."

The key verification items presented by KVV are as follows.

• Pre-Verification: Checks whether API parameter constraints such as temperature and top_p are properly enforced
• OCRBench: A 5-minute smoke test for multimodal pipelines
• MMMU Pro: Verifies vision preprocessing with diverse visual inputs
• AIME2025: A long-output stress test to detect KV cache bugs and quantization degradation
• K2VV ToolCall: Measures trigger consistency (F1) and JSON Schema accuracy
• SWE-Bench: An agentic coding test, though kept private due to sandbox dependencies

On the operational side, Kimi aims to fix root causes together with the vLLM, SGLang, and KTransformers communities, and provides pre-release validation so vendors can verify their stacks before launch. It also stated it will maintain a public leaderboard aggregating vendor results.

For evaluation costs, the full workflow was verified on 2 NVIDIA H20 8-GPU servers, taking approximately 15 hours on a sequential run basis. Streaming inference, automatic retry, and checkpoint resumption scripts were also optimized for long-duration inference.