Kimi Vendor Verifier - Verifying Accuracy of Inference Providers

Kimi Vendor Verifier(KVV) is an open-source tool aimed at distinguishing model's own limitations from engineering errors by verifying inference implementation deviations that can occur when serving open-source models on different infrastructures.

Based on the official API, it has published OCRBench 91.0, AIME2025 avg@32 98.4, and MMMU Pro Vision 78.8, along with the Temperature, TopP, and MaxTokens settings for each evaluation and the K2VV evaluation result files.

An investigation into benchmark anomalies reported by the community found that a significant portion stemmed from decoding parameter misuse, and in Thinking mode, Temperature 1.0 and TopP 0.95 are enforced along with content re-transmission verification.

The verification procedure consists of the following steps:

• In the Pre-Verification stage, check whether API parameter constraints are properly enforced
• Run a 5-minute smoke test of the multimodal pipeline with OCRBench
• Check Vision input preprocessing with MMMU Pro
• Detect KV cache bugs and quantization performance degradation in long-form output with AIME2025
• Measure trigger consistency and JSON Schema accuracy with K2VV ToolCall
• Perform agentic coding tests with SWE-Bench

The entire workflow takes about 15 hours when run sequentially on two NVIDIA H20 8-GPU servers, and includes streaming inference, automatic retries, and checkpoint resume mechanisms.

It also collaborates with the vLLM, SGLang, and KTransformers communities, and aims to spread an accuracy-first culture by providing a public leaderboard and early access so vendors can verify their stacks early.