TurboQuant Accuracy and Performance Evaluation

According to vLLM's comparison, --kv-cache-dtype fp8 was the most stable default for KV-cache quantization. It achieved 2x capacity with almost no accuracy loss, and the performance gap versus BF16 was mostly small.

• TurboQuant k8v4 only achieved around 2.4x savings, while throughput and latency consistently worsened.
• TurboQuant 4bit-nc is practical in environments with heavy KV-cache memory pressure, but it comes with accuracy and performance costs.
• k3v4-nc and 3bit-nc showed large accuracy drops in reasoning and very long-context scenarios, with latency and throughput also degrading significantly.

In conclusion, FP8 remains the safest choice for general-purpose serving, and TurboQuant is only worth limited consideration in edge or memory-constrained environments where memory is the bottleneck.