AI BriefingValkyr with TurboQuant applied
Key point
A TurboQuant inference path was added to Runtime Valkyr, achieving 120 tok/s on Gemma with an RTX 3090.
Details
Runtime Valkyr, based on the TRiP source code, was ported to Zig and headless Vulkan Compute Shader, and TurboQuant was attached as an optional inference path. It recorded 120 tok/s on an RTX 3090 with Gemma.
TurboQuant currently uses only Algorithm 1. Its configuration is RHT pre-conditioner + Lloyd-Max scalar quantization + global 4-bit codebook + norm-correction γ, and QJL (Algorithm 2) was excluded based on 5 independent reproduction results.
- sign-bit residual reduces bias but increases attention-score variance, so it was not adopted.
- Here RHT is not a random orthogonal transform but the Randomized Hadamard Transform.
- At 4-bit, the PPL of plain random rotation was 604, while RHT was 10.12.
- γ stores the original L2 / reconstruction norm to exactly match the L2 norm of the dequantized block.
The default setting is an asymmetric configuration of K=fp / V=TQ4. The TQ4 pack kernel matched 256/256 indices bit-for-bit with the CPU oracle and Python reference on a deterministic input ramp.
With Gemma 2B, max_pos 2048, the V cache shrank from 36 MiB → 4.6 MiB, plus an additional 2 MiB shared dequant scratchpad. Supported hardware spans Vulkan 1.3 GPUs in general, with one SPIR-V binary used per vendor.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Core Technologies for LLM Inference Optimization: Quantization, KV Cache, and Inference Chips
2026 AI Inference Hardware Innovations: New Architectures for Prefill/Decode Disaggregation and Solving Memory Bottlenecks
Pinterest Reduces Costs by Up to 30% with SSD Serving and Quantization in Manas Embedding Search Platform