NInfer Fork Released: Supports 555k Context on RTX 5090 with NVFP4 KV Cache and YARN

A fork developed to overcome NInfer's limitations supports a 555k token context window in an RTX 5090 environment by applying NVFP4 KV cache and YARN context extension techniques.

NVFP4 KV Cache Implementation

Unlike existing methods, it uses a custom MMA kernel that quantizes Q and K to NVFP4 and dequantizes V to BF16. It applies Hadamard rotation to suppress outliers and introduces a Fused append technique that quantizes K/V in-place during decoding. This reduces VRAM usage by 45% while maintaining performance without quality degradation on LongBench and AIME benchmarks.

Context Extension via YARN

Linear scaling was applied to match the RoPE settings of the Qwen3.8-27B model, extending the native 262k tokens to 555k (including vision) or 600k (text-only). Consistent quality was verified even when outputting 600k tokens, and it is projected that GPUs with 96GB or more can extend up to 8M tokens.

Host KV Cache Safety Net

To replace the existing LRU-based HostKvProvider, HostKVSafetyNet was introduced, using a fixed host arena and scatter-gather allocation. This safely handles concurrent restoration and improves memory efficiency through arena compression.