DetLLM Achieves Bit-Identical LLM Inference Across Hardware Using Integer-Only Arithmetic

DetLLM eliminates LLM inference nondeterminism by running the entire forward pass (matmuls, softmax, normalization, rotary embeddings) using exact integer arithmetic instead of floating-point, ensuring bit-identical outputs regardless of hardware or batch size.
In a test of Qwen3-0.6B across nine configurations on A100, H100, and various CPUs, integer runs produced a single unique hash, while floating-point runs produced nine different hashes due to rounding errors in addition order.
The approach maintains model quality with a WikiText2 perplexity of 20.72 (compared to 20.95 for fp16) and achieves decode speeds of up to 106.3 tokens per second on an A100 after kernel optimization.
Determinism is achieved by using dyadic scales for quantization and ensuring batch invariance through per-token activation quantization, with only the int8 matmul dispatching on hardware-specific instructions.

Technical Implementation

Uses integer operators from the I-LLM framework, requiring no fine-tuning, just a calibration pass and closed-form smoothing tricks.
Implements shift-based integer algorithms for Exp, softmax, RMSNorm, and SwiGLU, with precomputed int16 sine/cosine tables for rotary embeddings.
A TorchDispatchMode guard enforces the absence of floating-point tensors, and argmax breaks ties toward the lowest token id to ensure consistency.

The guarantee is tied to a specific prepared artifact because the float-to-int conversion process itself is not reproducible.
Prefill speed remains significantly slower than fp16 (about 0.13x), and the current implementation is a proof of concept for a single model size.