Breaking Down the Nondeterminism of LLM Inference

Large language models sometimes produce different outputs for the same input even under settings that are theoretically deterministic, like temperature=0. Concurrent execution on GPUs and floating-point non-associativity are commonly cited as the causes, but the author argues these alone don't fully explain it.

The key starting point is floating-point non-associativity. Even with the same numbers, changing the order of addition can change the result, and when large and small values are mixed together in addition, truncation can occur and errors accumulate. So changing the order of addition can produce different results.

However, the author doesn't immediately conclude from this that "LLM inference is therefore nondeterministic." In practice, repeatedly running the same input through something like a GPU matmul often produces bit-for-bit identical results, and the core kernels in the LLM forward pass are generally described as deterministic. In other words, the interpretation that the problem is simply due to "it being a GPU" is too crude.

Cases where concurrency truly causes nondeterminism are usually when multiple threads accumulate into the same location, as with an atomic add. In such cases, the accumulation order can vary depending on which thread finishes first. However, according to the author, the typical forward pass of an LLM has almost no such atomic adds, and most operations are implemented deterministically using approaches like batch dimensions, tree reduction, or semaphores.

So the author presents the view that "the kernels themselves may be deterministic, but the overall system is still nondeterministic." That is, at the inference server level, how requests are grouped, in what order they're processed, and which batches they're merged into can all shake up the final output. Even with the same request, if the server internally takes a different execution path, the result will differ from the user's perspective.

Ultimately, the message of this piece is that solving the nondeterminism of LLM inference requires looking not just at floating-point errors but also at server scheduling and batch composition methods. Only by understanding the true cause can one control the execution environment and achieve reproducible inference.