How Dynamic Batching Breaks AI Determinism at Temperature Zero

Measure the magnitude in fractions of a cent. If a bank account calculation rounds off a thousandth of a penny, the account holder never notices the missing fraction. But inside a large language model, a rounding error of that exact same magnitude—one thousandth of a decimal point—can completely rewrite a thousand-word response.[4]

This microscopic variance explains one of the most persistent mysteries in artificial intelligence deployment. Engineers routinely instruct models to be perfectly predictable, locking their settings to eliminate all randomness. Yet, when deployed at scale, these supposedly deterministic systems still occasionally change their answers to the exact same prompt.[4]

The culprit is not a software bug or a hidden layer of artificial creativity. It is a fundamental collision between the theoretical mathematics of machine learning and the physical reality of how silicon chips process numbers in parallel. The hardware itself forces a compromise.[4]

To understand why a locked model wanders, one must look at how commercial servers juggle thousands of simultaneous users. The mechanism that makes artificial intelligence economically viable is the exact same mechanism that breaks its reproducibility.[4]

The mechanics of greedy decoding

When a large language model generates text, it does not think in sentences. It calculates the probability of what the next single fragment of a word, or token, should be. The model assigns a mathematical score, called a logit, to every possible word in its vocabulary.[4]

Developers control how the model selects from this list using a parameter called temperature. A high temperature flattens the probabilities, allowing the system to pick lower-ranked words for a more creative or varied output. A temperature of zero does the exact opposite.[4]

At temperature zero, the model is instructed to use greedy decoding. It must always select the single token with the highest probability score, no matter how narrow the margin of victory. If the word "apple" scores 0.9001 and "orange" scores 0.9000, "apple" wins every time.[4]

In pure mathematics, this rule guarantees perfect determinism. If the input prompt is identical, the calculated probabilities will be identical, and the highest-scoring word will be identical. The model should produce the exact same sequence of words whether it is asked once or a million times.[4]

"In theory, greedy decoding is a deterministic function mapping an input sequence to an output sequence," notes the PyTorch reproducibility documentation. However, the documentation immediately warns that running the same function on parallel hardware introduces unavoidable numerical variations.[1]

The limits of floating-point math

Computers cannot store infinite decimal places. They represent fractional numbers using a system defined by the 1985 IEEE 754 standard, known as floating-point arithmetic. Modern artificial intelligence relies heavily on 16-bit floating-point numbers, or FP16, to save memory and speed up calculations.[2]

The FP16 format is highly efficient, but it has a strict precision limit. The smallest difference it can reliably distinguish, known as the machine epsilon, is roughly 0.000977. Any calculation that produces a fraction smaller than this threshold gets rounded off to fit into the available memory space.[2][4]

This rounding creates a critical mathematical quirk: floating-point addition is not associative. In standard algebra, adding A plus B, and then adding C, yields the exact same result as adding B plus C, and then adding A. In computer science, changing the order changes the rounding.[2]

If a processor adds a very large number to a very small number, the small number might be rounded out of existence entirely. If it adds several small numbers together first, they might accumulate enough mass to survive being added to the large number. The sequence dictates the sum.[2]

"Floating-point math is not associative, and the order of operations in parallel reductions depends on the launch configuration," the NVIDIA CUDA programming guide explicitly states. This hardware reality means that how you group the math changes the final answer.[3]

How GPUs sum millions of numbers

Generating a single word requires billions of mathematical operations. The core operation is matrix multiplication, which involves multiplying vast grids of numbers and then summing the results. A graphics processing unit, or GPU, performs these sums using a structure called a reduction tree.[4]

Instead of adding numbers one by one in a long line, a GPU pairs them off. It adds the first and second, the third and fourth, and so on, creating a new, smaller row of sums. It then pairs those sums together, building a tree that tapers to a single final number.[3]

The shape of this reduction tree determines the exact order of addition. A perfectly symmetrical tree pairs numbers evenly at every level. An asymmetrical tree might leave one number waiting while others are summed, introducing it at a different stage of the rounding process.[3][4]

If the tree shape remains constant, the rounding errors remain constant, and the final sum is perfectly reproducible. But in a live data center, the tree shape almost never remains constant. It shifts dynamically based on how much traffic the server is handling at that exact millisecond.[4]

This shifting architecture is what bridges the gap between hardware quirks and software unpredictability. The GPU is constantly rebuilding its mathematical pathways to accommodate the fluctuating volume of user requests, altering the rounding sequence with every structural change.[4]

The chaos of dynamic batch scheduling

Running a large language model for one user at a time is financially ruinous. To maximize the massive cost of silicon, server operators use dynamic batch scheduling. The system waits a fraction of a second to collect multiple user prompts and processes them simultaneously in a single batch.[4]

The size of this batch changes constantly. At 2:00 AM, a server might process requests in batches of 16. At 9:00 AM, during peak traffic, it might bundle 128 requests together. The batch size dictates the dimensions of the matrices being multiplied inside the GPU.[4]

When the matrix dimensions change, the GPU automatically optimizes its workload by changing the shape of the reduction tree. A batch size of 16 utilizes a different parallel summation strategy than a batch size of 128. The order of addition changes, and therefore, the rounding errors change.[3][4]

This means that one user's prompt will generate a slightly different internal mathematical sum depending on whether two other users happen to submit their prompts at the exact same millisecond. The users are isolated, but their shared hardware creates a microscopic numerical entanglement.[4]

The variance is tiny, usually altering the final probability score by less than a thousandth of a point. For the vast majority of calculations, this floating-point noise is entirely harmless. The most likely word remains the most likely word, and the output remains stable.[4]

The butterfly effect of a single token

The system breaks down when the model is genuinely torn between two words. If the neural network determines that "apple" and "orange" are equally valid next tokens, their probability scores might be separated by a margin smaller than the floating-point rounding error.[4]

When the batch size changes the reduction tree, the resulting rounding variance can bridge that tiny gap. The score for "orange" gets rounded up, the score for "apple" gets rounded down, and the rankings flip. The model, strictly following greedy decoding, selects the new winner.[4]

Once a single word flips, the entire generation diverges. Large language models are autoregressive, meaning they use their own past output as the input for the next step. The moment "orange" replaces "apple," the context changes, and the mathematical baseline for all future calculations shifts entirely.[4]

This cascade effect explains why two identical prompts can produce wildly different essays. The divergence did not start with a grand creative leap; it started with a single, invisible rounding error on the fourteenth word, triggered because a stranger on another continent submitted a prompt at the same time.[4]

Engineers can force perfect determinism by disabling dynamic batching, but doing so destroys server throughput and multiplies computing costs. The industry has largely accepted that absolute reproducibility is incompatible with commercial scale. The hardware demands a compromise, and mathematics pays the price.[4]