Why Temperature Scaling Distorts LLM Calibration Curves Under Top-p Sampling

Inside a machine learning laboratory in 2017, researchers plotting the confidence of deep neural networks against their actual accuracy noticed a recurring visual anomaly on their monitors. The reliability diagrams, which should have formed a perfect diagonal line representing honest self-assessment, instead bowed heavily downward.[2]

The networks were overwhelmingly overconfident, claiming ninety-nine percent certainty on predictions that were correct only eighty percent of the time. This observation birthed temperature scaling, a mathematical fix designed to force models to report their true odds.[2]

"Modern neural networks tend to be very poorly calibrated," noted Geoff Pleiss, co-author of the foundational 2017 paper on the subject. "We find that this is a result of recent architectural trends, such as increased network capacity and less regularization."

Today, that same fix is applied to large language models, which suffer from even more extreme overconfidence after undergoing reinforcement learning from human feedback. Engineers routinely dial up the temperature parameter to flatten the model's probability distribution and restore calibration.[6]

However, this adjustment collides with another standard generation mechanism: top-p, or nucleus, sampling. When these two operations interact, the mathematical guarantee of temperature scaling breaks down. The combination distorts the very calibration curve the temperature adjustment was meant to repair.[6]

The Logit Bottleneck

To understand this distortion, one must first trace how a language model assigns confidence to its vocabulary. The final layer of a transformer network produces raw, unbounded scores known as logits for every possible next word.[3]

Because these logits can be negative or arbitrarily large, they cannot be read directly as probabilities. They are pushed through a softmax function, which exponentiates and normalizes the scores so that they sum to exactly one across the entire vocabulary.[3]

The softmax conversion is highly sensitive to small numerical differences. A token scoring just two points higher than its neighbor in logit space becomes roughly seven times more likely after the exponential transformation.[3]

This rapid scaling is what makes modern models so decisive, but it also drives their inherent overconfidence. The exponential math pushes the probability of the top token close to absolute certainty, even when the model is effectively guessing between several plausible options.[6]

The training pipeline exacerbates this mathematical quirk. While base pretrained models are generally well-calibrated, the reinforcement learning phase rewards models for sounding authoritative. This pushes the Expected Calibration Error significantly higher, rendering the raw confidence scores entirely untrustworthy.[5]

Flattening the Curve

Temperature scaling intervenes directly at this bottleneck to correct the error. Before the logits enter the softmax function, they are divided by a single scalar value, known as the temperature. A temperature of 1.0 leaves the model in its native, overconfident state.[2]

A temperature greater than 1.0 compresses the raw logits, shrinking the numerical gaps between them. When the softmax function processes these compressed scores, the resulting probability distribution is significantly flatter, pulling mass away from the peak and distributing it across the alternatives.[2]

Crucially, dividing every score by the same positive number preserves their original ranking. The most likely token remains the most likely token, and the model's fundamental accuracy does not change. The adjustment only alters the stated confidence.

By tuning the temperature on a validation dataset, engineers can perfectly align the model's stated confidence with its empirical accuracy. This minimizes the calibration error and ensures that a token assigned an eighty percent probability is actually correct eight times out of ten.

If the generation process stopped there, the calibration would hold perfectly. But language models rarely output the single most likely token in a deterministic vacuum. Always choosing the top token, a strategy known as greedy decoding, causes the text to become repetitive and robotic.[1]

The Nucleus Mechanism

To prevent this degradation, systems employ stochastic decoding strategies that introduce controlled randomness. The most prevalent of these is top-p, or nucleus, sampling. Introduced by Ari Holtzman and colleagues in 2019, this method was designed to truncate the unreliable tail of the vocabulary.[1][4]

Nucleus sampling operates by sorting the tokens from most to least likely and calculating a running total of their probabilities. It keeps only the smallest set of tokens whose cumulative probability exceeds a predefined threshold, typically set between 0.90 and 0.95.[4]

The remaining low-probability tokens are discarded entirely, and the probability mass is redistributed among the surviving candidates. The model then randomly samples its next word from this truncated pool, ensuring that it never selects a completely nonsensical continuation.[4]

Because the size of the nucleus adapts to the shape of the distribution, it is highly sensitive to the model's confidence. If the model is certain, the nucleus might contain only two tokens. If the model is unsure, the nucleus expands to include dozens of plausible options.[1][4]

The Mathematical Collision

The distortion occurs because temperature scaling and nucleus sampling pull the probability distribution in opposite directions. When a high temperature flattens the probabilities to fix overconfidence, the top tokens lose a significant portion of their mass.[6]

Consequently, it takes far more tokens to reach the 0.95 cumulative threshold. The nucleus expands dramatically, ingesting a long, fat tail of low-probability candidates that would have been excluded under the model's native confidence levels.[4][6]

This expansion fundamentally alters the sampling environment. The model is now forced to draw from a candidate pool saturated with marginal tokens. Because the sampling algorithm must pick one of these tokens, the actual frequency of rare words in the generated output spikes.[6]

The mathematical elegance of temperature scaling assumes that the full probability distribution remains intact. By slicing off the tail after the distribution has been artificially flattened, nucleus sampling breaks the normalization that made the calibration accurate in the first place.[2][4]

The probabilities of the sampled tokens no longer match the calibrated confidence scores. The reliability diagram becomes inaccurate for the text that is actually produced, as the truncation step artificially inflates the likelihood of the worst tokens in the nucleus.[5][6]

Breaking the Normalization

This interaction explains why heavily scaled models often hallucinate or lose coherence, even when their internal metrics suggest perfect calibration. The flattened distribution invites the sampler to make riskier choices, degrading the factual reliability of the output.[6]

The free lunch of post-hoc scaling depends entirely on not sampling. For decision-making systems that take the mathematical maximum without generating text, the calibration holds perfectly. For generative systems that sample from the nucleus, the distortion is mathematically unavoidable.[6]

Engineers attempting to balance honesty and coherence must navigate this strict trade-off. Some pipelines attempt to apply the top-p truncation before scaling the temperature, though this introduces its own mathematical inconsistencies and complicates the caching of token probabilities.[6]

Others are abandoning temperature scaling entirely in favor of dynamic thresholding methods. Techniques like min-p sampling scale the cutoff point based on the maximum logit rather than relying on a fixed cumulative sum, providing a more stable boundary.

Ultimately, the collision between calibration and sampling highlights a fundamental tension in language model design. The mathematical operations required to make a model honest about its uncertainty actively undermine the mechanisms required to make its text read naturally.[6]