The H = -∑ p_i log_2 p_i Formula: How the Average Number of Bits Measures the Uncertainty of a Random Variable

It is a common intuition that the amount of information in a message is determined simply by its length—that a 1,000-word document inherently contains more information than a 10-word sentence. The foundational mathematics of communication contradict this directly. According to Claude Shannon's 1948 theorem, information is not a measure of volume, but a measure of surprise. If a long message is entirely predictable, it contains zero information. The actual data dictates that the true information content of any random variable is quantified by its entropy, a metric that calculates the average number of bits required to resolve the uncertainty of its outcome.[1][2]

The formula that defines this limit is $H = -\sum p_i \log_2 p_i$, where $H$ represents Shannon entropy, $p_i$ is the probability of each possible outcome, and the base-2 logarithm grounds the result in binary digits, or bits. This equation sets an absolute, mathematical floor on how tightly data can be compressed without losing fidelity. By multiplying the probability of a state by the logarithm of that probability, and summing across all possible states, the formula outputs the exact average number of yes-or-no questions needed to determine the variable's true state.[3][4]

To understand the mechanism, consider the simplest random variable: a fair coin flip. The probability of heads is 0.5, and the probability of tails is 0.5. Plugging these into the formula yields an entropy of exactly 1.0 bit. The data shows that it takes exactly one binary decision to resolve the uncertainty of a fair coin. Because both outcomes are equally likely, the system is at maximum uncertainty for a two-state variable, meaning the receiver learns exactly one full bit of information when the outcome is revealed.[4][5]

However, the evidence shifts when the probabilities become skewed. If a weighted coin lands on heads 99 percent of the time, the uncertainty drops precipitously. The formula calculates the entropy of this biased coin at just 0.08 bits. Because the outcome is almost entirely predictable, the average amount of surprise—and therefore the average information generated by each flip—approaches zero. A receiver watching this biased coin flip learns very little new information from each toss, allowing the data to be heavily compressed.[1][2]

This mechanism scales directly to complex systems, such as human language and text transmission. If the 26 letters of the English alphabet appeared completely at random, with each character having an equal 1-in-26 chance of occurring, each letter would carry an entropy of 4.70 bits. But empirical analysis of English text reveals strict statistical patterns: the letter 'q' is almost always followed by 'u', and vowels appear with highly predictable frequencies. These structural rules mean that a reader already possesses partial information about the next letter before it is even revealed, lowering the overall uncertainty of the sequence.[4]

By accounting for these probabilities, Shannon calculated that the actual entropy of the English language is only 2.62 bits per letter. This figure demonstrates that patterns reduce uncertainty, allowing a highly structured system to communicate complex messages using relatively little underlying information. Because the true entropy is nearly half of the theoretical maximum, English text contains significant redundancy. This redundancy is precisely what allows modern file compression algorithms, such as ZIP or gzip, to shrink text files to a fraction of their original size without losing a single character.[3][4]

The implications of this formula form the bedrock of modern digital infrastructure. As information theorist Tara Javidi notes, 'Shannon showed there is something like the speed of light, a fundamental limit.' The entropy rate of a data source dictates the absolute minimum channel capacity required to transmit it reliably, ensuring that 'Shannon entropy is a fundamental limit for how much we can compress a source, without risking distortion or loss.' No lossless compression algorithm can encode a sequence in fewer bits than its Shannon entropy.[1][2][4]

In machine learning, the formula is adapted to evaluate the quality of predictive models. Decision trees utilize a concept called information gain, which measures the reduction in entropy achieved by splitting a dataset along a specific feature. The algorithm calculates the entropy of the dataset before the split, and subtracts the weighted entropy of the resulting subsets. By selecting the split that maximizes this information gain, the model minimizes the remaining uncertainty, effectively isolating the most predictable subsets of data to improve classification accuracy.[5]

Similarly, neural networks rely heavily on cross-entropy loss to quantify the divergence between their predicted probabilities and the actual distribution of the training data. By penalizing confident but incorrect predictions exponentially more than uncertain ones, the network minimizes its internal entropy. As the model trains over thousands of epochs, it continuously tunes its internal weights to better reflect the true probability distribution of the dataset, driving the cross-entropy loss downward toward the theoretical minimum established by Shannon's original 1948 equation.[3][5]

Despite its universal application, the evidence supporting the formula's utility in certain domains remains thin. Shannon explicitly designed the metric to address the engineering problem of signal transmission, deliberately stripping away the semantic meaning of the message. The formula treats a sequence of random characters and a line of Shakespeare identically, provided their probability distributions match. This strict focus on syntactic probability means that Shannon entropy cannot distinguish between profound wisdom and complete gibberish, so long as both sequences are equally unpredictable.[1][2]

This semantic blindness limits the formula's ability to measure the true usefulness of information in biological or cognitive systems. While neuroscientists attempt to apply Shannon entropy to map the information processing capacity of the human brain, the data is noisy and the correlations are weak. The brain does not process information as a sequence of independent, identically distributed symbols. Instead, cognitive systems rely on context, memory, and physical embodiment, variables that resist clean categorization into discrete probability distributions.[3][6]

Furthermore, calculating the exact entropy of highly complex, continuous variables requires integrating over infinite probability density functions, a process known as differential entropy. In practice, this requires significant mathematical approximations, introducing uncertainty into the exact limits of compression for continuous analog signals like high-fidelity audio or video. Because a truly continuous signal can take on an infinite number of states, its absolute entropy is technically infinite, forcing engineers to rely on quantization and acceptable loss thresholds rather than the pure lossless limits defined by the discrete formula.[1][2]

The formula also assumes that the probability distribution of the source is stationary and fully known to both the sender and the receiver. In real-world communication networks, probabilities fluctuate dynamically, requiring adaptive compression algorithms that estimate entropy on the fly. The theoretical floor set by Shannon is rarely achieved in practice due to this operational overhead. Compressing data requires sending the compression dictionary alongside the message, adding a structural penalty that keeps real-world transmission rates slightly above the absolute mathematical limit.[4][5]

The $H = -\sum p_i \log_2 p_i$ formula remains a defining triumph of 20th-century mathematics. By proving that information is a quantifiable, physical property tied to probability, the equation transformed communication from a qualitative art into a rigorous science. The next frontier lies in reconciling this syntactic measure of uncertainty with the semantic meaning it carries, a challenge that continues to drive research in artificial intelligence and cognitive science. Until then, every bit of data transmitted across the globe remains bound by the limits Shannon calculated.[1][6]