How Normalized Entropy Strips Base Rate Illusions From Meta's Ad Rankers

In August 2014, a team of engineers at Facebook published a paper detailing the machine learning architecture behind their multi-billion dollar advertising business. The document bypassed the usual hype around artificial intelligence to focus on a mundane mathematical problem. The system's success relied entirely on a specific evaluation metric known as Normalized Entropy.[1]

At the time, Facebook was serving roughly three trillion ad impressions per month to over 750 million daily active users. The core task of the platform's ad ranker was binary classification: predicting whether a specific user would click on a specific ad.[1]

This prediction dictates the platform's revenue. In a standard online advertising auction, the winner is not simply the advertiser who bids the most money. The system multiplies the advertiser's cash bid by the estimated probability of a click to calculate the expected value of showing that ad.[2]

If the model's probability estimates are wrong, the auction breaks. A highly relevant ad with a low bid might lose to an irrelevant ad with a high bid, degrading the user experience and ultimately costing the platform money.[2]

To evaluate how well a model predicts these probabilities, engineers traditionally use a metric called log loss, or cross-entropy. Log loss penalizes a model based on how confident it was in a wrong answer.[3]

If a model predicts a 99% chance of a click and the user scrolls past, the log loss penalty is severe. If it predicts a 10% chance and the user clicks, the penalty is moderate.[3]

The Base Rate Illusion

However, raw log loss has a fatal flaw when applied to online advertising: it is highly sensitive to the underlying base rate of the data. The typical click-through rate for display ads hovers between 0.5% and 1.0%.

Because clicks are so rare, the dataset is massively imbalanced. If an engineer wants to achieve a low log loss, the easiest way is to evaluate the model on a dataset where the background click rate is extremely close to zero.[3]

"If almost all of the cases are of one category, then we can always predict a high probability of that category and get a fairly small log loss," notes a 2020 mathematical analysis of the metric.[3]

The extreme probabilities align with almost all of the cases, resulting in very few penalized mistakes. This creates a dangerous illusion for machine learning teams evaluating new neural network architectures.[3][5]

Imagine a social network changes its user interface, making ads slightly harder to see, which drops the platform-wide click rate from 1.0% to 0.1%. The ad ranking model's raw log loss will suddenly improve, making the model look vastly more accurate.[5]

In reality, the model has not learned any new patterns. The environment simply shifted to make predicting a non-click a safer mathematical bet.[5]

Isolating True Predictive Power

To prevent this base rate illusion from masking true model performance, Meta's engineers divide the model's log loss by the entropy of the background click rate. This resulting ratio is Normalized Entropy, sometimes referred to as Normalized Cross-Entropy.[1]

The denominator in this equation is the log loss of a theoretical baseline model. This naive model ignores all user features and simply predicts the average historical click rate for every single impression.[3]

By dividing the advanced model's score by the naive model's score, engineers isolate the actual intelligence the neural network adds. If a model achieves a Normalized Entropy of 1.0, it is performing exactly as well as a system that just guesses the average.[1]

"Unless your model is worse than predicting a constant, the denominator should be higher than the numerator, so it is usually between 0 and 1," explains the 2020 breakdown. A lower score indicates a genuinely smarter model.[3][4]

This normalization allows engineering teams to compare model architectures across completely different datasets. If one dataset has a background click rate of 5% and another has a rate of 0.5%, their raw log loss scores cannot be directly compared.[4]

However, if a model achieves a Normalized Entropy of 0.600 on the harder dataset and 0.699 on the easier one, the engineers know the model is extracting relatively more signal from the former, regardless of the baseline difficulty.[4]

Feature Engineering and Freshness

In their 2014 research, the Facebook team demonstrated that combining logistic regression with boosted decision trees reduced Normalized Entropy by 3.4% compared to a standard linear model. At the scale of trillions of impressions, that improvement translates to massive revenue gains.[1][4]

The researchers found that the most significant gains came from feature engineering, specifically transforming continuous variables into categorical ones. Each individual tree in the ensemble acts as a categorical feature, taking the value of the index of the leaf node that a specific ad impression maps to.[1]

Data freshness also proved critical to maintaining a low Normalized Entropy score. The team discovered that when a model is trained on data from one day and evaluated on subsequent days, its predictive accuracy degrades steadily as the delay increases.[1]

To combat this decay, modern ad rankers utilize online learning. They continuously update their linear classifiers in near real-time as new click data flows into the system, ensuring the model reacts to immediate behavioral shifts.[4]

Calibration and the Auction

While Normalized Entropy measures a model's predictive power, it is not the only metric required to run a functional ad auction. The model must also be strictly calibrated to reality.[1]

Calibration is the ratio of the average estimated click rate to the actual empirical rate observed in reality. "The less the calibration differs from 1, the better the model is," the 2014 paper states.[1]

A model can have an excellent Normalized Entropy score but terrible calibration. If a model perfectly ranks users from most likely to least likely to click, but divides all its probability estimates by two, its ranking power remains intact.[2]

In many machine learning applications, ranking is enough. For search results or movie recommendations, engineers often rely on Area Under the ROC Curve, which only cares about the final order of the items.

But in an ad auction, the exact probability matters. If the model divides the true probability by two, the expected value calculation yields half the actual value, causing the platform to underprice its inventory.[2]

"Area-Under-ROC is also a pretty good metric for measuring ranking quality without considering calibration," the Facebook researchers noted, but it cannot serve as the primary metric for an auction system where money changes hands.[1]

Industry Standardization

The mathematical framework detailed in 2014 has since become standard practice across the digital advertising industry. Platforms from LinkedIn to Pinterest rely on similar normalized metrics to evaluate their ranking systems.[2]

Some engineering teams prefer to express the exact same mathematics in reverse, using a metric called Relative Information Gain.

Relative Information Gain is calculated simply as one minus the Normalized Entropy score. While Normalized Entropy measures the remaining error, this alternative measures the percentage of uncertainty the model has successfully eliminated.

Whether framed as entropy or information gain, the underlying principle remains the same. As ad platforms transition from simple logistic regression to massive deep neural networks, the risk of models memorizing noise increases.[5]

By anchoring their evaluation to the background entropy of the data, platforms ensure that their algorithms are actually learning human behavior, rather than just riding the statistical waves of shifting base rates.[5]