Evidence Pack: The Mathematical Trade-offs Between the Wald, Score, and Likelihood Ratio Tests

In statistical modeling, the validity of a hypothesis is ultimately determined by the shape of the likelihood function—a mathematical curve that maps how probable the observed data is across different potential parameter values. The decision to accept or reject a claim hinges entirely on how one measures the distance between the peak of this curve and the restriction proposed by the hypothesis. This geometric evaluation dictates whether an observed effect is real or merely statistical noise.[1][4]

Statisticians have historically relied on three foundational methods to measure this distance: the Likelihood Ratio test, the Wald test, and the Score test. Known collectively as the "Holy Trinity" of maximum likelihood estimation, these three tests were developed between 1928 and 1948 to solve the exact same problem. Yet, they approach the likelihood curve from entirely different geometric perspectives, forcing researchers to choose between computational efficiency and mathematical precision.[1][3]

The core mechanism of all three tests relies on Maximum Likelihood Estimation (MLE). When a model is fitted to data, MLE finds the parameter values that maximize the likelihood function, representing the absolute peak of the curve. A hypothesis test then imposes a restriction—such as claiming a specific variable has an effect size of exactly zero. The tests evaluate how much the likelihood drops when forced away from the peak to this restricted value.[4][6]

The Likelihood Ratio (LR) test, introduced by Jerzy Neyman and Egon Pearson in 1928, is the most direct measurement. It requires the researcher to fit two separate models: one unrestricted model that finds the absolute peak, and one restricted model that forces the parameter to the hypothesized value. The test statistic is simply the vertical distance between these two points on the likelihood curve.[4][6]

Because it measures the actual drop in likelihood, the LR test is widely considered the gold standard for accuracy. As the UCLA OARC Statistical Consulting documentation explicitly notes, "The likelihood ratio test requires that you estimate both the unrestricted and the restricted models." This dual-estimation makes it highly precise, particularly in small samples, but historically made it computationally prohibitive for complex datasets.[4]

To bypass this computational burden, Abraham Wald introduced the Wald test in 1943. Instead of fitting two models, the Wald test requires only the unrestricted model. It measures the horizontal distance between the estimated parameter and the hypothesized value, scaling that distance by the curvature of the likelihood function at the peak. If the curve is steep, even a small horizontal distance suggests the restriction is highly unlikely.[1][5]

This single-model requirement made the Wald test the default output in almost all statistical software packages. When a researcher runs a standard regression, the software automatically computes the unrestricted model and uses the Wald test to generate the p-values for every coefficient simultaneously. As the SAS Help Center documentation explains, "The Wald test is based on the distance between the parameter estimate and the hypothesized value."[5]

However, the Wald test carries a significant mathematical vulnerability: it is not invariant to parameterization. If a researcher tests a hypothesis using a linear scale, and another tests the exact same hypothesis using a logarithmic scale, the Wald test can yield p-values differing by more than 0.05 purely due to the mathematical transformation. Furthermore, because it relies on the curvature at the peak to estimate the rest of the curve, it can become highly inaccurate in small samples where the likelihood function is asymmetrical.[1][3]

The Score test, introduced by C.R. Rao in 1948 and later generalized as the Lagrange Multiplier test, takes the opposite computational approach. It requires fitting only the restricted model. It then measures the slope, or gradient, of the likelihood function at that restricted point. If the restriction is close to the true peak, the slope should be nearly zero. If the slope is steep, the restriction is likely false.[1][4]

The Score test is uniquely powerful in econometrics and genetics, where fitting a massive unrestricted model with thousands of parameters is computationally impossible. By fitting only a simple baseline model, researchers can use the Score test to evaluate whether adding 500 new variables would significantly improve the fit, simply by checking the gradient at the restricted baseline.[3][4]

Mathematically, the relationship between these three tests is governed by asymptotic equivalence. In infinite samples, the likelihood function becomes perfectly symmetrical and quadratic. Under these ideal conditions, the vertical drop of the LR test, the scaled horizontal distance of the Wald test, and the slope at the restriction of the Score test all converge to yield the exact same test statistic.[2][3]

But real-world data is rarely infinite. In finite samples—often defined in simulation studies as fewer than 500 observations—the likelihood curve frequently skews, causing the three tests to diverge. The Wald test tends to underestimate the true uncertainty, leading to false positives, while the Score test can sometimes be overly conservative. The Likelihood Ratio test remains the most robust, as it does not rely on quadratic approximations.[2][6]

The persistence of the Wald test as the software default is largely an artifact of 20th-century computing limits. When fitting a single logistic regression took 12 hours of mainframe time in 1985, avoiding a second model fit was a practical necessity. Today, a standard 3.2 GHz desktop processor can fit complex models in under 50 milliseconds, eroding the primary advantage of the Wald and Score tests for standard datasets.[1][7]

The choice between the Holy Trinity is therefore no longer dictated by hardware, but by the specific constraints of the data. The next frontier in this mathematical debate lies in high-dimensional machine learning, where models with billions of parameters make even a single unrestricted fit computationally ruinous, forcing researchers to develop entirely new approximations that bypass the classic trinity altogether.[7]