Metropolis-Hastings Cancels Impossible Denominators to Sample Posterior Distributions

Rejection sampling can map a complex probability distribution by drawing random points inside a bounding box and discarding those that fall outside the target curve. But as the number of dimensions grows, the bounding box becomes mostly empty space, and rejection sampling discards almost everything it draws.

The Metropolis-Hastings algorithm solves this dimensional curse by abandoning the bounding box entirely. Instead of drawing independent points, it builds a Markov chain—a sequence where each new sample depends only on the current one, allowing the algorithm to wander toward high-probability regions and stay there.[8]

The core problem this algorithm solves lies at the heart of Bayesian statistics. To update beliefs based on new data, statisticians must calculate a posterior distribution, which requires dividing by a marginal likelihood. For models with more than three or four parameters, calculating that denominator requires an integral over all possible parameter values—a mathematical operation that is analytically impossible.[7]

In 1953, physicist Nicholas Metropolis introduced a workaround for symmetric distributions, which statistician W.K. Hastings generalized in 1970. They proved that the impossible denominator is actually irrelevant if you only care about the relative probability of two points. Because the denominator is a constant across the entire distribution, it cancels out when you divide the probability of one state by the probability of another.[3][8]

As data scientist Brian Keng notes in his analysis of the method, Metropolis-Hastings is "a method that allows us to sample from any probability distribution, provided we can compute the value of a function that is proportional to the density of the target distribution."[8]

The algorithm begins by selecting an arbitrary starting point in the parameter space. From there, it uses a "proposal distribution"—often a standard Gaussian bell curve centered on the current location—to suggest a random step to a new parameter value.[1][4]

Once a new state is proposed, the algorithm calculates the acceptance ratio. This metric divides the unnormalized probability of the proposed state by the unnormalized probability of the current state, factoring in any asymmetry in the proposal mechanism.[5]

If the acceptance ratio is greater than 1.0, the proposed state is strictly more probable than the current one. The algorithm accepts the move immediately, stepping uphill toward the denser regions of the target distribution.[6]

The mathematical brilliance of Metropolis-Hastings lies in how it handles proposed steps that go downhill. If the acceptance ratio is less than 1.0—say, 0.3—the algorithm does not automatically reject the move. Instead, it draws a random number between 0.0 and 1.0 from a uniform distribution.[2][6]

If the random draw is lower than the acceptance ratio (0.3), the algorithm accepts the inferior step. If the draw is higher, it rejects the step, stays in its current position, and records that current position as the next sample in the chain.[1][4]

This probabilistic filter prevents the Markov chain from getting permanently trapped on a single local peak. By occasionally accepting downhill moves, the algorithm explores the entire parameter space. Over 10,000 or more iterations, the frequency with which the chain visits any specific region becomes perfectly proportional to that region's true underlying probability.[5][7]

The efficiency of this exploration depends entirely on the variance of the proposal distribution. If the variance is set too small, the proposed steps are tiny. The algorithm will accept nearly 100 percent of them, but it will take an impractically long time to traverse a wide distribution, a failure mode known as high autocorrelation.[3][8]

Conversely, if the proposal variance is too large, the algorithm frequently suggests wild leaps into low-probability wastelands. The acceptance ratio for these leaps approaches zero, causing the algorithm to reject almost everything and freeze in place, recording the same state hundreds of times in a row.[1][7]

Finding the mathematical sweet spot requires tuning the acceptance rate. For a single-dimension target distribution, theoretical results show the optimal proposal variance achieves an acceptance rate of roughly 0.44.[3]

As the number of dimensions increases, that ideal threshold drops. For high-dimensional models with 20 or more parameters, the optimal acceptance rate converges to exactly 0.234. Hitting this specific fraction ensures the chain takes steps large enough to explore, but conservative enough to avoid constant rejection.[3][7]

Even with perfect tuning, the initial samples in the chain are heavily biased by the arbitrary starting point. Practitioners typically discard the first 1,000 to 5,000 iterations—often up to 50 percent of the total run—as a "burn-in" period, using only the subsequent samples for their final analysis.[2][4]

The primary limitation of Metropolis-Hastings emerges in highly correlated or multimodal distributions. If a target distribution features two isolated probability peaks separated by a vast, near-zero probability valley, the algorithm struggles to cross the gap.[2][6]

Because the algorithm relies on a random walk, the probability of proposing a sequence of steps that successfully navigates a deep valley is vanishingly small. The chain may spend its entire run mapping only one peak, leaving the researcher entirely unaware that the other exists.[5][8]

When faced with these pathological geometries, modern statisticians often abandon the random walk entirely. Instead, they turn to advanced variants like Hamiltonian Monte Carlo, which calculates the gradient of the probability space to slide efficiently toward high-density regions without relying on blind proposals.[7]