Why Exponentiating Log Regressions Underestimates Means and How Duan's Smearing Fixes It

The mathematical distortion that corrupts predictive models of skewed data occurs at the exact moment an analyst converts a logarithmic prediction back into original units. When a researcher exponentiates the output of a log-transformed regression, they expect to recover the arithmetic mean of the target variable. Instead, they recover the median.[4]

This silent substitution happens because of Jensen's inequality, a geometric rule dictating that the expectation of a nonlinear function is not the function of the expectation. In fields like healthcare economics, where a few extreme values drive the true average, discarding the right tail artificially erases massive amounts of projected cost.[4]

To correct this retransformation bias, statisticians rely on Duan's smearing estimator, a nonparametric method introduced in 1983. Rather than assuming the model's errors follow a perfect bell curve, the smearing estimator calculates a correction factor directly from the empirical residuals, restoring the arithmetic mean without parametric fragility.[1]

The problem begins with the shape of real-world data. Variables like medical expenditures, real estate prices, and biological concentrations rarely distribute themselves symmetrically around a central average. They are heavily right-skewed, featuring a dense cluster of low values trailing off into a long, sparse tail of extreme highs.[3]

The Geometry of Skewed Data

Ordinary least squares regression struggles with this asymmetry. The algorithm minimizes squared errors, making it highly sensitive to extreme outliers in the right tail. Furthermore, OLS assumes that the variance of the errors remains constant across all predicted values, a condition known as homoscedasticity.[3]

In skewed datasets, variance typically scales with the mean. A prediction of a $1,000 medical bill might have an error margin of $200, while a prediction of a $100,000 bill might carry an error margin of $20,000. This heteroscedasticity violates the core assumptions of linear regression, rendering standard errors and confidence intervals invalid.[3]

To stabilize the variance and pull in the extreme outliers, analysts routinely apply a natural logarithm transformation to the dependent variable before fitting the model. The regression then operates in log-space, where the relationships between variables are linear and the residuals are roughly symmetric.[2]

The model learns the conditional mean of the log-transformed outcome. If the target variable is price, the regression engine outputs the expected value of the natural logarithm of the price. To make these predictions interpretable to decision-makers, the analyst must transform the outputs back into raw dollars.[2]

The intuitive, naive approach is to simply exponentiate the predicted log value. However, exponentiating the mean of a log-transformed variable does not produce the mean of the original variable. Because the exponential function is strictly convex, the transformation fundamentally alters the statistical property being measured.[4]

Jensen's Inequality and the Retransformation Trap

Jensen's inequality formalizes this distortion. The theorem states that for any convex function, the expected value of the transformed variable is strictly greater than the transformation of the expected value. Consequently, the naive exponentiation systematically underestimates the true arithmetic mean.[1]

Instead of the arithmetic mean, the naive retransformation yields the geometric mean, which corresponds to the median of a log-normal distribution. For a business forecasting average revenue or a hospital projecting total patient costs, predicting the median rather than the mean guarantees a massive financial shortfall.[3]

Early statistical software attempted to fix this underestimation using a parametric correction. If the residuals in log-space follow a perfect normal distribution, the exact arithmetic mean can be recovered by adding half the variance to the prediction before exponentiating.[4]

This parametric adjustment relies on the mathematical properties of the log-normal distribution, where the expected value equals the exponentiated sum of the mean and half the variance. When the normality assumption holds, the correction flawlessly bridges the gap between the median and the arithmetic mean.[4]

In practice, empirical residuals rarely achieve perfect normality. If the error distribution has heavier tails than a normal distribution, the parametric correction overcompensates. Exponentiating a variance term that is inflated by a few extreme outliers results in wildly exaggerated predictions on the original scale.[4]

Duan's Nonparametric Solution

In 1983, statistician Naihua Duan published a solution in the Journal of the American Statistical Association. Working with data from the RAND Health Insurance Experiment, Duan needed a way to retransform healthcare cost predictions without relying on the fragile assumption of normally distributed errors.[1]

Duan introduced the smearing estimator, a nonparametric correction factor that bypasses theoretical distributions entirely. Instead of using the variance parameter, Duan's method leverages the actual errors the model made during training. The estimator smears the empirical residuals across the predictions to calibrate the scale.[1]

The calculation requires only a few steps. First, the analyst extracts the log-scale residuals for every observation in the training data. Next, they exponentiate each of these residuals. Finally, they calculate the simple arithmetic average of these exponentiated values.[1]

This average serves as the smearing factor. To generate a corrected prediction, the analyst multiplies the naive exponentiated prediction by this single, global multiplier. Because it uses the empirical error distribution, the smearing estimator exactly recovers the arithmetic mean regardless of the underlying distribution's shape.[1]

The magnitude of this correction is often staggering. Researchers at the University of York note that in empirical applications modeling healthcare costs, the smearing factor typically falls between 1.5 and 4.0. Ignoring the retransformation bias in these models would underestimate average costs by up to 75 percent.[3]

The Homoscedasticity Constraint

Despite its elegance, Duan's basic smearing estimator carries one strict requirement: the log-scale residuals must be homoscedastic. The global smearing factor assumes that the distribution of errors is identical across the entire feature space, applying the exact same proportional correction to every prediction.[3]

If the data exhibits heteroscedasticity in log-space, a single global factor introduces a new layer of bias. For instance, if the variance of log-costs is much higher for elderly patients than for young patients, applying an average smearing factor will over-predict costs for the young and under-predict for the elderly.[3]

To handle heteroscedastic errors, econometricians use segmented smearing. The dataset is partitioned into distinct subgroups based on the variables driving the variance. The analyst then calculates a separate smearing factor for each subgroup, applying the specific multiplier only to predictions within that segment.[3]

Segmented smearing requires sufficient sample sizes within each subgroup to calculate a stable empirical average. If the heteroscedasticity is driven by continuous variables rather than discrete categories, analysts may run a secondary regression of the exponentiated residuals against the predictors to generate a dynamic smearing factor.[3]

Two-Part Models and Machine Learning

Modern alternatives to log-transformed OLS exist, most notably Generalized Linear Models. A GLM with a log link function and a Gamma distribution models the arithmetic mean directly, avoiding retransformation bias entirely. However, GLMs can struggle to converge when the data contains extreme outliers or complex zero-inflation.[2]

Two-part models remain the standard for healthcare data, which features a massive spike of true zeros representing patients who incurred no costs. The first part uses logistic regression to predict the probability of any cost occurring, while the second part models the positive costs.[2]

When the second part of the model utilizes a log-transformed OLS regression, Duan's smearing estimator remains an essential component. The final expected cost for a patient is the predicted probability from the first part multiplied by the smear-corrected prediction from the second part.[2]

The retransformation trap extends beyond classical statistics into modern machine learning. Data scientists frequently apply log transformations to target variables before training gradient boosting machines or neural networks, seeking to prevent extreme values from dominating the loss function.[2]

These algorithms learn the conditional mean in log-space, meaning their naive exponentiated outputs suffer from the exact same metric-warped interpolation bias. Applying Duan's smearing estimator to the outputs of a random forest or a neural network restores the arithmetic mean just as effectively as it does for linear regression.[2]

The persistence of retransformation bias highlights a fundamental tension in predictive modeling. Transformations that make data mathematically tractable for algorithms inherently warp the geometry of the predictions. Correcting that distortion requires explicit, deliberate mathematical intervention.[4]

Duan's smearing estimator endures because it solves a complex geometric problem with empirical simplicity. By trusting the actual errors the model made rather than a theoretical curve, it provides a robust bridge between the mathematical convenience of log-space and the financial reality of original units.[1]