Why Multi-Step Forecasts Require Diebold-Mariano HAC Adjustments to Prevent False Model Comparisons

Data scientists and econometricians deciding which forecasting model to deploy into production face a strict mathematical boundary when evaluating out-of-sample performance. When they backtest a model that predicts multiple steps into the future, the errors those models produce are never independent. A failure to account for this dependence guarantees that random noise will masquerade as a statistically significant improvement.

The standard procedure for comparing two models is to calculate a loss differential, typically the difference in their squared or absolute errors. If Model A has a lower mean squared error than Model B, the forecaster must determine if that gap is real or merely a byproduct of the specific test sample.

For a one-step-ahead forecast, this is straightforward. Assuming the models are well-specified, the errors from predicting a single period ahead are largely uncorrelated. A standard paired T-test can reliably evaluate whether the mean loss differential is statistically distinguishable from zero.

The architecture of the problem changes entirely the moment the forecast horizon extends beyond a single period. If a central bank predicts inflation four quarters ahead, the forecast made in the first quarter shares unknown future shocks with the forecasts made in the second and third quarters.

This overlap creates a mechanical dependence in the data. The shock that occurs in the fourth quarter was invisible to all the multi-step forecasts made leading up to it, meaning they will all miss the target in a correlated direction.

The Mechanics of MA(h-1) Autocorrelation

The mathematical consequence of this overlapping information set is a specific form of serial correlation. "The optimal h-step-ahead forecast errors exhibit serial correlation up to lag (h-1) under a general non-stationary autoregressive data generating process," notes a 2026 preprint by Xiaoqian Wang at Monash University.[1]

This means that if a model is forecasting six months into the future, its errors will inherently follow a Moving Average process of order five, or MA(5). The error recorded today is mathematically tethered to the errors recorded over the previous five periods.

This correlation structure exists even if the forecasting model is perfectly specified and optimal. It is not a symptom of a bad model, but rather a fundamental property of predicting across an interval where new information continuously arrives but cannot be incorporated into past predictions.

When practitioners ignore this property, the statistical machinery breaks down. A naive T-test assumes that every observation in the sample provides independent information, dividing the sample variance by the square root of the total number of observations to compute the standard error.

Because the multi-step errors are positively correlated, the effective sample size is drastically smaller than the raw observation count. The naive calculation severely underestimates the true variance of the loss differential, producing an artificially tiny standard error.

Why Naive T-Tests Fail

Dividing the mean loss differential by this artificially small standard error inflates the resulting T-statistic. A difference in model performance that is entirely driven by chance will suddenly appear highly significant, crossing the standard threshold of 1.96 with ease.

"Without HAC errors, autocorrelated forecast errors inflate significance," explains a 2025 technical guide from MetricGate. The guide notes that for a one-step horizon, loss differentials are serially uncorrelated under the null hypothesis, but for any horizon greater than one, the overlap-induced autocorrelation must be corrected.[2]

The magnitude of this distortion is severe. In a sample of 300 observations with strong serial correlation, an ordinary least squares T-statistic of 4.81 can drop to 2.14 once the dependence is properly accounted for, according to an analysis by Ryan O'Connell.[6]

Without the correction, a researcher might overstate the precision of their estimate by more than a factor of two. This leads directly to false discoveries, where a complex, computationally expensive model is chosen over a simple baseline despite offering no genuine predictive advantage.[6]

The illusion of predictive superiority is particularly dangerous in financial markets and macroeconomic policy, where deploying an overfit model can trigger catastrophic capital misallocation. The forecaster believes they have found a signal, when in reality they have only found correlated noise.

The Diebold-Mariano Framework

The definitive solution to this problem was introduced in 1995 by econometricians Francis Diebold and Roberto Mariano. Their framework, now known as the Diebold-Mariano test, explicitly accommodates the autocorrelation structure inherent in multi-step forecast comparisons.

The test evaluates the null hypothesis that the expected loss differential between two competing forecasts is exactly zero. It makes no assumptions about whether the models are nested, and it supports highly flexible, asymmetric loss functions.[2]

"If the null hypothesis of equal predictive accuracy is true, the forecast errors from the competing models are exactly the same and perfectly correlated," notes a lecture series by Umberto Triacca at SSSUP. The test accounts for this by replacing the naive variance with a long-run variance estimator.

This long-run variance incorporates not just the variance of the individual loss differentials, but also their autocovariances up to the relevant lag. By adding these covariance terms back into the denominator, the test shrinks the T-statistic back to its proper, mathematically justified size.

The test statistic is then compared against a standard normal distribution, provided the sample size is sufficiently large. If the absolute value of the statistic exceeds the critical value, the forecaster can legitimately conclude that one model outperforms the other.

Newey-West and HAC Adjustments

To actually compute the long-run variance required by the Diebold-Mariano test, practitioners rely on Heteroskedasticity and Autocorrelation Consistent standard errors. The most widely used estimator was developed by Whitney Newey and Kenneth West in 1987.

"Instead of estimating the variance of the sample mean using only the variance of individual returns, Newey-West also considers covariance between returns separated by different lags," explains a 2026 analysis by Quant Atlas.[3]

The Newey-West estimator applies a Bartlett kernel, which assigns declining weights to the autocovariances as the lag distance increases. Correlations at nearby lags receive greater importance in the calculation than correlations further into the past, reflecting the decaying nature of the overlap.[3]

This adjustment is completely agnostic to the specific form of the error process. It is asymptotically valid under any form of serial correlation and heteroskedasticity, making it highly robust when the exact structure of the forecasting errors is unknown.[6]

Crucially, the adjustment only changes the standard errors. It uses the original loss differential estimates, requiring no re-estimation of the underlying forecasting models. It simply provides a wider, more honest confidence interval for the comparison.[6]

Bandwidth Selection and Finite Samples

Implementing the Newey-West correction requires the forecaster to select a bandwidth parameter, which dictates how many lagged autocovariances to include in the sum. For an h-step-ahead forecast, the theoretical autocorrelation strictly cuts off at lag h-1.

In practice, however, researchers often use slightly larger bandwidths to ensure the variance matrix remains positive semi-definite in finite samples. A common heuristic is to set the bandwidth to 1.5 times the horizon, or to use a data-driven rule based on the sample size.[5]

Even with corrected standard errors, the Diebold-Mariano test can over-reject the null hypothesis when the out-of-sample evaluation period is short. The reliance on asymptotic normality breaks down when the sample size drops below roughly 100 observations.

To address this, Harvey, Leybourne, and Newbold proposed a modified test in 1997. The correction adjusts the statistic to account for the finite sample bias and compares the result against a Student-t distribution with n-1 degrees of freedom rather than a standard normal distribution.

Alternative approaches have also been developed to handle severe distortions. "The Kiefer-Vogelsang approach also uses a 't test,' with possibly many autocorrelations included in the HAC variance estimate, but the limiting distribution is not Normal," notes Andrew Patton in a Duke University paper.[4]

Implications for Modern Machine Learning

Despite decades of econometric consensus, this property remains a massive blind spot in modern machine learning. Data scientists frequently compare deep learning forecasting models using raw Root Mean Squared Error metrics, declaring victory over baselines without any formal statistical testing.

When tests are applied, they are almost universally naive paired T-tests or standard cross-validation routines that assume independent folds. "No conformal prediction methods have taken this potential dependence into account in their methodological construction," Wang observes regarding the distribution-free inference literature.[1]

Ignoring this framework leads directly to the replication crisis currently plaguing time-series machine learning. Models that appear to achieve state-of-the-art performance on benchmark datasets frequently fail in production because their apparent superiority was merely an artifact of uncorrected autocorrelation.

The mathematical reality is unavoidable. Until the evaluation pipeline explicitly adjusts for the overlapping information sets inherent in multi-step predictions, any claim of predictive supremacy remains statistically hollow. The true test of a model is not its raw error rate, but its performance scaled by the honest, adjusted variance of its environment.