Evidence Pack: The Accuracy and Limits of AI Weather Forecasting Models

Every six hours, meteorologists at national forecasting agencies face a choice that dictates evacuation orders, flight groundings, and agricultural planning. They must decide whether to issue severe weather warnings based on the latest model runs. Historically, that decision relied entirely on physics-based numerical weather prediction (NWP) models running on massive supercomputers. But when the next operational cycle initializes at 00Z or 12Z, forecasters at the European Centre for Medium-Range Weather Forecasts (ECMWF) and the World Meteorological Organization (WMO) will increasingly weigh a fundamentally different input: forecasts generated in seconds by artificial intelligence. This marks a structural shift in how humanity anticipates the atmosphere, moving from solving equations to recognizing patterns.[3][6]

To understand why this shift is happening, one must look at how traditional forecasting works. NWP models like the ECMWF's High Resolution (HRES) system or the American Global Forecast System (GFS) divide the atmosphere into a three-dimensional grid. They use complex differential equations—fluid dynamics and thermodynamics—to calculate how air, heat, and moisture move from one grid box to the next. This requires immense computational power; a single 10-day global forecast takes hours to run on a supercomputer, severely limiting how many scenarios an agency can simulate before a storm makes landfall. The physics are rigorous, but the computational bottleneck is absolute.[1][6]

AI weather models, such as Google DeepMind’s GraphCast, Huawei’s Pangu-Weather, and ECMWF’s own Artificial Intelligence Forecasting System (AIFS), abandon the physics equations entirely. Instead, they use deep learning architectures—like Graph Neural Networks (GNNs) or 3D Vision Transformers—trained on decades of historical weather data. The models learn the statistical relationships between atmospheric states. When fed the current weather conditions, they recognize the pattern and instantly output the next state. They do not calculate how the wind will blow; they remember how it blew the last time the atmosphere looked exactly like this.[1][2][3]

The mechanism is purely data-driven. Pangu-Weather, for instance, formulates atmospheric pressure levels into cubic data and applies a hierarchical temporal aggregation algorithm to predict the weather in steps. Because it does not calculate physics equations at runtime, the inference cost is negligible. Pangu-Weather can generate a 24-hour global forecast in 1.4 seconds on a single NVIDIA Tesla-V100 GPU—more than 10,000 times faster than the operational IFS. This speed unlocks the ability to run massive ensembles on standard hardware, democratizing access to high-quality meteorological guidance.[1]

Speed alone would be a novelty if the accuracy were poor, but the data shows the opposite. In deterministic forecasts, AI models are now beating the best physics-based systems on standard meteorological metrics. The benchmark metric for medium-range forecasting is the Root Mean Square Error (RMSE) of the 500 hPa geopotential height (Z500)—essentially, how accurately the model predicts the pressure patterns that steer global weather systems. Lower numbers indicate a forecast that more closely matches reality.[1][2]

According to peer-reviewed benchmarks published in Nature, the traditional ECMWF HRES model carries a 5-day Z500 RMSE of 333.7 m²/s². Pangu-Weather reduces that error to 296.7 m²/s². This represents an 11.1% reduction in error compared to the world's leading physics model, and a 35.8% improvement over earlier AI iterations like FourCastNet, which scored 462.5 m²/s². The models are not just matching the legacy systems; they are actively widening the performance gap at medium-range lead times.[1][5][6]

GraphCast demonstrated similar dominance. In a head-to-head benchmark published in Science in December 2023, the DeepMind model outperformed the ECMWF HRES on 90% of 1,380 verification targets. It proved particularly adept at predicting upper-level wind and geopotential height at 5-to-10 day lead times. Running on a single Google TPU v4, it produced a complete 10-day global forecast in under one minute, predicting the track of severe storms days before traditional models locked onto the threat.[2]

This leap in accuracy and speed fundamentally changes the economics and accessibility of early warning systems. When a model runs in seconds on a single GPU rather than hours on a supercomputer, meteorological agencies can generate massive ensembles—running the model hundreds of times with slight variations in the initial conditions to calculate the exact probability of a hurricane track or a heatwave. This translates directly into earlier, more confident warnings for communities in the path of extreme weather, shifting the operational focus from computing the forecast to communicating the risk.[4][6]

Yet, the mechanism that makes AI models so fast also introduces their primary limitation: unphysical smoothing. Because they optimize for average error across a massive dataset, AI models tend to blur their predictions when uncertain. A 2026 WMO intercomparison project noted that while AI systems generate highly accurate 10-day forecasts, they systematically underestimate the intensity of record-breaking extreme events—like unprecedented rainfall or localized severe storms—that fall outside their historical training data. When the atmosphere does something it has never done before, the AI struggles to predict it.[6]

The evidence remains thin on fine-scale surface variables. While AI models excel at broad atmospheric patterns, they can struggle with the localized physics of wind and precipitation. For example, while Pangu-Weather's temperature forecasts exceed traditional models, independent evaluations have found that its performance on specific surface wind speeds can sometimes lag slightly behind the ECMWF baseline. The AI models know what the weather usually does, but they do not inherently know the laws of physics that govern what the weather must do.[1][6]

To bridge this gap, the next frontier is hybrid forecasting. The WMO's Weather Prediction Model Intercomparison Project (WP-MIP), initiated in late 2024, is currently assessing how to combine AI guides with physically based models. By using AI to rapidly generate the broad atmospheric patterns and traditional NWP to resolve the fine-scale physics of extreme events, agencies hope to capture the benefits of both paradigms. This approach aims to eliminate the unphysical smoothing of pure AI while bypassing the computational bottlenecks of pure physics.[6]

ECMWF has already moved its AIFS model to operational status, publishing its outputs alongside traditional forecasts. Forecasters now compare the AI's rapid pattern recognition against the physics model's rigorous calculations. As Tim Palmer, a WMO IMO medallist, noted regarding the shift, "I believe that in the future, weather forecasting will be a mixture of data-driven AI and physics-based numerical weather prediction." As national agencies continue to integrate these tools, the decision of whether to issue a warning will increasingly rely on a consensus between human expertise, fluid dynamics, and artificial intelligence.[3][6]