Model Skill

Model Skill

Time series (1981-2006) of Anomaly Correlation of ECMWF 500 hPa Height Forecasts

Once a climate model is tuned and running, it can be tested and evaluated, much as weather models are. In both cases, model results are compared to observations. And, in both cases, model skill has improved significantly over the past thirty years.

Looking first at weather models, this figure from the European Centre for Medium-Range Forecasts (ECMWF) shows forecast skill for their medium-range NWP model since 1981. It depicts skill (measured by the anomaly correlation) of the 500-mb height forecast for 3, 5, 7, and 10-days in advance. The top line in each color band is skill in the Northern Hemisphere, and the bottom line is skill in the Southern Hemisphere.

Two things are immediately apparent in the graph:

  1. Model skill has increased over the years. For example, if you look at a 5-day forecast, model skill has improved from about 0.60 to about 0.87 (where 1.0 is perfect correlation) over 30 years.
  2. The model has until recently, been much more skillful in the Northern Hemisphere.

This difference in model skill resulted from better initial conditions for weather forecasts in the Northern Hemisphere than the Southern Hemisphere. Until recently, there were more observations in the Northern Hemisphere. Now, satellite data is optimized and the initial conditions have the same quality.

Note that while forecast skill has improved over the last 30 years, the trend has flattened out since about 2003. This could be due to a variety of reasons, including uncertainties in the initial conditions, parameterization biases, and inherent weather predictability issues resulting from internal atmospheric dynamics (i.e., chaos).

Climate Skill Score for Each Version of CCM and CAM, Based on NMSE [normalized mean square error, right] and SVR [scaled variance ratio, left] for the 200-mb Height Field

Climate models have skill scores similar to those for weather models.

This graphic shows a skill score for successive generations of the atmospheric model component at NCAR over the past 30 years. It is based on the 200-mb height field for the Northern Hemisphere and calculated as one (1) minus the mean square error normalized to the variance of the analyzed observations. The data show that skill has climbed steadily from the very low skill score of the original Community Climate Model (CCM0) in the early 1980s. But, similar to weather forecast models, the skill score for climate models also seems to be flattening out. It is possible that we may reaching some kind of limit on how skillful these kinds of models can be.