Issues in the Design of Multi-Model Ensembles
An ensemble prediction system needs to sample all significant sources of forecast error. Uncertainties in the initial conditions - related to the limitations of the observational network - and in the formulation of the analysis and forecasting system itself have an impact on forecast quality. Due to a limited understanding of the sources of error, it is difficult to sample all sources of error in a satisfactory manner in a single ensemble prediction system. In such systems, one thus needs ad-hoc procedures to arrive at an ensemble spread that is representative of the error of the ensemble mean. The combination of similar quality ensembles from different centers, which use somewhat different components of the observational network, different data-assimilation procedures and different forecast models, leads to an improved sampling of the uncertainties in the forecasting system. As a consequence, a multi-center ensemble is naturally more reliable than a single-center ensemble. Since we combine ensembles with fairly independent error, the statistical resolution of the combined ensemble will also be superior. These assertions will be demonstrated with results from the North American Ensemble Forecast System (NAEFS). The statistical calibration of the bias and dispersion of ensembles will lead to improved reliability only. Given a sufficiently long hindcast data set,a stable observing system and a stable climate, one can in principle arrive at a perfect calibration of an ensemble. Calibration is particularly important for single-ensemble systems. For multi-center ensembles, with frequent changes in the participating ensemble systems, a proper and beneficial calibration of the ensemble is non-trivial especially for longer-range forecasts. For bias correction, this will also be illustrated with results from the NAEFS. Having more than two participating ensemble systems of similar quality, we will eventually arrive at high-quality reliable ensembles. As more centers contribute, they should - to ease resource problems - perhaps each contribute less members. This will likely shift research and development work towards the development of products for specific user groups.
Correction of Atmospheric Dynamical Seasonal Forecasts Using the Leading Ocean-forced Spatial Patterns
Atmospheric variability forced by slowly-varying boundary conditions contains information useful for seasonal predictions at sufficiently long lead times. Unfortunately, the spatial structure of the forced signal is usually model dependent. A statistical approach to correct the ensemble forecasts is formulated based on the regression of GCM's leading forced spatial patterns and the observed anomalies. This technique is applied to the winter forecasts of four Canadian models. The performance of the corrected forecasts is assessed by comparing its cross-validated skill with that of the original GCM ensemble mean forecasts. This postprocessing technique is able to improve the forecast skill of two major atmospheric patterns, i.e., the Pacific/North American (PNA) pattern and the North Atlantic Oscillation (NAO), as well as the Canadian winter precipitation. The corrected forecasts predict the Canadian winter precipitation with statistically significant skill over the southern prairies and a large area of Quebec-Ontario, comparing to almost no skill in the uncalibrated GCM ensemble seasonal predictions.
Status and Upgrade of NAEFS and NCEP Global Ensemble Forecast System
NAEFS currently combines bias corrected NCEP and CMC global ensemble forecasts, every 6 hours out to 384 hours, twice per day from 00UTC and 12UTC, and totally more than 40 members at each initial time. The main products include bias corrected probabilistic (10%, 50%, 90%) forecasts, ensemble mean, mode, spread forecasts, and anomaly forecasts as well. For CONUS, there are downscaled probabilistic forecasts at 5km NDFD resolution for near surface variables. New NCEP Global Ensemble Forecast System (GEFS) has been planned to implement in 2009. This implementation will expect to improve NCEP GEFS model forecasts and the quality of NAEFS (The North American Ensemble Forecast System) products. The NAEFS is operationally joined multi-model ensemble forecast system since 2006, which combined NCEP and CMC's ensemble forecasts after post process, such as bias correction and downscaling. This upgrade mainly includes following changes: 1). Increasing horizontal resolution from T126 to T190 out to 384 hours. Increasing model resolution will expect to have better forecasts for first 3-5 days. 2). Using 8th order horizontal diffusion instead of 4th order of current operation. Experiments demonstrated the higher order diffusion will promise better ensemble forecast and ensemble spread which allows waves interaction and propagation in reality. 3). Adding stochastic perturbation scheme to account for random model errors. This process will increase ensemble spread and reduce RMS errors, especially for longer forecast. Overall, NCEP new GEFS will enhance our model forecasts and NAEFS products significantly according to two 2-month retrospective experiments which are November- December 2007, and August-September 2008.
The Canadian seasonal forecast and the APCC exchange.
In this talk, we will first describe the Canadian seasonal forecast system. This system uses a 4 model ensemble approach with each of these models generating a 10 members ensemble. Multi-model issues related to this system will be describes. Secondly, we will describe an international multi-system initiative. The Asia-Pacific Economic Cooperation (APEC) is a forum for 21 Pacific Rim countries or regions including Canada. The APEC Climate Center (APCC) provides seasonal forecasts to their regional climate centers with a Multi Model Ensemble (MME) approach. The APCC MME is based on 13 ensemble prediction systems from different institutions including MSC(Canada), NCEP(USA), COLA(USA), KMA(Korea), JMA(Japan), BOM(Australia) and others. In this presentation, we will describe the basics of this international cooperation.
North American Ensemble Forecasting System (NAEFS): Bias Removal and Multi-Model Ensemble
The North American Forecasting System (NAEFS) currently combines the U.S. National Weather Service (NWS) and Meteorological Service of Canada (MSC) ensemble forecast systems. Previous studies show clear benefits to merging the two ensembles. A running mean bias removal scheme is applied on the NAEFS ensemble for a number of variables and pressure levels. Here we present verification results against upper air observations comparing the NAEFS, NWS and MSC raw ensemble forecasts with the respective bias corrected version of each. The bias, dispersion and Continuous Ranked Probability Scores (CRPS) of both raw and bias corrected systems are compared. A bootstrap method is used to compute 90% confidence intervals on the differences in scores for both systems. It is found that the bias removal scheme improves the bias and the CRPS of both the NWS and MSC ensemble systems when compared to upper air observations. However the effect on the combined NAEFS system is relatively small in CRPS term. In fact, it is found that the raw NAEFS combined forecasts (40 members) have similar scores than the bias corrected ones. This may highlight the limits of the current simple bias removal framework and/or the efficiency of the multi-model approach.
Seasonal Probability of Precipitation Forecasts Using a Weighted Ensemble Approach
A weighted ensemble (WE) method is revisited and employed to issue an improved seasonal probability of precipitation (POP) forecast. Nine boreal summer time seasonal precipitation hindcasts obtained from the APEC (Asia-Pacific Economic Cooperation) Climate Center (APCC) multi-model ensemble system are used to assess the suitability of the WE approach for seasonal POP predictions. Due to its performance-based selective nature for assigning weights, the WE method produced marginally superior seasonal POP forecasts compared to the conventional approach.
Probabilistic Week-Two Forecasts of Temperature and Precipitation Using a Multi-Model Ensemble System
Experimental temperature and precipitation forecasts for week-two were developed and are run operationally at
the Climate Prediction Center (CPC) of the US National Centers for Environmental Prediction (NCEP). This is
a collaborative project with the NCEP Environmental Modeling Center, Environment Canada's Canadian
Meteorological Centre and the National Meteorological Service of Mexico. The temperature and precipitation
forecasts from the NCEP Global Ensemble Forecast System (GEFS) model and the Canadian Meteorological
Centre ensemble model are being combined to produce a single probability density function, which can be
used to assess the probability of exceeding various climatology thresholds, including extremes. Lead-
dependent, model-dependent bias corrections are calculated for the models from the two centers prior to
making the seven-day mean forecasts. In addition, corrections can be made to the derived multi-model
ensemble probability density function for week-two, accounting for forecast skill and ensemble spread.
Verification of the forecasts indicates increases in the reliability of forecasts after the correction of the
probability density function (PDF), especially for the tails of the distribution. This is an indication that though
bias corrections reduce the errors of the ensemble mean, they cannot correct errors in the ensemble spread.
Therefore, PDF correction produces little change in forecast skill for the center of the distribution, but significant
improvement in the skill of the tails of the distribution. Though the skill of the forecasts is similar for each
ensemble model from the two centers, an equal-weighted combination of multiple models should improve
forecasts by improving the ensemble coverage of possible short-term climate fluctuations.