5  Forecast performance relative to two benchmark models

To assess the out-of-sample forecast performance of the indicator model, we estimated two benchmark models for comparison: (i) a univariate ARIMA (2,1,3) model of employment growth and (ii) a two-variable vector autoregression (VAR) model including employment growth and changes in the composite index of leading indicators.[10]

To evaluate the forecast performance of the indicator model and the two benchmark models, we generated out-of-sample forecasts using a fixed length, rolling window, time varying coefficient approach for the period 1996Q1 to 2001Q4. With this technique, we first estimated all three models over the period 1987Q2 to 1995Q4. The estimated coefficients from each model were used to forecast employment growth one quarter ahead. The models were then rolled forward one quarter and re-estimated. We repeated this process until the last observation was reached. This led to 24 one-quarter-ahead out-of-sample forecasts.

Out-of-sample forecasts of employment growth from the indicator model and the two benchmark models, plotted in Figure 4, all appear to follow actual employment growth quite closely. To compare the forecast performance of the three models more formally, we calculated the mean absolute forecast error (MAE), the root mean squared forecast error (RMSE) and the U-Theil for each model. The smaller the values of the MAE and the RMSE are, the more accurate, on average, the forecast of a model. The smaller the value of the U-Theil, the better the model performs compared to a naïve forecast of no change.[11] The results are reported in Table 3.

Figure 4 - Out-of-sample forecasts of employment growth

Table 3 shows that the indicator model outperforms both the VAR and ARIMA models in terms of out-of-sample forecast ability. All three evaluation criteria, the MAE, the RMSE and the U-Theil statistic, are lower for the indicator model than for the two benchmark models. The VAR model outperforms the ARIMA model in terms of the RMSE and U-Theil. The composite index thus appears to be particularly useful in forecasting employment changes.

* in thousands

Finally, Table 4 reports the 2x2 confusion matrices for the three models. The confusion matrix records the number of times a model correctly predicted the direction of next period employment growth out of sample.[12] The upper diagonal (upper-left) element records the number of times a model correctly predicted an increase in employment growth, while the lower diagonal (lower-right) element reports how often the model correctly forecast a decrease. For example, the indicator model correctly predicted fourteen rises and five declines in employment growth. The off-diagonals report the number of times a model missed the direction of employment changes. The lower-left (upper-right) off-diagonal element records the number of actual moves in employment growth that were up (down) while the predicted changes were down (up).

The results in Table 4 suggest that the indicator model and the VAR model forecast the direction of next period employment growth reasonably well. The indicator and VAR models have five false signals each, which is better than the ARIMA with nine false signals. The indicator and VAR models correctly forecast the direction of employment almost 80 percent of the time compared to about 63 percent for the ARIMA. Both the indicator and VAR models predicted employment growth to increase when it actually fell more often than they predicted employment growth to fall when it actually increased. The out-of-sample forecast performance of the indicator and VAR models suggests that the composite index of leading indicators is a good predictor of employment changes.