The Treasury

Global Navigation

Personal tools

Treasury
Publication

Treasury's Forecasting Performance: A Head-to-Head Comparison - WP 06/10

3.2  Forecast performance for Budget current year

Another useful evaluation of Treasury’s relative forecast performance is to focus on the current year Budget forecasts, since getting an accurate picture of the current economic outlook is a critical input for Budget decision-making and fiscal strategy. Figure 3 presents the average relative rank and Figure 4 presents the RMSE for each forecaster’s GDP and CPI current year forecasts made in April/May to coincide with when Treasury typically finalises its Budget forecasts. Once again, the Consensus performed well on both measures and was in the lower left areas of Figures 3 and 4. Treasury’s forecast performance showed an improvement on both GDP and CPI for current year only forecasts.

Table 5 presents the average relative rank and RMSE for each forecaster. The Mean once again had the best performance for GDP on an average relative rank basis, with the Consensus coming in second. Treasury’s ranking improved to fourth placing on the average relative rank basis, but came in second on the RMSE measure. Major forecasting groups again were outperformed by private sector forecasters on the average relative rank basis, but are comparable on the RMSE measure. For CPI, Treasury was sixth on the average relative rank basis and ninth on the RMSE basis, better than the 10th placing when evaluated across all periods.

Figure 3 – Average relative rank for Budget current year forecast
Figure 3 – Average relative rank for Budget current year forecast.
Figure 4 – RMSE for Budget current year forecast
Figure 4 – RMSE for Budget current year forecast.

Table 5 – Average RMSE and relative rank for Budget current year forecast

Table 5 – Average RMSE and relative rank for Budget current year forecast.

The improvement in Treasury’s forecast performance for the Budget current year compared to all forecast horizons indicates that Treasury is better at current year forecasts, and underperforms for year ahead forecasts. In the case of GDP, there was a large improvement in forecast performance from seventh to fourth on the average relative rank, and 13th placing to second on the RMSE measure. The big improvement on the RMSE measure was due to the exclusion of the large forecast errors in the year ahead forecasts relating to the 1998 recession. Treasury’s CPI forecast performance also improved but remains in the middle (sixth out of 12) or the bottom half of all forecasters.

3.3  No one consistently outperforms the Consensus

Similar to the findings of the international studies cited in the introduction, no individual forecaster in New Zealand consistently outperforms the Consensus (or the Mean) for both GDP and CPI. Forecaster 5 had the highest percentage of out-performance for GDP beating the Mean 44% of the time, yet only placed fifth overall on the average relative rank basis. For CPI, Forecaster 6 had a higher rate of out-performance beating the Consensus 47% of the time, putting it in second place overall on the average relative rank basis. Forecasting Group 2, the worst performer for GDP forecasts, only managed to beat the Mean 18% of the time. Forecasting Group 1, which had the worst performance for CPI forecasts, beat the Consensus 23% of the time. Treasury managed to beat the Mean 30% of the time for GDP, and Consensus 33% of the time for CPI.

Figure 5 – Frequency of GDP relative ranking over all evaluation periods
Figure 5 – Frequency of GDP relative ranking over all evaluation periods.

Figures 5 and 6 present the frequency of each forecaster’s relative ranking over all evaluation periods based on either a top 3, middle or bottom 3 placing. Despite the Mean having the best overall performance for GDP, three other forecasters had more top 3 placings. Because the Mean helps smooth out the extreme forecasts, it provides a more reliable and stable forecast performance which is the main reason for its overall top position. Forecasting Group 5 had the most top 3 placings for GDP, yet placed 12th on the average relative rank and 11th on the RMSE basis.

For CPI, Forecaster 4 had the highest number of top 3 placings, even though they came in sixth overall based on the average relative rank. The Consensus had the best overall performance even though over half of the individual forecasters had more top 3 placings. This highlights the importance of consistency in forecast performance.

Treasury had a top 3 placing 11 times out of 40 for GDP forecasts, and 14 times for CPI forecasts. Treasury’s overall forecast performance for CPI was dragged down by a high frequency of bottom 3 placings (10 times).

Figure 6 – Frequency of CPI relative ranking over all evaluation periods
Figure 6 – Frequency of CPI relative ranking over all evaluation periods.

3.4  Forecasting turning points is difficult

Despite the importance of being able to predict turning points and in particular recessions at an early stage, overseas studies find that the ability of forecasters to predict recessions is poor. For example, Loungani (2001) evaluated the performance of Consensus Forecasts of real GDP growth for a large number of industrialised and developing countries for the time period 1989 to 1998 (calendar years). Only two of the 60 episodes of recessions (defined as any calendar year in which real GDP declined) that occurred over the sample were predicted a year in advance, two-thirds remained undetected by April of the year in which the recession occurred, and in about a quarter of the cases the forecast in October was still for positive growth (albeit small). Loungani notes that the predictive failure could arise either because forecasters lack the requisite information (in terms of reliable real-time data or reliable models) or because they lack the incentives to predict recessions.

The only recession period in this study was 1998, and consistent with Loungani’s finding, none of the individual forecasters picked the recession even only a few months out. Figure 7 presents the GDP forecast revisions for calendar 1998. Even by April of that year, most forecasters were expecting reasonable growth with a range of 1.6% - 3.5%, with the Consensus at 2.3%. Treasury at that stage was forecasting growth of 2.4%, close to the Consensus but a large downward revision from the previous forecast of 3.9%. Although subsequent forecasts were revised downwards, it was not until October of that year that most forecasters were predicting a recession. In fact, by August, only 2 of the forecasters covered by the Consensus Forecasts were picking a recession. Treasury at that stage, in a one-off forecast update in August, had revised down its forecast to -0.5%. In the September edition of Consensus Forecasts, almost half of the 13 forecasters surveyed were predicting a recession, but the Consensus remained positive at 0.3%.

Figure 7 – GDP forecast revisions for 1998

Forecasters find it equally difficult to predict the peak of the economic cycle. The peak of GDP growth over the evaluation period was in 2004 at 4.8%. Twenty months out, the range of forecasts was 2.1% - 3.6%, with the Consensus at 2.7%. Fourteen months out, the forecast range was actually revised downwards to 1.9% - 3.2%, although the Consensus remained at 2.7%. By April 2004, the Consensus was revised up to 3%, but the forecast range of 2.5% - 3.6% still did not encompass the actual outturn. It was not until July that the forecast range included the actual outturn.

Page top