Staff and teams are writing in their individual capacity and the views are not necessarily a "Treasury" view. Please read our disclaimer.

The Treasury regularly carries out an assessment of the accuracy of our biannual forecasts: the Budget Economic and Fiscal Update (BEFU) and the Half Year Economic and Fiscal Update (HYEFU). These forecasts underpin the Treasury’s fiscal advice and the Government’s fiscal outlook, as well as outlining the economic context in which Ministers make decisions. Therefore, it is desirable that these forecasts are as accurate as possible.

Broadly speaking, a “good” forecast consists of two features. First, the forecast should be unbiased: the forecasts do not tend to systematically over or under-predict economic outturns. A set of forecasts is said to be unbiased when the average of forecast errors, defined as the differences between the actual outturns and the forecast values, is zero.

The second criterion is that the deviation of the forecast errors from their mean is small. This way, one can be confident that even if a forecast error arises, it would be within a plausible range. To measure this deviation of forecast errors, we use the root mean square errors (RMSE).[1]

We apply these two criteria to three variables of interest, namely, the average annual percentage change in real GDP growth, CPI inflation, and the average percentage change in gross tax revenues. The sample period is 1990 – 2017.

We assess the accuracy from multiple aspects: over time, relative to other forecasters, and across different forecast horizons.[2]

In terms of unbiasedness, there is not enough evidence to conclude that the Treasury has consistently over- or under-predicted the variables evaluated in this Special Topic. This result holds across forecast horizons ranging from 1 quarter before the actual outturn, to up to 20 quarters. For example, Figure 1 shows the 95% confidence interval around forecast errors for real GDP growth, i.e. we are 95% certain that the true forecast error at each forecast horizon lies within the blue band. Since the confidence interval includes zero at each forecast horizon, we cannot be certain that the true error is non-zero, so we cannot conclude that the forecast errors are biased.

Figure 1: Mean error of real GDP growth forecasts and 95% confidence bands

Figure 1: Mean error of real GDP growth forecasts and 95% confidence bands

Source: Treasury calculations

In terms of forecast performance over time, we find that the Treasury’s forecasts compare favourably to other forecasts available at the time the Treasury’s forecasts were finalised in most years between 2000 and 2017 (Figure 2). Moreover, since 2013, there has been a steady and marginal improvement in forecast accuracy in GDP growth and inflation. These results might be due to several factors: the absence of large surprises in recent years, the informational advantage Treasury holds over other forecasters, as well as improved forecasting experience over time.

Figure 2: One-year ahead forecast errors in GDP growth

Figure 2: One-year ahead forecast errors in GDP growth

Comparisons of forecasts from different organisations are complicated as organisations do not all finalise their forecasts at the same time.  As a result different information sets will have been used in compiling the predictions in any collection of forecasts at a point in time.  In the comparison that follows, forecasts from other organisations are those that were available at the point in time the Treasury’s forecasts were finalised. 

This means that the Treasury forecasts have the most timely data (as new data will have become available after the other forecasts were finalised).  Given this advantage, the result that the Treasury’s forecasts are among the most accurate is reassuring.

In terms of cross-sectional performance relative to other forecasters, there is evidence that the Treasury outperforms the consensus of other forecasters in real GDP growth and inflation, across both short and long forecast horizons. Figure 3 shows the comparison with respect to real GDP growth. As one can see, the relative accuracy of the Treasury is especially significant in the long-term forecast horizon.

Figure 3: Average RMSE of real GDP growth forecasts among forecasters, 2000 – 2017

Figure 3: Average RMSE of real GDP growth forecasts among forecasters, 2000 – 2017

Source: Treasury calculations

To test whether these results are statistically significant, rather than due to chance, we use Diebold-Mariano (DM) test to assess the relative forecast accuracy in real GDP growth and CPI inflation between the Treasury and fifteen other forecasters for the period 2000 – 2017 (Table 1). In general, the results show that the Treasury’s performance in forecasting GDP in the long-term (i.e. 7 quarters ahead) is more accurate than other forecasters. In addition, Treasury’s forecasts of inflation over the 1-quarter, 5-quarters, and 7-quarters ahead horizons are also more accurate.

Table 1: Diebold-Mariano test of relative forecast accuracy between the Treasury and consensus forecasts

Table 1: Diebold-Mariano test of relative forecast accuracy between the Treasury and consensus forecasts

Source: Treasury calculations

Notes

  1. [1] where N is the number of assessed forecasts.
  2. [2] The other forecasters are: ANZ, ASB, BNZ, Deutsche Bank, Economist Intelligence Unit, First NZ Capital, HIS Global Insight, IMF, Infometrics, JP Morgan Chase, NZIER, OECD, Reserve Bank, UBS, and Westpac