Appendix 2: Fan chart data and methodology

Approach

The use of fan charts to indicate uncertainty in fiscal forecasts has been used inter alia by fiscal authorities in the United States and United Kingdom (Congressional Budget Office, 2006; Office of Budget Responsibility, 2010). A broadly similar method is implemented here.

The approach taken is to construct confidence intervals by looking at the average historical error of the variables which are inputs for the CAB indicator. These variables are revenue, expenses, output gap, unemployment rate, revenue elasticities and nominal GDP.

This is done using data from Treasury's forecasts made at each Budget from 1994 to 2009. Therefore, the results rely on the assumption that data from forecasts for 1994 to 2009 form a representative sample for future forecast errors.

A key assumption is that forecast errors in the future will be unbiased and normally distributed. The assumption of unbiased forecasts (ie, symmetric confidence intervals) is a matter of judgement. Tax revenues have been underestimated, on average, over the sample period. However, the hypothesis that there is no bias cannot be rejected at the 5% significance level. Moreover, the sample period is dominated by the boom of the last decade. There is reason to think that forecast errors are cyclical, underestimating revenue in an expansion and overestimating revenues in a contraction. It will be helpful to get more data which will indicate if there is an opposite bias in during a downturn. A further alternative would be to subjectively skew risks based on judgement of the current balance of risks (as done by the Bank of England for inflation forecasts).

The root mean square error (RMSE) is used to estimate the standard deviation of the error distributions:

for each i = 0, …, 4which is the number of years ahead in the forecast. Confidence intervals are computed by assuming the future forecasts errors are normally distributed with zero mean and standard deviation of RMSE_i (ie, Ê_{2010 + i} ~ N(0,RMSE_i²) )

Uncertainty about these assumptions, combined with limited data series, means that results should be interpreted as only very approximate.

Revenue

There are two data sources for the tax revenue forecast errors. The first is a series of one-year-ahead forecasts of tax receipts (ie, cash) over 1972 to 2009. The second is for revenue (ie, accrual-based) over 1994 to 2009 for forecasts made up to three years ahead. The latter series is also adjusted for tax policy changes which were made after the forecasts were done. The latter data series is used (ie, revenue, adjusted for policy changes) as it is more fit for purpose. However, the longer time series of cash tax receipts is useful as a check that the error dispersion over 1995 to 2009 is reasonably consistent with a longer series.

Forecast errors (E_t) are measured as a nominal difference between forecast revenue () and actual revenue (R_t) as a percentage of the actual:

In Appendix Table 2, the summary statistics are shown which are used to construct the confidence intervals. Appendix Table 3 shows the results for the longer series which shows that errors are significantly larger as would be expected since they are cash, not accrual, and unadjusted for policy changes.

Expenses

Uncertainty around future expenses uses data on errors in core Crown primary expenses over 1994 to 2009. These are not adjusted for policy changes. While clearly there is a close relationship between expenses and policy, there will be uncertainty about transfer payments (where these are indexed to economic variables), take-up rates and underspends by departments (where expenses are governed by appropriations which defines a maximum, but not a minimum, limit on expenses).

In Appendix Table 4, it can be seen that forecast errors are about half of the magnitude of 1-year-ahead errors for tax revenues. But 2- and 3-year ahead errors are much higher and biased which will reflect policy changes. Because the 1-year-ahead errors would be expected to be minimally effected by policy changes (since these are generally made at one year intervals through the annual Budget), it seems reasonable to put greater weight on that value. If the assumption is made that outyear forecast errors, adjusted for policy change, would follow a random walk, the error dispersion can be estimated with the following formula: for N-year ahead forecasts. (The derivation of this formula can be found in standard financial mathematics textbooks, eg, Campbell et al, 1996). This suggests that as a reasonable working assumption, the confidence intervals for the 2- and 3-year ahead expense forecasts should use a standard deviation of 2.1% and 2.6% respectively. These are the values used for constructing the confidence intervals.

Output gap

The output gap is unobservable and hence there is no “actual” value from which to measure the error. However, a proxy can be derived by looking at the distribution of revisions to the Treasury's official estimate of the output gap. Unlike revenue and expenses, these revisions apply to history as well as over the forecast horizon. The Treasury only began reporting the output gap in 1997, so there is a limited time series. To compute a proxy for error, the official Treasury forecasts in real time are compared with the most recent estimate (Budget 2010). The root mean square errors are used to form the standard deviation of the confidence intervals, and are reported in Appendix Table 7.

Unemployment rate

Errors for the unemployment rate are measured by comparing the forecasts with the actual. The actual is defined as the original vintage data released by Statistics New Zealand since errors attributable to statistical revisions following forecasts would complicate the analysis. Confidence intervals use the RMSE to estimate of the standard deviation of the error distribution and are shown in Appendix Table 7. The Okun's law relationship is used to estimate the structural rate of unemployment for a given probability distribution of the output gap.

GDP

A similar approach is used for nominal GDP as for the unemployment rate. Errors with respect to the original vintage data are found for growth rates in nominal GDP (again, to avoid issues with statistical revisions). Summary statistics are shown in Appendix Table 5.

Elasticities

Quantifying the uncertainty in the elasticity estimates is problematic as many values are assumed rather than econometrically estimated. One case where there is an econometric estimate is for the standard error of the elasticity of the wage bill with respect to the output gap (by Girouard and André, 2005). Girouard and André find an average standard error of 0.2 using cross-country panel data which is used to calibrate the New Zealand parameter. Making the arbitrary assumption that the standard errors for other elasticities (both base-to-output gap and revenue-to-base) are of this magnitude, then the standard error for the combined revenue-to-output gap elasticity would be approximately 0.3.[4] This is used to calibrate the standard deviation for the confidence intervals for all elasticity values used. This would suggest an that a 95% confidence interval around an elasticity parameter of 1.0 would be (0.4,1.6).

Covariances

Since the approach used is to construct a confidence interval for the CAB based on estimates of the errors of the constituent parts, an assumption is required for the covariances between the error distributions. Appendix Table 5 shows the sample correlation coefficients for the 1-year-ahead errors.

* significant at 10% level; ** significant at 5% level; *** significant at 1% level

These sample estimates are used in the computation of the CAB confidence interval. While many of the sample coefficients are not statistically different from zero, they appear to reflect economically plausible relationships and are the best point estimates available.

The elasticity errors are assumed to be independent distributions since there is no empirical means of estimation, nor any theoretical reason to believe the case is otherwise.

Summary of assumptions

The assumed standard deviation for the distribution of errors for each variable is summarised in Appendix Table 6.