Tim Ng, Deputy Secretary to the Treasury
The Treasury has revised the child-poverty projections it provided to the Government in December 2017.
As outlined in the Treasury's 17 January 2018 media release, the previous estimate was based on code that included a modelling error (Coding error in the Treasury's poverty-reduction projections).
"We have confirmed that our error in the microsimulation modelling affected both our assessment of the Families Package announced in December 2017 and comparisons with the previous Government's Family Incomes Package announced in May 2017," says Deputy Secretary to the Treasury Tim Ng.
"In the table below, the figures we provided to the Government in December 2017 are in the 'As published' row. Correcting for the coding error, our best advice at that time should have been the estimates in the 'Revised' row.
| Families Package (Budget Update December 2017) |
Family Income Package (Budget 2017) |
|||
|---|---|---|---|---|
| Decrease | Percentage | Decrease | Percentage | |
| As published (Dec 2017) | 88,000 | 48% | 49,000 | 27% |
| Revised | 64,000 | 41% | 33,000 | 21% |
This data corrects the coding error identified in January 2018. No other changes have been applied.
"No new coding errors were found during our internal review.
"As we stated in January, the error does not affect the number of people who will be helped by the Government's Families Package, the amount of extra income they will receive, or the fiscal impact of the Package. Our analysis continues to show that the Families Package will substantially reduce the number of children living in low-income households.
"Our mistake is deeply regrettable, and I repeat the Treasury’s apology for it," Mr Ng says.
Meanwhile, Secretary to the Treasury Gabriel Makhlouf has confirmed that the independent review of the cause of the error, and possible improvements to the Treasury’s microsimulation modelling framework and associated quality-assurance processes, is under way.
Background#
The Treasury's projections for the previous Government of the effect on child poverty of its Family Incomes Package, announced in May 2017, included some calculations necessary to correct weaknesses in how the model estimated the effects of the Accommodation Supplement. An internal technical review of the calculations for the May 2017 announcement suggested improvements were needed.
In late November and early December 2017, a module was developed to further improve the Accommodation Supplement analysis. This was applied to both the previous Government's package and the current Government's Families Package. The coding error occurred in this "add-on module".
The error involved transposing two terms. The relevant extracts of the original code and the correction are in this table ("SLP" is "Supported Living Payment" and "SPS" is "Sole Parent Support"):
| Incorrect |
... SPS_Num = sum((P_Benefits_SPS_Amount_Abated > 0)), SLP_Num = sum((P_Benefits_SLP_Amount_Abated > 0)), Benefit_AS = sum((P_Benefits_Accommodation_Abated > 0)*((P_Benefits_JSS_Amount_Abated + P_Benefits_SPS_Amount_Abated +P_Benefits_SPS_Amount_Abated)> 0)) ... |
|---|---|
|
Correct |
... SLP_Num = sum((P_Benefits_SLP_Amount_Abated > 0)), SPS_Num = sum((P_Benefits_SPS_Amount_Abated > 0)), Benefit_AS = sum((P_Benefits_Accommodation_Abated > 0)* ((P_Benefits_JSS_Amount_Abated + P_Benefits_SPS_Amount_Abated + P_Benefits_SLP_Amount_Abated)> 0)) ... |
Modelling improvements#
As a separate exercise from correcting the original coding error, the Treasury has built a new version of the add-on module using an improved quality assurance process – independent co-production. This involves two people developing the analysis independently, and cross-referencing their results until they agree. This significantly reduces the risk of errors. This new module also includes methodological improvements.
The Treasury had intended to use this new module to update the child-poverty projections using 2015-16 Household Economic Survey* (HES) data (in addition to the 2014-15 HES data used for the December 2017 projections). This would have incorporated more recent information with a larger sample size, and thereby reduced the statistical margin of error in the projections. However, in discussions with Stats NZ it was agreed that the 2015-16 HES data needs further review before being used for such projections.
In the absence of suitable data for 2015-16, the Treasury has updated the projections incorporating HES 2012-13 data and some new supplementary data from the Ministry for Social Development. The estimates based on this improved modelling are in Treasury report T2018/395 of 28 February 2018, which is now available on the Treasury website. The projection based on "pooled data" estimates that there will be 54,000 (35%) fewer children in low-income households as a result of the Families Package. (The equivalent projection for the Budget 2017 Family Incomes Package is a reduction of 27,000 (17%).)
The Treasury has advised the Government that these and all of the projections provided to date should be used with caution and are not suitable for benchmarking, or for assessing progress towards or setting child-poverty reduction targets. This is because of the limitations of the input data and the models, and the large statistical margins of error. For example, the 54,000 projection above has the smallest statistical margin of error and constitutes the Treasury's current best advice on child-poverty projections, but that estimate is within a wide range (between 42,000 to 73,000).
In addition, projections of relative child-poverty are inherently volatile. They are likely to change substantially when new survey data becomes available and when actual and forecast economic conditions change. This volatility applies even in the absence of modelling improvements or policy changes.
* Access to the Household Economic Survey data was provided by Stats NZ under conditions designed to give effect to the security and confidentiality provisions of the Statistics Act 1975. The results presented here are the work of the Treasury, not Stats NZ.
Related Material#
- Child Poverty Error Information Release (15 March 2018)