The Treasury has advised the Government of a coding error in the Treasury’s modelling of projected changes in child poverty.
“The error in our microsimulation modelling affects our assessment of both the Families Package announced in December 2017 and comparisons with the previous Government’s Family Incomes Package announced in May 2017,“ says the Secretary to the Treasury, Gabriel Makhlouf.
“The error likely led to an overstatement of the projected impact both packages would have on the reduction of child poverty.
“It affects our projections of the number of children expected to be in low-income households, and the number to be lifted out of poverty*, by 2020/21,” Mr Makhlouf says.
“The extent of any change in the projections on child poverty is still being determined. Because the error applies equally to comparisons with the previous Government’s Family Incomes Package, the estimated relative impact of the two packages is essentially unchanged,” Mr Makhlouf says.
“The error does not affect the number of people who will be helped by the Government’s Families Package, the amount of extra income they will receive, or the fiscal impact of the Package. Our analysis continues to show that the Families Package will substantially reduce the number of children in low-income households.
“We are remodelling the projected impact of both packages, and we’re aiming to have revised projections available in the second half of February.
“This is a deeply regrettable mistake and I apologise for it on behalf of the Treasury. The Treasury holds itself to high standards and I’m disappointed to have not met those standards here.
“We are reviewing our quality-assurance procedures to minimise the risk of this kind of error in the future. I am also commissioning an independent review of the cause of the error and possible improvements to our microsimulation modelling framework and associated quality-assurance processes,” Mr Makhlouf says.
* Defined as living in a household with an income less than 50% of median equivalised household income before deducting housing costs
ENDS
Media Enquiries
Please contact: [email protected]
Technical note#
The Treasury’s projections for the previous Government of the effect on child poverty of its Family Incomes Package, announced in May 2017, included some calculations necessary to correct weaknesses in how the model estimated the effects of the Accommodation Supplement. An internal technical review of the calculations for the May 2017 announcement suggested improvements were needed. (Accordingly, the projections produced for the previous Government’s Family Incomes Package no longer represent our best professional advice.)
In late November and early December 2017, a module was developed to further improve the Accommodation Supplement analysis. This was applied to both the previous Government’s package and the current Government’s Families Package. The coding error occurred in this “add-on module” – in a single line of about 1000 lines of code.
The quality-assurance (QA) process for the add-on module included an independent review of the methodology by a senior statistician outside the Treasury’s microsimulation modelling team, multiple layers of code review, and an independent replication of each stage by two modellers. No issues were identified during this process.
This QA process, however, is not as rigorous as independent co-production, which is used for modifications of the core microsimulation model. Independent co-production involves two people developing the analysis independently, and cross-referencing their results until they agree. This significantly reduces the risk of errors, but takes longer and was not possible in the time available.
The analysis now under way will include independent co-production of the updated add-on module as described above. The Treasury is aiming to provide the results of the analysis, including its updated child-poverty projections, to the Government in the second half of February 2018.
This amount of time is needed to correct the model, re-run the analysis, and include a scheduled update with data from the Household Economic Survey 2015/16.
The overall cost of the Families Package, the number of families eligible and the amounts they are eligible to receive under the Package are not affected by this error. The add-on module was used only to help estimate the distributional impacts of the Families Package and compare it to the previous Government’s package, not for costing purposes.
Independent review#
The Secretary to the Treasury is commissioning an external review of the cause of the coding error and possible improvements to our modelling framework and associated QA processes.
The review will consider the underlying causes of the coding error, and whether improvements could be made to the Treasury’s microsimulation modelling framework to enable it to better respond to requests for similar advice in the future.
The review will assess the Treasury’s QA practices in the context of this piece of advice, and provide an assessment of the fitness for purpose of the Treasury’s data and microsimulation modelling framework more generally.
The intention is to complete the review by mid-March 2018. The Treasury is currently (17 January 2018) working to appoint an external reviewer with expertise in both the technical constraints in this type of modelling and the policy purposes for it.
Projections being revised#
The tables below contain the information that the Treasury is now in the process of revising.
Table 1 Projected reduction in number of children in low-income households under the Families Package
Table 1 Projected reduction in number of children in low-income households under the Families Package
Table 2 Projected reduction in number of children in low-income households under the Budget 2017 Family Incomes Package
Table 2 Projected reduction in number of children in low-income households under the Budget 2017 Family Incomes Package
Note for Tables 1 and 2:
Access to the Household Economic Survey data was provided by Statistics New Zealand under conditions designed to give effect to the security and confidentiality provisions of the Statistics Act 1975. The results presented here are the work of the Treasury, not Statistics New Zealand.