The Treasury

Global Navigation

Personal tools

2  Previous literature

There is a sizeable international literature comparing survey measures of income to tax-based administrative measures of income.[3] Indeed, because of the recent linking of several surveys to administrative data in New Zealand, there is now a modest number of New Zealand papers on the differences between survey and administrative data. Of these, the most closely related to ours are Suei (2016), who compares the New Zealand 2013 Census to the IRD data we use within the IDI, and Hyslop and Townsend (2016a, 2016b) who compare the IRD data to New Zealand's Survey of Family Income and Employment (SoFIE).

The techniques used in Suei (2016) are different to those used in our paper because Census income is recorded in bands rather than in precise amounts as in HES, IRD and SoFIE. On the whole, Suei finds similar results for Census-IRD comparisons as we find for HES-IRD comparisons, that is, a strong correlation between survey and administrative measures of income. However, whereas we find HES income is on average slightly higher than IRD income, Suei finds that Census income is on average slightly lower than IRD reported income.[4] Part of this difference is due to the way we treat missing records in IRD. In our paper, we treat people with an IRD number but no recorded income in a year as earning zero income in that year, whereas Suei treats their IRD income as unknown. However, when we side-step this issue by subsetting to people with positive reported incomes in both sources, we still find higher average incomes in the HES data compared with the IRD data.

Hyslop and Townsend (2016a) also find similar results comparing SoFIE and IRD incomes as we do comparing HES and IRD incomes. The key difference between their results and ours is that they find a slightly closer relationship between their data sets then we do, and whereas we find HES incomes tend to be slightly higher than IRD incomes, they find SoFIE measured incomes tend to be slightly lower. For example, they find that mean SoFIE earnings are between 1% lower to positive 3.5% higher than mean IRD earnings (depending on the sample), whereas we find mean income in HES are 1.5-6% higher than mean IRD incomes (depending on the year).[5] In addition, Hyslop and Townsend find log SoFIE earnings are 0.02-0.04 log points lower than log IRD earnings, whereas log HES earnings are about 0.02 log points higher than log IRD earnings. Hyslop and Townsend find reliability ratios for

SoFIE-IRD that are somewhat higher than what we find, with SoFIE reliability ratios of 0.83-0.85 and IRD reliability ratios of 0.87-0.91, whereas we find reliability ratios of 0.78 for HES and 0.82 for IRD.[6]

Somewhat relevant to these comparisons is Ball (2016). Though not the main purpose of his paper, Ball compares the samples (but not the different sources of earnings) in the IDI-linked HES, SoFIE and Census 2013 data. Ball shows that the linked HES sample of people has higher IRD reported income than the linked SoFIE and Census sample’s IRD income.

Other papers in New Zealand comparing survey and administrative data include Samoilenko and Law (2014) who found very large discrepancies between reported KiwiSaver enrolment in SoFIE and administrative data from IRD (with KiwiSaver enrolment under-reported by 50%). Chapple and Crichton (2012) (in part) compare Household Labour Force Survey (HLFS) reports of benefit receipt to MSD records for the same individuals and find significant numbers of beneficiaries (based on MSD records) are not reporting Unemployment Benefit receipt in the Household Labour Force Survey (HLFS). As we will see, these results are consistent with what we find when comparing HES to MSD benefit records.


  • [3] Studies comparing survey and administrative reports of income for the same people go back at least to Miller and Paley (1958) who match data for about 3,900 respondents to the 1950 US Census and compare this to IRS data. They found that median wage and salary income for matched families was about 4.6% ($3,570 vs. $3,412) higher in IRS data than Census data. Prominent early papers include Pischke (1995), Bound and Krueger (1991) and Bound, Brown, Duncan and Rodgers(1994). Recent contributions include Abowd and Stinson (2013), Kapteyn and Ypma (2007) and Britton, Shephard and Vignoles (2015). Bound, Brown and Mathiowetz (2001) provide an overview of measurement error in survey data including the effects on estimates, methods for correcting for measurement error and the empirical evidence from validation studies on measurement error's nature and extent.
  • [4] Census income is recorded in bands, so mean differences are not computed. However, Suei finds that 33% of people are in a lower income band for Census-measured total income compared with the tax data, while 25% are in a higher band in Census. This is also despite missing investment income information in the IRD data. Suei argues that "substantial conceptual differences between the two sources [of income] may be a key contributing factor" to these differences (Suei, 2016, p. 23).
  • [5] See Table 12.
  • [6] The reliability ratio is a measure of agreement between two measures of the same phenomenon. They are calculated as the ratio of covariance between the two measures to the variance of each measure with reliability ratios closer to 1 indicating higher agreement. More details and discussion on reliability ratios can be found in Appendix F.
Page top