The Treasury

Global Navigation

Personal tools

2.2  Data treatment and quality

The primary population for BOS is all private-for-profit firms with at least six employees. From that population, a stratified random sample of around 7,000 firms is selected each year. In 2011 there was a total of 5,352 useable responses.[4] After excluding an additional 168 firms that gave an inconsistent answer to the core routing question in the international engagement module, the sample was reweighted to reflect the original population of 35,499 firms.

Analysis of the quality and internal consistency of the BOS11 survey responses suggests a number of issues:

  1. The number of firms earning income from overseas residents visiting New Zealand (eg, income from tourism or education exports) is likely to be understated, as these firms do not view their activities as “generating overseas income”. Despite an explicit instruction that overseas income includes “significant income from overseas residents visiting or studying in New Zealand”, of the 201 firms that reported substantial tourism revenue in Module A (> 25% of total revenue) only one third identified themselves as having current overseas income in the later module.[5]
  2. Comparison of Module A and Module C also shows that the distinction between firms with past overseas income, and those that have never generated overseas income is unclear. Almost 10 percent of the firms that state in 2011 that they have never earned foreign income, had previously said that they either were currently generating overseas income or had in the past (Module C, 2007). A similar proportion of those whose BOS Module A responses could be tracked back to 2005 had recorded earnings from exports in at least one of the past six years.[6]
  3. Item non-response is significant throughout the survey, but is particularly problematic among those firms with past overseas income. This is likely to be driven by recall difficulties - non-response rates are highest for questions relating to the firm's past activities and much lower for questions relating to intentions for the future.[7]
  4. A similar problem is seen when firms with no past overseas income are asked for details on their future intentions. While response rates for basic questions around the level of interest in future overseas income are answered well, details such as the motivation behind this interest, the countries they would target, and the barriers they perceive, are frequently missing. Failure to answer these more detailed questions is related to firms' level of interest - non-response rates are lower among firms that have actively taken steps towards international engagement than for those that note only that they are interested in exploring the options - suggesting that non-response occurs because firms have not previously considered such details.
  5. Item non-response is also relatively high for questions on the barriers that firms face in generating overseas income. This may be associated with the response categories provided, which do not give firms the option of reporting “no significant barriers”. The 2015 module explicitly allows firms to select ”no significant difficulties”. This option was the most frequently selected response among firms currently selling to overseas market, selected by around 34 percent of these firms (Statistics New Zealand Infoshare).

In response to the first issue, and in recognition that the barriers faced by firms dealing with foreign customers in New Zealand are likely to differ substantially from those engaging offshore, we include, where relevant, controls for those industries that we expect to be more likely to operate onshore (retail trade, accommodation and restaurants, road and rail transport, museums and zoos, sports and recreation activities, and education) to consider whether responses differ for these “onshore” industries.

In general, our approach to item non-response is to calculate proportions over the firms that provide a useable response. For example, if 40 firms respond “yes”, 40 firms respond “no” and 20 firms fail to respond, the proportion saying “yes” would be reported as 0.5 (40/(40+40)) rather than 0.4 (40/(40+40+20)). Thus, we are implicitly assuming that item non-response is not correlated with the firm's “true” response. For those groups where response rates are particularly low (eg, questions on past and future activities), this implicit assumption is not likely to hold. However, in these cases observed responses may still be a reasonable representation for important subsets of firms: those with recent experience and those that are actively looking at generating overseas income in future. Appendix A reports item non-response rates for all questions used in this note. Where raw non-response rates exceed 5 percent for a given item, this detail is included in the table or figure notes.[8]

We also implement an adjustment to the question on barriers to generating overseas income among those who are currently engaged. Firms were presented with 12 possible response options for this question but did not have the option to state that they did not face any specific barriers. We therefore assume that firms that are consistently good responders but do not provide an answer to the barriers question would have taken a “no major barriers” option had one been available. We generate a new response category and allocate to it all firms that provide internally-consistent responses to at least 13 of the preceding 14 questions but fail to answer the question on barriers. This reduces the missing rate from seven percent to four percent.[9]

The combined effect of these corrections leads our results to differ slightly from the official BOS results reported by Statistics New Zealand. However the differences are very small - in almost all cases where a comparable figure can be constructed from the official release, our estimates differ by less than two percentage points. The exception is reported reasons for complete exit from overseas markets, where differing treatment of non-responses combines with a high non-response rate leading to noticeable differences between our results and the official release. Given the small sample and high rates of non-response, these results should be treated with particular caution.

Finally, in interpreting firms' responses to the survey, it is worth noting that not all questions cover the same period of time. Whether a firm is classified as having current, past, or no overseas income depends on a question referring to their income over the last financial year. This definition is carried through most of the survey, but there are a few key exceptions. Most significantly, firms that are currently engaged in income generation are requested to identify barriers they have faced over the last three financial years. This may generate some disconnect in the analysis that follows, as we link responses to the barriers questions to other information including the source of overseas income. If firms are indeed responding to the barriers question by including barriers that they faced in earlier years, and their characteristics have changed in the interim (eg, they may have added or dropped one or more form of income generation) we may be mistaken in linking their reported barriers to their current activities. More generally, if some barriers were sufficient to prevent firms from undertaking a certain activity, this may misrepresent the link between barriers and activities.

Notes

  • [4] All firm counts reported in this note have been random rounded base three in accordance with Statistics New Zealand confidentiality requirements. The BOS sample includes both an annual, representative cross-section and a top-up sample which is designed to maintain a panel component for longitudinal analysis or, in 2007, to facilitate the move from ANZSIC96 to ANZSIC06 industry classifications (see Fabling &Sanderson 2016) for further detail on the BOS sample). The main analysis in this paper uses only the core, cross-sectional samples, while the longitudinal analysis of entry rates also incorporates the additional top-up samples.
  • [5] These two answers are not entirely inconsistent, as firms may be including revenue from domestic tourism in Module A. However, this is unlikely to fully explain the gap between the Module A and Module C responses.
  • [6] In general these export earnings made up a relatively low proportion of total sales.
  • [7] Raw non-response rates are as high as 36% for a text question on the year in which firms first earned overseas income, compared to 9% for a yes/no question on whether the firm is interested in generating overseas income in future.
  • [8] Raw non-response rates tend to be slightly lower than weighted ones, as non-response is more common among small firms. Reported standard errors reflect the variation within the responses, and have not been adjusted to account for non-response.
  • [9] This adjustment is only implemented for current exporters as firms that do not have overseas income are not asked enough questions to be able to judge respondent quality. Comparisons of reported barriers between current exporters and non-exporters use the unadjusted responses for consistency. Reducing the threshold for respondent quality has little impact on results as few firms lie between 10 and 12internally-consistent responses.
Page top