2.2  Dealing with missing data

Merging demographic and employment data with GST sales and purchases data highlighted several issues. First, there were enterprises that had GST sales information but no employment data for the entire period they existed, or conversely had employment data but no GST sales information. Because it was not possible to form a measure of firm labour productivity when either employment or GST sales data were missing for the entire period the firm existed, these firms were dropped.[8] Second, some enterprises had partial information on employment or GST sales for part of the period the firm was recorded as existing. When this occurred during the middle of a firm’s existence the missing data observations were filled if at all possible using historical imputation. For example, a firm in existence between 1994 and 2003 with GST sales for the corresponding period but missing employment data in 1996 and 1997 would have the missing 1996 and 1997 employment data filled using employment in 1995. If historical imputation was not possible we then impute using data from subsequent years.

A partial explanation for situations similar to the example above is that some firms fail to respond to the Annual Business Frame Update (ABFU) questionnaire, despite the firm still operating (as indicated by the firm filing GST sales of $30,000 or greater). The non-response rate for the ABFU is estimated to be between 10% and 15%. The approach to imputing missing values is one that has been adopted by Statistics New Zealand in other contexts.

Cases occurred where a firm was in existence in the BD but there was no recorded information on total employment and GST sales or purchases at the beginning or end of the firm’s life. In these situations the firm was deemed to be an entrant in the first period that either employment or GST was available and was ‘ceased’ in the period following the last observation for either employment or GST sales.[9]

Table 1 shows the numbers of firms deemed to be in operation for each year between 1994 and 2003 and the percent of missing observations on sales, purchases and total hours in each year that are subsequently filled as described above. Missing observations on total hours are more prevalent than those on sales or purchases. Across all years 22% of observations on total hours are missing[10] as compared to 11% for each of sales and purchases. The portion of firm-year observations with missing information on any one of sales, purchases or total hours is 29%. Fewer observations are missing in 1994 and 2003 than in other years, particularly on sales and purchases. This is likely to be due to the fact that potential missing observations for firms existing in previous (in the case of 1994) and later years (in the case of 2003) cannot be observed and therefore cannot be taken into account in the process used for imputing missing data.

Table 2 shows the totals of all firms’ sales, purchases, value-added and hours worked after filling in every year. Total value-added ranges from between 80.9 and 96.2 percent of GDP over the period. Total sales and total purchases in each year are on average approximately 3.5 and 2.5 times total value-added respectively. Total hours worked per year averages 2,730 million for the entire period (or around 1.4 million full time equivalent employees[11]) which represents on average around 90 percent of economy-wide total hours worked (ranging from about 83.3 percent in 1994 to 94.5 percent in 1999).