3.  Survey Methods

Table 2 summarises the survey methods used for the six surveys discussed in this paper.

WEF

The WEF uses a combination of hard data, and responses to their Executive Opinion Survey. Three components of the ‘Market Efficiency’ pillar in the ‘Global Competitiveness Index’ are examples of the hard data used: the number of procedures and time required to start a business are both drawn from the World Bank database discussed below, and imports (as a percentage of GDP) are also used under the ‘Competition’ sub-index. The responses to the Executive Opinion Survey range from 1 to 7, while each hard data variable was converted to the 1-7 scale using the following formula:

This implicitly assumes that the highest and lowest sample values are the maximum and minimum values which this variable can take in the population. That is, they assumed that the country sample represented the entire population of reference.

The Executive Opinion Survey was designed to capture the expert opinions of business leaders on the most important issues affecting their working environment. The business executives were asked to respond in regard to their own business experience, and to rank their perception on a scale of 1 to 7, with 1 representing the worst-case scenario.

The WEF’s partner institutes—typically consisting of leading research or academic institutes, business organisations, national competitiveness councils, or any other recognised professional entity committed to improving the competitiveness condition of their economy—lead the survey at the national level in accordance with the guidelines set by the WEF. The partner institutes were asked to select a sample of business executives to take part. Specifically, all respondents had to be a CEO or equivalent, unless the business had greater than 500 employees, in which case the participant could hold any of the company’s top five management positions. The companies sampled had to represent the main sectors of the economy, in proportion to its percentage share of total GDP. A master list of companies (compiled from a list of companies registered for a telephone line or companies registered with the national statistical office or tax authorities) was grouped by economic sector, geographical region and size, before sampling from each subgroup. However, the selection of firms to be interviewed within each group could not be completely random, in part because there was a preference for interviewing business executives who have an international perspective, which often meant selecting large companies.

The New Zealand sample consisted of 47 participants. Table 1 below, Panel A sets out the distribution of respondents by firm size, while Panel B provides a breakdown of the distribution of all firms in New Zealand by size (number of employees). It is obvious from a comparison of Panel A with Panel B that the sample drawn for the Executive Opinion Survey is not representative of the actual population of firms in New Zealand. This may be a direct consequence of the preference stated above for the sample to consist of business executives who have an international perspective. So while there may be advantages to surveying business executives with an international perspective (discussed below), there are also disadvantages, as the resulting sample may not be representative of the population of interest.

Table 4: Comparing the distribution of firms sampled in the WEF survey to the population of firms in New Zealand

IMD

Like the WEF survey, the IMD data are made up of hard data, taken from international and regional organisations and private industries, and survey data drawn from their Executive Opinion Survey. The survey of 113 questions was sent to top and middle management in each country or region who explicitly deal with international business situations. The survey questions were designed to measure competitiveness as it is perceived by the business executives. Four thousand executives responded in 2005 for an average of approximately 67 responses per economy.[17]

The survey was sent to executives who represent a cross-section of the business community in each country or region. The distribution reflects a breakdown of industry by sectors: primary, manufacturing, and services, and they state that, in order to be statistically representative, they select a sample size proportional to the GDP of each economy. This could mean that they end up with a very small sample for New Zealand compared to other countries (they do not report sample sizes for each country). If they do want to be statistically representative, the proportion of business executives (out of the entire population of business executives in the country) actually sampled should be higher the smaller the population of business executives, to maintain the same level of confidence in the results. The respondents were asked to evaluate the present and expected competitiveness conditions of the economy in which they work and have resided during the past year, drawing from the wealth of their international experience, and “thereby ensuring that the evaluations portray an in-depth knowledge of their particular environment”.

For each survey question, the IMD calculates the average value for each economy, then the data are converted from a 1-6 scale to a 0-10 scale. Finally, the survey responses are transformed into their standardised values (STD values), from which the rankings are calculated. The STD values are calculated using the following formula:

These STD values enable criteria which were originally scaled differently to be used in the computation of the overall, factor, and sub-factor indicators. The sub-factor rankings are determined by calculating the weighted average of the criteria STD values that make up the sub-factor. All of the hard data have a weight of 1, whereas the survey data are weighted so that the survey accounts for one-third in the determination of the overall ranking (reflecting the two-thirds to one-third split of hard data to survey data in the criteria). For 2005, each survey criterion has a weight of 0.5. When data are unavailable for particular economies, the missing values are replaced by a STD value equal to zero.

The partner institutes of the IMD performed a similar task to those of the WEF, supplying data from national (or regional) sources and helping distribute the survey questionnaire. The IMD states that “A long-established collaboration with Partner Institutes also helps ensure that the data is reliable, accurate, and as up-to-date as possible”. The partner institute in New Zealand is the New Zealand Institute of Management Inc., Wellington.

Gwartney and Lawson

As mentioned in the previous section, Gwartney and Lawson use both IMD and WEF survey data as components in the areas of interest for this paper (area 4: ‘Freedom to trade internationally’, and area 5: ‘Regulation of Credit, Labour, and Business’). They also use hard data from various sources including the IMF, the World Bank, and the OECD.[18] Each component (both from survey or hard data) was placed on a scale from 0 to 10 that reflected the distribution of the underlying data. This was achieved using the following formula for components comprised of hard data:

where V_i represents a variable of interest for country i – for example the mean tariff rate for country i. This is similar to the WEF computations in that they are assuming that the country sample represents the entire population of interest.

However, for four components included in area 5 which were constructed using hard data, alternative scaling methods were used. For example, for the component ‘Ownership of banks – percentage of deposits held in privately owned banks’, a country was given a score of 10 if privately held deposits totalled between 95% and 100%. When private deposits constituted between 75% and 95% of the total, a rating of 8 was assigned, and so on. Thus, for some of the components, the authors have chosen the categories using subjective opinion, and assigned scores to these categories. For the survey data, the ratings given by the IMD and WEF were rescaled to range from 0 to 10. The component ratings within each area were then averaged (using equal weights) to derive ratings for each of the 5 areas.

Heritage Foundation, Index of economic freedom

The Heritage Foundation and Wall Street Journal also assign categories to score each factor. That is, some subjective assessment was used. For example, if a country’s banking system received a score of 3, this means that its banking and financial system displayed most of the characteristics for level 3, according to the author’s assessment of the data. All of the subjective assessments, however, are justified by reference to hard data. The main sources of data include the Economist Intelligence Unit Country Commerce, Country Profile and Country Report, official government publications of each country, the US Department of Commerce Country Commercial Guide, the OECD, the IMF, and the World Bank.

Discussion

While using business evaluations of the regulatory system has the advantage of being able to capture enforcement issues, there are some obvious drawbacks to doing so. For example, since the respondents were asked to rank only their own country in the WEF and IMD surveys, the scale used by business people in particular countries might be more sensitive to government intervention. As a result, they may rate their nation more severely than others. Unfortunately, there is no way to test this. The authors of the WEF survey state that they have made great efforts to reduce this “perception bias” by phrasing the survey questions in such a way that asks respondents to compare their own environment to a world standard, rather than thinking in national terms. Also, Pryor (2002) states that sampling business people with international experience may mitigate this problem somewhat, as they might have some implicit basis of comparison.

The WEF also tested whether the judgements of respondents were affected by a country’s general economic climate, the argument being that in times of recession, business executives will tend to be more pessimistic and may exaggerate the extent of regulations. However, the WEF found no significant relation between either a country’s real growth rate, or the change in the growth rate, to changes in the average level of responses across the survey questions. Thus a country’s general economic climate did not appear to affect the responses.

For each survey question, the WEF also compared the standard deviation of answers within a country to the standard deviation of answers across all countries. In those countries with high within-country variance of responses on many questions it is hard to interpret the country averages, independently of the possible reasons for the variances. As expected, the within-country consensus was higher for cross-cutting business environment indicators and lower for measures where there would be variation within the country across companies and clusters. Thus the WEF concluded that “the country averages, then, capture meaningful differences across countries in competitive circumstances, while limiting idiosyncratic biases that would result if there were only a handful of responses per country”. However, studying the country averages may also lose some valuable information. For example, if respondents from one sector are consistently rating their country worse on some score than respondents in another sector, we might find this information useful from a policy perspective. All of the surveys considered here only report the country averages. They also carried out a “data consistency test”. This test consisted of determining, for each country, whether at least 40 questions out of 57 had lower within-country variance than the cross-country variance. One hundred and ten countries passed this test.

Also, for both the WEF and IMD surveys the samples are small and their degree of randomness can be questioned (Pryor 2002). Plus, the respondents were limited to high level business executives, who presumably have experience in dealing with governmental regulations. Given their attitudes toward governmental regulations, however, they may potentially exaggerate or downplay the extent of these regulations.

Some indication of the quality of the data can be gained by comparing answers to similar questions asked by both the WEF and the IMD. In almost all cases the correlation coefficients were high and significant. Even when the wording of the question differed there was sufficient agreement between the two sources for Pryor (2002) to conclude that “I felt the data reflected the same underlying reality”.