World Bank

The World Bank data are based entirely on published information, informed by research of laws and regulations, with input and verification from more than 3,000 local government officials, lawyers, business consultants, and other professionals who routinely administer or advise on legal and regulatory requirements. The data are standardised (to enable comparisons across countries) by asking the respondents to reply with regard to very specific case studies. However, this could be at the expense of losing information. For example, when asked about the procedures, time and costs involved in starting a business, respondents were asked to consider the situation for a business that:

• is 100% domestically owned;
• operates in the country’s most populous city;
• has 5 owners;
• has start-up capital of 10 times income per capita;
• performs general industrial or commercial activities;
• does not perform foreign trade activities;
• does not qualify for investment incentives;
• has up to 50 employees one month after the commencement of operations (all of them nationals);
• has a turnover at least 100 times income per capita; and
• has a company deed 10 pages long.

The logic behind these criteria is simple: the World Bank wants to compare like with like. However, nations might have “typical” businesses that differ in many ways from the World Bank’s hypothetical business, and the regulatory structure in those nations could be designed with regard to typical native business models. By being so specific about the hypothetical business, the World Bank is improving the integrity of its index from one perspective, but loses information when it fails to account for the diversity of business models within and between nations (and, therefore, the experience of starting typical businesses).

OECD

The OECD also use objective data based on a survey of civil servants in national administrations that have knowledge and responsibilities related to the relevant policy areas.

Their overall indicator is aimed at measuring the degree to which policies promote or inhibit competition, in areas of the product market where technological and market conditions make competition viable. Thus, the OECD is making 2 assumptions when constructing their index:

1. that they can “measure” or “know” which policies promote or inhibit competition, and

2. that they know which areas of the market can be competitive.

The OECD indicators only account for formal government regulation (as do the World Bank indicators). This makes them comparable across countries but also implies some limitations. ‘Informal’ regulatory practices, such as administrative guidance or self-disciplinary measures of professional associations, are captured to a very limited extent, and the way in which regulations are applied by enforcement authorities, which the OECD admit can have a considerable impact on competition, is also only reflected in a relatively minor way in the indicators.

As with the Heritage Foundation and Gwartney and Lawson indices, the OECD convert cardinal statistics and qualitative information into discrete, ordinal categories. The often qualitative nature of the data and the need to aggregate different regulatory provisions involved a certain amount of arbitrary judgement in the construction of the detailed indicators. That is, the scoring procedure often involved some subjective judgement. While any mistaken judgements may have had an influence on the ranking of countries in the individual regulatory provisions, the OECD argues that as long as they are not systematic it is unlikely that they can affect significantly the values of the summary and overall indicators of regulation, due to the large number of provisions (156 of them) included in the analysis.

Also, as with the WEF and Gwartney and Lawson calculations, the OECD assumed that the country sample represented the entire population of reference: ie, the least restrictive and the most restrictive provision in the country sample were assigned the values 0 and 6 respectively.

The OECD indicators, while providing detail at the industry level, may not fully cover the range of regulations within a country as they only cover regulations in some industries for some of the indicators. For example, the question in which respondents were asked about restrictions on businesses to enter markets was only asked in relation to the network sectors. The OECD state that the existing indicators could be expanded to incorporate a range of additional economic information. For example, the sectoral coverage of the indicators could be increased by expanding the number of sectors over which indicators such as ‘the scope of public enterprises’, ‘legal barriers to entry’, and ‘barriers to foreign ownership’ are calculated. Currently, only the retail distribution, transportation, and telecommunications sectors are included.

Also, the OECD discuss how new data could be used to refine some of the low-level indicators. For example, data on the number of hours that retail outlets are typically able to trade could be useful in determining the extent to which retail trade is regulated. Another example involves incorporating data on producer support for agriculture into the indicator of barriers to trade.

Nicoletti and Pryor (2005) show evidence that the OECD economy-wide indicators and the Fraser Institute indicators are highly correlated, despite the different methods used in their construction.

Aggregation Methods

In constructing indicators of regulation, the aggregation procedure must involve weighting the individual components or areas of regulation to form the overall indicator. In principle, the weights should reflect the relative impact of the different policies on the policy objective in question.

The OECD have tried various weighting schemes in the construction of their summary indicators. They concluded that there appears to be only some differences in the ranking of countries (according to the stringency of employment protection legislation) as one moves from a uniform weighting scheme, to subjective weighting and, finally, to a statistically-defined weighting scheme. However, these small differences may have some impact in analytical studies of the impact of employment protection legislation on economic performance, to the extent that the rank position (as opposed to the actual summary values) of countries are used.

The OECD also used a random weighting technique to construct confidence intervals around their indices based on principal components analysis. That is, they used 10,000 sets of randomly generated weights to construct 10,000 overall indicators for each country.

The principal components technique used by the OECD reveals, within each regulatory domain, families of detailed indicators which are most associated with different underlying (unobserved) factors. That is, the underlying detailed indicators are grouped into factors based on the data, rather than relying on judgment as to what components are included in each factor. Principal component analysis is appealing, therefore, since the aggregation of the detailed indicators is data-based and ensures that the resulting summary indicators account for a large part of the cross-country variance of the detailed indicators. It also means that the aggregation is independent of prior views on the relative importance of each regulatory provision.

However, principal component analysis is sensitive to any modifications in the basic data (ie, the weights may change with any change in the data). The results are also likely to be sensitive to the presence of outliers, which may introduce a spurious variability in the data. Data limitations may also imply difficulties in the statistical identification and the economic interpretation of the unobserved factors. The OECD checked the robustness of the results by excluding a few outlier countries.

The principal component analysis of the four detailed indicators describing outward policies ran into some problems. Due to the limited country coverage of some of the basic data, the focus had to be restricted on a few dimensions of outward-oriented regulations, not necessarily fully representative of the countries trade and investment policies. In addition, the cross-country variance of the detailed indicators was much smaller than in the other domain of regulation. As a result, little correlation was found among the indicators.

The same set of weights which were found using principal component analysis in the OECD 1998 report were also used for their 2003 indictors. According to the OECD, maintaining consistent weights in the different estimation periods is an important pre-requisite for making meaningful comparisons of indicator values in different years. The use of the same weights implies that any changes in the indicators over time are entirely due to changes in regulations and not in the aggregation process.

Gwartney and Lawson have also experimented with several weighting methods ranging from the subjective views of “experts” to principal component analysis. They found that in most cases, the choice of weighting method exerts little impact on the rating and ranking of countries. As a result, they decided that it is best to keep the procedure simple and transparent by use of the simple average procedure. By doing this, they do not mean to imply that all components and areas of economic freedom are equally important.

The Heritage Foundation also weights all ten factors equally in the construction of its overall Economic Freedom Index, as they argue, in contrast to Gwartney and Lawson, that all of the factors are of equal importance. They state that this is the common-sense approach, which is consistent with the purpose of the Index: to reflect the economic environment in every country. The index is not designed to measure how much each factor adds to economic growth. In the 2004 Index, Richard Roll concluded that equally weighting the factors reveals as true a picture of economic freedom as the best weighting scheme that statistics can devise (weights based on principal components analysis).

However, while the correlation between the 2 indices (one based on equal weighting and the other based on principal components weighting) was very high (R² = 0.98—a good argument, according to Roll, in favour of preserving equal weights as they have the advantage of simplicity), the 2 indices did generate somewhat different country rankings. For example, Guatemala moved down 32 positions when using the principal components method rather than equal weights, and quite a few countries changed in rank by at least 10 positions.

As one of the main recommendations of this paper is to look at individual components rather than the overall indicators, the weighting method used does not become an issue (except to the extent that people ignore this recommendation and focus on the consolidated index). Another recommendation may be to compare groups of countries rather than focusing on individual rankings. The OECD identified two broad country groupings (using their constructed confidence intervals) with clearly distinct regulatory regimes—a “relatively liberal” group (which includes New Zealand) and a “relatively restrictive” group. The rest of the OECD countries (”middle-of-the-road” countries) were not statistically different from these two groups.

Kaufmann et al (1999) also warn against concentrating solely on individual rankings compared to other countries. The rather large size of the confidence intervals they constructed makes it clear that small differences in point estimates of governance across countries are not likely to be statistically significant.[19] As a result, they suggest that users of this data should focus on the range of possible governance for each country as summarised in the confidence intervals. For two countries at opposite ends of the scale of governance, whose confidence intervals do not overlap, it is reasonable to conclude that there are in fact significant differences in governance between these two countries. For pairs of countries that are closer together and whose confidence intervals overlap, “one should be much more circumspect about the significance of estimated differences in governance between two such countries”.