5.  Concluding remarks

Indices of regulatory impact will enter policy discussions as they provide convenient, if equivocal, evidence. As noted earlier, the surveys discussed in this paper generally show New Zealand in a flattering light according to the aggregate indices. What weight should they be given, though, when determining reform agenda? And what do they not say about the impact of regulation?

Assessment is comparison. To say that something is performing well or poorly implies a comparison. When we say that New Zealand is performing well or poorly, what are we comparing performance to? Comparison needs an alternative scenario, a counterfactual. There are at least three possible choices of counterfactual. We can assess current performance in relation to:

• an ideal outcome;
• past performance; or
• the performance of others.

These alternatives have advantages and disadvantages. A comparison with an ideal outcome presents two problems. First, how do we define the ideal? And second, if the ideal is unachievable in practice, how useful is a comparison with it? Maybe the answer is to compare performance not with a theoretical ideal, but with a practical optimum—the best potential outcome given limits to our knowledge and resources. But the problems of definition and measurement remain.

Comparisons with past performance seem natural enough, but are they sufficient? Given experience and uncontrollable events, we ought to expect changes in performance. Concentrating on comparisons with the past tells us where we have gone, but begs the question of where we ought to be. We are led back to comparisons with an elusive ideal or potential.

A third way, the pragmatic thinking underlying various kinds of assessment from school examinations to international indices, is to compare performance among cohorts. How has an individual performed compared with others? If we assume that members of a cohort face similar problems, resources and constraints, then differences in performance tell you how well or poorly you did in relation to others in similar circumstances.

This kind of comparison also seems natural to us. People esteem themselves based on differential characteristics within cohorts. And, if any number of sporting contests can show, people like their group to outdo others. But as natural as it might seem, using this approach to assess nations’ regulatory performance is problematic. When it comes to something like regulation, can there ever be (to continue the sporting analogy) a “level playing field”?

New Zealand, and other countries, face particular constraints and possess particular resources. Where these are historical, or natural, or otherwise outside the current control of nations, what are comparative statistics measuring? Are they saying more about how well countries have done, or are they simply measuring the impact of uncontrollable circumstances? What, for example, does New Zealand’s score in an international index of the competitiveness product markets say about the wisdom of our regulations rather than about our remoteness or the size of our population? What resources should New Zealand spend trying to improve its rank on this index rather than trying to achieve other goals?

Notwithstanding these questions, we argue that these indices, these surveys, provide useful information. They can supplement historical comparisons: they offer information about possibilities today that single-country historical studies do not. Other countries are different from New Zealand, but not so different that we can completely disregard carefully drawn comparisons. These surveys must say something about the performance of our regulation. The question is figuring out what that something is.

Surveys such as these ought to be part of debates on regulation in New Zealand. A number of points must be kept in mind, however:

• Surveys must be credible. This paper has discussed surveys by reputable organisations using published methods. Surveys that do not disclose methods deserve no regard.
• Surveys must be as informative as possible. Surveys that provide more detailed results are likely to be more useful than those that provide only aggregate results.
• Even credible surveys have limitations. These limitations will, since the surveys are conducted by reputable organisations, be stated alongside their methods and results, but an awareness of limitations must be carried into any policy discussions that use the survey results. All results are conditional on these limitations.
• Greater weight should be placed on results that are independently confirmed by different surveys. On the other hand, idiosyncratic and contradictory results warrant closer study.
• As mentioned above, the aggregate indices do not tell the whole story. Often the most valuable information (and the greatest flaws) of a survey will lie in the details.
• All surveys are subjective. When composing an index, or selecting what to measure and how, or framing survey questions, authors of surveys must make subjective judgements. It follows that the results of any survey are largely a function of the subjective judgement of those undertaking it. Numbers can give a false impression of objectivity. But in economic matters, quantitative analysis is often simply subjective, qualitative analysis expressed in numbers.
• A corollary to this last point is that surveys assume some ideal, even implicitly. In the case of the Heritage Foundation, the ideal (economic freedom) is spelled out, but all of the surveys considered in this paper are based on an idea of what regulatory systems should look like.
• As mentioned, the surveys usually overlook the benefits of regulation. And these benefits depend on the policy goals of the regulation in question. Unless we know what goals our and other countries’ regulations are designed for, and how well they are meeting these goals, we cannot know whether any particular rank is a good or a bad thing.
• Country-specific constraints should be taken into account when interpreting surveys. For example, in a small economy, industries may be naturally more concentrated than the same industries in larger economies. Location, distance, population density, and local culture may also materially affect scores and ranks.
• Finally, for better or worse, we ought not to get too concerned with small changes in the results of these surveys. A good score is no excuse for complacency, and a bad score is no excuse for despair. Good regulatory practice is about aiming to achieve a potential, however elusive, and it is not about climbing a league table. However, consistent trends may matter, large changes in rankings may matter, and the failure to improve poor rankings over time may matter. These things may matter, not because getting better ranks is an end in itself, but because they could indicate problems that are adversely and unnecessarily affecting the well-being of New Zealanders.