Difficulties with measurement of output and productivity

The measurement of output is a crucial part of calculating productivity and assessing performance for any type of organisation. National accountants measure the performance of entire economies in the commonly quoted measure GDP. National accounts give two main measures of the total output of an economy, current price GDP and constant price GDP. The first gives the total monetary value of the output of the economy each year, using the prices prevailing in that year; the second removes the effects of price changes and represents the actual or real volume of output produced in the economy. Constant price, or real volume, measures are typically more interesting from an economic growth perspective because they identify when an economy produces more goods and services. In this paper, when we refer to output, we will mostly be referring to real output. We are also more concerned with growth in output rather than the actual level of output.

For many government services, output growth is assumed equal to input growth...

Typically, the output from public services has been measured poorly in national accounts, where in many cases changes in input levels (often hours worked) are used as a proxy for changes in output levels. This means that the growth rate for the government’s actual production is not observed and is simply approximated by the growth rate in the number of hours worked. Such a framework is deficient because it does not allow for changes in the amount of output produced per input. This means productivity, the amount of output per unit of input, is assumed to be constant. Labour productivity growth was 2.6% per year during 1988 to 2005 in the well-measured market sector of the economy and there is no reason to believe that productivity growth should be zero in the public sector (Statistics New Zealand 2006).

…which means productivity is constant over time.

Constant productivity assumptions are concerning because of two main reasons. First, growth in GDP is used widely to judge how well an economy is doing over time and in comparison with other economies. Analysing economic growth is particularly important in forming and adjusting economic policy. Constant productivity assumptions for a large proportion of the economy, such as the public sector, may cause understatement or overstatement of growth and could result in misleading conclusions, particularly in cross-country comparisons where the size and structure of public sectors vary widely. The second main reason is that, without real output and productivity measurements, we have no idea how well the government is using its resources. Without productivity growth measures we cannot tell whether tax payer funds are being put to better or worse use than in the past.

For many government non-market services, the unit of output is difficult to define…

The constant productivity assumption has been made in the past because the measurement of output and productivity in the public sector suffers two prominent difficulties. The first is defining the units of output. For some services, this is easier; for example, an output in health could be the completion of an individual operation or a consultation with a general practitioner. For others, it is more difficult; for example, New Zealand’s defence forces contribute to a safer environment for all New Zealanders but it is very difficult to quantify the output of defence. Health is an example of an individual service, as the output is delivered to individual consumers, whereas defence is a collective service where the output is consumed by the entire economy simultaneously. The relative ease of defining and measuring output in individual services has meant most progress in output measurement has been made for individual services.

Often, individual public services have private sector equivalents and the same output methodology should apply to both. For example, in private sector education, the amount spent on fees would be adjusted for price inflation to get a real volume of output. This would essentially amount to a volume measure based on the number of pupils (potentially with a quality adjustment) and a similar methodology could be applied to public sector education. In the case of government departments providing consultancy services, such as providing advice to ministers, methods used for calculating real output for privately supplied consultancy services could be applied analogously.

…and the value of output is hard to measure without market prices.

The second major difficulty with output measurement is the lack of market price information, due to public sector output being provided free or for an economically insignificant amount. This means that it is difficult to attain a value for each service. In the market sector of the economy, the price of a good or service is taken to represent the marginal value to the consumer; in the non-market environment, we have no such information. This has meant that in current price GDP, the total monetary value of public sector output is calculated from the sum of the input costs (thus assuming zero net operating surplus or profit).

The lack of price information means it is difficult to add together diverse units of output. To create measures of aggregate output, we need to sum together the various outputs a service produces. For example, in calculating total health output, we need to somehow combine the number of heart operations with the number of ingrown toenail removals. These two outputs have different values to the consumer and it is misleading to simply add one heart operation to one toenail operation to get a total of two operations. In a market situation, where the value to the consumer is represented by the price, aggregate output is calculated by multiplying the number of operations in each category by the price in each category before adding each type together. This weights each output by its value in the total aggregate output. Since public services are usually provided at no or below cost, other means of weighting each output are needed.

The ideal practice would be to weight outputs using a measure that represents the marginal value of the service to the consumer. In health, for example, this could be done using a measure like quality adjusted life years (QALYs), which gives an estimate of the total benefit of an intervention to the patient in terms of added life years. However, comprehensive sets of QALY information are often lacking. An alternative for services such as health or education is to use prices from private health care and education as proxies. This would assume that private education and health operated in competitive markets and that the quality of private and public provision is the same.

In the place of value, cost of production is often used.

In practice, it is difficult to estimate what value each output has and typically, in summing together public service outputs, cost weights are used. This means that each different type of output is weighted by the cost of providing that output before the outputs are added together. By doing this, we are using the relative per-unit costs as a proxy for the relative per-unit values to the consumer. In a competitive market, where all output is allocated until marginal cost is equal to marginal value, this may be valid. However, for a public service that might be under or over allocating services, this is not ideal and should be avoided if actual value weights are available.

Outputs often change in quality over time and this must be accounted for.

Measurement of output is also complicated by the changing nature of some outputs over time. Outputs often change in quality and this should be accounted for; for example, heart operations may be augmented by new technology and techniques, resulting in better outcomes for patients. Some quality change can be accounted for by differentiating output into categories that are as homogenous as possible. Doing this means that quality adjustment is recognised in relative volume changes between output categories. When it is no longer practical to further differentiate output, then quality adjustment can be recognised by marking up or down the volume of output according to a quality index. This effectively treats higher quality output as a higher quantity of output. For example, a higher quality heart operation could be worth 1.5 original heart operations.

Quality adjustment is a complex issue and remains problematic (not just in this area but in the construction of price indices, where it is important to distinguish between changes in price due to quality changes and “pure” price changes). Some aspects of the public service are more amenable to quality adjustment than others; for example, QALY measures can be used as an indicator of the quality of health care. For education, the amount of improvement in a cohort’s test results could be used to measure quality change, although this is complicated by the other influences on test results outside the control of the school system. In comparison, there is no obvious form of objective measurement of changes in the quality of policy advice provided to ministers.

Despite the difficulties involved, recently there has been pressure to improve public sector output measurement in the national accounts. The Atkinson Review commissioned by the United Kingdom’s national statistician and Eurostat’s Handbook on Price and Volume Measures in the National Accountsprovide advice to statistical agencies and set new standards for measuring output in national accounts (Atkinson 2005, Eurostat 2001). Both strongly discourage the use of inputs as a proxy for outputs in calculating output growth from public services in constant price GDP. For individual services, such as health and education, output measures based on input proxies are no longer deemed acceptable by Eurostat. For collective services, where output continues to be very difficult to measure, Eurostat and the Atkinson Review both concede that input measures may still be the most appropriate, provided they are as comprehensive as possible.

EU members must now use direct output measures.

Decisions made by the European Commission mean that European Union members are required from 2006 by legislation to use the standards set by the Handbook on Price and Volume Measures in the National Accounts (Decision 98/715/EC and 2002/990/EC). This means that from 2006, European Union members must use comprehensive direct output measures for individual services. The OECD will also launch a project in October 2006 to provide more detailed international guidelines for the development of volume measures of non-market output, in particular for education and health.[1]

Productivity

With output and input measures, we can calculate productivity...

The movement toward better output measurement of public services in the national accounts means that productivity has become to a limited extent, and will become increasingly, measurable using national accounts data (as has become routine for the majority of the market sector of the economy; see Black 2003, Statistics New Zealand 2006). The other ingredient needed to calculate productivity is input information. Depending on the productivity statistic required, we need the amount of labour input, the amount of capital input and the amount of input goods and services. The Atkinson Review recommends that, for public sector productivity calculations, the input measures should be as comprehensive as possible covering all three areas.

…however, input measurement requires care too.

All inputs must be measured in real volume terms either by measuring the input directly (ie, recording the number of hours worked) or by deflating expenditure on inputs by an appropriate cost index. The measurement of input from capital is often problematic and is often measured by capital consumption – the depreciation of fixed capital. This is not ideal, as it does not include the opportunity cost of the capital being employed in the public sector, and the Atkinson Review recommends that the appropriate measure is the flow of capital services. This involves adding an interest charge on the entire owned capital to capital consumption. It is also important to consider quality changes in the inputs in productivity calculations, as output may change due to different-quality workers, machines or intermediate goods and services being employed. However, as with outputs, making quality adjustment is difficult.

The ONS has recently calculated productivity growth in the public sector by subtracting the growth in all inputs, including intermediate goods and services, from the growth in gross outputs. Comparing gross output growth with total input growth provides an indication of how the amount of output the public is getting for each dollar of expenditure is changing when the effect of input price movements are removed. This type of measure is different from the type of measure typically used when considering productivity growth in the market sector. For the market sector, value added (gross output minus intermediate inputs) is usually compared with labour growth to get labour productivity, and then with both labour and capital input growth to calculate multi-factor productivity. The difference means that public sector productivity growth calculated in this way is not comparable with growth rates calculated for the private sector.

Macro-level productivity indicators do not tell the whole story.

It is also important to note that productivity calculated in this way is a residual of two difficult measurements – inputs and outputs – both of which may include considerable error. Productivity calculated in this way should not be used alone to judge a service’s performance, particularly given the politically sensitive nature of productivity conclusions. These macro-level indicators should feed into more comprehensive productivity studies that use micro-level information and look at changes in outcomes. The Atkinson Review recommended a process of “triangulation” where the productivity = outputs/inputs equation is looked at from multiple angles to build up an overall view of productivity.