Adjusting for quality
In the ONS’s first productivity measures, there was little treatment of quality improvement.
The ONS’s productivity measurements for general government and for health resulted in controversy, which was part of the motivation for the Atkinson Review. This controversy was in part due to disagreement about how output should be measured, where a major criticism concerned quality adjustment. Following the Atkinson Review, the ONS has made considerable attempts, in cooperation with spending departments and using expert advice, to improve the measurement of quality of output.
Education
This occurred first in a productivity analysis for education that presented both experimental input treatments and also experimental output series with new treatments of quality (UK Centre for the Measurement of Government Activity 2006a). The analysis aimed to improve on the quality measure used in the United Kingdom national accounts that was based on the past trend in General Certificate of Secondary Education (GCSE) exam results. This trend adjustment had not been updated since the measure was first introduced in 1998.
The ONS looked at new quality adjustment methods for education.
To remedy this situation, the ONS tested a number of different potential adjustments for quality and for the increasing value of education, adjustments that were based on suggestions from the Department for Education and Skills (2005). The first simply looked at more recent trends in GCSE results data. The second used progress between the Key Stages of the English compulsory education system, where a cohort’s Key Stage test results were compared with their results at the previous Key Stage (Key Stage examinations are usually taken at age 7, 11, 14 and 16). The third involved adjusting output to account for the increasing value of education as the economy becomes more productive, the idea being that as real wages increase, education provides more benefit to the student. To do this, the ONS made an adjustment based on the trend in real wages in the economy. The ONS discussed potentially using teaching assessment or class size as a measure of quality but used neither. Teacher assessment grades were found to be unsuitable because of frequent changes in the assessment method and the effect of class size on students’ performance is heavily debated in the international literature.
Adopting different adjustment methods gave a wide range of productivity growth rates.
To analyse productivity growth, a number of different methods of calculating the amount of input into education services were investigated, including changing the treatment of the cost of labour and using capital services instead of consumption. The total input measure was relatively insensitive to the method used. Combining the various input measures with the various quality and value adjusted output measures created a wide range of potential productivity growth rates from -2% per year to 2% per year over the 1998 to 2004 period, highlighting the significant differences alternative methods create.
It must be noted that the quality adjustments the ONS considered were based on the outcomes of the education system; namely, test results. Using outcomes, like test results, can be misleading due to the multiple contributors to outcomes, although this problem is reduced when a cohort’s improvement in standardised test is used. Ideally, the contribution from education service would be separated out from other inputs using regression analysis. However, the data requirements for such analyses are often prohibitive. Adjusting for the increasing value of education in a growing economy is more part of a human capital approach to examining the value of education output. A comprehensive human capital approach would examine the difference in lifetime earnings of people with different levels of education to obtain a value for education. Once again, data requirements for such a treatment are large and would also involve significant lags. The use of regression analysis and human capital are discussed by Christian and Fraumeni (2005).
Health
New methods of quality adjustment in health were investigated.
A major criticism of the article on health productivity by Lee was that the measures did not contain adjustments for changes in quality. The United Kingdom Centre for the Measurement of Government Activity (2006b) responded to this by producing new measures with various adjustments for quality. The Atkinson review also recommended that health care should be attributed a higher value as the economy becomes more prosperous and healthy individuals can earn higher real wages than previously. In this work, the ONS presented a wide range of productivity estimates, starting from those implied by the national accounts output measures unadjusted for quality, and using a range of different input series. These estimates implied that productivity decreased by between 0.6% and 1.3% per year over the 1995 to 2004 period.
The new adjustments changed the productivity growth rate from negative to positive.
The output measures from the national accounts were then adjusted for quality using a number of indicators including: survival rates, measured increases in patient health due to intervention, life expectancy, waiting times, improved outcomes from primary medical care, patient experience, and the use of value weights instead of cost weights for drugs used to control cholesterol (statins). Combining the quality adjusted measure with the input measures gave a range of 0.2% to -0.5% per annum productivity growth over the period from 1999 to 2004. Following this, an adjustment was made for the increasing value of health due to rising real wages. The third set of estimates indicated productivity growth of between 0.9% and 1.6% per year. This work demonstrates the considerable effect adjusting for quality and value can have, changing the measured productivity growth during the period from negative to positive.
The work on quality of health output was based on research carried out by the Department of Health and by the Centre for Health Economics at the University of York and the National Institute of Economic and Social Research (Department of Health 2005, University of York and National Institute for Economic and Social Research 2005).
In this improvement of health output measurement, the use of value weights instead of cost weights for statins presents an interesting example. The Atkinson Review recommended that the ONS and the Department of Health should identify treatments where marginal valuation and cost weights were likely to be very different. The Department of Health looked at statin use in the treatment of heart disease. Instead of cost weights for these drugs, the Department of Health used value weights based on added life years as a direct result of statin therapy. The value of taking the drug was calculated from estimates of the number of life years gained by patients taking the drugs. In this case, the value weight was over four times greater than the unit cost of the drug, increasing the overall output of the National Health Service.
Both the education and health articles presented a wide range of productivity estimates that made conclusions about performance difficult. In fact, following the publication of the health article, the ONS drew criticism from the Economist (Economist 2006) for publishing “confusing measures of NHS productivity”. However, the criticism seems unwarranted given that these articles were published as ongoing research into the measurement of National Health Service productivity, and the range of measures and methods were published to create discussion and public debate and stimulate work in the field. The ONS recognises that it will take a good deal of discussion and debate before a definitive measure of quality change can be accepted and applied as a standard. The experience of the ONS highlights the length of time required to build comprehensive and fully representative measures of output.
