6.2  Evaluation

Good information about the effectiveness of policy and the management challenges facing agencies is important in any public management system – particularly those looking to assess whether interventions have had an anticipated impact on outcomes. However, the New Zealand public management model has poor systems for evaluating and reporting the effectiveness of policy implementation (State Services Commission and Treasury, 2003). Arguably, until this is addressed it is unlikely that the model will not effectively support a managing for outcomes environment – primarily because without learning from effective evaluations agencies will not be in a position to appropriately shift resources from poorly performing programmes and policies to those that are more effective.

6.2.1  Evaluation in the New Zealand public management system

A recent report on evaluative activity in the New Zealand public management sector, undertaken by representatives of the State Services Commission and Treasury (2003), found that there is: considerable variation in the extent and quality of planning and use of evaluative activity across the New Zealand State sector [p. 14].

Examples of good practice were identified in terms of both deciding what evaluative activity to undertake, and the use of evaluative activity. The authors also suggested that three broad areas need to be addressed if the quality of evaluation across the public sector is to improve. These areas are: a culture of inquiry; poor coordination and prioritisation; and capability.

It could be argued that issues in these areas have arisen because no clear expectations around capability were expressed in the implementation of the New Zealand model – in part because there was a view that evaluation was so obviously important that agencies would build the capability they require in-house[8].

At the same time, central agencies have tended to distance themselves from any overarching analysis of strategic (and cross-sectoral) policy areas, and in some cases anecdotal evidence suggests that Treasury actively discouraged evaluation in the early to mid 1990s by supporting lower cost policy initiatives. This has shifted over time, and new budget bids must now have an evaluation plan associated with them. The fact remains, however, that no central capacity for evaluating policies deemed to be significant to the government exists, and this can create a disconnect between policy, resourcing and accountability decisions. For, how can a system effectively manage for outcomes when it does not evaluate the effectiveness of policies and programmes?

Central agencies have suggested that the issues identified be addressed by central agencies: building a culture of inquiry, initially by setting expectations of agencies and chief executives; promoting more coordinated evaluation activity; and supporting agencies to build the capability they require to undertake evaluation (pp. 32 – 36).

6.2.2  How can evaluative activity be promoted?

How evaluative activity can be built into a public management system is an issue a number of jurisdictions have attempted, and are attempting, to resolve. Boyle and Lemaire (1999) have identified three “waves” of governments seeking to introduce evaluation to their public management systems (drawing off work by Derlien):

• first wave – countries sought to institutionalise evaluation to improve effectiveness;
• second wave – evaluation was seen as a tool of public accountability via the budgetary process;
• third wave – countries are looking to both the first and second waves with the hope that evaluation will provide for good governance as well as accountability (pp. 1 – 3).

In comparing these three waves Boyle and Lemaire (1999) identified that four basic issues can influence how evaluative activity is built into a public management system. They suggest that these include the availability of sound data systems, a history of social science, the existence of appropriate capability and capacity, and the timeframe available for institutionalisation of evaluative activity. They also suggest that in developing a public management system that provides for evaluation it is important to take wider environmental issues into account, in order to create a balance between supply of and demand for evaluation. This links to the notion that all aspects of a public management system need to align if outcomes are to be efficiently and effectively achieved.

Bringing all of the points made above together, Boyle and Lemaire (1999) have identified a range of criteria to assist in determining how an evaluation regime can be provided for within a public management system.

As the appendix suggests, there would appear to be a need to provide for more demand and supply of evaluative activity across the New Zealand public management system. Demand needs to be driven by both Ministers and the Parliament, with both allowing for evaluation to be undertaken within a learning environment. Agencies that could assist in building this demand may include the Audit Office and central and lead agencies.

It would also appear that all policies and programmes should be subject to evaluative activity of some kind. This may simply involve monitoring of activity levels and simple performance indicators. In other instances, generally where a programme has been funded in a particular way, or where the policy or programme is likely to have a significant impact upon New Zealanders, then a full evaluation of those policies and programmes should be undertaken.

Finally, evaluation should be liked into a range of other public management systems, including the budget process, to allow for the information generated to influence subsequent decisions.

6.2.3  Conclusion

Evaluative activity – assessing whether interventions have had the anticipated impact – is a key component in a results-oriented system. The New Zealand public management system does not have a strong history of evidence-based decision-making. As noted, recent work by the central agencies has suggested that before evaluative activity can be embedded in the public management system a culture of inquiry needs to be generated. Officials have proposed that, initially, this be generated by central agencies setting expectations and using persuasion.

This section supports the need for generating a change in culture. It also suggests, however, that evaluative activity needs to be anchored in both the legislative and executive branches of government, and that some consideration needs to be given to how cross-cutting issues can be evaluated. It may be that central agencies will need to take a lead role in the latter area. Finally, this section also suggests that evaluative activity should be linked into other aspects of the public management system. In part, this should support the wider processes but it should also ensure that the findings of individual evaluations can be acted upon.

6.3  Performance indicators

“Good” performance indicators will be integral to effective accountability and evaluation systems. There are, however, inherent difficulties associated with developing “good” indicators. For example, Smith (1995) has suggested that unintended behavioural responses of performance indicators will include:

• tunnel vision – an emphasis by management on phenomena that are quantified in the performance measure scheme, at the expense of unquantified aspects of performance;
• suboptimisation – the pursuit of narrow local objectives by managers, at the expense of the objectives of the organisation as a whole;
• myopia – the pursuit of short term targets at the expense of legitimate long term objectives;
• measure fixation – an emphasis on measures of success rather than the underlying objectives (e.g. the waiting list);
• misrepresentation – the deliberate manipulation of data so that reported behaviour differs from actual behaviour;
• misinterpretation – although in possession of all the facts, bounded rationality might cause the controller systematically to misinterpret them, and to send the wrong policy signals to the agent;
• gaming – the deliberative manipulation of behaviour to secure strategic advantage; and
• ossification – organisational paralysis brought about by an excessively rigid system of performance evaluation.

Within a public management system where “accountability” is given a high weighting as a principle these perverse incentives can be strengthened, particularly in terms of seeking to reduce the level of specification used. It may be, however, that the issues could be addressed by providing for the use of “appropriate” indicators (i.e. those defined by the Auditor-General (2002)as those which are relevant to stakeholders; understandable, transparent and traceable; and complete), placing short-term expectations in a longer-term context, encouraging staff to take ownership of data coupled with increased audit, and assessment processes that take into account the nature of the organisation.

6.3.1  Different indicators for different organisations?

It has been suggested that different forms of organisation will need to consider different performance indicators, and that control levers will often differ depending on the nature of services delivered by an agency [see Controller and Auditor-General, (2002); and Cunningham and Harris, (2001)].

Focusing on results and action controls (i.e. outcomes and outputs respectively), it is possible to see that there could well be some organisations where it might be possible to focus on outcome assessment rather than output assessment, if some of the issues identified about attribution can also be addressed. On the other hand, it could be that the links between output quality indicators and immediate outcome indicators will be so closely linked that they could act as proxies for each other.

6.3.2  Performance indicators for outputs

Services delivered in the public service vary greatly. They range from the development of policy advice through to assessing biosecurity risks at the border or case managing beneficiaries. It is not, therefore, possible to identify common performance indicators across these outputs.

The OECD (2000) has identified a matrix that could assist in identifying what indications are most appropriate in looking to develop output performance indicators for different types of activity. It identifies possible indicators agencies should focus on when assessing output delivery: quality, quantity, timeliness and cost. In doing so, the matrix suggests that not all indicators will be applicable for all outputs.

For example, where outputs are relatively standardised and production processes are well-articulated an organisation should be able to focus on quantity, cost and timeliness indicators rather than quality indicators. Conversely, where outputs are not able to be standardised, and production processes tend to be less well articulated an organisation is more likely to focus on quality and timeliness indicators.

A good example of the latter would be the provision of policy advice, whilst an example of the former might be the processing of residence applications. The latter example creates some difficulties when we look towards a managing for outcomes environment: in part, because it leads to a question of how a results focus can be engendered if the quality of decisions are not used as an indicator of performance.

In other words, how can those organisations that have become dependant upon the use of performance indicators to drive cultural change now promote a managing for outcomes environment if they are not also coupling performance measures (i.e. for accountability reasons) with outcomes? There is no easy answer here, except possibly to look back to the cultural web outlined in chapter three and seeking change tools within that framework.