The Treasury

Global Navigation

Personal tools

Treasury
Publication

Demonstrating Performance: A Primer for Expenditure Reviews (June 2008)

Condition #7: Demonstrably Reduce Need by Improving Outcomes

The ultimate and only reason to fund an intervention is to improve outcomes. Impact measures validate interventions by showing how outcomes improved, and for whom. Ongoing funding is warranted only while there is every expectation – preferably a level of proof – that the results promised by its proponents were achieved.

Impact is gauged against goals set in #2, using the same measures used to establish need for intervention (#1). Intermediate outcomes specific to the intervention are often assessed in parallel. Impact is assessed by assessing outcomes for situations in which the intervention was, and was not, delivered (see Appendix 3 on comparison groups).

“It is far better to be roughly right, than perfectly ignorant”

Two main methods are used to demonstrate improved outcomes:

  • Impact is assessed directly when end outcomes are highly measurable, and robust comparison groups allow us to gauge the reduction in need due to the intervention.
  • Near term results and intermediate outcomes are assessed to confirm interventions at least induce some changes predicted by proponents of the intervention. (This method gives fast feedback and triangulation, and improves measurability and attribution.)

Using both methods often improves the results’ credibility with decision-makers:

  1. End outcomesjustify ongoing delivery (but give little insight into how to improve).
  2. Intermediate and near-term outcomes give early, more specific warnings of issues.
  3. Multiple measures build support when individual measures are weak or challenged.
  4. Multiple measure help test the logic model (and may suggest other approaches).
  5. With output information, results help leaders gauge problems and how to respond.
A Strategy That Works! (2003 vs. 1999)
Fatal accidents Down 20-28%
Hospital bed-days Down >22%
Safety related tickets Up 10-400%
Exceeding 100 km/h Down 14%
Speed related deaths Down 8-14%
Rear seat belts worn Up 40%
Deaths avoidable by belt Down 28%
Breath tests Up 23%
Drunk in charge (per test) Down 30%
Results Show Success AND Help Managers Improve Coverage
Results Show Success AND Help Managers Improve Coverage.

Remember …

  • Leaders must pay special attention to results that challenge ‘conventional wisdom’.
  • Acknowledging (or anticipating) the need to change often works better than denial.
  • A decision to change is not a bad result; the worst result is a failure to change.
  • Multiple measures and repeat measures build confidence that the impact is real.
  • Impact measures are only as good as the counter factual used to produce them.
  • Impact differs across groups and areas. If feasible, disaggregate results (see above).
  • Impact measures must focus on outcomes and groups used to establish need in #1.

So … Do Major Interventions Perform Credibly?

Strong commitment is needed from ministers, boards and managers to back the evidence and sustain the outcomes focus of major policies and programmes. This is particularly true when results do not sit well with prevailing opinions, activities or budgets.

Credible performance stories[5] can be told if major interventions worked as planned. The results chain lays out what was expected from the programme (see #2). Quantitative and qualitative measures – while not comprehensive or conclusive - should reveal whether major expectations are being met.

Credible performance stories mirror the shape of this paper, ie:

  • Indicators will show the intervention addresses needs that remain relevant today
  • Major results sought, goals and measures were clear, and are being reported on
  • Managers have been delivering major outputs efficiently, while maintaining quality
  • People or institutions with the need got the vast majority of the output produced[6]
  • And most importantly of all, outcomes improved as predicted in areas of need

Good leaders strive to protect what works, while improving or changing what doesn’t. Good leaders always look for new ways to achieve their mission: ‘improve outcomes’.

Parts of a ‘good news’ story from transport are presented in #7: ‘A strategy that works!’ Good news stories typically focus on outcomes, and efficient production and coverage.

‘Mixed news’ stories are more valuable, as they show where improvement is needed. Appendix 1 illustrates some performance stories that mix ‘good news’ and ‘bad news’.

Leaders seldom take issue with positive results or positive reviews. But optimism bias or distorted reporting can make ministers, boards and managers conclude things are going well when stark evidence exists to the contrary. Warnings of problems include:

  • sudden silence about measurement approaches (eg, that had been hyped)
  • burying bad news (sometimes while grandstanding on selected, positive results)
  • lengthy debate on the merits of ‘bad’ measures (without action to improve them)
  • sudden termination of measurement systems, or personal attacks on ‘measurers’
  • regressive focus on output measures, without demonstrating results for citizens.

When results are not as expected, it is crucial to look first at outcomes and ask why. If efficiency, delivery or coverage is an issue, perhaps the intervention can be improved. Assess risk, and then decide whether to redesign, downsize, replace or stop the intervention. Too often, bad news breeds indecision (perhaps masked as ‘review’).

One hallmark of great leaders is their use of contrary information to create the sense of urgency needed to improve public services. Think about it. Good news just encourages staff to preserve what they do, or do even more of the same. Bad news takes us all out of our comfort zone, and can be used to drive management, innovation and progress.

Notes

  • [5]Three examples are presented in Appendix 1. Examples of measures are presented on page 5
  • [6]If this is not so, an essential precondition to improved outcomes has not been met
Page top