4 Assessing Risk, Uncertainty and Intangibles
4.1 Risk and Uncertainty
The previous chapter detailed the valuation of expected future costs and benefits to enable a comparison of the net benefits for alternative options. There will always be some difference between these estimates and the actual costs and benefits that are eventually realised. These differences can arise because of inherent biases in developing the cost benefit analysis or because of risks or uncertainties that were not foreseen at the time the analysis was undertaken.
The cost benefit analysis should include a risk assessment to determine how exposed each option is to future uncertainty. For example, if uncertain or highly variable costs are over- or under-stated by 5%, how much would this impact on the choice of preferred option?
The extent of the risk assessment should be determined on a case-by-case basis, depending on the scale of the proposal, the degree of uncertainty in key costs or benefits, and the risk tolerance of stakeholders. A comprehensive risk assessment should include:
- establishing the context for managing risks, including environmental risk drivers, organisational objectives and risk management capability, and how much risk stakeholders are prepared to accept
- identification of all the risks that could impact on objectives
- analysis of the risks by considering the sources of risk, their consequences and the likelihood that the consequences will arise
- determining if each risk can be accepted or needs to be treated
- consideration of actions, before and during implementation, to treat (avoid, mitigate, transfer) the risks, and
- communication to decision-makers so that they can make fully informed decisions based on the potential impact of risks and bias on each alternative option.
4.2 Optimism Bias
Risk can take many forms. The most familiar form of risk to outcomes is that the estimated future costs arising from the given proposal are overly conservative, or the benefits are overly optimistic. That is, the analysis does not fully reflect the possibility of cost-overruns, short-falls in demand or implementation timing delays. There is a demonstrated, systematic, tendency for analysts to exhibit optimism bias when preparing spending proposals.
Optimism bias can be reduced by making explicit adjustments to key assumptions and variables to improve estimates, for example by explicit adjustments to increase costs and reduce, or delay, benefits.
Both of these approaches assume that that the amount of bias and the level of overall risk can be reasonably estimated. However this is often not possible. Generally a sensitivity analysis approach is preferred and should also be used to test and demonstrate how the results of the analysis vary as individual assumptions, costs or benefits are changed.
4.3 Sensitivity Analysis
Sensitivity analysis is a form of quantitative analysis that examines how net present values, total cost, or other outcomes vary as individual assumptions or variables are changed. This approach can be used to test the robustness of the analysis as well as allowing for optimism bias and uncertainty about future cash flows.
Sensitivity analysis can help draw attention to those factors that require especially careful assessment or management. This analysis can address two key questions. Would the proposal still be worthwhile pursuing if some of the key assumptions do not eventuate? Are there actions that can be taken to reduce the risks before accepting a particular option? Sensitivity analysis can help in forecasting uncertainty and in assessing and treating project risks.
The sensitivity analysis needs to be well designed and clearly presented. The analysis should give a realistic picture of the extent to which the selected option is still worthwhile pursuing even if there are significant changes in key variables.
The decision about which form of sensitivity analysis to undertake and the effort to invest should be made on a case-by-case basis, depending on the scale of the proposal, the degree of future uncertainty around key costs or benefits, and the risk tolerance of stakeholders.
Sensitivity analysis can also be used as part of the monitoring and risk management regimes that form part of the implementation. This should ensure that the risks key variables are subject to are effectively monitored and managed.
- AS/NZS 4360:1999 Australian/New Zealand Standard “Risk Management”, http://www.standards.co.nz/default.htm.
- Flyvbjerg, B. & Holm, M.S. & Buhl, S. (2002). ‘Underestimating Costs in Public Works Projects: Error or Lie?’ Journal of the American Planning Association, Summer 2002, Vol. 68 Issue 3, pp.279-96.
- Refer to UK Green Book, Chapter 5 at http://greenbook.treasury.gov.uk/chapter05.htm for further detailed guidance on managing optimism bias.
- Ministers agreed that quantitative risk analysis should be used as the basis for appropriations and ongoing access to finding for implementing major IT projects, unless relevant Ministers agree that the size and nature of the project does not warrant this approach. Major IT projects are defined in Cabinet Office Circular (01)4 “Monitoring Regime for major Information Technology (IT) Projects” at http://www.dpmc.govt.nz/cabinet/circulars/co01/4.html.