Providing advice amongst the uncertainty of the COVID-19 pandemic created new analytical challenges for the Treasury and other government agencies. This blog describes how the Treasury advised from an economic perspective on options for the Government’s health response. It explains the exceptional context in which this advice was developed and how our assessment tools and principles were used. We hope this will support constructive discussion and improve future advice.
COVID-19 challenged traditional economic analysis
In February 2020, the rapid emergence of the COVID-19 pandemic highlighted the risk of the New Zealand health system and critical public sector functions becoming overwhelmed if COVID-19 took hold in New Zealand. While there was significant uncertainty, initial modelling suggested that more than two-thirds of the population could become infected, with up to 14,400 deaths, if there was little or no intervention.
By March 23, when New Zealand moved into Alert Level 4, there was sufficient information to know that if a response was not forthcoming, it was likely that the virus would take hold in New Zealand. While there was not enough information to know exactly what the impacts would be, we did know that without action there were significant risks of substantial health, economic and social costs.
The restrictions on economic and social activity under the Alert Level system aimed to curtail the spread of COVID-19 and these associated negative impacts. However, on the flipside, it was also clear that there would be significant economic and social costs from the restrictions themselves.
The challenge this posed for policymakers was weighing the costs of restricting economic and social activity against the costs from the spread of COVID-19 if activity was not restricted. An approach that economists typically use when there are different policy choices, with different mixes of costs and benefits, is Cost Benefit Analysis (CBA). CBA quantifies and monetises the different types of costs and benefits so that they can be aggregated and compared across policy options. The relative probability of different costs and benefits under different policy responses is typically taken into account.
The Treasury used CBA principles to inform our judgements about the need for containment measures, including that the costs of initial containment would be far outstripped by the costs of an uncontrolled outbreak. While limited information and the timing of these rapid decisions meant we were only able to consider “order of magnitudes” of costs and benefits, these principles helped us to structure our thinking.
The Treasury’s work in the early stages of the pandemic particularly focussed on understanding the potential economic impacts of COVID-19 and the related response measures. We modelled several scenarios to reflect the high degree of uncertainty we faced. This modelling showed large falls in economic activity and increased unemployment, and also assessed the impact of additional fiscal support.Subsequent work looked at the risk of lost jobs and falling incomes, compounding the impacts on mental health of social distancing and the risk of a disproportionate impact on existing vulnerable groups.
While we used the principles of CBA, there were both practical and methodological limits on conducting a formal, traditional CBA for three main reasons: uncertainty, complexity and valuation.
The COVID-19 crisis is novel. A significant analytical challenge at the beginning of the pandemic was radical uncertainty, with very extreme outcomes unable to be ruled out, and little to no evidence on their likelihood. The possibility of extremely negative health outcomes spurred action, but there was not enough information to know with certainty the best course of action.
There were a range of uncertainties at the beginning of the pandemic:
- There was uncertainty about fundamental disease characteristics, including how far and fast the outbreak would spread, and both immediate and on-going health impacts on the New Zealand population. Undetected transmission and a risk of losing the ability to bring the outbreak under control were real possibilities.
- Compounding this uncertainty was that the immediate and more medium-term economic impacts — both domestically and globally — could be modelled but not be known with precision.
- There was also uncertainty about the impacts of policy responses. For example, the impact of different Alert Level settings on the rate of transmission was uncertain, as was the behavioural response.
Further complexity arose from the asymmetric nature of these uncertainties. That is, we had the challenge of assessing actions that would cause quantifiable economic costs, to avoid highly uncertain but potentially devastating and irreversible health costs.
In the face of such uncertainty, there are limits to using fully quantified and monetised CBA, particularly where learning from observation, adaptation, keeping options open and sequential decision-making were needed.
In particular, inaction in the face of uncertainty about the outbreak dynamics and effectiveness of control measures risked irreversibly ruling out some choices and forcing others. For example, waiting for more information and conducting a comprehensive CBA could have resulted in the virus spreading to the extent that the health system was overwhelmed, or a later lockdown was less effective in containing the virus. With the risk of uncontrolled infection, undetected community transmission or super spreader events, the problem risks being shifted to the future and amplified.
Decisionmakers moved quickly, recognising that buying time for more information to emerge had a high option value and was the ‘least-regrets’ approach.
With high levels of uncertainty, a CBA can present spurious precision, even with sensitivity testing. Impacts that suggest trade-offs in the short term may turn out not to be trade-offs over time. For example, a stringent response with a higher up-front economic impact may only be needed for a relatively short period compared to voluntary measures or a targeted ‘least-cost’ approach. More severe restrictions have greater short-term economic and social impacts but may reduce longer-term economic damage if they successfully shorten the duration of a lockdown and avoid cycling back into more restrictive measures.
The second limitation to the use of formal CBA was the complexity in decisionmakers having a range of interconnected objectives. There is no simple trade-off between health outcomes and economic activity: they are interdependent. Healthy populations result in stronger economies, and a larger economy has more resources to devote to improving and extending lives. We would have had an economic downturn, no matter what path we took to contain the virus. Even without the official restrictions, many people would have still chosen to isolate in order to protect themselves, and we would still have been exposed to a weakened international economic environment.
Differential distributional impacts also increased complexity. The impacts of COVID-19 and response measures were known early on to have uneven impacts on people and groups. For example, a poorly controlled outbreak would affect the health of older people and those with comorbidities the most. However, a strict lockdown may exacerbate existing inequalities across ages and occupation groups, including those unable to work remotely, and may create additional social issues for some people.
In order to produce CBA, the challenge of complexity can be addressed with detailed modelling work, specifying objective functions, outlining causal links, identifying proxy indicators, parsing evidence and estimating likelihoods. This requires careful modelling, including to take formal account of uncertainties and risks, which takes time.
A key question is whether time invested in formal analysis will be timely enough to inform better decisions. In the first phase of the response, during the imposition of Alert Levels 4 and 3 nationally, the extent of formal modelling needed to address the trade-offs and complexities adequately was infeasible due to the uncertainties COVID-19 brought.
The third challenge in using a CBA approach was valuation.
There are challenges in quantifying the value of the wide range of costs and benefits of different strategic policy responses to COVID-19, including the need for a number of ethical and moral judgements in valuing health benefits, mortality, harm reduction and the impacts on social cohesion, .
There are, for example, competing methodologies for putting a value on the lives saved from restricting the spread of COVID-19, which lead to significantly different CBA results from different containment policies. Two commonly-adopted methods to value health and life outcomes are the concepts of Quality-Adjusted Life-Years (QALYs), which are used by Pharmac to make choices on the funding of medical treatments, and the Value of Statistical Life (VoSL), which is used in road safety investment cases, and is based on surveys of what people are willing to pay for marginal improvements in traffic safety measured in terms of risk to life.
QALYs tend to be lower than VoSL, particularly for a pandemic that most significantly impacts the older population, as it adjusts the value of a life saved by the amount of expected remaining years of life and sometimes also for the expected ‘quality’ of those years. VoSL analyses tend to justify more restrictive measures, while QALY analyses are more likely to suggest that the economic and social costs outweigh their benefits.
While both methods have value in their specific decision-making contexts, it is an untested leap to assume they would be valid for a CBA approach to COVID-19 decision-making where impacts on life and health are quite different in causal origin and incidence across the population.
Treasury CBAx guidance advises that the validity of quantification tools is very dependent on context. Both the QALY and VoSL methods for valuing lives deal with marginal risks and may underestimate the value people attach to the health harm caused by COVID-19 or the value of avoiding an erosion in trust that would result from a ‘treatment lottery’ in an overburdened hospital.
A learning regulatory response was required
The scale and immediacy of the projected impact of COVID-19 meant a non-traditional regulatory response was needed initially. With agencies across the public sector, the Treasury recommended urgent action. On 23 March, Cabinet suspended the Regulatory Impact Analysis (RIA) regime — its ‘peacetime’ expectations for ordinary analysis, disclosure and consultation for policy decisions.
Alternative policy evaluation frameworks and tools were adopted, informing quick action to preserve options, protect critical public services, maintain social cohesion, and promote collective purpose.
The risks generated by the temporary suspension of the RIA regime were addressed by the use of sunset clauses that would enable fitness-for-purpose assessments prior to any extension of regulation. Public sector agencies were asked to include key information on impacts in Cabinet papers and to engage with affected parties. Learning and adaptation resulted in a more robust framework for enabling safer economic activity while limiting transmission risk. For example, feedback from businesses navigating the essential services rules resulted in changes to enable more activity and better targeting of risk mitigation.
Due to the urgency of the required response, and the challenge in identifying the ‘optimal’ strategy ex-ante, officials prioritised protecting existing essential services and targeting risk. The high transmissibility and uncertain prevalence of COVID-19 meant inaction could create a possibly irreversible outcome. To preserve future options, a precautionary approach was taken — while economic activity can recover, lives lost cannot be restored.
Scenarios and sensitivity analysis of economic impacts
The Treasury quantified the economic costs of COVID-19 and policy responses where feasible and practical. From March 2020 the Treasury was estimating and quantifying the direct economic effects of the Alert Levels. Initial scenarios were publicly released in April, and May, 2020. Further estimates were published in the Treasury’s Budget Economic and Fiscal Update (BEFU), with a range of scenarios.
For example, the BEFU assumed that two weeks at Alert Level 3 compared to Alert Level 2 would have resulted in approximately $1.4 billion of lost GDP, driven by the closure of public-facing businesses, with restrictions on public transport, early childhood education and school capacity also limiting workforce availability. Two weeks at Alert Level 2 compared to Alert Level 1 would cost approximately $900 million, driven by physical distancing requirements that affected public transport and workplaces, limiting workforce participation and productivity.
Importantly, the scenarios signalled the potential economic impact from COVID-19 and set a new baseline from which to assess regulatory choices.
This enabled advice to consider the economic impacts of remaining at higher Alert Levels for longer (in terms of lost output relative to pre-COVID-19 GDP).
For example, different scenarios showed the economic effects of multiple escalations to higher Alert Levels. This was viewed as likely to be more damaging than successfully maintaining low or no infection prevalence and allowing economic and social activity to resume more quickly and durably.
In addition, given the publication lag in the main official macroeconomic indicators, officials developed sources of raw high-frequency data (such as card transactions and transport movements) to monitor activity and assess impacts in closer to real-time. The New Zealand Activity Index was created as a summary high-frequency indicator to gauge the immediate impacts of the Alert Levels. This high frequency data helped to inform the refinement of the Treasury’s economic impact estimates as the crisis unfolded.
These quantitative approaches were limited to the direct ‘first-round’ economic impacts of the regulatory response. They were not able to provide insight into whether the economic impacts on firms and individual might get worse over time. The longer-run scarring consequences of firm distress and failure and labour market detachment are complex and good evidence on these effects is not yet available. To help understand these effects, we tested perspectives with bank economists, businesses, unions, iwi, and overseas counterparts, as well as with lead policy agencies.
Understanding the dynamics of COVID-19 transmission
To assess intervention points and likely behavioural responses, we also sought to ground advice in epidemiological and microeconomic understandings of the virus and regulatory responses.
Early and relatively simple epidemiological models have been criticised for the inaccuracy of their case, hospitalisation, and death rate projections. They can however serve as a baseline case of uncontrolled outbreak. Against these baselines, modelling different actions and their relative impact has helped to narrow the range of feasible options and quantify the impacts on transmission, based on the interactions between the underlying virus biology, human behaviour and regulatory controls.
As the pandemic progressed, deterministic approaches became less useful. This was particularly the case in New Zealand, where the early border closure limited cases and contact tracing workloads to very low numbers compared with overseas. Time-series techniques fitting curves based on observed case data helped inform near-term predictions, while structured modelling approaches evolved to incorporate detailed data on actual cases, their genomic characteristics, and testing, contact tracing and isolation system performance. The Treasury worked closely with modellers such as Te Pūnaha Matatini on modelling to inform key policy decisions such as the probability of elimination during an outbreak, to judge the impact of different public health interventions, and to assess the likelihood of whether further strict controls would be required.
What next for CBA and our response to COVID?
A big challenge with applying a precautionary principle to policy — particularly regulatory policy — is when to switch back to ‘normal’ policy practices and when more conventional policy tools, like CBA, will support decision-making.
Should another new community case emerge that cannot be connected to the border, there will still be uncertainty about the optimal timing, stringency, and duration of the response. However, the uncertainty is reducing over time with more evidence about the previous episodes. The extent of undetected transmission and patchy or lagged health surveillance information mean that taking time to trace out the wider costs and benefits to inform a decision needs to be balanced with mapping out the potential scenarios and modelling and qualitatively estimating best-guess and worst-case outcomes.
Decisions will always be based on a mix of qualitative and quantitative information and expert judgement. The balance between these will shift, depending on the nature of the decision and the information available.
The route back to ‘normal’ economic arrangements will depend on the evolution of the virus and the measures underway to combat it, including vaccine deployment. Factors within the control of the Government include combinations of more targeted containment, vaccination strategies, therapeutics, and reliable testing and quarantine of travellers.
Do we now have enough certainty to use formal CBA?
The prospects for CBA to helpfully inform decision-making are now greater. We have learned more about the nature and sources of transmission and mortality risk, as well as the impacts of infection control measures, and the likely performance of the response system in the event of an outbreak. Our initial actions have bought us time to address complexity in our interconnected objectives. Together, these developments reduce the limitations on CBA that were sketched out at the beginning of this post.
Consistently applied CBA (or a variant, such as switching point analysis, complemented with multi-criteria analysis) could be more helpful now for some types of decisions. The best place to apply more fully articulated CBA is likely to be with a series of incremental regulatory decisions based on quantitative risk assessments. These could include, for example, the progressive opening of the borders, the use of face coverings in certain situations, or alternative options for testing or vaccination. These may be ‘marginal’ decisions, but they still generate potentially major impacts and risks, requiring the assessment and quantification of a wider set of wellbeing impacts.
Cost-benefit analysis can usefully include wider wellbeing impacts, which are many and varied in the case of COVID-19. Examples include the psychological impacts of unemployment and job insecurity, impacts on mental health, and education quality.
Choices also create uneven impacts, so incidence and distributional consequences need to be incorporated.
When it is necessary to consider optimal regulatory design choices and assess cost-effective combinations of measures for a given scenario, CBA provides a transparent process for exposing a wide set of objectives, bringing in quantification where possible, standardising measurement, and identifying key gaps in knowledge or ongoing uncertainty. Importantly, sensitivity analysis within a CBA framework can provide a sense of which design choices — and which uncertainties — matter most.
CBA may also bring benefits in terms of scrutiny and structured input to adaptive policymaking. The Treasury encourages ongoing ex-post assessment, particularly where ex-ante analysis has not been completed. This will help substitute for normal expectations of policy evaluation, consultation, and transparency when complexity, uncertainty, and particularly irreversibility prevents timely ex-ante assessment.
As part of returning to business-as-usual, the temporary suspension of the RIA regime ended in July. RIA requirements have also been amended in order to provide for a more enduring and targeted exemption where a ‘declared’ emergency action is still needed. Ex-post regulatory reviews or sunset clauses may also be considered as tools to manage the risks associated with RIA exemptions in the future.
The challenges of uncertainty, complexity and valuation meant that CBA was of limited value in the early stages of the COVID-19 pandemic, when quick decisions were needed and risks were asymmetric. Nonetheless, a range of quantitative information was used where available, including both economic and health impacts, which were incorporated into advice to provide a sense of the order of magnitude impacts of decisions. Qualitative information from key stakeholders added insight and enriched judgements, and we are grateful to the many individuals and organisations who shared their insights and wisdom so freely at what was, for many, a very difficult time.
As time has passed and the context changed, there is now a greater opportunity for CBA to more helpfully inform decisions on the COVID-19 response. Uncertainty is still pervasive but has reduced and we understand much more about the impacts of COVID-19 regulatory decisions.
As a result of COVID-19, we have learnt – and are continuing to learn – how to adapt our analytical tools and advice to reflect different types of uncertainty, including through integrating input from stakeholders with more quantitative analysis. As for New Zealand as a whole, COVID-19 has challenged the Treasury to adapt and work differently.
 Modelling conducted in the early days of the pandemic suggested large numbers of hospitalisations and deaths:
- Wilson and Baker in March 2020 suggested a range of 44 – 64 percent of the population could have become infected with 8,600 – 14,400 deaths. Available online at: https://www.health.govt.nz/system/files/documents/publications/report_for_moh_-_covid-19_pandemic_nz_final.pdf. Date accessed: 08 Dec 2020.
- Modelling by Te Pūnaha Matatini in March 2020 suggested that up to 89 percent of the population could have become infected and 1.67 percent could have died if spread were not controlled. Available online at: https://cpb-ap-se2.wpmucdn.com/blogs.auckland.ac.nz/dist/d/75/files/2017/01/Supression-and-Mitigation-New-Zealand-TPM-006.pdf. Date accessed: 08 Dec 2020.
 Available online at https://www.treasury.govt.nz/publications/tr/treasury-report-t2020-973-economic-scenarios-13-april-2020. Date accessed: 2 February 2021
 Diana Cook, Phil Evans, Hana Ihaka-McLeod, Kara Nepe-Apatu, Jez Tavita and Tim Hughes (July 2020) He Kāhui Waiora: Living Standards Framework and He Ara Waiora, COVID-19: Impacts on Wellbeing. New Zealand Treasury. Available online at: https://www.treasury.govt.nz/publications/dp/dp-20-02. Date accessed: 13 Jan 2021.
 John Kay and Mervyn King (2020) Radical Uncertainty: Decision-Making Beyond the Numbers, W. W. Norton & Company.
 Davies and Grimes (2020) COVID-19, lockdown, and two-sided uncertainty. Available online at: https://www.tandfonline.com/doi/full/10.1080/00779954.2020.1806340. Date accessed: 08 Dec 2020.
 This appears to be reflected in emerging cross-country evidence. See for example IMF (2020) World Economic Update, A Crisis Like No Other, An Uncertain Recovery, Available online at: https://www.imf.org/en/Publications/WEO/Issues/2020/06/24/WEOUpdateJune2020. Date accessed: 08 Dec.2020.
 Jensen, Kirsten; Thompson, Christopher. Valuing Impacts. Policy Quarterly, [S.l.], v. 16, n. 1, mar. 2020. ISSN 2324-1101. Available at: https://ojs.victoria.ac.nz/pq/article/view/6357. Date accessed: 08 Dec.2020. doi: https://doi.org/10.26686/pq.v16i1.6357.
 There is a vast literature on the use of CBA in moral and ethical questions. See for example Bennis et al (2010) Available online at: http://home.uchicago.edu/~bartels/papers/Bennis-Medin-Bartels-2010a-POPS.pdf. Date accessed: 08 Dec 2020.
 The QALY itself is not given a monetised value as such but is derived from the Government’s previous funding decisions as a way of consistently treating the health benefits of treatment choices. Available online at: https://pharmac.govt.nz/assets/cost-utility-analysis-explained-2015-08.pdf. Date accessed: 08 Dec 2020.
 See, for example, Greenstone, Michael and Nigam, Vishan, Does Social Distancing Matter? (30 March 2020). University of Chicago, Becker Friedman Institute for Economics Working Paper No. 2020-26, Available at SSRN: https://ssrn.com/abstract=3561244 or http://dx.doi.org/10.2139/ssrn.3561244.
 See, for example, Wilkinson, Bryce (9 April 2020). Quantifying the wellbeing costs of COVID-19. Available online at: https://www.nzinitiative.org.nz/reports-and-media/reports/research-note-quantifying-the-wellbeing-costs-of-covid-19/document/647. Date accessed: 28 January 2021.
 While people tend to strongly prioritise health over wealth, these priorities appear to change as the experience of COVID-19 deaths and income losses evolves. Available online at: https://cepr.org/sites/default/files/news/CovidEconomics22.pdf#Paper3. Date accessed: 08 Dec 2020.
 These focused on governance mechanisms to identify unintended impacts within decision papers and through scrutiny in Parliament. The revised arrangements include a new framework for impact analysis in emergency situations and have been formalised in Cabinet Circular CO (20)2. Available online at: https://dpmc.govt.nz/publications/co-20-2-impact-analysis-requirements-html. Date accessed: 08 Dec 2020.
 Available online at: https://www.treasury.govt.nz/publications/tr/treasury-report-t2020-973-economic-scenarios-13-april-2020 and https://www.budget.govt.nz/budget/forecasts/befu2020.htm. Date accessed: 28 Jan 2021.
 Available online at: https://www.treasury.govt.nz/publications/research-and-commentary/new-zealand-activity-index. Date accessed: 08 Dec 2020.