The Treasury

Global Navigation

Personal tools

Treasury
Publication

Can Population Projections be Used for Sensitivity Tests on Policy Models? - WP 03/07

2  A framework for assessing demographic sensitivity tests on policy models

Figure 1 shows the steps involved in demographic sensitivity testing. The population projections are generally carried out by the relevant statistical agency. Future paths for fertility, mortality, and migration are chosen. These are entered into a population projection model, such as the standard ‘cohort components’ model,[1] and future paths for population size and structure are derived. These paths are the ‘population variants’ referred to in the population projections literature. Complete descriptions of the variants consist of variables giving the size of each age-sex group, in each year of the projection. Many derived variables are, however, produced, such as dependency rates, numbers of school-age children, or total population size.

Figure 1 – Demographic sensitivity tests for policy models
Demographic sensitivity tests for policy models

Users of policy models generally take the population variants as given. Some models require all the detail produced in the population projections. Typical health expenditure models, for instance, require population numbers for every age-sex group. Other models require only a few derived variables. Some macroeconomic models, for instance, require nothing more than numbers for the total and working-age population. Paths for the required variables are entered into the policy model and the results compared, in an attempt to learn something about the sensitivity of the model’s results to demographic uncertainty.

Under what conditions does this procedure in fact provide informative results? Table 1 presents a simple typology of cases arising during sensitivity testing, and shows admissible conclusions under each case.

Table 1 – Coverage of possible scenarios and the implications for sensitivity testing
Coverage of empirically possible scenarios Conclusion about sensitivity of policy model to demographic assumptions
Case Fertility, mortality, and migration Population variables used in policy model Outcome variables from policy model
1 Wide Wide Wide Sensitive
2 Wide Wide Narrow Insensitive
3 Narrow Wide Wide Sensitive
4 Narrow Wide Narrow Insensitive
5 Narrow Narrow Wide Sensitive
6 Narrow Narrow Narrow No conclusion possible

In Case 1, coverage of empirically possible scenarios for fertility, mortality, and migration is wide. In other words, the sets of fertility, mortality, and migration assumptions that are entered into the population projection model in Case 1 jointly cover a broad range of plausible conditions. In Case 1, coverage of possible scenarios for population variables is also wide. This is likely when coverage of possible fertility, mortality, and migration scenarios is also wide, since changes in population size and age structure are completely determined by changes in fertility, mortality, and migration. Finally, in Case 1, coverage of possible outcomes from the policy model is wide. Entering different population scenarios into the policy model gives substantially different outputs. The correct conclusion is that the policy model’s results are sensitive to demographic assumptions. Uncertainty over future demographic variables carries through to uncertainty about the model results.

Case 2 is identical to Case 1, except that the range of outcomes from the policy model is narrow. Entering different population inputs into the policy model has little effect on the model outcomes, even though the population inputs cover a wide range of possible cases. The model user is entitled to infer that the model is insensitive to demographic assumptions. This is the result modellers generally prefer.

In Cases 3 and 4, coverage of possible fertility, mortality, and migration scenarios is narrow. Coverage of possible scenarios for the population variables used in the policy model is, however, wide. This combination of wide and narrow coverage does arise in practice. One example is when low and high migration assumptions differ markedly, and the only population variable used in the policy model is total population size. Wide and narrow coverage of the outcomes from the policy model lead to the same conclusions about the sensitivity of the model in theses cases as they do in Cases 1 and 2.

In Case 5, the narrow coverage of fertility, mortality, and migration carries through to coverage of population variables. Coverage of policy model outcomes is, nevertheless, wide. The fact that outcomes from the policy model vary substantially even when the population scenarios vary relatively little implies that the model is definitely sensitive to demographic uncertainty.

Finally, in Case 6, coverage is narrow for fertility, mortality, and migration, and for the population variables, and for model outcomes. The narrow coverage of model outcomes is consistent with the model being insensitive to the demographic assumptions, but it may simply reflect the fact that the population scenarios entered into the model covered only a small proportion of the plausible range. No conclusion about the policy model’s sensitivity to demographic assumptions is therefore possible.

Cases 1 and 2, in which wide coverage of possible fertility, mortality, and migration scenarios ensures wide coverage of possible population scenarios, never occurs when working with population variants. As later sections of this paper illustrate, this is because the set of plausible scenarios for fertility, mortality, and mortality trajectories is too large and multi-dimensional to be adequately represented by a small number of population variants.

Work with population variants only leads, then, to Cases 3-6. Whether or not modellers draw the correct conclusions from these cases depends on which cases the modellers believe to have occurred. The extent to which the policy model produces a wide range of outcomes during demographic sensitivity testing is readily observable. Modellers who encounter Cases 3 or 5 and observe a wide range of outcomes are therefore likely to assume that one or other of these cases has occurred; if the modellers are unfamiliar with the limitations of population variants, they might also assume that Case 1 has occurred. Similarly, modellers who encounter Cases 2 or 4 and observe a narrow coverage of possible outcomes are likely to assume that one of Cases 2, 4, or 6 has occurred.

Table 2 – Interpretation of results from a sensitivity test
Case that actually occurred Case that modeller believes occurred Modeller’s interpretation of the sensitivity test
3 or 5 1, 3, or 5 Correctly concludes that model sensitive to demographic uncertainty
4 2 or 4 Correctly concludes that model insensitive to demographic uncertainty
4 6 Incorrectly concludes that test uninformative
6 2 or 4 Incorrectly concludes that model insensitive to demographic uncertainty
6 6 Correctly concludes that test uninformative

Table 2 shows the possible combinations of cases and modellers’ beliefs, and the consequences for the correctness of the modellers’ interpretations. The first row of the table shows what happens when Cases 3 or 5 occur. Regardless of whether the modellers believe that Case 1, 3, or 5 has occurred, they still conclude, correctly, that the policy model is sensitive to demographic uncertainty.

The second and third rows show combinations occurring under Case 4, when the policy model produces only a narrow range of outcomes. If the modellers assume that Cases 2 or 4 have occurred, they conclude, correctly, that the model is insensitive to demographic uncertainty. If, however, the modellers assume that Case 6 has occurred, so that the narrow range of outcomes is simply a result of a narrow range of population scenarios, they conclude, incorrectly, that the test is uninformative.

Finally, the fourth and fifth rows of the table show combinations occurring under Case 6, when the range of population scenarios and model outcomes are both narrow. If modellers overlook the narrow range of population outcomes, and assume that Cases 2 or 4 have occurred, they conclude, without warrant, that the model insensitive to demographic uncertainty. If they assume that Case 6 has occurred they reach the correct but unhelpful conclusion that the test is uninformative.

Case 6 is evidently the most problematic. It does, however, often arise in practice. The following four sections describe some examples.

Notes

  • [1]Typically, a path for fertility, mortality, or migration is specified using a single variable: mortality paths, for instance, are often specified using life expectancy. To carry out population projections, entire schedules of age-sex-specific rates are needed. These schedules are derived from a model relating overall levels to underlying rates (see, for instance, Lee and Carter 1992). The discussion in this paper implicitly treats such models as part of the overall population projection model.
Page top