Using Integrated Administrative Data to Identify Youth Who are at Risk of Poor Outcomes as Adults (AP 15/02)

Abstract#

This paper summarises findings from an analysis of integrated administrative data seeking to identify the characteristics of young people aged 15 to 24 who are most at risk of poor longterm outcomes. The research is part of a broader 'social investment approach' by government agencies seeking to target services more effectively towards those most at need and reflects the recognition that such an approach requires better evidence about who these at-risk groups are. The analysis identifies those characteristics in the administrative data that are most predictive of a range of future poor outcomes and how this changes over the course of a young person's entry into adulthood and identifies groups of young people at particular risk at different ages.

The analytical paper is being released alongside an A3 document which summarises the results of the analysis, and a set of detailed excel tables. See links below.

Disclaimer#

The views, opinions, findings, and conclusions or recommendations expressed in this Analytical Paper are strictly those of the author(s). They do not necessarily reflect the views of the New Zealand Treasury, Statistics New Zealand or the New Zealand Government. The New Zealand Treasury and the New Zealand Government take no responsibility for any errors or omissions in, or for the correctness of, the information contained in this Analytical Paper.

The results in this report are not official statistics, they have been created for research purposes from the Integrated Data Infrastructure (IDI) managed by Statistics New Zealand.

Access to the anonymised data used in this study was provided by Statistics NZ in accordance with security and confidentiality provisions of the Statistics Act 1975. Only people authorised by the Statistics Act 1975 are allowed to see data about a particular person, household, business or organisation and the results in this paper have been confidentialised to protect these groups from identification.

Careful consideration has been given to the privacy, security and confidentiality issues associated with using administrative and survey data in the IDI. Further detail can be found in the Privacy impact assessment for the Integrated Data Infrastructure available from www.stats.govt.nz.

The results are based in part on tax data supplied by Inland Revenue to Statistics NZ under the Tax Administration Act 1994. This tax data must be used only for statistical purposes, and no individual information may be published or disclosed in any other form, or provided to Inland Revenue for administrative or regulatory purposes.

Any person who has had access to the unit-record data has certified that they have been shown, have read, and have understood section 81 of the Tax Administration Act 1994, which relates to secrecy. Any discussion of data limitations or weaknesses is in the context of using the IDI for statistical purposes, and is not related to the data's ability to support Inland Revenue's core
operational requirements.

Executive summary#

Purpose of the paper#

This paper summarises findings from an analysis of integrated administrative data aiming to identify the characteristics of young people aged 15 to 24 who are most at risk of poor long-term outcomes. The work is part of a broader emphasis by government agencies to target services more effectively towards those most at need and reflects the recognition that such an approach requires better evidence about who these at-risk groups are.

The Treasury has identified the need for the state sector to play a particular role in helping the most disadvantaged to participate in society and the economy and has noted the importance of agencies doing this through working innovatively and collaboratively across agency boundaries.[1] This has driven the development of a 'social investment' approach to decision making about government investment in social services. The social investment approach involves “using information and technology to better understand the people who need public services and what works, and then adjusting services accordingly”.[2]

This work represents one in a number of steps towards implementing a social investment approach. The overall work programme was led by the Ministry of Education, with the analysis summarised in this paper being led by the Treasury's Analytics and Insights team, in collaboration with a number of other agencies, and using integrated administrative data held in Statistics New Zealand's Integrated Data Infrastructure (IDI).

The results of this analysis are also described in the accompanying A3 document entitled 'Youth at risk: Identifying a target population', produced by the Ministry of Education. This paper provides a general description of the process adopted, presents a descriptive analysis of the populations of interest, summarises the results of the modelling work undertaken and describes the target populations identified through the project. It also provides some guidance to assist with interpreting the A3 document.

Research objectives#

The aim of this work is to identify which risk factors between the ages of 15 and 24 are most strongly associated with poor long-term outcomes at ages 25 to 34, identify target populations between the ages of 15 and 24 who are most at risk of experiencing poor long-term outcomes and identify some of the larger fiscal costs associated with those target populations.

Data and methods#

The study uses the Integrated Data Infrastructure (IDI), which brings together information from a wide range of government departments. Records are linked using name and date of birth. The data is anonymised and used only for research purposes.

The main analysis is a birth cohort analysis, which focuses on those born between 1 July 1990 and 30 June 1991, who can be observed through to age 21 in the data set. The analysis describes the key characteristics and outcomes that could be observed for this cohort at various ages and the various service use patterns and outcomes that were experienced by different subgroups within this population. The future outcomes of this birth cohort out to age 35 are also estimated using a statistical record linkage technique, in which data for an older birth cohort is linked to that of the 1990/91 cohort.

This is complemented by an analysis of the characteristics and outcomes of the current youth population, defined as being aged 15 to 24 as at the end of December 2013. We are able to describe these young people's interactions with selected social services up to the end of December 2013. Projected future outcomes and selected service costs are also estimated for this population using data for other birth cohorts and statistical record linkage techniques.

Limitations and caveats#

The study has a number of limitations and caveats:

• The scope of the study is limited by the nature and breadth of the information collected in agencies' administrative systems and included in the IDI. For example, the administrative data used in this work provides only a partial picture of childhood adversity, service use and service costs.
• The population coverage errors, linkage errors and biases present mean that the results are unlikely to be completely accurate and should be viewed as providing broad estimates of scale.
• The methods used to estimate future outcomes and costs are designed to provide a comparative picture of future outcomes and costs for different population subgroups, but they have some significant limitations. These estimates should not be viewed as forecasts of the actual outcomes and costs that will be incurred in the future.

While the results highlight the power of using integrated administrative data in new and innovative ways, this is the first time some of the data has been used in this way, and as such, these results should be considered as preliminary and will need further testing and development over time.

The caveats and limitations are discussed in more detail later in the paper.

Key findings#

• Integrated administrative data can be a powerful tool for government and other agencies to identify at-risk groups in the population. Limitations in some of the data mean that the findings of this analysis need to be treated with some caution. However, the results provide a useful insight into the lives of at-risk youth. The data used for this type of analysis will continue to improve over time.
• A number of characteristics can be identified throughout a person's early life that are predictive of future poor outcomes including early contact with government agencies such as Child, Youth and Family (CYF), demographic characteristics and geographic location, characteristics of the young person's caregiver and early outcomes evident in data from the education, corrections, welfare and health systems. These can be used to quantify risk at an individual level and to identify the size and characteristics of at-risk groups of young people at different ages.
• The characteristics that are predictive of future outcomes change over time. As young people progress into early adulthood, poor future outcomes become directly evident through contact with the benefit, corrections and health systems. Whilst it becomes easier to predict poor outcomes as a young person ages, these outcomes may become more difficult to influence.
• It is possible to identify groups of at-risk youth at different ages using a small set of identifying characteristics. However, these predictions are by no means perfect. Young people who are identified as being at risk are highly likely to have poor future outcomes, but a large number of people have poor outcomes despite not falling into one of these defined groupings.
• In general, geographic location is strongly associated with risk of poor outcomes, with location-based measures such as the New Zealand Deprivation Index (NZDep) and territorial authority area being important predictors of risk, even controlling for other observed characteristics. Youth at risk of poor outcomes tend to be concentrated in specific areas such as the Far North, Kawerau, Opotiki and Wairoa. However, it is important to note that the largest numbers of at-risk youth still live in larger urban centres such as Manukau, Waitakere, Hamilton and Christchurch.

1  Introduction#

1.1  Purpose#

This work makes use of integrated administrative data collected from across a number of government agencies in the Integrated Data Infrastructure (IDI) and managed by Statistics New Zealand to ensure the security and confidentiality of people's information. The analysis is part of a broader direction by government agencies to work in a more collaborative way to better target social services where they are most needed. This study is focused on the youth population, defined as people aged 15 to 24, and seeks to identify specific groups within this population that are at particular risk of poor longer-term outcomes. The outcome measures used in the study and the approach to measuring risk are discussed below.

1.2 Outcome measures#

A set of measures were identified from data available in the IDI, and based on the Youth Outcomes Framework,[3] across the domains of:

• Enjoying Economic Opportunity
• Engaging and Achieving in Education
• Maintaining Good Health
• Enjoying Safety and Security.

Outcomes considered in the analysis were derived from educational, health, corrections and welfare outcomes data. The measures used were considered to be the best that could be developed in the time available and represent a broad, but not exhaustive, range of poor outcomes that negatively impact on the lives of young people. In many cases, improved measures are likely to be able to be developed as the data in the IDI continues to develop and expand. Measures were defined as follows:

• Not achieving:
  • at least an NCEA level 2 qualification by age 23 (for those turning 15 or 16)
  • at least a level 4 qualification by age 23 (for those turning 17 to 21).
• Use of mental health or addiction services between ages 20 and 22 inclusive (for those turning 15 to 20).
• Receiving a custodial or community sentence between ages 25 and 34 inclusive.
• Being on a benefit for five years or more between ages 25 and 34 inclusive.

The preferred approach was to analyse youth outcomes at ages 25 to 34 when possible. The 'mental health or addiction services' outcome measure was measured at ages 20 to 22 simply because the data needed to estimate it at ages 25 to 34 was not available. Over time, historical data will become more complete in the IDI, and such compromises will not be necessary.

1.3 Approach to defining the 'at-risk' populations#

In order to define target 'at-risk' populations, a multi-stage process was adopted that:

1. identified the factors most associated with the outcomes of interest, calculated estimated risk scores for each individual and identified an 'at-risk' population
2. identified groups of people (or 'clusters') with similar identifying characteristics within this 'at-risk' population
3. defined and described a small set of proposed target populations that broadly matched the identified clusters.

The at-risk groups and associated target populations are then able to be described according to their level and type of risk, key predictors of risk and associated fiscal welfare and corrections costs.

Unless otherwise stated, people were identified on their birthday when they turned 15 to 22, so characteristics for 15-year-olds, for example, covered the time leading up to that birthday when they were 14 or younger. Characteristics at age 22 were used for ages 23 and 24.

A person was considered to be 'at risk' where they were in the top 5% estimated risk group for at least one of the four outcomes identified above. An additional multiple risk measure was also used based on a person having high risk across multiple outcomes measures. This was defined by ranking the estimated risk scores of each of the four outcomes, averaging these ranks for each person and then taking the 15% of the population considered to be most at risk according to their average rank.

1.4 Report structure#

The structure for the rest of the paper is outlined below:

• Section 2 describes the data and methods used, including any limitations and caveats.
• Section 3 describes the expected outcomes for young people with different characteristics.
• Section 4 describes the results of the regression modelling exercise undertaken to identify the key predictors of poor outcomes for the youth population and describes the at-risk populations.
• Section 5 describes the characteristics and definition of the target populations identified by the project.
• Section 6 provides some guidance in interpreting the accompanying A3 document.
• Section 7 concludes.

2 Data and methods#

2.1  Data description and limitations#

The study uses the Integrated Data Infrastructure (IDI), which was developed and is maintained and held by Statistics New Zealand. The data is held in a secure environment and made available to bona fide researchers under strict conditions. The IDI includes a wide range of survey and administrative data from across government agencies. This study uses data sourced primarily from the Ministry of Social Development (MSD - related to benefits, CYF care and protection and youth justice), the Department of Corrections (sentencing), the Ministry of Education (schooling and tertiary study participation and achievement), the Department of Internal Affairs (birth and death registrations), the Ministry of Health (mental health and addiction service usage and mental health pharmaceuticals), Inland Revenue (salaries and wages) and the Ministry of Business, Innovation and Employment (movements into and out of New Zealand).

Data is rounded to a multiple of three to protect confidentiality, and small cells are suppressed. As a result data in tables and figures may not add exactly to totals.

A number of potential data issues were outlined in Treasury Analytical Paper 15/01,[4] and are relevant to this study also. Some of these issues are summarised here:

• The IDI includes information on children who were referred to CYF by the New Zealand Police because they had broken the law. However, the proportion of young people who have contact with the New Zealand youth justice system as a whole is higher than reported here. This is because the vast majority of apprehensions by the Police are dealt with by caution or warnings or by the Police Youth Aid Section and reflects the system's emphasis on diverting young offenders who commit lower-level offences away from formal youth justice processes where possible.
• Benefit data can be used to identify periods when children and young people are supported by a benefit as a child, periods when they are the primary recipient of a benefit in adulthood and periods when they are supported by a benefit as the partner or spouse of a primary benefit recipient. The benefit data covers the period 1 January 1993 onwards, so for the 1990/91 cohort, this only observes whether the child is supported by benefits after around age two.
• Benefit data has been used to identify the parents and caregivers of the young people in the study populations, which means that some information about parents or caregivers, such as their corrections sentencing history or their qualifications is only known for children who have been supported by a benefit at some stage. For the 1990/91 cohort, this is about 50% of young people.
• A child or young person's contact with CYF for care and protection reasons can be divided into a number of different levels of contact[5] depending on the highest level of intervention. Administratively derived measures of the proportion of children who have had a finding of abuse or neglect may not provide a reliable measure of the real occurrence of child maltreatment however. They will reflect both variations in the extent to which children who experience maltreatment are notified to CYF as well as the uncertainty inherent in making a determination that maltreatment has occurred. In addition, the CYF data is incomplete in the early 1990s, and therefore some of the estimates of service use for this period will be understated.

Service cost data

This paper includes estimates of some of the largest fiscal costs that are associated with different groups of individuals when they are aged 25 to 34. The costs included cover benefit payments and the costs of serving sentences administered by the Department of Corrections. Only these costs were analysed because our method of estimating costs at ages 25 to 34 uses data for an earlier birth cohort (those born in 1978/79), and information on the use of other government services is not available for that earlier cohort.

All cost estimates used in this study are CPI adjusted to December 2014 dollars.

Benefit costs

Benefit costs were categorised into three groups - Tier 1 (main benefits), Tier 2 (supplements) and Tier 3 (additional support for people in hardship). Working for Family tax credits, student allowances and student loans were not included in the study.

In the early childhood period of the study, the Tier 1 benefits included Domestic Purposes Benefit, Widow's Benefit, Unemployment Benefit, Sickness Benefit, Invalid's Benefit, Orphan's and Unsupported Child Benefits, Independent Youth Benefit and Emergency Benefit. More recently, following changes to the benefit system in 2013 (and some earlier changes), Tier 1 benefits have included Jobseeker Support, Sole Parent Support, Supported Living Payment and their subcategories as well as Youth Payment and Young Parent payment.[6]

Tier 2 supplements include Accommodation Supplement, Family Tax Credit (not including payments made by IRD), Disability Allowance, Orphan's Benefit or Unsupported Benefit, and Foster Care Allowance. Tier 3 additional supports included Funeral Grant, Special Needs Grant and Temporary Additional Support.

Corrections costs

The costs of serving custodial and community sentences were calculated by multiplying the length of each sentence (taking the days actually served) by an average cost per day from a table of average per day sentence costs provided by the Department of Corrections.[7] The sentences and orders for which cost data is available include prison, remanded in custody, supervision and related sentences (including extended supervision orders and intensive supervision), community detention, community work, other community sentences, home detention, parole, post-detention conditions, released to home detention and released with conditions.

The average cost figures provided by the Department of Corrections related to the last four financial years. In this analysis, the cost figures for those four years were averaged (giving more weight to recent data) and applied historically (after adjusting for inflation). The cost estimates include both direct and indirect costs. Note that average per person costs are not the same as marginal costs, and therefore the figures used in this analysis cannot be used to calculate the aggregate costs that could be added or saved by increasing or decreasing the total numbers of persons serving sentences.

2.2 Populations of interest#

Two study populations were used in the analysis. A sample birth cohort was defined based on people who were born between 1 July 1990 and 30 June 1991 and who were therefore aged 22 as at 30 June 2013. This single 'cohort population' was tracked over time and their risk of poor future outcomes estimated at each year of age from 15 to 22, using the regression modelling approach described below. Characteristics at ages 23 and 24 were not able to be observed for the 1990/91 birth cohort. Information observed at age 22 was used at these ages. Future outcomes were projected beyond age 24 using statistical matching to individuals from an earlier birth cohort, as discussed below.

A second study population was also used, capturing people who were aged 15 to 24 on 31 December 2013, were eligible to live in New Zealand on a permanent basis and were living in New Zealand for at least six months during 2013.[8] Once the modelling, clustering and target population definition processes were undertaken on the cohort population, the December 2013 (or 'current') population was used to describe the characteristics of the identified target populations. This current population provides a better view of the size and characteristics of at-risk individuals at a recent point in time than the 1990/91 birth cohort population would.

The birth cohort population was based on people born in 1990/91 because the coverage of the various data sets included in the IDI meant their characteristics and outcomes could be tracked up to the age of 22, covering most of the ages of interest in the Youth Funding Review. This was also the first birth cohort for which near-complete school enrolment data was available from the Ministry of Education covering the years when the children were aged 14/15 and above (ie, 2006 and subsequent years). The selection of a cohort based on a 1 July to 30 June year is also consistent with the practice of aligning age to school years.

The criteria for the 1990/91 birth cohort population were intended to select all children who were living in New Zealand as permanent residents during the 2003 to 2007 period, when they were aged 12/13 to 16/17. We selected children who met at least one of the criteria of:

• being enrolled at a New Zealand school as a domestic student for some or all of the years from 2003 to 2007
• having an income tax payment record in 2005-08
• having a benefit paid to them or on their behalf in 2005-07
• being part of the National Health Index population in 2006-07.

In addition, they had to:

• be in New Zealand for at least three years of the period from 1 January 2003 to 31 December 2007 (in total, rather than continuously)
• be born in New Zealand or have permanent residence entitlement through some other means (those with temporary residence visas were excluded).

Defining the birth cohort population in this way has these effects:

• We miss a small number of children purely because a link could not be established between their administrative data records.
• We do not include people who were away from New Zealand for much of 2003 and 2007 but were continuously resident at earlier or later phases of their lives.
• We include some people who were overseas for a substantial part of their childhood or young adulthood. These individuals will be missing from the administrative data sets in earlier and/or subsequent years and will appear to have had no contact with the welfare, child protection or corrections systems. We are able to identify when these people were overseas but do not remove them from the study population.

The second study population comprises children and youth who were aged from 15 to 24 years at 31 December 2013, who had New Zealand citizenship or permanent residence entitlements and were living in New Zealand for at least six months during 2013.[9]

There are 289,540 people aged 15 to 19 and 292,210 people aged 20 to 24 in our current population. These numbers represent 93% and 91% respectively of Statistics New Zealand's estimates of the resident populations in these age groups in the December 2013 quarter. Our study populations are smaller because we exclude temporary residents, those who were out of New Zealand for six months or longer in 2013 and those who could not be linked to the key data sets in the IDI.

2.3 Estimating future outcomes and costs using statistical matching#

A statistical record linkage technique was used to help estimate the likely longer-term outcomes of the study populations. This process is discussed in detail in Treasury Analytical Paper 15/01 in the context of earlier analysis of ICD data and is only summarised briefly in this paper.

The approach involved linking data for an older birth cohort (specifically the July 1978 to June 1979 birth cohort) to the data for the 1990/91 birth cohort population to project outcomes for this latter population. Records were linked on the basis of benefit receipt and corrections sentencing rates and patterns when aged 16 to 21 years inclusive as well as on gender and ethnicity. Observed outcomes and costs experienced by the 1978/79 cohort were then used to estimate the outcomes and costs of the 1990/91 cohort up to age 35.

Matching individuals rather than population groups gives us the flexibility to estimate costs for very different subsets of the population. This is particularly important when we are looking to identify specific target populations for investment decisions. The statistical matching method uses real patterns for individuals over time with very similar observed characteristics up to a certain age.

The approach assumes longitudinal patterns of benefit receipt and corrections sentences can be moved around in time from one cohort to another and that, conditional on a set of 'early indicator' matching variables, these patterns remain relevant to later cohorts. The success of this depends on how well we establish good matching criteria and on how relevant these are for forecasting future outcomes. The range of variables used in the matching process also had some significant omissions, such as region and NCEA achievement. As a result, some caution must be taken with analysis based on these characteristics. Differences in groups defined by these characteristics are probably more diluted than the differences in other group comparisons.

We have also not accounted for differences in macro-economic conditions experienced by the 1978/79 cohort and those that may be faced by the 1990/91 cohort in future years. As a result, future outcome estimates will in part reflect the particular patterns of labour demand and unemployment that have occurred over the last 20 years. Ideally, we would like to remove the effects of these macro-economic fluctuations and have a more constant underlying macro-economic picture underpinning the analysis. This remains an issue for further investigation.

Long-run shifts in New Zealand's social assistance policies could also influence the success of the cohort matching if they have affected the outcomes of different birth cohorts very differently. Ideally, we would adjust individuals' outcomes to remove the effects of any secular trends that are external to the individual but affect the outcomes of the cohort as a whole. In practice, however, it may be difficult to do so in an objective way using the data currently available.

2.4 Approach to the identification of target populations#

Identifying risk factors and predicting risk

Logistic regression models were run against the four outcome variables described in section 1.2, covering the welfare, health, education and corrections domains. Over 60 potential risk factors derived from a number of administrative data collections were included in the modelling exercise. Models were run at each year of age for females and males separately. Logistic regression with a forward selection was used to construct a model based on a reduced set of risk factors that were most predictive of each outcome measure. These factors are listed in Appendix 1, along with an indication of the number of models the factor was included in at each year of age.

This process allowed us to identify the key risk predictors for each age/gender combination and calculate an estimated risk score for each individual in the target population. The estimated risk score was used to define an 'at-risk population' according to the above criteria, which could then be used to identify target populations with a higher than average probability of being at risk of poor longer-term outcomes.

As discussed in section 2.3, long-term outcomes were estimated using statistical matching. These were then modelled against characteristics that were directly observed in the data, and this may dilute the relationships between the characteristics and outcomes in the models. Since matching was undertaken on a limited set of characteristics, it is possible that this may not affect all characteristics equally. As such, some caution should be taken when interpreting the relative strength of the modelled relationships.

Defining and describing target populations

For each age group, a cluster analysis was undertaken identifying groups of individuals within the 'at-risk population'. Multiple correspondence analysis was firstly used to redefine the key categorical predictors from the regression modelling into a smaller number of continuous variables, and these were then used to identify a number of clusters at each year of age for females and males jointly.

The youth population was next split into the late teen population (aged 15 to 19) and the early 20s population (aged 20 to 24). Five fairly distinct groups of people with similar characteristics and at particular risk of poor outcomes were identified within each of these age groups. For the early 20s population, risk was defined primarily using the welfare and corrections outcomes measures, as health and education outcomes could have already occurred at these ages, potentially conflating the risk and outcomes measures.

The identification of target population groupings was informed by the factors that were most predictive of poor outcomes in the regression analysis in Step 1 and the clusters identified in Step 2. They were constructed using the following guiding criteria:

• Parsimony – target populations should be able to be identified using only a few criteria.
• Separation – overlap between target populations should be minimised.
• High sensitivity – most people identified as being at risk should fall into at least one target population.
• High specificity – most people identified as not being at risk should fall outside of the target populations.

2.5 General caveats and cautions#

The process of matching records is probabilistic and creates some level of error, as there are likely to be some cases where individuals cannot be matched (and appear in the data with less service use than actually occurred) as well as cases where individuals have been wrongly matched (and appear in the data with inaccurate estimates of service use).

The data covers a specific time and cohort, and some care must be taken in generalising results to the experience of more recent cohorts of children. Some cohorts born more recently have had a higher likelihood of being notified to CYF, partly because of administrative changes related to family violence events attended by Police. This is described in further detail in Treasury Analytical Paper 15/01.

There are also possible biases for those young people who have spent any lengthy period of time outside of New Zealand between ages 15 and 22. The characteristics of these people, including any outcomes achieved, are less likely to be visible in our data, as any contact with government agencies may happen outside of New Zealand. It may look like these people fail to gain qualifications, avoid prison sentences or benefits and do not access health services where these things happen out of New Zealand.

To some degree, this is controlled for by including an indicator in the modelling when a young person is out of the country for the entire previous year. However, there may be some biases introduced that may be better controlled for by including more sophisticated measures of time outside of the country or by treating this group differently, possibly excluding some from the analysis. There is no single approach that would be better, however, and more thinking may be needed on this issue for future work.

3 Which young people experience poor outcomes?#

This section presents a descriptive analysis of the 1990/91 birth cohort's characteristics and outcomes observed between ages 15 and 22, alongside their projected outcomes beyond that age. The population is described using a range of factors we expect to be predictive of poor future outcomes. These factors are a selected subset of the factors included in the regression modelling exercise described in the next section. The full set of factors included in the modelling is listed in Appendix 1.

Socio-demographic characteristics

Outcomes by gender and ethnicity are given below in Table 1 for the 1990 cohort population. Young men are somewhat more likely than young women to have poor educational outcomes and considerably more likely to have been sentenced for a criminal offence. Women are slightly more likely to have a poor mental health outcome and considerably more likely to experience long-term benefit receipt in their late 20s and early 30s. Young people of Asian ethnicity are less likely to experience poor outcomes across all domains, while Māori youth have relatively poor outcomes across all but the mental health domain, where they are more or less on a par with European youth. Outcomes for young Pasifika people tend to be better than for Māori but worse than other ethnic groups. The exception to this is mental health service use, where Pasifika rates are low compared to most ethnic groups.

Results are also presented by New Zealand Deprivation Index (NZDep) deciles. NZDep is a geographically defined measure of socio-economic deprivation.[10] Scores are associated with each meshblock in New Zealand and defined in such a way that a 10th of the New Zealand population fall into each decile group. The population living in the least deprived areas in New Zealand are categorised as decile 1, while those living in the most deprived areas are categorised as decile 10. Table1 shows outcomes for the youth population living in each NZDep decile at age 15. Unsurprisingly, there is a clear gradient of outcomes across NZDep deciles, with those living in higher decile areas being progressively more likely to experience poor outcomes. The exception to this is the mental health outcome measure, with little difference across deciles (ranging from 18% in decile 2 to 22% in decile 8).

* NZDep is calculated here based on the young person's identified location at age 15.

Location in New Zealand

Outcomes by territorial authority (TA) area are presented in Appendix 2 Table 1. Some caution needs to be exercised when interpreting these results as some areas are small, and results are for a single birth cohort only. Outcomes could vary considerably across cohorts for these small areas. For this reason, we avoid commenting on areas with fewer than 100 people in the cohort. Amongst larger TA areas, a few things stand out however.

A number of areas stand out for the high proportion of people in the cohort at age 15 who were expected to have poor educational outcomes. In Buller, Opotiki, Ruapehu and Waitomo districts, around two-fifths of 15-year-olds failed to achieve level 2 qualifications by age 23, while around four-fifths failed to achieve a level 4 qualification by age 23. This compares to around one-quarter and three-fifths respectively in the general population and around one-fifth and one-half respectively in Auckland City.

Two territorial authorities stand out not only for poor educational outcomes but for poor outcomes across a range of domains. Kawerau and Wairoa have similar expected educational outcomes to the districts just discussed, but around one-fifth (19% and 22% respectively) of 15-year-olds in the cohort were expected to be sentenced to a custodial or community sentence, and almost one-quarter (23% and 24% respectively) were expected to be on benefit for five years or more between the ages of 25 and 34. The corresponding figures for both measures were about 7% in Auckland City and around 9% across New Zealand overall.

One territorial authority, Carterton, stands out as having particularly high use of mental health services with 34% of the 1990/91 birth cohort using mental health services between ages 20 and 22 (compared to 16% of the Auckland cohort and 20% across New Zealand).

Located overseas

As discussed earlier, where young people spend significant periods of time outside of New Zealand, we may not observe changing characteristics or outcomes in our data. Table 2 looks at this in greater detail by comparing the future outcomes at each age for those in the 1990/91 cohort by whether they were out of New Zealand for the entire year or not. As a result of the way the cohort was defined, limiting the population to people enrolled in schooling around the age of 15 or 16, very few people in our cohort were overseas in the years they turned 15 or 16. Over time, from age 17 progressively, more people moved overseas, with around 4% being overseas for the whole year in the year they turned 22.

Because of the way the outcomes were constructed, those young people who were overseas in each year show very different outcomes for our different outcome measures of interest. In the case of educational outcomes, lack of contact with the educational system in New Zealand can lead to poor outcomes being inferred when people are not seen to be gaining qualifications. Around 90% or more of those who were overseas for the full year from age 17 through to age 22 did not achieve level 4 qualifications in New Zealand by age 23 compared to around 60% of those who remained in New Zealand. Those who were overseas at ages 17 or 18 were unlikely to achieve level 2 qualifications in New Zealand (84% did not compared to 24% of those remaining in New Zealand). However, this rate was lower for those who were overseas at older ages (eg, 53% of those who were overseas at age 22).

For other outcomes, extended periods overseas were associated with a reduced likelihood of poor outcomes. In this case, lack of contact with the corrections, welfare or health systems is used to infer the lack of a poor outcome. In particular, very few of those who were overseas after age 15 used mental health services, consistent with the focus of the measure on an earlier age (20 to 22 compared to 25 to 34 for the welfare and corrections outcomes).

Table 2 focuses on people who were overseas for an entire year. Those who were overseas for a substantial part of the year will also have had a reduced likelihood of completing a New Zealand qualification or using benefit, health and corrections services.

Childhood risk factors

By age 15, a number of risk factors may be evident through contact with agencies such as CYF and Work and Income. A number of these potential childhood risk factors are outlined in Table 3 below, alongside future outcomes.

The table shows a clear association of time spent supported by a benefit as a child and future outcomes. There is a clear gradient between the proportion of time spent on a benefit as a child and the likelihood of future poor outcomes across all domains. Those who were mainly supported by a Sole Parent Support benefit (or its equivalent) were particularly likely to experience poor outcomes. This could be due to other risk factors associated with receipt of a Sole Parent Support benefit or could be a reflection of the high likelihood of receipt of such benefits being associated with a considerable length of time on a benefit.

The table also provides information on the parents' or caregivers' educational attainment from benefit data.[11]Where a parent has never been on a benefit, no qualification information is collected, and even for those who have been on a benefit, information may not have been collected or may not be up to date. The collection of information could also be connected to other, unobserved characteristics such as the time the caregiver spent on a benefit or the time since they were last on a benefit. As such, the results are difficult to interpret. Nevertheless, conditional on the caregiver having received a benefit and having their qualifications information recorded, higher caregiver qualifications tend to be associated with lower probabilities of poor outcomes. As before, this pattern is less clear with mental health outcomes.

Schooling

Characteristics associated with schooling can also be predictive of future outcomes. Whilst these change between ages 15 and 18, most characteristics do not change after age 18. As such, Table 4 outlines the school-related characteristics of the youth cohort population at age 18 and their projected outcomes.

Around two-thirds of the cohort population were still at school in the year they turned 18. These young people were considerably less likely to be on a benefit long term, to have a corrections sentence or to access mental health services in the future. Not surprisingly, they were much more likely to have gained level 2 qualifications by age 23 and also more likely to have gained level 4 qualifications by age 23.

A small number of young people were overseas at age 18, and for this group, it was unclear whether they were still enrolled at school. While they were unlikely to gain qualifications in the future, they were also unlikely to access mental health services, receive a corrections sentence or be on a benefit, indicating that many probably stay overseas long term and are consequently not captured in the administrative data.

The characteristics of the school the young person most recently attended is also closely associated with a number of outcomes measures. Those young people whose most recent school was a private school or a high decile state-funded school[12] were less likely to experience a range of poor outcomes than those who had attended other schools. As with the earlier analysis of NZDep, the lack of a linear relationship between socio-economic status and mental health service utilisation is reflected in the school decile analysis.

Almost all young people attending a special school failed to achieve level 2 qualifications by age 23 or level 4 qualifications by age 23. Around one-third accessed mental health services, while almost two-thirds were on a benefit long term between ages 25 and 34. Those young people who had accessed special education services experienced similarly high levels of poor outcomes across the education, welfare and mental health domains, while very few received a corrections sentence.

Indicators of behavioural issues at schools are also expected to be associated with poor outcomes, and a history of truancy, suspensions and stand-downs[13] can be seen to be strongly associated with poor outcomes across all domains. Almost three-quarters of 18-year-olds had achieved level 1 NCEA qualifications by age 18, over one-half had achieved level 2 qualifications and around one-eighth had achieved level 3 qualifications.

Tertiary education

Tertiary qualifications are clearly likely to be related to educational outcomes, but tertiary study is also expected to be linked with broader social outcomes. Table 5 shows participation and outcomes of tertiary study[14] by age 22 alongside the broader set of outcomes analysed.

One-fifth of the 22-year-old youth population had not achieved a qualification by age 22, while around one-tenth had only achieved a level 1 qualification. Not surprisingly, almost all of these groups failed to achieve a level 2 or 4 qualification by age 23. By construction, nobody who had achieved a level 2 or 3 qualification (around half of the cohort) at age 22 could have failed to achieve this level by age 23, and similarly, none of the approximately one-quarter of young people who had achieved level 4 to 8 qualification at age 22 were recorded as having a poor educational outcome under either measure.

In general, high qualifications and high-level tertiary qualification enrolments are associated with good outcomes across most domains. This is most clearly seen in corrections sentencing with 5% or fewer of those with level 5 qualifications or above at age 23 having a future corrections sentence or long-term benefit receipt between ages 25 and 34. Mental health outcomes show a less strong declining trend with higher qualifications. However, 28% of those with no qualifications at age 22 used mental health services between ages 20 and 22, whilst only 10% of those with level 8 qualifications did so. In the case of tertiary enrolments, people who had been enrolled in level 1 or 2 tertiary qualifications had the highest rates of mental health service use, at 26% and 27% respectively. Those enrolled in level 1 tertiary qualifications were most likely to be on a benefit long term, whilst those enrolled in level 2 tertiary qualifications were most likely to receive a corrections sentence. Enrolment in low-level tertiary qualifications could be a proxy for early school leaving and a failure to achieve this level of qualification at school.

The longer someone is in tertiary study before age 22, the less likely they are to experience poor future outcomes across all domains. Only one-quarter of those with three years' tertiary study by age 22 failed to achieve a level 4 qualification by age 23, while 15% used mental health services between ages 20 and 22, only 3% received a corrections sentence between ages 25 and 34 and only 4% received a benefit long term in the same age range.

Work and welfare

We would expect employment and earnings to be important predictors of future outcomes across multiple domains. In addition, both time on a benefit and type of benefit are likely to be closely linked with multiple outcome domains. For example, receipt of the Supported Living Payment – Health Condition or Disability may be strongly associated with outcomes across the health, welfare and education domains. Table 6 below presents selected measures of employment, earnings and benefit receipt, by future outcomes, at age 22.

At age 22, almost one-quarter of young people in the 1990/91 birth cohort had no salary, wage or self-employment earnings.[15] As might be expected, this group had poor future outcomes across all domains, with three-quarters not achieving level 4 qualifications by age 23 and one-fifth expected to be on a benefit for five years or more between ages 25 and 34. In general, progressively higher earnings were associated with improved outcomes. Of those earning over $40,000 at age 22, only 2% were expected to be on a benefit long term, 6% were expected to receive a corrections sentence and 14% were expected to use mental health services.

On the other hand, two-thirds of those earning over $30,000 at age 22 failed to achieve level 4 qualifications by age 23. This is perhaps not surprising given that this level of earnings is likely to be strongly associated with full-time work that would preclude full-time study. Lower earnings are consistent with part-time work that, in many cases, would be undertaken alongside full-time study.

An alternative measure of engagement that combines both a labour market and education dimension is the proportion of time spent not in employment, education or training (NEET) since age 16. Around one-half of the youth population spent either no time or less than 10% of time between ages 16 and 22 as NEET. Both of these groups had relatively good outcomes across all domains. Another one-fifth spent between 10% and 25% of their time as NEET, while around one-quarter spent more than 25% of their time as NEET.

As might be expected, there is a clear relationship between time spent NEET and the expectation of poor outcomes across all domains. Of the 4% of young people who had spent three-quarters or more of their time as NEET between ages 16 and 22, almost 40% had accessed mental health services by age 25, more than one-third were sentenced to a custodial or community sentence between ages 25 and 34 and over 40% were on a benefit for more than five years between the same ages. Not surprisingly, almost all this group failed to achieve level 2 and level 4 qualifications by ages 18 and 23 respectively.

Benefit receipt was also closely associated with outcomes. Not surprisingly, progressively more time on a benefit was particularly strongly associated with future time in receipt of a benefit, but there were also strong associations with other outcomes. Most benefit types were also strongly associated with poor outcomes across multiple domains. People who had mainly spent time on a Jobseeker Support - Training benefit at age 22 (10% of all young people) had a lower probability of poor educational outcomes than other beneficiaries and even than those who had not spent time on a benefit. They also had considerably better outcomes across other domains than those on other benefits.

Not surprisingly, those 22-year-olds whose main benefit since age 18 was either Jobseeker Support - Health and Disability or Supported Living Payment - Health and Disability were more likely than other groups to have used mental health services between ages 20 and 22, but the latter groups also had relatively poor educational outcomes and were expected to be on a benefit long term between ages 25 and 34. While they were very few in number, those who had mainly been on a Young Parent Payment benefit at age 22 had a high likelihood of poor outcomes across all domains and were particularly likely to receive a corrections sentence between ages 25 and 34.

Early corrections contact

Early contact with the corrections system, either through contact with CYF Youth Justice or through receiving a custodial or community sentence as an adult, would be expected to be closely linked with the chances of being sentenced to a custodial or community sentence between ages 25 and 34. However, these may also be important predictors of outcomes across other domains. Table 7 shows expected outcomes by contact with the corrections system up to age 22.

Around 5% of young people had been referred to CYF Youth Justice by age 22. Not surprisingly, this group were particularly likely to receive a corrections sentence between ages 25 and 34 (almost half compared to 7% of those who had not been referred to Youth Justice by age 22). Of the 1% of youth who had a CYF Youth Justice placement by age 22, around two-thirds received a community or custodial sentence between 25 and 34. Both of these groups (especially those with a placement) also had poor outcomes across other domains with large numbers having failed to achieve level 2 and level 4 qualifications by ages 18 and 23 respectively and large numbers also being users of mental health services (41% and 58% of those with a referral and a placement respectively).

Almost one-tenth of young people had received a community sentence by age 22, while around 3% had received a custodial sentence. Not surprisingly, being sentenced by age 22 was highly associated with being sentenced between ages 25 and 34, with longer sentences progressively increasing this probability. Over 80% of those young people who had spent a year or more on a custodial sentence by age 22 received a corrections sentence between ages 25 and 34.

As with youth justice contact, being sentenced at an early age was associated with poor outcomes across all domains, although length of sentence (particularly for custodial sentences) did not have as strong a relationship with other outcomes as it did for the chances of reoffending.

Early use of mental health services

Early indicators of mental health issues could be a strong predictor of future outcomes across multiple domains, with mental health issues potentially affecting participation in education and the labour market. Mental health issues, and especially alcohol and drug addiction, are also potentially linked with criminal offending in various ways. Table 8 shows four indicators of poor mental health at age 20 by future outcomes.

One in 20 young New Zealanders who were born in the year to the end of June 1991 had used alcohol or drug addiction services by age 20, while almost 10% had used some other sort of mental health services.[16] In both cases, this was associated with poor educational, welfare and mental health outcomes. Not surprisingly, more than three-quarters of young people who had accessed these services by age 20 continued to access them between ages 20 and 22. Almost one-quarter were on a benefit long term between ages 25 and 34 (compared to fewer than one-tenth of other young New Zealanders). Young people who had accessed alcohol or drug addiction services by age 20 were particularly likely to serve a future corrections sentence, with 40% doing so between the ages of 25 and 34 compared to 7% of those who hadn't accessed these services.

Broader indicators of substance abuse and other mental health issues were able to be derived using a wider set of health data including prescriptions for mental health-related pharmaceuticals. Around one-tenth of young people were identified as having a history of substance abuse and one-quarter of having mental health issues according to these expanded measures. Both measures show a relationship with all outcome domains, although in most cases, this was more muted than the earlier measures.

Early parenting

Early parenting could be associated with poor outcomes for a number of reasons. In particular, childcare could affect the ability to participate in employment or education. Table 9 shows outcomes by early parenting status at age 22, whether the young person was a parent before age 19 and the nature of any interaction with CYF with regard to that child or children.

In total, 13% of the 1990/91 birth cohort had had a child or children by age 22, while 5% were a parent before age 19. Becoming a parent at an early age was associated with a high likelihood of poor outcomes across all domains, with those who became a parent by 19 being slightly more at risk than those who had a child for the first time in the following three years.

Where a young parent had a child that was subject to a CYF care and protection notification (4% of the birth cohort), outcomes were particularly poor. Almost one-third of these young people had a corrections sentence between ages 25 and 34, while almost two-fifths were on a benefit long term during this period, and a similar number accessed mental health services between ages 20 and 22. Where there was a finding of abuse or where there had been a Police/family violence notification, these figures were even higher, and for the 228 parents whose children had been placed in care by age 22, outcomes were particularly poor. Two-fifths of this group had a poor corrections outcome, over one-half were on a benefit long term and almost three-fifths used mental health services. Two-thirds had not achieved NCEA level 2.

4 Predicting poor outcomes#

In this section, regression analysis is used to more systematically identify which characteristics observed in the administrative data between ages 15 and 24 were most strongly associated with a higher likelihood of poor outcomes as an adult. The aims of this analysis are predictive in nature, seeking to identify the factors most strongly associated with poor outcomes and to use these to predict risk and identify target populations for investment decision making. The analysis does not seek to understand the causes of poor outcomes and does not answer underlying questions of causality. Just because a factor predicts poor outcomes does not necessarily mean the underlying concept causes those outcomes.

We are restricted to using existing administrative data sources from each agency. We are also limited by the confines of the data collected in the IDI for the cohort we are studying. This meant, for example, that health data relating to early childhood was not able to be used in the study. As such, the results are less definitive in terms of the relative importance of various factors. Nevertheless, the analysis is useful for understanding which interactions with government agencies at particular stages of a young person's life are more strongly associated with poor outcomes later on.

As discussed above in section 2, logistic regression models were run for the 1990/91 birth cohort for each year of age from the year they turned 15 to the year they turned 22 (ie, from the 2005/06 July to June year to the 2012/13 year). Predictive factors were selected for the models on the basis of a forward selection approach.[17] Models were also run separately for males and females on the basis that different risk factors were likely to be important for each gender. Models were run separately for our four outcome measures, with models on the health and educational outcomes being run up to ages 20 and 21 respectively to avoid the predictive factors becoming conflated with the outcome measures. These outcome measures are measured at an earlier age than the welfare and corrections measures.

4.1 Regression model factor selection and estimation#

Predictive factors selected in the models

Appendix 1 Table 1 highlights the number of times each characteristic is selected across the different models at each age. This gives a broad idea of the characteristics that are important in predicting risk as someone ages through the late teen years and into their 20s. Care needs to be taken in interpreting the importance of these selections however.

The choice of factors to include in a forward selection modelling procedure is heavily dependent on the factors already selected in the model. Where a factor is highly correlated with another factor already included, it may not add much to the model and hence not be selected for the final model. With very slightly different data, the reverse may be true. For example, duration spent on a benefit is closely associated with the type of benefit, and each may be related to future time on a benefit. In cases where the duration is slightly more predictive of future benefit receipt and hence added to the model first, benefit type may not be included, even though it is also predictive of future receipt. With slightly different data (and possibly depending on what other variables are already added to the model, for example, use of mental health services or early parenting status), benefit type may be included but not duration.

Nevertheless, there are some interesting patterns in the risk factors selected for the models. Broadly speaking, as people age from 15 to 22, we have more information about them that can be used to predict future outcomes. At age 15, there were 42 potential factors used in the modelling, while by age 18, there were more than 60. With an increase in the number of potential factors, more factors were generally selected for the models. On average, 15.6 factors were used per model at age 15, increasing to around 21 at ages 18 to 20. While fewer factors were used in the models at ages 21 and (especially) 22, these were ages at which only welfare and corrections outcomes were being predicted, with fewer models run as a result.

Some specific patterns are evident in the table and are worth pointing out:

• Some factors are clearly predictive across all outcomes and most ages. The most prominent of these is ethnicity, which is included in all 58 models, but 'Notified to CYF care and protection as a child' was included in 56 models (and all models up to age 20), 'Maternal caregiver education/benefit status' was included in 53 models (and all models up to age 19), and 'Referred to youth justice' was included in 48 models.
• The only factors not included in any model were the 'Early parent (before age 19)' indicator and the 'Had own child in placement or with maltreatment finding' indicator. The former may be highly correlated with some benefit types, while the latter is closely linked to other indicators regarding interactions around the young person's child(ren), many of which were included in a few models.
• As might be expected, characteristics relating to school-level qualifications were mainly important during the mid to late teenage years. The NCEA level 1 achievement indicator was used in all models at age 16 but no models after age 18, whilst levels 2 and 3 were most important at ages 18 to 20. Having been stood down from school was a significant factor for most models at ages 15 to 20, having been suspended from school was important at ages 15 to 16 and being recorded as being truant from school was important to most outcomes at age 17. Having received special education services was predictive in at least half of the models at all ages. School decile was important in most models at ages 15 to 17.
• A number of factors were constructed relating to the enrolment and completion of tertiary qualifications, and these measures were included in various models from ages 18 to 21.
• Simple yes or no indicators of employment were included in five of the models at ages 15 and 16, but the level of earnings became more important as a predictor of outcomes by the later teen years. Depending on the model, the factor selected related to the previous year or the previous two years. However, the variables are highly correlated, and the distinction may not be meaningful. Time spent NEET was important from ages 17 to 21 (not being available prior to age 17), with different factors constructed that covered different time periods. Indicators of benefit status, type and duration were included in all models from age 18, the minimum age of eligibility for most types of benefit.
• Factors related to the young person's caregiver were particularly important at the younger ages. Having a caregiver with a community sentence was included in almost all models through to age 18, while having a caregiver with a custodial sentence was included in half of the models at age 15.
• Unsurprisingly, accessing mental health services or being sentenced to a community or custodial sentence at any age were important predictors of poor mental health and corrections outcomes respectively. However, they were also broadly predictive of poor outcomes across other domains. In the case of corrections sentences, whether the sentence was custodial or community appears to be of limited importance in predicting outcomes. However, accessing alcohol or drug services appears to predict outcomes quite differently from accessing other mental health services, with the latter being much more broadly predictive across multiple outcomes domains.

Model discrimination

The area under the receiver operating characteristic (ROC) curve indicates how well each model is able to differentiate between those young people at each age who go on to have poor outcomes as adults and those that do not. The ROC statistic is a measure of how well a logistic regression model fits the data. Specifically, it measures how well the model discriminates between those with and without the outcome of interest.

The areas under the ROC curves for each of the 54 models that were run are given in Table 10 below. The model that fitted least well was that predicting future mental health outcomes for 15-year-old females (ROC statistic of 0.64), while the models that fitted the best were generally those predicting a corrections sentence or longterm benefit receipt at ages 20 to 22 (ROC statistics consistently above 0.8). The average across all 54 models was 0.80, indicating that the models were generally good at predicting who would experience a poor future outcome.

Comparing females to males, there was little difference in the ROC statistic, with the models for females having slightly better fit in general but only marginally so. Consistent with both more information becoming available over time (often closely linked to the outcomes of interest) and increasing proximity to the outcome period, predictions generally improved as a person aged. Average ROC statistics increased from around 0.75 at age 15 to almost 0.9 at age 22.

Some future outcomes also appear to be easier to predict at an early age than others. Averaged across ages 15 to 19 (ages 20 to 22 are excluded since not all outcomes were modelled), corrections and welfare outcomes had higher ROC statistics than the other two outcomes on average and at each year of age. Across all ages, the use of mental health services was clearly the most difficult to predict, with ROC scores considerably smaller than for other outcomes. This is perhaps not surprising given the earlier descriptive analysis, which showed less clear differentiation in mental health outcomes across key socio-demographic characteristics such as ethnicity, deprivation decile and school decile.

High ROC statistics at the older ages (especially 19 years and over) reflect the availability of measures that are closely related to the outcomes being modelled (for example, benefit receipt), as well as the close proximity of the age at which outcomes are measured. At age 19, for example, it is relatively easy to predict whether somebody will achieve a level 4 qualification by age 23, as qualifications achieved up to age 19 are known, as is the level of any current study being undertaken at that age.

* Level 2 qualifications were modelled at ages 15 and 16, and level 4 at older ages.

4.2 Predicting risk and defining populations with high risk#

The regression modelling allowed us to construct an equation for each individual that could be used to allocate them a risk score for each outcome of interest based on their age and gender as well as a wide range of other characteristics. Individual data is anonymised, and as such, it is difficult to use an individual risk score to target services. For this reason, the main purpose of this study is to identify target populations with high risk of poor outcomes across our outcome domains based on a small set of identifiable characteristics. The first stage in getting to these target populations was to construct a measure or measures of broader risk. We constructed two measures that were used in the remaining analysis:

• Risk across multiple poor outcomes – at each year of age, the population was ordered according to their estimated risk score for each outcome and assigned a rank. These ranks were then averaged, and the population was ordered according to this average rank. Following a fairly arbitrary delineation, the 5% of the population with the highest average ranks were defined as being at extreme risk, while the 10% with the next highest average ranks were defined as being at high risk.[18]
• Extreme risk of at least one poor outcome – the ranks constructed in the previous process were used to identify the 5% of the population at greatest risk of a poor outcome on each outcome measure, ie, at extreme risk of that outcome. A person was considered to be at risk where they were at extreme risk for at least one of the four outcomes.

The process of calculating risk scores and ranks and identifying general risk measures was repeated for both the 1990/91 birth cohort population and the December 2013 population. The focus of the descriptive analysis in the remainder of the report uses the December 2013 population as its basis, although outcomes measures and costs are inferred from the equivalent 1990/91 population either according to level of risk or target populations, as defined in the next section.

Risk across multiple poor outcomes - characteristics and expected outcomes

Table 11 below shows the different demographic characteristics of the current population according to different levels of estimated risk across multiple outcomes as defined above. Almost two-thirds of the youth population defined as being at extreme risk are Māori, as are a little over half of the high-risk population. This compares to 13% of the remaining youth population defined as having low to moderate risk. A little over half of the extreme-risk population are male, while almost half live in low socio-economic status areas, as defined by small area deprivation (NZDep) deciles 9 and 10. Only 20% of the low to moderate-risk population live in these areas.

The distribution of the high-risk population across New Zealand territorial authority areas is given in Figure 1. The numbers behind this figure are presented in Appendix 3 Table 1.

Figure 1: Percentage of the December 2013 youth population at high risk across multiple poor outcomes by territorial authority area (Auckland territorial authorities expanded)

The top four regions in terms of this youth risk measure are Kawerau, Opotiki, Far North and Wairoa districts (with 42%, 31%, 30% and 30% high risk respectively). These areas have relatively small youth populations, however, and as such, only 4% of high-risk young people live in these areas. This compares to large urban areas such as Manukau City, where almost 10% of all high-risk young people live, and Waitakere City, where 5% live.

Not surprisingly, given the modelling was designed to predict poor future educational, welfare, mental health and corrections outcomes, high-risk groups had considerably poorer expected future outcomes and higher projected future welfare and corrections costs. Table 12 shows these projected outcomes by estimated overall risk.

Table 12 shows a clear pattern of increasing risk of poor outcomes with increasing estimated overall risk (for example, almost all of the extreme-risk population fail to achieve NCEA level 4 by age 23 compared to half of the low to moderate-risk population). Risk is nevertheless difficult to predict. While relatively high proportions of the high-risk population are expected to go on to be supported by a benefit long term between ages 25 and 34 (26% and 40% respectively compared to 5% of the lower risk population), this only represents a little over half of the 57,000 people expected to be long term on a benefit. Even low proportions of poor outcomes amongst the large low to moderate-risk population can equate to large numbers of people.

Similarly, while expected welfare and corrections costs for the extreme-risk population are almost 10 times higher than those of the low to moderate-risk population, large numbers of those estimated to have low to moderate risk are estimated to have high future welfare and corrections costs. Table 13 shows estimated future costs in broad dollar bands by level of estimated risk for the 1990/91 birth cohort population at age 20. Whilst only a small proportion of young people with predicted high or extreme risk are expected to have zero future corrections and benefit costs, reasonably large numbers (albeit a small percentage) of those with low to moderate risk are expected to have future costs in excess of $200,000.

Extreme risk of one or more outcomes - characteristics and expected outcomes

Tables 14 and 15 below show the demographic characteristics and expected outcomes respectively where the population is divided according to whether a person is at extreme risk of one or more outcomes or not. Around half of the at-risk youth population are Māori compared to 15% of the remaining youth population.

As expected, the at-risk population has poorer expected outcomes than the not-at-risk population across all outcome measures. When compared to the high and extreme-risk populations identified above, the at-risk population expected outcomes generally lie between the two, as we might expect (ie, poor outcomes are more likely than for the high-risk population but less likely than for the extreme-risk population). The exception to this is the 'No level 4 qualifications by age 23' outcome for which the at-risk population have slightly better outcomes than either the high-risk or extreme-risk populations (88% compared to 90% and 95% respectively). It's possible that this outcome is less predictive of extreme risk than it is of lower levels of broader risk.

Figure 2: Percentage of the December 2013 youth population at extreme risk of one or more poor outcomes by territorial authority area (Auckland territorial authorities expanded)

The distribution of the at-risk population across New Zealand territorial authority areas is given in Figure 2 (see Appendix 3 Table 1 for the numbers behind the figure). The regions with the largest proportion of their youth population at extreme risk of one or more outcomes are the same as those using the multiple risk measure (Kawerau, Opotiki, Far North and Wairoa districts with 30%, 22%, 22% and 21% predicted to be at risk respectively). As for the alternative risk measure, the largest absolute concentration of at-risk young people is in Auckland, with around one-sixth living in Manukau City, Waitakere City or Papakura District.

Table 16 shows estimated future costs for the at-risk and not-at-risk 1990/91 birth cohort populations at age 20. As with the earlier table showing level of risk across multiple outcomes, there is not a perfect link between estimated risk and expected costs. While most of the not-at-risk population are estimated to have low future costs, almost half of those with high ($100,000 and over) projected costs are predicted to be not at risk at age 20.

Figures 1 to 4 in Appendix 4 show the distribution of costs in greater detail, plotted against the multiple outcome risk measure for the 1990/91 birth cohort population at age 20. Figures 1 and 2 in Appendix 4 show the distribution in terms of absolute numbers of young people (for those with projected costs of less than and more than $100,000 in total costs between ages 25 and 34 respectively), while Figures 3 and 4 show the distribution expressed as a percentage of young people with each estimated level of risk. These graphs tell a similar story to Table 16 above:

• In absolute terms, most people with low projected costs have low to moderate estimated risk. At the same time, there are more people with low to moderate estimated risk with projected costs of more than $100,000 than people with high risk or extreme risk.
• In relative terms, there is a far higher proportion of people with high or extreme estimated risk with projected costs of greater than $100,000 or greater than $200,000, especially in the case of those with estimated extreme risk.

Comparing risk measures

As might be expected given the differences in characteristics of the populations defined by the different risk measures outlined above, the different measures cover slightly different populations. The degree of overlap in the risk populations at age 15 and 21 are examined in Tables 17 and 18 below. In both cases, there is a high degree of overlap between the populations, with 83% and 82% respectively of the birth cohort population being considered as not being at high risk of poor outcomes using either measure.

An additional 9% and 8% of the 15-year-old and 21-year-old populations respectively were considered to be at risk on both measures. The remaining 8% and 10% of the population respectively were considered to be at risk under one of the outcome measures but not the other. Almost all of those considered to be at extreme risk overall were at extreme risk of at least one poor outcome, particularly at age 15.

How does estimated risk change over time?

We might expect that someone who is predicted to be at risk of poor outcomes at age 15 is likely to be still at risk at older ages, and to some degree, this is true. More than 90% of those who were estimated to have low to moderate risk of poor multiple outcomes at age 15 were still expected to have low to moderate risk at age 21.

As illustrated in Table 19, however, there is some shifting of risk categories over time. Over half of those who were estimated to be at high risk of poor outcomes at age 15 were estimated to have low to moderate risk at age 21, and only one-quarter of those estimated to be at extreme overall risk at age 15 were still estimated as being at extreme risk at age 21. Four-fifths of this latter group were considered to have low to moderate risk at age 21.

Table 20 shows a similar story using the risk measure based on extreme risk of one or more poor outcomes. In this case, 94% of those who were not at risk at age 15 were still considered to be not at risk at age 21. On the other hand, only half of those who were estimated to be at risk at age 15 were still considered at risk at age 21. These changes over time are likely to reflect the influence of a number of factors, including new information about a young person's life becoming available as they get older, changes in people's lives and circumstances and the challenges inherent in using data of any sort to predict future outcomes.

5 Target populations#

5.1  Approach#

For each age group, a cluster analysis was undertaken identifying groups of individuals within the at-risk population, defined as being at extreme risk (top 5% of population risk) of at least one outcome measure. Multiple correspondence analysis was used to redefine the key categorical predictors from the regression modelling into a smaller number of continuous variables, and these were then used to identify a number of clusters at each year of age for females and males jointly.

The youth population was next split into the late teen population (aged 15 to 19) and the early 20s population (aged 20 to 24), and we sought to identify a small number of target populations within each of these age groups. The aim was to identify clearly defined groups at risk of poor outcomes that aligned as closely as possible with the estimated risk from the regression analysis. Target population groupings were informed by the factors that were most predictive of poor outcomes in the regression analysis outlined in the previous section, as well as the clusters identified through the correspondence and cluster analysis, and constructed using the following guiding criteria:

• Parsimony – target populations should be able to be identified using only a few criteria.
• Separation – overlap between target populations should be minimised.
• High sensitivity – most people identified as being at risk should fall into at least one target population.
• High specificity – most people identified as not being at risk should fall outside of the target populations.

In the end, five groups were identified in each age range. Between them, these groups covered a majority of the at-risk population, and there were no additional clearly identifiable groups that met the criteria above. For the early 20s population, risk was mainly defined using the welfare and corrections outcomes measures, as health and education outcomes could have already occurred at these ages, conflating the risk and outcomes measures.

5.2 Target population descriptions and criteria#

The 10 target populations identified are described in Table 21 below along with the criteria by which they can be identified. The two measures of overall risk defined in the previous section are also given. As discussed above, the clusters were primarily designed to align with our at-risk measure, based on someone having extreme risk (being in the top 5% of the population) on at least one outcome measure. Nevertheless, these groups also tend to have a high probability of having high risk across multiple outcomes (our other overall risk categorisation). Around two-thirds or more of each of the 10 target populations are considered to be at extreme risk of one or more poor outcomes, while for most groups, three-quarters or more are predicted to be at high risk across multiple outcomes. The two groups that are the exception to this are 'Teenagers with health, disability issues or special needs' (aged 15 to 19) and 'Long-term disability beneficiaries' (aged 20 to 24). Both of these groups tend to have extreme risk of a poor welfare and education outcomes, but are far less at risk of poor corrections outcomes than other groups. A breakdown of target populations by territorial authority is included in Appendix 5.

5.3 Target population overlap and coverage#

Despite the attempt to identify target populations that not only predict risk well and are identifiable through a few simple criteria but also are mutually exclusive from each other, in practice, these objectives tend to counteract each other, and a trade-off is necessary to reach a balance across the objectives. Figures 3 and 4 below illustrate the degree of overlap between each target population and the other target populations identified in the same age range using Venn diagrams as well as the overlap between the target populations and the at-risk population (those with extreme risk of one or more poor outcomes). These figures relate to the December 2013 population, while overlaps and coverage of the 1990/91 birth cohort population are illustrated in Appendix 6.

The Venn diagrams were designed to be area proportionate such that the area covered by each part of the diagram relates to the size of the populations that meet the relevant criteria. This was accomplished using ellipses instead of circles to represent the different populations.[19] Whilst each Venn diagram is designed to be area proportionate within itself, the diagrams are only broadly comparable with each other, and caution should be taken inferring areas as being equivalent across different diagrams.

Figure 3 shows the various 15 to 19-year-old target populations. Between them, the target populations cover 72% of the total at-risk population, while around 36% of people classified as being in at least one target population do not meet the definition of being at risk. Despite this latter percentage being higher than we might like, these young people may have higher risk than the average person in the not-at-risk population. Between 43% and 60% of the target populations also fall in at least one other target population. The former, with the lowest overlap, is the 'Teenagers with health, disability issues or special needs' group, while the latter, with the highest overlap, is the 'Teenage girls supported by benefits' group.

Figure 4 shows the various 20 to 24-year-old target populations. These target populations cover 82% of the total at-risk population, while 25% of people classified as being in at least one target population do not meet the definition of being at risk. Both of these are an improvement on the younger target populations, reflecting the improving ease of prediction with age. Overlap between target populations is also considerably lower at the older ages, ranging from 7% to 41% (for the 'Long-term disability beneficiaries' and 'Jobseekers in poor health with CYF history' groups respectively).

Figure 3: Target population overlaps December 2013 population, ages 15 to 19

Figure 4: Target population overlaps December 2013 population, ages 20 to 24

5.4 Target population projected outcomes and costs#

Expected outcomes for each target population group are outlined in Table 22. As we might expect, young people in target populations are considerably more likely to experience poor outcomes than young people who are not in a target population.

For 15 to 19-year-olds, 'Teenagers with health, disability issues or special needs' are most likely to not achieve a level 2 qualification (75%), 'Mental health service users with stand-down or CYF history' are most likely to use mental health services (52%), 'Teenage boys with Youth Justice or Corrections history' are most likely to receive a corrections sentence (46%), and 'Teenage girls supported by benefits' are most likely to be on benefit longterm (62%).

Among 20 to 24-year-olds, 'Jobseekers in poor health with CYF history' and 'Long-term disability beneficiaries' are most likely to not achieve a level 2 qualification (71%), 'Jobseekers in poor health with CYF history' are most likely to use mental health services (75%), 'Young offenders with a custodial sentence' are most likely to receive a corrections sentence (67%), and 'Long-term disability beneficiaries' are most likely to be on benefit longterm (72%).

Average annual expected corrections and benefit costs between ages 25 and 34 are given in Table 23 below for each target population, as well as for the population not covered by a target population at those ages.

Results largely match what we might expect given the population definitions and earlier results. Target populations have higher expected costs than young people not in a target population across all groups. Some groups have particularly high expected corrections costs ('Teenage boys with youth justice or corrections history' and 'Young offenders with custodial sentence') or high income support costs ('Teenagers with health, disability issues or special needs', 'Teenage girls supported by benefits', 'Jobseekers in poor health with CYF history', 'Sole parents not in full-time employment with CYF history' and 'Long-term disability beneficiaries').

While the 'Young offenders with community sentence and CYF history' group is defined by contact with the corrections system, expected corrections costs are not especially high (around $25,000 per person per annum), and total expected costs are lower than for other target populations at ages 25 to 34. Similarly, at the earlier ages, the 'Teenage boys with youth justice or corrections history' group and the 'Mental health service users with stand-down or CYF history' group have lower expected total costs than other target populations.

In considering this information it is important to recognise that the welfare and corrections costs identified only represent a partial picture of the direct fiscal costs and wider societal costs of poor outcomes.

6 Interpreting the A3 document#

The accompanying A3 document titled 'Youth at risk: Identifying a target population' presents the results of the analysis in some detail. This section provides some general guidance to interpreting the results.

6.1  Identifying poor long-term outcomes (page 1)#

The four outcome measures used to define poor long-term outcomes are outlined under the broad headings of 'Economic opportunity', 'Safety and security', 'Education' and 'Good health'.

6.2 The risk factors most associated with those outcomes (15-year-olds) (page 1)#

The regression modelling undertaken in Step 1 described above relates a large number of risk factors to each of the four outcome measures used. The statistical strength of this relationship can be assessed according to the order in which the factors were selected by the forward selection procedure in the regression modelling - this procedure progressively adds factors to the model according to the additional explanatory power that factor contributes to the model at each stage of selection.[20]

The factors listed in the A3 document present an example of those five factors that are considered to be most predictive of each outcome using age 15 as an example. These are the factors added earliest in the modelling procedure for the regression models of 15-year-olds (and therefore that explain the most variation in outcomes, conditional on other variables already added). These are compared across models for both females and males, with extra weighting afforded factors that are highly predictive for both females and males. In saying this, all factors listed were significant predictors for both males and females. This list should be considered broadly indicative of the factors that are most important in predicting poor future outcomes for 15-year-olds.

Factors are different for different outcomes, but some factors are highly predictive across multiple outcomes. Being notified to CYF as a child was highly predictive of poor outcomes across all four domains, while ethnicity was significant across three domains (all except for having no level 2 qualifications by age 23), as was being stood down from school (all except for being on a benefit for more than five years). Having a caregiver with benefit receipt and/or low qualifications, receiving special education services and spending a long time on a benefit as a child were all highly predictive across two outcomes areas.

These findings should be interpreted with some caution for a number of reasons. Most importantly, whilst the association between a factor and a future outcome means that that factor may be a useful predictor of future outcomes, it does not necessarily mean there is a causal relationship between the two.

Additionally, factors are identified as being highly predictive in the modelling if they add something on top of the factors already selected for the model. Where a number of factors are highly correlated with each other, only one may be selected for the model even though the relationships may be complex, and correlated factors may also be independently highly predictive and bear an important relationship to the outcome of interest.

Finally, we have a limited set of observed predictive factors we can use from administrative data. In many cases, these factors may merely be acting as a proxy for other, unobserved factors that we are unable to measure. As young people enter their adult years, more information becomes available that can be used to determine their risk of poor outcomes. In many cases, this is a direct early indicator of the outcome of interest (for example, long-term benefit receipt in the late teen years is a direct measure of early long-term benefit receipt - the 'economic opportunity' outcome measure.

6.3 Identifying those most at risk (page 1)#

As discussed above, the regression modelling allows estimated risk scores of future poor outcomes to be calculated for a 'current' population' as at 31 December 2013. An estimated risk score of poor outcomes across multiple domains was calculated using a person's average ranked risk across the four domains. These average ranks were themselves ranked and the top 5% of young people selected and categorised as being at extreme risk, with the next 10% of individuals categorised as high risk.

The table in the A3 document contrasts the demographic characteristics and projected future outcomes of these high-risk groups with the rest of the youth population and the total youth population. Extreme-risk individuals were more likely to:

• receive a benefit for more than five years between ages 25 and 34 – 40% compared to 26% of those at high risk and 5% of youth not identified as high risk
• be given a custodial or community sentence between ages 25 and 34 – 46% compared to 24% of those identified as high risk and 5% of those not at high risk
• not achieve a level 2 qualification by age 23 (72% compared to 58% of those at high risk and 18% of other young people) or a level 4 qualification by age 23 (95% compared to 90% of those at high risk and 55% of other young people)
• use mental health or addiction services or mental health pharmaceuticals when aged 20 to 22 – 58% compared to 36% of those at high risk and 16% of those not at high risk.

High-risk individuals were more likely to be Māori and to live in areas of relatively high deprivation and were likely to have higher future corrections and benefit costs. The map and table at the bottom of the page shows that, whilst over a quarter of high-risk youth live in Auckland, young people outside of the main centres tend to be more likely to be high risk, particularly those living in the Gisborne, Northland, Hawke's Bay or Bay of Plenty regions.

6.4 Characteristics of at-risk groups by age (page 2)#

The demographic characteristics and risk factors associated with poor outcomes change with age, as illustrated on page 2 of the A3 document. Predictors that were important at age 15 are not necessarily as important at age 20, as other potentially more predictive factors become available. Whilst indicators such as being stood down from school, having a CYF notification as a child or having been supported by a benefit for a long time as a child were predictive of poor outcomes at age 15, by age 20, indicators related to personal experience in the corrections system (through being sentenced), long-term benefit receipt or time out of employment, education or training (ie, NEET), using mental health services and a lack of qualifications to that point became most important. These latter measures are more closely aligned with the outcomes measures used and could be seen as direct early indicators of these outcomes.

6.5 Target populations 15 to 19 years and 20 to 24 years (pages 3 to 6)#

Targeting investment toward those who need it most based on an individualised risk measure is often difficult to accomplish, either due to practical considerations, such as the efficiency of focusing efforts at a distinct geographic area or community, or due to data limitations that could restrict the ability to calculate individualised risk. As such, it may be necessary for investment to be targeted at specific target populations identified through a smaller set of identifiable characteristics. Some potential target populations (five for people aged 15 to 19 and five for people aged 20 to 24) were identified based on regression modelling, clustering and descriptive analysis using the approach outlined above.

These populations are described in pages 3 to 6 of the A3 document - with pages 3 and 5 describing the demographic characteristics and outcomes for each population at ages 15 to 19 and 20 to 24 respectively and pages 4 and 6 indicating their geographic location.

This represents one potential way the model could be used to target services. Numerous other approaches will be equally valid, depending on the nature of investment being considered. Although an attempt was made to make the target populations as separate as possible while still being highly predictive of risk, there is some overlap between target populations, especially in the 15 to 19 age range, where the construction of the target populations relies on a wide range of characteristics.

In broad terms, the five target populations at ages 15 to 19 are constructed around a mixture of age and gender criteria alongside childhood indicators of risk such as contact with CYF or youth justice and, in the late teens, benefit receipt. Target populations in the 20 to 24 age range are largely specified on a smaller number of criteria defined largely by a history of benefit receipt and corrections sentences. This is both consistent with the fact that only corrections and welfare outcomes are able to be modelled for most of these ages and that benefit type is related to other domains, such as ill health and disability.

7 Conclusions#

This paper has presented findings from an analysis of Statistics New Zealand's Integrated Data Infrastructure. The analysis looked at the characteristics of young people aged 15 to 24 who were at risk of poor outcomes as adults across welfare, corrections, education and health domains, and attempted to define useful and identifiable target populations at high risk of experiencing these poor outcomes.

The work was undertaken by the Analytics and Insights team at the Treasury in collaboration with other government agencies. It fed into a stream of work being led by the Ministry of Education, resulting in the production of the accompanying A3 document. The analysis presented here represents one of a number of early steps towards using a more data-driven approach to prioritising social assistance initiatives and evaluating the effectiveness of social assistance programmes. The analysis has highlighted a number of characteristics that are predictive of future poor outcomes. Examples include early contact with government agencies such as Child, Youth and Family (CYF), caregiver qualifications and benefit status, geographic location and participation and early outcomes in the education system. These can be used to quantify risk at an individual level and to identify the size and characteristics of at-risk groups of young people at different ages.

The characteristics that are predictive of future outcomes change over time. As young people progress into early adulthood, poor future outcomes become directly evident through contact with the benefit, corrections and health systems. This, combined with the proximity of the outcomes period we are seeking to predict, means that it becomes easier to predict poor outcomes as a young person ages. At the same time, however, these outcomes may become more and more difficult to influence.

It is possible to identify groups of at-risk youth at different ages using a small set of identifying characteristics, such as benefit type and duration, corrections sentencing information and information on a person's early contact with government agencies such as CYF. These predictions are by no means perfect however. Those young people who are identified as being at risk are much more likely to have poor future outcomes than those who aren't, but a large number of people have poor outcomes despite not falling into one of these defined groupings. Approaches to targeting services should be flexible enough to offer services based on particular individual circumstances as well as broad characteristics.

One useful way of targeting services is to focus on specific areas with higher concentrations of at-risk youth. However, there is a tension between targeting those services at areas where a high proportion of youth are at risk (such as Kawerau or Opotiki) and larger centres where large number of at-risk youth live (such as Manukau or Waitakere).

All of the findings in the paper should be treated with some caution given the various caveats associated with the data and methods used as well as the early stage of this type of analysis. There is some scope to improve the results in future, taking advantage of the improvements that are being made to linked administrative data and refining the analytical and estimation methods.

Appendices#

Appendix 1: Predictive factors at different ages#

The table below outlines the number of models in which a predictor was selected for inclusion in the modelling at each age.[21] Each predictor could be used in up to eight models (up to age 20), one for each of the four outcomes for both males and females. A zero in the table indicates that the factor was not included in any of the regression models at that age, while a missing value indicates that the factor was not available for inclusion, generally because that type of information was not available at that age.

* At age 21, only three outcomes were modelled, and factors could be included in only six models (rather than eight). At age 22, only two outcomes were modelled, and factors could be included in only four models.

† The 22-year-old models were also applied to the population at ages 23 to 25.

Appendix 2: Outcomes by territorial authority#

s = suppressed for confidentiality reasons.

Appendix 3: Estimated risk by territorial authority, 2013 youth population#

Appendix 4: Distribution of projected costs by estimated risk of poor outcomes at age 20#

Appendix 4 Figure 1: Projected welfare and corrections cost distribution by risk level at age 20, 1990/91 birth cohort (numbers of people - $0 to $100,000 only)

Appendix 4 Figure 2: Projected welfare and corrections cost distribution by risk level at age 20, 1990/91 birth cohort (numbers of people - over $100,000 only)

Appendix 4 Figure 3: Projected welfare and corrections cost distribution by risk level at age 20, 1990/91 birth cohort (percentage of each risk level population - $0 to $100,000 only)

Appendix 4 Figure 4: Projected welfare and corrections cost distribution by risk level at age 20, 1990/91 birth cohort (percentage of each risk level population - over $100,000 only)

Appendix 5: Target populations by territorial authority, 2013 youth population#

s = suppressed for confidentiality reasons.

s = suppressed for confidentiality reasons.

Appendix 6: Target population overlaps - 1990/1991 birth cohort#

Appendix 6 Figure 1: Target population overlaps 1990/91 cohort, ages 15 to 19

Appendix 6 Figure 2: Target population overlaps 1990/91 cohort, ages 20 to 24