Child protection services are in a process of ongoing reform, including the Vulnerable Child Reforms of 2011-2014 and, more recently, the creation of the new Ministry for Vulnerable Children | Oranga Tamariki in April 2017. The collective name for the review process is the CYF (Child, Youth & Family) Review. It includes the Modernising Child, Youth and Family Expert Panel reports, associated Cabinet papers, and third-party reports such as Ernst & Young’s.
The prevention policies proposed by the CYF Review are under development. The basic logic is sound: to create a range of preventive services, sensitive to the actual needs of families, that are well coordinated and effective, so that the harm caused by child abuse can be prevented from occurring. However, if we dig down into the details of what is proposed so far, we find multiple points of tension between this vision and the proposed policies. This Briefing Paper critiques one aspect of the CYFS Review process: how outcomes are measured. A wider analysis of the process is in my longer report.
Measuring outcomes
The CYF Review conflates the prevention of child abuse and neglect with the government’s new social investment approach. Reductions in renotifications, future welfare and criminal justice liability are stated as the desired outcomes of social investment in this context, but these all have tenuous relationships with reductions in the incidence of child abuse. Even with the use of the integrated data infrastructure that can track individuals through time, the ability to draw inferences about child abuse prevalence and the effects of it over long time gaps with administrative data is problematic. Much child abuse is never reported, so it doesn’t make it into the data to begin with. Because of this, predictive models based on reported cases are likely to struggle with predictive accuracy. Many factors not well captured in administrative data influence a life over time. Accordingly, making realistic conclusions about the efficacy of services (either in relation to child abuse, or to other poor outcomes) many years after they have been provided is difficult. Constructing the success of prevention as a reduction in economic costs to the state also contains implicit and questionable criteria about what constitutes a ‘good life’, as well as an assumption that independence is the ideal state/family relationship. It assumes any kind of state support is undesirable, despite the fact that many people need support to parent, and raising of children can be made either easier or harder by the broad policy landscape it happens within.
How success is measured in social investment requires careful consideration. The attraction of the new metric-driven social investment paradigm is that the effects of a standardised prevention programme can be measured quantitatively, then compared to future outcomes. While evidence is very important, the type of evidence obtained and the nature of the outcomes deemed desirable requires careful scrutiny. There are two reasons this approach may not result in effective preventive services. Firstly, the way that the effectiveness of prevention programmes is measured may not adequately capture their full effects. Secondly, such programmes are alone unlikely to reduce the more general causes of child abuse and neglect relating to deprivation and community need.
For example, if a programme succeeds in improving the relationships between parents and their children, but the local school provides poor education and the local community provides few jobs, then those children could end up on benefits and the programme will be deemed a failure. Similarly, if a programme succeeds in boosting the confidence of a sole parent and her parenting improves accordingly, but she gets involved with a violent partner who prompts a notification to child protection services, then (again) the programme will be deemed a failure. Conversely, if improving economic conditions improve the life outcomes of a child, this could be incorrectly ascribed to the impacts of a programme, because the ability to control for multiple other factors across a person’s lifespan is tenuous, especially with administrative data that may not capture all relevant influential factors. Administrative data only records points of contact with administrative systems, and relies often on third party information.
The appropriateness of using benefit receipt as an indicator of success in life is also problematic – if a disabled child and her parents require benefits for their whole life, is this a ‘failure’ within the current social investment paradigm? To avoid these misattributions of causal influence, it is important to incorporate a range of evaluative measures, both short and long term, and including both administrative, self-reported quantitative, and qualitative perceptions of people receiving services. Careful public debate about the hoped for outcomes and their assumptions about what ‘good outcomes’ look like is also important.
Some of these issues were explored by Ernst & Young in a commissioned report on the development of an actuarial model to support the social investment approach. The authors of the Ernst & Young report take quite a different approach to the CYF Review to conceptualising both vulnerability and the outcomes used to evaluate success. Firstly, they ‘emphasise the need to consider the child in the context of their environment: parents, carers, siblings, family, and community, including whānau, hapū and iwi where relevant. This requires measurement of the wellbeing of these entities in a child’s life as they are key contributing factors for the child’s own wellbeing’ (p. 5). This focus on the child in context – one whose wellbeing is delivered essentially via their families and communities, as opposed to a stand-alone individual – flows through the report. This differs in tone from the CYF Review, which considers children as essentially separate from their family and community context.
There is also a difference in their definitions of desirable outcomes. The CYF Review focuses closely on reduction in welfare, criminal justice and notifications as the desired long term outcomes that would be used as measures of success. The Ernst & Young report expands the range of outcomes, and how they are measured, to include some that relate more broadly to social need and wellbeing – not just child abuse prevention. In light of the difficulties of inferring connections between administratively derived variables over time, the Ernst & Young report suggests short-term outcome measures as well as long-term. Its proposed actuarial model ‘seeks to understand the development of risk, need, outcome and cost over the short and long term for individuals. The actuarial model proposed includes a measure of liability, a measure of well-being and a measure of need’ (p. 172). It specifically recommends that ‘that non-financial measures … should be used to help put financial measures in context where possible’ (p. 175).
By including short- and long-term outcomes, a focus on wellbeing, as well as measures derived from close to the subject, the Ernst & Young report offers a more fine-grained and realistic approach to what should be taken as evidence of success. Crucially, it connects the broader social conditions relating to poverty and community wellbeing to the experiences of children and includes them as part of what is needed to promote wellbeing including prevention of child abuse and neglect. They point out that to build an actuarial model that considers all these factors would be challenging, because administrative data only reflects the points where people come into contact with services, so it might miss some people and not cover community factors very accurately.
The Ernst & Young report addresses some of the criticisms relating to context and measurement found in the social investment approach but it is difficult to ascertain if the government is following their more complex approach. This matters because how outcomes are measured will influence how policy is developed, delivered and assessed. Ultimately, it matters because it will impact on child abuse and neglect prevention.
