The Allegheny Family Screening Tool
Predictive Risk Modeling in Child Welfare in Allegheny County
Since August 2016, the Allegheny County Department of Human Services (DHS) has used the Allegheny Family Screening Tool (AFST) to enhance our child welfare call screening decision making process with the singular goal of improving child safety.
The AFST is a predictive risk modeling tool that rapidly integrates and analyzes hundreds of data elements for each person involved in an allegation of child maltreatment. The tool can rapidly integrate and analyze these data, housed in the DHS Data Warehouse,
and create a synthesized visualization of the information. The result is a ‘Family Screening Score’ that predicts the long-term likelihood of future involvement in child welfare. By combining the insight gained through the score with other traditionally
gathered information, a better prediction can be made of the long-term likelihood that the child will need to be removed from the home in the future.
According to the algorithm, the higher the score, the greater the chance of future out-of-home placement. When the score is at the highest levels, meeting the threshold for ‘mandatory screen in,’ the allegations in a call must be investigated. In all
other circumstances, the information summarized by the score does not replace clinical judgment but rather provides additional information to assist in the call screening decision making process. The Family Screening Score is not used to make investigative
or other child welfare decisions and is not shared beyond call screening.
Background
Utilizing Integrated Data from the DHS Data Warehouse
Since 1998, DHS has used its Data Warehouse to collect and store confidential data related to individuals who receive
services through DHS.
In 2014, with support from the Richard King Mellon Foundation, DHS launched a highly competitive Request for Proposals (RFP) process to
solicit proposals to explore how we could better use these data to
improve decision making through predictive risk modeling.
The contract was awarded to a team from Auckland University of Technology (AUT) led by the co-director of the Centre for Social Data Analytics, Rhema Vaithianathan, and including Emily Putnam-Hornstein from the University of Southern California, Irene
de Haan from the University of Auckland, Marianne Bitler from the University of California – Irvine, and Tim Maloney and Nan Jiang from AUT.
Over the next months, DHS and the AUT team explored various applications of predictive analytics and decided to focus initial efforts on child welfare and develop a tool that would assist in decision making when allegations of maltreatment are received.
Predictive Analytics and Child Welfare
The video below explains our partnership with Rhema Vaithianathan in the development and implementation of a predictive risk model for child welfare.
Prior to August 2016, any allegation of maltreatment (a referral) received at the child abuse call center required staff receiving the referrals – and their supervisors – to manually access a myriad of data and information to help decide whether or not
to investigate the allegation (‘screen in’ and investigate or ‘screen out’ and offer relevant community resources).
In the absence of standardized protocols for using the data or for systematically weighting the information, an analysis found that 27% of highest risk cases were being screened out and 48% of the lowest risk cases were being screened in.
We reasoned that by using technology to gather and weigh all available pertinent information we could improve the basis for these critical decisions and reduce variability in staff decision-making.
The Allegheny Family Screening Tool
The design and implementation of the AFST was a multi-year process that included careful procurement, community meetings, a validation study, and independent and rigorous process and impact evaluations. In addition, the resultant model was subjected to
an ethical review prior to implementation. Tim Dare, Professor of Philosophy at the University of Auckland and Eileen Gambrill, Professor of Child and Family Studies at the University of California-Berkeley conducted the independent ethical analysis. Their report,
Ethical Analysis: Predictive Risk
Models at Call Screening for Allegheny County, concluded the tool is ethically appropriate, particularly because its accuracy exceeded the alternatives at the time and there would be ethical issues in not using the most accurate measure.
DHS issued Ethical Analysis: Predictive Risk
Models at Call Screening for Allegheny County - DHS Response thereafter.
Details of this process and data related to the implementation of the screening tool are provided in the AUT report Developing Predictive Models to
Support Child Maltreatment Hotline Screening Decisions - Allegheny County
Methodology and Implementation, released in May 2019.
Evaluation
Formal evaluation of the Allegheny Family Screening Tool implementation process and impact began in 2015, following a DHS-issued RFP for assessments in these areas. Two separate contracts were awarded.
Hornby Zeller Associates, Inc., a public sector evaluation, research and consulting firm, was awarded the task of examining the process by which the tool was implemented. The firm analyzed how the tool has changed the experience of call screeners, the
practice and policy implications of its use, and perceptions and reactions to the tool. Their report, Allegheny County Predictive Risk
Modeling Tool Implementation: Process Evaluation, was published in January 2018.
Jeremy Goldhaber-Fiebert, Assistant Professor of Medicine at Stanford University, examined the impact of the tool on the accuracy of decisions, disparity rates, overall referral rates and workload and more. A summary of the impact evaluation and the full evaluation report were published in April 2019.
We continue to refine the model based upon these evaluation results, frequent analysis and ongoing feedback from call screening staff,
AFST Examined
DHS-generated Content
DHS Response to 'Automated Inequality' by Virginia Eubanks
January 31, 2018
Ms. Eubanks set out to examine the development and use of the Allegheny
Family Screening Tool (AFST). Consistent with other decisions we have made to promote rigor, transparency and accountability in our implementation of this new tool, we invited the author into our agency. Unfortunately, her piece has numerous inaccuracies
and several key points require correction.
Allegheny Screening Tool - Frequently
Asked Questions and Answers
DHS Office of Data Analysis, Research and Evaluation, July 2017
Developing Predictive Risk Models to
Support Child Maltreatment Hotline Screening Decisions
DHS Office of Data Analysis, Research and Evaluation, March 2017
Local Reaction
We will use all resources to keep
children safe
Pittsburgh Post-Gazette, March 23, 2018
This letter to the editor responds to letter from child welfare advocate Richard Wexler.
An agency that works, helping kids
and their families (op-ed)
Pittsburgh Post-Gazette, February 11, 2018
National Reaction
Hospital Injury Encounters of Children Identified by a Predictive Risk Model for Screening Child Maltreatment Referrals - Evidence from the Allegheny Family Screening Tool
JAMA Pediatrics, August 2020
Want Less-Biased Decisions? Use
Algorithms
Harvard Business Review, July 26, 2018
Can Big Data Help Save Abused Kids?
Reason.com, February 2018
This article discusses efforts to use predictive analytics in preventing child abuse.
Can an Algorithm Tell When Kids are in Danger?
The New York Times, January 2, 2018
This article takes an in-depth look at Allegheny County’s use of predictive analytics. This article also appeared in the Irish Examiner.
Actionable Intelligence
University of Pennsylvania’s Actionable Intelligence for Social Policy
This video highlights the
Allegheny Family Screening Tool as one of the projects across the country that use integrated data systems (IDS) to improve social programs.
Predictive Analytics in Child Welfare
U.S. Department of Health and Human Services Office of the Assistance Secretary
for Planning and Evaluation, November 2017
This report examines the potential benefits and pitfalls of predictive analytics and provides advice for progress.
How Machine Learning Can Improve
Public Sector Services
The Regulatory Review, October 2017
This article discusses how Allegheny County government digitized its records and uses big data analysis to improve health and human services.
Predictive Analytics in
the Child Welfare System, Starting with the Basics
Alliance for Racial Equity in Child Welfare and the Center for the Study of Social Policy
This webinar discuss the development and use of predictive analytics in child welfare.
Using Integrated Data Systems to Improve
Case Management and Develop Predictive Modeling Tools
Annie E. Casey Foundation, 2017
Four reports explore the value of using Integrated Data Systems to improve outcomes for individuals and families.