Imagine trying to get to know a child through photographs and a short update once or twice a year, compared to interacting with that same child on a daily basis. The opportunity to interact frequently provides a more clear and complete picture of the child’s safety, development, and well-being. Such is the case with data as well: Collecting data over multiple points in time — or longitudinally — allows for a more thorough understanding of the complexities of a child and family’s experience with the child welfare system when compared to data collected at a single point in time that provides only a snapshot of information. 

Point-in-time data not only is limiting, it often results in an undercount of whatever is being summarized, whether that is the total number of children impacted by the child welfare system each year or the total number of children on a caseworker’s caseload. For example, a caseworker may have eight children on their caseload at a single point in time (i.e., the last day of a month) but may have conducted investigations involving 20 children over the course of the full month.

Tracking data longitudinally, however, ensures that every incidence, outcome, service, family, and child is counted, making it easier to identify causal relationships and answer more impactful evaluation questions. For example, data may show that a particular intervention is linked to better reunification outcomes. Longitudinal data can be used to answer questions such as, “What happened right before reunification?” or “Which programs are most effective at preventing children from re-entering out-of-home care?”

Analyzing data longitudinally does not necessarily require cutting-edge technology or teams of data scientists. To the contrary, a basic list of facts with associated dates and identifiers can be linked together to address important research and evaluation questions simply and succinctly. Federal data collection systems are making progress toward collecting, comparing, and analyzing data. State, county, and tribal child welfare systems — no matter the size — also have a capacity to use longitudinal data. 

Longitudinal data helps child welfare agencies answer questions about what approaches are most effective in keeping children safe and reducing the need for foster care. This brief shares foundational elements and key opportunities for collecting and using longitudinal data based on the experiences of three jurisdictions working to increase their use of data to track performance, inform interventions, and improve child and family well-being.¹

Foundational elements

Current federal data collection requirements generally are designed for funding compliance, not performance management. As such, they represent the bare minimum of data needed to track child outcomes and are primarily focused to gauge safety and permanency — and do not tend to measure well-being indicators. Longitudinal data for specific programs should track information from inception through completion, and ideally capture evidence related to implementation, participation, and effectiveness. Longitudinal assessment of the child welfare system as a whole should begin with data about hotline calls and continue by capturing data about investigations, prevention services, out-of-home placement, and post-exit outcomes. 

User friendly, standardized data collection methods

Data quality and accuracy are critical. Frontline staff typically are the people required to enter child and family data and — often saddled with competing demands — can view the task as an encumbrance. Data systems therefore should be set up as user-friendly as possible, with easy-to-navigate interfaces and without extraneous data fields. When possible, data entry forms and processes should be automated, standardized, and designed to decrease caseworker burden.  

When the County of Santa Clara (Calif.) Social Services Agency (SSA) first launched its Cultural Brokers program, data were collected in a spreadsheet with more than 20 columns, which made reliable entry difficult. After transitioning to a more user-friendly format, program staff have found that data are more complete. The county plans to further improve its process by implementing an online data entry form, which program staff expect to result in even more reliable data.

It is important to remember, however, that pre-defined, entry-form categories sometimes can be too limiting. For example, describing allegations via long-form narrative is almost always more informative and illuminating. San Francisco Human Services Agency (HSA) is developing methods for grouping and categorizing variables (such as geographic location and time between events), which are hard coded into their data warehouse. This will make extracting and analyzing data more efficient and consistent. Ideally, data entry is inherently built into established business processes and is not a distinct task. 

Safeguarding updates and changes

Data can and will change over time, and databases should be set up to accommodate these changes. For example, a child’s race, ethnicity, or Indian Child Welfare Act status may have been reported incorrectly initially and needs to be corrected. Databases should be set up so that historical data are not overwritten when new information is entered.

Use unique identifiers for children and adults

Creating unique child-specific identification numbers that are consistent across programs and across time is key to effectively leveraging longitudinal data. Unique identifiers also should be used for caregivers, providers, and reporters. It may be helpful to review the circumstances under which a person may be assigned multiple identifiers and the process by which duplicates and conflicts are resolved. This can be a particular issue at intake, such as when a child may receive a new identifier but is later identified as a child already in the database. Some systems assign a new identifier each time a child re-enters care, which is not a recommended practice because it inhibits a full understanding of a child’s experience, resulting in a missed opportunity to understand what improvements can be made to improve outcomes in the future. 

Philadelphia Department of Human Services (DHS) designed a study using longitudinal data to better understand why out-of-home care entry rates consistently were so high, particularly among children of color, and to inform interventions to reduce those rates. The dataset, which includes both child- and community-level data, allowed staff to track what happens to each case in terms of disparities in system involvement over time. Philadelphia DHS found that poverty was ubiquitous among families reported to its hotline and within neighborhoods where the majority of residents were families of color. Of all reports to the DHS hotline, nearly four in five were reports about neglect (which often is associated with poverty-related stressors such as housing, food insecurity, and childcare needs), and 93% of children reported to the hotline did not require a formal DHS safety service. These findings suggested that cross-system interventions outside of DHS are needed that are better equipped to address poverty, concrete resources, and other well-being concerns. When DHS was creating its longitudinal dataset, matching and joining cases across different prevention and safety services was complex and time-consuming. To avoid this issue in future analyses, the agency is working to develop a unique child identification number across the social service continuum.

Disaggregate data to monitor and address disparities

America’s history of institutionalized racism and classism has led to disparities in society that manifest themselves at all key decision points in the child welfare system. Given the entrenched nature of these biases, achieving racial equity and ensuring anti-discriminatory practice within the child welfare system will require a variety of strategies. As a first step, child welfare systems must disaggregate data by key attributes of the child and family, including age, race, ethnicity, Indian Child Welfare Act status, tribal enrollment, gender identity, LGBTQ+, and socioeconomic status/income.  This will allow systems to monitor and address disparities as a matter of course in all data evaluations, informing the development and implementation of strategies to achieve equity. 

Develop a common understanding of data and evaluation

Program staff need to understand the value that longitudinal data provide, and data analysts need to understand that programs and families are more nuanced than datasets typically can capture. When both program staff and data staff understand each other’s perspective, they can partner more effectively to create robust systems and protocols for data collection and analysis.

Key opportunities

Connect data across systems — with appropriate safeguards

Integration with external systems — such as vital records, healthcare, housing, public benefits, and education — can provide valuable insight and data relevant to child welfare. Concerns about confidentiality and surveillance are valid anytime data is collected, particularly when linking to datasets and systems outside of child welfare. Marginalized families experience increased surveillance and therefore are more likely to be subject to intrusive investigations that  result in fear, trauma, and distancing from service providers. Child welfare agencies seeking to connect families to voluntary prevention services (such as those provided through the Family First Prevention Services Act) will need to reassure them that receiving those services does not put them at risk of losing their children. 

As connections between datasets are created, care should be taken to securely store the data, limit the number of fields collected, and limit the number of users who can access the data. Data should be used to support children and families and, whenever possible, families should be given access to the data collected about them and be treated as partners throughout the process in deciding what data to collect, how to collect it, and how to interpret the data.

Dedicate IT staff to create longitudinal datasets

With help from dedicated IT staff, the San Francisco HSA is creating a data warehouse that spans numerous social services programs, including: child welfare; the Office of Early Care and Education; the Department of Disability and Aging Services; and public benefit programs such as Medi-Cal (California’s Medicaid health care program) and CalWORKs (California’s public assistance program for eligible families). An HSA program support analyst oversees the project and serves as a “translator” between the IT staff and program staff. The goal is to set up the data warehouse in such a way that programmatic questions can be addressed. Integrating data from across social services programs will permit questions to be addressed that can result in better service delivery. For example:

• Data from the Child and Adolescent Needs and Strengths (CANS) instrument, collected by the Department of Public Health, can be integrated with child welfare data and examined to see whether there are any associations with child welfare outcomes to inform the development of targeted interventions. In the future, linking child welfare data with public benefit data will permit research on program participation and child welfare involvement.
• Families enrolled in CalWORKs are eligible to enroll in the Parents as Teachers home visiting program. Linking datasets and following families over time will permit a better understanding of which families are more or less likely to participate in the home visiting program and the relationship between program participation and child welfare involvement.

Drive decision making with internal priorities

Child welfare systems often are driven by external factors, such as federal data collection parameters and deadlines, Program Improvement Plans, consent decrees, and news stories. Strong data leaders can help their agency use data to drive decision-making in the direction of the agency’s internal priorities.

In effect, internal longitudinal data communicates to agency leadership on a regular — even real-time — basis as to where the system stands on key indicators, as opposed to having to wait years for information to become available or wait to learn about performance from the federal government. In addition to potentially improving services for children and families, this positions the agency to correct problems early and potentially avoid consequences altogether. When an agency can focus on its goals and use data to achieve those goals, it is much better positioned for success than one overwhelmed by status reports and competing priorities.

Philadelphia DHS is conducting a study of factors affecting high school graduation rates among children in its care, which requires combining data from child welfare and public education systems. In the exploratory data analysis phase, Philadelphia DHS is identifying factors that potentially promote or thwart high school graduation. The school system already shares regular reports with DHS with real-time information on which students have high rates of absences, suspensions, and expulsions. If the study finds additional indicators that predict dropping out, the information could be used to identify earlier, more targeted interventions.

Involve community members

Incorporating the perspectives of community members — particularly people with lived experience in the child welfare system — is an important amplification to administrative datasets, and helps to ensure that racial equity is at the core of all data collection and analysis efforts. Ideally, community voice is incorporated early in conversations about data collection and evaluation questions, as well as in conversations about the value of longitudinal data, client consent and protection of privacy, interpretation of findings, and development of strategies to improve outcomes. 

The County of Santa Clara SSA is incorporating the perspectives of family members who have participated in its Cultural Brokers program in developing an evaluation of it. The Cultural Brokers program provides families interacting with the Department of Family & Children’s Services (DCFS) access to a community liaison who can help them navigate the child welfare system (such as explaining mandates, policies, and laws). By participating in the evaluation, families are able to help define program success and influence the process so that the evaluation will assess program elements and outcomes that they consider most important. From the very beginning of the Cultural Brokers program, staff collected data on who was offered the program and, of those, who participated. Staff plan to match families that do and do not participate to various characteristics in order to construct a comparison group. Given that they are able to connect to longitudinal data in California’s CWS/CMS system, the evaluation should be able to compare outcomes such as re-referral rates, time to referral closure, and out-of-home care entry rates between families that do and do not participate in Cultural Brokers. 

Administrative data usually are limited to categorical variables and often cannot tell the story behind the data. Recognizing this, Philadelphia DHS has supplemented its longitudinal study with qualitative data collection, partnering with the University of Pennsylvania to conduct over 100 interviews with families and staff about their experiences and perspectives regarding system involvement. Administrative data alone sometimes can perpetuate pre-existing narratives. Speaking directly with families, on the other hand, provides opportunities to learn and also reduces the risk of bias in administrative data. Allison Thompson, senior research officer for Philadelphia DHS, said “one thing that came out of our project was a recognition that data have the potential to reinforce bias and do reinforce bias. Currently, we only collect sex at birth as a binary variable. We don’t yet collect gender or gender identity. We’re forced to speak to male and female as a binary construct, but that’s not how all youth or families are identifying. That’s a limitation.” Similarly, collecting racial identity based on observation without speaking to families is a limitation. To address these limitations, Philadelphia DHS is engaging with the Center for the Study of Social Policy to assess and modify its data collection and policy procedures to reduce bias and ensure greater equity.

Leverage data system modernization efforts

Federal regulations issued in 2016 laid out new requirements for state and tribal child welfare systems to build modular, interoperable systems that leverage data from external state systems (such as Medicaid). CCWIS (Comprehensive Child Welfare Information System) provides an opportunity to take full advantage of modern technology (including cloud computing and affordable data storage), and modern application. CCWIS must, at minimum, be able to collect data that can be analyzed longitudinally. Beyond complying with federal regulations, CCWIS should support agency efforts to develop more comprehensive, useful longitudinal databases.

Start somewhere

Conducting longitudinal analyses may seem intimidating at first. Breaking down the process into manageable steps can make it less daunting. Agencies can start by selecting an outcome of interest, determining what the required data elements are to measure that outcome, examining available data, and making changes to data collection processes if needed. If an agency does not have the required expertise to conduct longitudinal analyses, it may be helpful to partner with a local research organization or university for support.

Demonstrating the positive impact of one’s work with children and families is rewarding. Longitudinal analyses can both demonstrate that impact and reveal areas for program improvement.

1 This brief is based on interviews with Allison Thompson, Senior Research Officer, Philadelphia Department of Human Services, and Ian Ganhinhin, Data Scientist, City of Philadelphia Office of Children and Families, June 3, 2021; Doug Thompson, Program Support Analyst, San Francisco Human Services Agency, June 9, 2021; and Rocio Abundis, Prevention Bureau Manager, Bertha Reyna, Cultural Broker Program Manager, and Idauli Tamarin, Senior Research and Evaluation Specialist, Office of Research and Evaluation, County of Santa Clara Social Services Agency, July 15, 2021.