How did Washington, D.C., develop its approach to meaningful continuous quality improvement?
Over the past several years, Washington, D.C.’s Child and Family Services Agency (CFSA)1 has employed several strategies to advance transformation, moving from a focus on foster care to a focus on implementing family support and strengthening interventions that prevent entry into out-of-home care. The development of a robust, thoughtful continuous quality improvement (CQI) system has been an essential element in this transformation. Since 2012, CFSA’s work has been driven by Four Pillars, each of which is associated with evidence-based strategies and specific outcome targets that are tracked on the agency’s quarterly CQI scorecard. The agency’s CQI process has also supported progress towards exiting the consent decree under which it has been operating since 1989, and successfully submitting and receiving approval for the nation’s first Title IV-E Prevention Plan.
Planning and development
CFSA undertook the revamping of its CQI system deliberately over the course of several years.
Structural changes
In fiscal year 2017, CFSA merged three offices (Office of Agency Performance, Office of Quality Assurance, and Office of Quality Improvement) into one: the Performance Accountability and Quality Improvement Administration (PAQIA), which sits within the Office of Planning, Policy, and Program Support. This restructuring permitted all evaluation and CQI activities to occur under the umbrella of one division, allowing for more effective data collection, analysis, and reporting. The mission of the PAQIA is to “create a continuous learning environment for consistent use of data that helps improve agency processes, procedures, and functions.”
The Four Pillars are foundational to CFSA’s work, embedded in daily practice, and an integral part of the agency’s CQI system. Indicators of performance for each of the four pillars are tracked in a quarterly scorecard:
- Narrowing the front door: Remove children only when necessary.
- Temporary safe haven: Permanency planning begins the day a child enters care; children are placed with families whenever possible.
- Well-being: All children have the right to a nurturing environment that supports healthy growth and development, physical and mental health, and academic achievement.
- Exit to permanence: Children exit care as quickly as possible to a safe, permanent home or life-long connection; youth in care actively prepare for adulthood.
Self-assessment
With guidance from Chapin Hall at the University of Chicago, CFSA conducted a thorough self-assessment of its CQI structure and processes. Findings from this assessment informed plans for the development of the CQI system. The assessment included strengths, challenges, and action steps related to:
- Strategic objectives and theory of change.
- Foundational administrative structure (including organizational commitment, systemic approach to CQI, and workforce development and capacity).
- Collection and analysis of quality evidence (including data integrity and dissemination).
- Feedback and adjustment.
Approach
Rather than develop and implement a CQI system using a top-down approach (setting up a governing body and determining strategies at a leadership level), CFSA employed a developmental approach. This involved discussions with individual units about their existing CQI processes to understand strengths and needs, and incorporating staff expertise into the design of CQI processes. In addition to learning from staff, this approach increased staff engagement with CQI. Through these discussions, CFSA identified two primary areas of focus: understanding and using theories of change, and (2) soliciting and incorporating stakeholder feedback.
Learning organization
A key component of CFSA’s progress in CQI has been building agency culture to support growth, innovation, learning, and improvement. CFSA has shifted from being an organization perceived as punitive to one in which people feel safe sharing and learning from results that do not meet targets. In the past, some staff perceived that they would get in trouble if they presented data showing unfavorable performance. CFSA leadership took steps to model the type of environment needed to allow for learning without judgment, and that model cascades throughout the organization. CQI is discussed in an environment focused on learning from the data, and accountability for performance has spread throughout all levels of the organization.
We needed a governing body and a system to help determine what our CQI system would look like. We realized we had to approach it from the ground up instead of the top down. How do we enhance what we’re currently doing at the program level? How do we take what we have and add onto that?
– Michele Rosenberg, Deputy Director, Office of Planning, Policy and Program Support, CFSA
From theory to action
CFSA initiated a range of activities to support CQI implementation and spread throughout the agency.
Building staff capacity for CQI
The agency needed to develop sufficient staff capacity for CQI. CFSA has spent the last two years training a variety of staff (including program managers, supervisors, analysts, and deputies) to understand and engage with data. Trainings cover the fundamentals of CQI, effective use of data, and data visualization. “Data bootcamps” feature instruction on data quality, creating effective spreadsheets, and synthesizing and presenting complex information in accessible ways for lay audiences. CFSA’s PAQIA staff conduct CQI-related work (such as monitoring data integrity, creating and maintaining dashboards, and writing internal and public-facing reports) and serve as internal agency resources on all CQI matters.
CQI is built into CFSA’s routine. For example, staff meet with supervisors of each unit and conduct mini quality reviews on a weekly basis. During these meetings, supervisors present two open cases and discuss a consistent set of questions regarding case activities (such as conducting key interviews and assessments) to accelerate progress and positive outcomes.
Partnerships
PAQIA staff have developed strong partnerships with staff in other areas of CFSA to infuse and embed CQI in all aspects of program and agency functioning. CFSA’s program operations leadership and staff embrace and value CQI—a buy-in that is critical to a successful CQI system. Management and data quality liaisons from each program area are engaged, including Entry Services, Program Operations, Administration Services, Office of the Attorney General, and Office of Well Being. As a result, the agency has become significantly less siloed—staff work across program areas and partner toward common goals—and program staff throughout the agency have stepped up as champions for the CQI process.
Data quality and integrity
CFSA conducted an initial inventory of all data collection activities agency wide and found that staff were using 60 different manual databases in addition to the District’s Statewide Automated Child Welfare Information System, FACES. These databases were housed in individual program areas and only certain people had access to them, so it was not possible to get a full picture of performance. In addition, various program areas defined metrics differently so it was not possible to compare performance across areas. As a result, CFSA established a data quality committee (the goal of which is to infuse data quality as an integral part of agency practice and culture) and a data integrity committee (the goal of which is to ensure that all data are collected and reported on in a reliable and valid way).
Using CQI to assess and address mental health service needs
CFSA continues to look for ways to strengthen children’s placement stability and permanency outcomes. Using its CQI processes, the agency learned that the accessibility of mental health services was lacking. Evidence from multiple sources indicated that mental health service needs—which are critical for the stabilization of children, youth, and families, both to maintain and to move to permanency—were not being met. Staff talked with families to gain an understanding of their experiences and examined measures such as the number of youth awaiting services, wait times, and therapist turnover at the Department of Behavioral Health. The agency decided to try bringing mental health therapists in house to decrease wait times and increase access, and will be monitoring outcomes to see whether the change is resulting in increased placement stability and permanency rates.
Public accountability
CFSA created a public-facing dashboard to provide community members with up-to-date information about agency performance. The dashboard contains quarterly statistics, including the number of children in foster care, the number of children being served in their homes, and indicators related to each of the Four Pillars. For each measure, the dashboard includes a definition, performance metrics, and an explanation as to why it is important to track. The webpage housing the dashboard also includes links to agency reports, including the 5-Year Child and Family Services Plan and the Annual Progress and Services Report.
Community engagement
As part of the rollout of the dashboard, CFSA hosted a town hall attended by families, resource parents, community agency representatives, and other key stakeholders. Its purpose was to share progress and solicit feedback. Moving forward, CFSA plans to hold town halls on a routine basis. CFSA further engages the public through a network of community-based agencies that provide direct services to children and families. The agency’s five-year prevention plan was informed by input from the five Healthy Families/Thriving Communities neighborhood collaboratives.
Effective CQI within a child welfare agency requires building a culture where data is not punitive, where we are learning from each other. We have labored to create a culture where no matter what data we share, we’re going to listen, be reflective, and ask the important questions—but not blame. Shifting our culture has been key.
– Stephanie Jones Peguero, Administrator, Performance Accountability and Quality Improvement Administration, CFSA
Impact
Examples of CQI positively impacting families and staff include the following:
Putting families first
In October 2019, CFSA’s five-year Family First Prevention Plan became the first in the nation to be approved. CFSA’s Family First plan increases the availability of evidence-based practices to prevent foster care entry and support reunification (including parenting and home visiting programs, mental health treatment, and substance use treatment). It includes detailed plans for ongoing CQI processes using internal and external supports.
Developing the new Comprehensive Child Welfare Information System (CCWIS)
CQI principles are being used to support CFSA’s development of its new Comprehensive Child Welfare Information System (CCWIS). With the guidance of an external facilitator, program staff mapped their processes on detailed value stream maps and identified steps that add value and those that do not. Staff revised their processes as needed based on this close examination, and these revised processes are being incorporated into the design of CFSA’s new CCWIS.
Resources for CQI system development
- A 2012 information memorandum from the Administration for Children and Families, Establishing and Maintaining Continuous Quality Improvement (CQI) Systems in State Child Welfare Agencies, includes CQI background information and resources specific to child welfare agencies.
- The Center for State Child Welfare Data at Chapin Hall’s 2014 document, Principles, Language, and Shared Meaning: Toward a Common Understanding of CQI in Child Welfare, provides an overview of CQI for child welfare agencies, including core principles, shared language, and examples of interpreting evidence.
Approaching the consent decree’s “finish line” and optimal service delivery
After more than 30 years under the LaShawn consent decree, CFSA staff and leaders are eager to exit the lawsuit and operate as a self-regulating system. CFSA must meet 23 standards in the Exit and Sustainability Plan (known in CFSA as “the finish line”) and used several CQI processes to review data, make meaning of it, and develop and assess strategies for improvement. Another important and ongoing effort at CFSA is the Quality Service Review (QSR), one of CFSA’s primary qualitative approaches for the continuous quality improvement of service delivery and implementation of CFSA’s Practice Model. The QSR assesses how system partners work together as a team to ensure that services for children and families are tailored and appropriate to their needs. This case-specific and system wide process includes reviews of hard copy case records in addition to face-to-face and telephone interviews with team members.
At the end of the qualitative review by program area, data is aggregated, and the results shared with program staff and leadership. Subsequently, a strategy plan is developed and monitored based on the results, with the goal to effectuate change and improve at the next review. This allows CFSA to obtain a comprehensive picture of systemic strengths and areas in need of improvement.
1 Content developed from interviews with Child and Family Services Agency staff: Stephanie Jones Peguero, Administrator, Performance Accountability and Quality Improvement Administration, and Marc Elliott, Project Manager, on 1/31/20; and Michele Rosenberg, Deputy Director, Office of Planning, Policy, and Program Support, on 2/25/20.