This article by Patrick Lawler, Youth Villages CEO, was written as part of a series on “Defining Positive Outcomes.”
Moving From Pass/Fail to Continuous Progress
It’s hard to fully understand the effects of interventions that aim to address several life challenges at once. But it can help to transition from all-or-nothing assessments to more incremental measures.
By Patrick Lawler Dec. 15, 2016
Lizzie was never in foster care but probably should have been. Her family was 
chaotic: her father in prison, her mother working low-paying jobs. She was sexually abused in middle school but received no support or treatment.
Despite her struggles, Lizzie graduated from high school with a 3.8 GPA. She was accepted to a four-year college and dreamed of becoming a veterinarian. Referred by her high school counselor, who saw both her potential and her challenges, Lizzie entered Youth Villages’ YVLifeSet program, which helps transition-age youth successfully move into adulthood. Since 1999, Youth Villages has helped more than 10,000 young people through YVLifeSet. From the beginning, we’ve tracked program participants’ outcomes, which provide important information on the effectiveness of our services.
During her first semester in college, Lizzie began dating a young man with troubles of his own. The relationship became serious and Lizzie found out she was pregnant. Subsequently, the relationship turned volatile, and Lizzie’s boyfriend became physically abusive. Frightened, she left college and moved out of state, seeking safety. Later, she learned she was still responsible for the financial aid she received that first semester. Through it all, she had the support of YVLifeSet staff in two different states, who assisted her with basic needs and helped her develop the skills to keep her and her child safe, access needed services, and manage her financial situation.
Which outcomes are important for Lizzie, and what represents progress for her? During the program, she began rebuilding her life and attempted to reengage in school, but she was waylaid by financial constraints, the birth of her child, and her continued involvement with the child’s father. She was able to maintain stable employment but moved frequently. She ultimately left the program with stable employment and adequate housing, but was not in school and had significant debt.
If Lizzie had sought services from a reproductive health center, the appropriate desired outcome might have been delaying another pregnancy until she was ready and able to care for a second child. If she received services focused on domestic violence, the expected outcome could be related to safety for Lizzie and her child. YVLifeSet’s primary objective is more comprehensive: to help young adults achieve their goals and live successfully. Which outcomes should such a program be expected to achieve when serving young adults with complex life challenges? And what outcomes did our young participants achieve because of program participation?
For programs like YVLifeSet, which aim to impact many different domains, establishing evidence of program effectiveness is particularly challenging. For many years, we’ve measured outcomes of participants like Lizzie against benchmarks from studies and from our own experience to render a judgement about the effectiveness of our services. These benchmarks measure whether or not program participants have attained a certain level of success in several important domains, including:
- Employment and earnings
- Housing stability and economic well-being
- Criminal involvement
- Education
- Health and safety
- Social support
In 2010, we sought to better understand participants’ outcomes in these domains by launching the largest randomized controlled trial (RCT) of a program for this population, to measure the impact of the program against the outcomes of those who didn’t participate.
The RCT was large (sample size of 1,322) and lengthy (eight years from planning to reporting year one impacts). Evaluations of this size and scope are expensive, time-consuming, and not feasible for many social programs. Fortunately, we received significant philanthropic encouragement and support to conduct the study.
We found that the program had significant positive impacts among youth in several important domains (employment and earnings, housing stability and economic well-being, health and safety), although it did not have impacts in other areas (criminal involvement, education, and social support). Given the diverse population of young adults who can be well-served by the program, that finding was nevertheless nothing short of amazing.
But this RCT has also led us to rethink our definition of success, both for young adults and for the program. Recently, we’ve sought to better understand outcomes less in terms of benchmarks and more in terms of a progress-oriented lens.
Not all young adults have challenges or goals in every domain, so our current measures of program effectiveness may not truly represent our work with all types of young adults. The benchmark approach also may miss real progress by young people. For example, reducing arrests from once a month to once in six months is tangible progress with measurable benefits, but is not adequately captured with our current all-or-nothing measures.
Moving from a pass/fail benchmark-based system to a progress-oriented framework that measures incremental growth might allow us to better understand the starting point for each young adult, how much progress they make toward their goals during program enrollment, and how well that progress is sustained and extended after discharge. That information, in turn, could provide a clearer picture of program effectiveness.
In today’s nonprofit environment, we’re all seeking the often-elusive proof of impact. Maybe we should turn our attention away from the thumbs-up, thumbs-down mentality often associated with RCTs, and instead focus on what else we can learn from RCTs and other types of rigorous evaluation. We should consider not only how information from rigorous evaluation reflects on our past and current efforts, but also how it can be best employed to improve programs, thus securing more positive trajectories for service recipients in the future. Rather than framing evaluation results as a pass/fail dichotomy, we should create a continuous learning environment, both within agencies and in the broader social services field.
Agencies that serve people facing complex challenges should have ongoing outcome evaluation processes that allow them to determine their effectiveness and provide information to guide program improvement efforts. Insights should be shared with others, and the most promising approaches should be replicated and tested repeatedly, with learning occurring through each repetition. At Youth Villages, we are continuing to mine our RCT data for insights into who benefited, who didn’t, and why, to more precisely focus our interventions and identify areas needing program enhancement.
Lizzie’s outcomes, at this point, are mixed. She has maintained stable housing, kept part-time employment, and gained access to healthcare through the Affordable Care Act. However, she also remains in the abusive relationship and has a second child. Twice she attempted to pursue further education, through online vocational-technical training and another four-year program, but she is not currently in school. Lizzie’s case demonstrates why outcomes measurement is so challenging and complex, and why we need to go about it in a nuanced and continuous way.
How can we best help young people like Lizzie find positive trajectories? We can focus on them as individuals, recognize the barriers they face, and measure their progress toward their goals. And as an agency, we can best help by concentrating our energies on continuously improving our programs; by learning, though rigorous, focused evaluation, what strategies and interventions are most effective; and by sharing that knowledge with others who are committed to this work.
Patrick Lawler is the founder and chief executive officer of Youth Villages. Under his leadership, the nonprofit organization has grown from two small residential facilities serving 40 youth in Memphis to a national leader in the child welfare community, now helping more than 22,000 families every year and operating in 71 locations across 12 states and the District of Columbia.