Philanthropy Roundtable recently spoke with James Sullivan, co-founder of the Wilson Sheehan Lab for Economic Opportunities (LEO) and professor of economics at Notre Dame University, and Heather Reynolds, managing director at LEO. LEO helps workforce nonprofits do a better job evaluating their outcomes through research. Ultimately, its goal is to help policymakers and philanthropists allocate their resources toward programs that have been proven effective.
Q: Can you quickly introduce us to LEO’s mission, history and focus?
Sullivan: LEO is a research center here at Notre Dame that partners with local social service providers and government agencies nationwide to build evidence to understand what works in terms of lifting individuals and families out of poverty.
The idea is that this evidence can be used to inform policymakers about how to allocate scarce resources, improve outcomes for the disadvantaged and inform social service providers about how to implement more effective programs. Policymakers and philanthropists can then allocate their resources toward programs that will improve outcomes.
We’re celebrating our tenth anniversary this year. Bill Evans is the co-founder along with me. We’re both economists here at Notre Dame. About 11 years ago, Bill and I got connected with the leadership at Catholic Charities USA, who were providing feedback from their member agencies about frustrations regarding how they articulate the impact of their work. How do they demonstrate to funders the effectiveness of their programs? That started a conversation where we talked about the researchers who are interested in addressing these same questions.
I think the realization we had from those conversations was that there was a market failure, and there are providers across the country who want to understand the impact of their work. Researchers at universities are really interested in these questions, but these two groups aren’t coming together to address them. That was the original impetus for LEO: to address that market failure and bring these two groups together.
Q: Why is the evaluation of outcomes so important in looking at these different nonprofits? Why is it important to know what works and what doesn’t?
Reynolds: My part of the story is that I started on the provider side. I’m a social worker by training, and I spent 17 years at Catholic Charities Fort Worth — the last 13 leading the organization.
I think I should just give you an example of why this sort of evaluation is so important, which is really causal evidence building. Many nonprofits, such as Catholic Charities Fort Worth, run a financial literacy program where local thinkers and investment experts come in and teach clients about financial literacy. We also had programs to get people jobs, and similar support. We tracked impact as a measurement of output.
How many people attended our financial literacy classes? How many people came through our job training programs? That is very common in the nonprofit space. Taking it a step further, a lot of organizations will then say things like, “Okay, our goal was to teach people to be more financially literate. Our goal was to help get people jobs. Well, what happened?”
We might say, “Oh, we got all these people jobs,” which is another measure of an outcome of impact. We might look at how many of them still have their job six months later, and then other pointed measurements.
The difference between that and what LEO does is that LEO offers an opportunity for us to capitalize on the sad fact that most nonprofits have more people to help than there are resources to help them. LEO can help that nonprofit fairly allocate resources where some people get the service, and some people don’t because of limited capabilities slots. Then, we study what happens to both groups. Instead of just saying, “Oh, we got this many people jobs, and they kept their jobs for this number of months,” we’re able to have two groups of people who look the same, but some got the help of the nonprofit and some did not.
What that allows us to do is know precisely whether that service is what was effective. Because oftentimes, going back to my Catholic Charities example, we might show great employment outcomes, but that’s because the Omni Hotel just opened up in Fort Worth, Texas, and hired 50 of our clients, something both the groups of those we helped and those we did not help would benefit from. Being able to do causal evidence building instead allows us a more precise way of identifying whether an intervention has the outcome we intended for it to have.
Q: Based on your experience and research, what are some of the top attributes of nonprofits that operate well in this space? What have you found?
Reynolds: There are a couple of things we’ve observed that have been key. One is that leadership really matters. We have found that a lot of the leaders we work with are concerned with understanding their impact. The pain of doing research is worse than the pain of not knowing why they got into this work in the first place, which is to make a difference and to really understand that level of impact.
We spend a lot of time working with organizations that have a level of innovative services they have developed. That can be anything from a small entrepreneur in Nashville, Tennessee, to a large multi-service organization in Chicago. The size of an organization is not as important as its leadership, the innovation of programming, and the desire to find real solutions that will drive forward change in the social service sector.
If we have the right partner, then we need to find the right project with that partner. That involves looking for organizations that have a very well-defined program or service to be studied, and clear outcomes of what they want to see happen. Then the organization will have an excess demand for services and a willingness to conduct this level of a randomized control trial.
Those are the typical things we look for in a partner and a project. We continue to be wowed, again and again, by our partners who are uber committed to understanding and unlocking the potential of what they do by building evidence alongside it.
Q: Can you think of one or two compelling stories of what nonprofits are doing with that sort of research?
Sullivan: One of the strongest predictors of spending a majority of your life in poverty is not to graduate high school. A startling realization we had here when we talked with some nonprofits is that after you reach a certain age, you can no longer get a high school degree.
There are state laws which prevent accreditation for high school degrees after you age out. You can get a GED, but there’s a lot of research that shows pretty convincingly that a GED is not the same. It doesn’t lead to more stable employment or higher earnings in the way a high school diploma does.
We have been working with a partner at the Excel Centers that is basically running a high school program and accreditation for older adults in states where this is possible. We partnered with them to measure the impact of the program, since these tend to be vulnerable individuals who have unstable employment or a criminal justice record. Excel Centers provide childcare and other case management support alongside the high school program.
We measured the program’s impact over the long term on factors, like the earnings and employment of graduates, and showed that it increased earnings by nearly 40%. Excel Centers then used this data to convince other states to invest in the program and allow for changes in the laws that will allow accreditation of high school degrees for adults.
Q: For both of you personally, what motivates you in the work you’re doing?
Sullivan: I’m an economist, and I got into this business because I thought the tools that economists use, particularly the data that they use and the models, provide the best hope for improving the way policymakers and leaders make decisions. In particular, I thought that with an understanding of economics, we could help policymakers and providers have a bigger impact on improving the lives of the disadvantaged.
I feel like we’re just getting started and the potential is much greater than I had ever imagined.
Reynolds: I’m committed to this work because when you start your social work career, you’re really excited about the industry. Then you work to change lives, and then even the industry itself, and it’s really rough.
You’re hamstrung a lot because so many of your dollars are expected to go to service, which in theory, is a great idea. But that really doesn’t leave a lot of room for what is typical in every other industry: research and development dollars, and dollars put toward innovation and trying new things.
Shortly after I became CEO of Catholic Charities, I read an article in our local newspaper about a homeless services CEO who was retiring. He had served in that role for over two decades, but he was quoted in the newspaper as saying he believed that the homeless in Fort Worth were no better off on the day he retired than on the day he began.
I remember thinking that was the saddest thing I could ever hear: that you dedicate your whole career to making things better, and you don’t really feel like they are. That became a rallying cry, both when I was at Catholic Charities, but personally through my career. We’re called to make the world better. I began to know what made a difference based on our partnership with LEO.
Q: If money were no object for your organization, what would you envision for that future?
Reynolds: We would double the number of projects we’re running. Second, we would be taking findings from one place and replicate that in five other communities across the country. We’d be testing whether it just worked in that one site, or whether it scales at several other places.
Sullivan: Ultimately, we want to have a broad impact. Evidence presents an opportunity to do that, but not if the results get buried. You really need to have a mechanism to take this evidence and inform providers and funders all across the country.
Reynolds: One beautiful example is the Dave Thomas Foundation. It focuses on foster care and adoption. They were giving out resources to help organizations place hard-to-place children for adoption, older children in particular. But then what they ended up doing is starting a program.
They ran a randomized control trial with the program and they found that it worked. Then, because of that, instead of just giving money to a variety of organizations across the nation, they gave money to say, “Who wants to do this program? Because this is what the evidence says works.”
Now, the Dave Thomas Foundation is making a huge difference in finding adoptive homes for hard-to-place kids.
