Promoting effective charitable giving: a Q&A with Yale economist Dean Karlan

Anyone who has ever given to charity likely endured a flood of email solicitations over the holidays from nonprofits seeking support. Choosing a cause to support is just the first step in charitable giving, warn experts. Discriminating donors should ask whether a nonprofit produces evidence of its effectiveness.

ImpactMatters, an initiative launched in December by Yale economist Dean Karlan and his former student, Elijah Goldberg, ’ 15 B.A , helps donors identify effective charitable organizations and shows nonprofits how to use and produce sound evidence of their impact.

Karlan recently spoke with YaleNews about this project.

What is the mission of ImpactMatters?

Our mission is to help nonprofits learn how to use and produce better evidence of their impact and to help donors know which nonprofits do this most effectively. We’re trying to influence both spaces.

We want to help donors know which nonprofits will be good stewards of their money — meaning careful, thoughtful organizations that pay attention to what works and produce appropriate evidence of their impact. We want to help nonprofits with feedback on how to improve their use and production of evidence in order to guide them to become better organizations.

To do this, we conduct an impact audit of the nonprofit’s impact, which, similar to an accounting audit, provides a short-run assessment of a nonprofit to certify to stakeholders and guide its internal management.

What distinguishes ImpactMatters from other nonprofit evaluators?

There is a spectrum of nonprofit evaluators out there, but there is not anyone who’s doing what were doing. We’re different in a few dimensions: One is that there is no review system that I’m aware of that gives the nonprofits feedback. Most just provide external reviews. Second, with one exception, none of the nonprofit evaluators focus on evidence of impact. They are focused on things like financial data and the nonprofit’s board of directors. They put a very heavy emphasis on self-reported data on overhead, which is misplaced and can lead to some really bad incentives for nonprofits.

The one exception is Give Well, but it’s focused strictly on international poverty issues, does not provide feedback to nonprofits, and only makes a handful of recommendations as it is focused on comparing across charities. We see our effort as complementing (and complimenting) their approach.

What aspects of nonprofits do the impact audits evaluate?

The impact audit answers two big questions. First, impact: Does the nonprofit change the world? Second, operations: Does the nonprofit do what it says it does?

In terms of impact, we look at evidence on outcomes and cost. What is the nonprofit’s “theory of change” — in other words the chain linking the nonprofit’s activities to positive changes in the world — and is that chain logical and sufficiently supported by evidence?

For example, if a nonprofit says it provides bed nets to households to prevent malaria, then the first step in evaluating the logic of the story is to ask whether bed nets are an effective means to fight malaria, whether households already have bed nets, and whether malaria is a problem in the area in question. It’s a basic thought process to evaluate the theory behind a nonprofit’s work.  The next question would be to ask why households don’t already have bed nets. Where is the market failure that makes subsidizing bed nets a sensible thing to do?

Then we evaluate the evidence that supports the theory of change. That evidence can come from afar. It could come from examining other organizations and other research and mapping that evidence to the nonprofit’s program or it could come from the nonprofit itself.

We also look at the cost to achieve that impact. You can imagine a scenario in which the theory of change makes sense but doesn’t really add up to something that is exciting from a donor perspective. A program might work but it’s not creating all that much impact relative to the cost.

Evaluating operations is simpler. We look at data provided by the organization, interviews, and other information sources to confirm that the nonprofit is actually delivering the activities and outputs it claims. Similarly, we look to make sure the nonprofit is meeting some key criteria in terms of making sure those activities and outputs are delivered at high quality.

Under operations, we also look at learning and transparency. We want to reward groups that are learning how to improve internally. Similarly, we want to reward organizations that publish data on their impact and operations, and share experiences and methods with outsiders, both successes and failures, so that other organizations can learn from them.

How many nonprofits have you evaluated so far?

Four have passed and we have posted those evaluations on the website.

We sign non-disclosure agreements with nonprofits. This means no public shaming for those that do not pass. This is done specifically to protect them, so that we do not lose out on opportunities to help charities just because they may not pass. The downside risk is simply not passing and not receiving the accolade, and not finding the feedback useful. This allows us to engage with nonprofits that likely will not pass — but still want the feedback and advice, and hopefully coach them to be better organizations. 

The final product is a seal of approval with two extra pieces to it. One is a fairly long report that helps to explain our logic, what we have reviewed, and the good and the bad of what we saw. We also provide a private management letter that is probably harsher than the public report that offers feedback and points out areas of improvement.

We recognize that if you post areas of improvement, you’re making the nonprofit vulnerable and that is not the goal here. We’re aware of the public relations issues that nonprofits face in discussing their effectiveness. The nonprofits that are certified are excellent organizations, and we want the certification to be unqualified, as with most accounting audits.

What should people consider when deciding where to donate their money?

Put simply, people want to give to nonprofits that will do the most public good with their money. How one defines “the most public good” can be quite personal, of course. We are not looking to pass judgment on what that is, but rather aim to answer whether a nonprofit is using and producing evidence of impact for their goal, whatever that goal is. Unfortunately, the most popular rating systems have absolutely nothing to do with a nonprofit’s impact. We hope to influence that.

Is information about overhead costs or administrative expenses useful in evaluating a nonprofit’s effectiveness?

Overhead information is almost completely useless and misunderstood because it has almost nothing to do with the nonprofit’s impact. Why “almost”? Because there are cases of fraud, charities that spend almost the entirety of their proceeds on their fundraising and executive salaries. But these are very much the exception. And I’m willing to bet that the types of people who are giving to fraudulent nonprofits do not know to look up the overhead ratios in the popular ratings systems that use overheads.

If somebody is solely concerned about doing the most good for the most people with their donations, then what sort of nonprofits should they support?

Most people don’t approach it that way. Most people, instead, have a cause that they are passionate about, such as improving education in America or fighting poverty in developing countries. We want to give people a set of choices and say to them, “If that’s where your passion is, then here’s how you can do the most good with that passion.”

(Image courtesy of Shutterstock)