Beyond the Head Count: Measuring Lasting Change

Background-Beyond Headcount-01How is our community better because of our work?

That may be the toughest question a nonprofit ever has to answer. “We can’t get stuck in outcomes only,” says Humber College Fundraising Management professor Ken Wyman. “As governments cut back and load more responsibility onto the nonprofit sector, we need to know whether we’re creating effective solutions. Does spending $10,000 on services for a homeless person prevent $50,000 in emergency room use, for instance?”

Chasing the data for such widespread, complex issues is challenging and expensive. “Look at the medical model of testing,” Ken continues. “They follow people for 20 or 30 years at the cost of millions of dollars. One solution when evaluation is too complex or expensive to do on your own is to partner with academics who can devote some research time to you. They can bring skills, student labour and professional understanding to your review.”

Susan Pigott of Ashoka Canada also emphasizes the importance of a long timeline. “If you’re trying to accomplish large scale systemic change, measurement must be long term,” she affirms. Ashoka’s entire philosophy is built on that commitment. Intentional long-term relationships with Ashoka Fellows allow the organization to assess the long-term impact of Fellows’ initiatives according to carefully developed criteria. But smaller charities and funders struggle to establish such criteria, especially when time and other resources are limited.

Higher need, abstract impact

Winnipeg philanthropy consultant Joan Blight of Strategic Philanthropy sees a direct correlation between the nature of the need that a program targets and the difficulty of measuring its impact.

“Impact on needs that rank higher in Maslow’s hierarchy is much harder to measure,” she notes. “Fortunately, some funders understand that individual development measures (such as self-actualization) are much more qualitative. It’s crucial to know your potential funder and whether they understand and value the impact your charity is striving for. An experienced philanthropist will see the whole range of needs, but people who are just starting out may be in a different, less sophisticated place.”

Impact emphasis rare among smaller funders

The larger foundations in Philanthropic Foundations Canada (PFC ) believe that funders must do more than just demand impact evaluation from the charities they support. “We get to better measurement by starting with a lot of dialogue among funders,” explains PFC’s president Hilary Pearson. “What do we want to accomplish? How can we play a part in bringing about a big change?”

She sees a trend among the larger foundations in PFC. Generally, she explains, the larger they are, the greater their staff capacity. That sometimes goes hand in hand with a more strategic approach to systemic change that may involve working together, creating coalitions and even operating programs.

“On the other hand, many small funders don’t have capacity themselves to be demanding or sophisticated about measures,” she admits. “In the $5- to $10-million range, foundations typically don’t have staff to collect and analyze reports. They do want to know what happened and whether the money was applied for the purpose for which it was given. But at that level, they very seldom ask or hear what difference is made.”


Think decades, not years

Inexperienced funders – or those whose own resources are limited – miss the distinction between results (or outputs), outcomes and impacts, she continues. Impact is the hardest thing to measure, particularly for complex, intractable problems like poverty, homelessness and substance abuse. And both charities and funders need to understand the extended timeframe that impact measurement requires.

Hilary cites Calgary’s 10-year plan to end homelessness as an example. “We’ll see its impact over time. The outcome will be fewer people being permanently homeless. The impact will be that they get jobs, they’re better off, they send their children to school. But you still want to know – was the C that we chose to do better than the A or B we considered? You can never know for sure. Impact measurement will always be approximate.”

Impact a collective responsibility

PFC members are in good company as they move towards viewing impact as a collective accomplishment. The United Way and community foundations now pay a lot of attention to working together to share skills and expertise, notes PwC corporate responsibility director James Temple. Both use the insights of the Vital Signs community mapping process to inform their work. Like funders, charities must work together to gauge their full impact, James believes. “They must ask what community vitality looks like, understand their own role in their community ecosystem, then play to their strengths to create change.”

Today, that’s starting to happen more and more through collaborations like Innoweave (an initiative of The J.W. McConnell Family Foundation) and the Community Knowledge Exchange (developed by the Ontario Trillium Foundation and Community Foundations Canada). Although it is not yet common to see funders helping charities evaluate impact, these thought leaders may well be launching a trend that will benefit funders, charities and service users alike.