Measuring Impact: an idea whose time has come

Impact measurement is the current sector zeitgeist. Hugh Wilson finds charities embracing it to keep funders happy and arguments over the measurement of data, but ultimately, the benefits of good impact measurement are significant and the idea is here to stay


Some say it started in the 1990s with Comic Relief and the Big Lottery Fund. Others say that the first (short lived) wave of dot com millionaires in 2001 and 2002 forced the change. Many would cite more recent origins, from the financial collapse of 2008 to the first mutterings about something called The Big Society.

But whenever the steely-eyed focus on impact measurement began, one thing is for certain. For the third sector, it’s an idea whose time has come. For those who fund charities and those who commission the sorts of services charities increasingly compete to deliver, the vague assumption that charities know best and will spend money wisely is no longer enough. Large donors and public commissioners want evidence that a charity can deliver real and lasting change, rather than figures showing the number of posters it can put up.

“When we first started, 23 years ago, there was very little impact measurement going on,” says Sam Matthews, acting chief executive of Charity Evaluation Services (CES), an organisation which provides support and advice on evaluation systems for the voluntary sector.

“What’s changed is not necessarily just funder lead, but clearly there are very strong external drivers coming from funders. At one time funders were quite happy to get a Christmas card, and then – perhaps driven by the Big Lottery Fund, BBC Children In Need and large funders like that – they started requesting more information. And what they started wanting to know is what difference you are making.”

Keeping funders happy
According to the New Philanthropy Capital’s (NPC) Making an Impact report, three-quarters of charities now say they measure the impact of their work and nearly three quarters (74 per cent) of these have invested more in measuring results over the last five years.

The report suggests that keeping funders happy is still the main driver of these efforts. Over half (52 per cent) say they have increased their measurement efforts in order to meet funder requirements, while only one in twenty (5 per cent) state service improvement as their main motivation

David Pritchard, head of measurement and evaluation for NPC, believes the report is a good reflection of the state of the sector. “Certainly now we do less of trying to convince people that measuring their impact is a good idea than we would’ve done five or ten years ago,” he says. “We spend more time now taking people through the process of
actually doing it.”

“We use the analogy that it’s medicine that donors and funders have required, but medicine that charities are actually beginning to see is good for them.” Pritchard says that charities have reacted to the need for better impact measurement in the way the public have reacted to – say – official pronouncements on the need to exercise.

Everybody knows it’s good for them but while some are already in the gym every night, others are still trying to decide what exercise suits them best, and even whether they can afford the training gear. A stubborn few are stuck on the couch hoping it all might go away.

What’s universally acknowledged beyond the couch potato minority is that it won’t go away. Impact measurement is here to stay. Large donors and public funders will only want to see more evidence of money well spent in years to come – and lives changed as a result – rather than less. The 25 per cent of charities who do no impact reporting
(according to Making an Impact) are living on borrowed time.

Ned Wills, global director of the Laureus Sport for Good Foundation, which supports over 140 sports-based community programmes around the world, says measuring impact trend is one charities need to get to grips with. “Measuring impact is increasingly becoming a crucial element of the charitable mix. Increasingly well informed funders are rightly asking for proof of the value of their investment. This is a trend that I expect to continue,” he says.

Wills says impact measurement does not need to complicated, but it does need to become more embedded in each organisation. “It needs to be part of the culture. Effective measurement will help an organisation to focus on its fundamentals – what are we trying to do, how effective are we, what improvements can we make to ensure efficient and impactful delivery?’”

Wills says that for years a lot of the information collected from Laureus grantees was limited to qualitative data, for example case studies of particular individuals, rather than to outcomes – measuring the number of actual life changes taking place as a result of programmes as a whole.

“This data is increasingly important for funders as it allows us to assess the impact of our grant on the people we exist to help and add the necessary support required to global transfer knowledge and best practice to those who require it. Data will also help us to assess and promote the impact of sport at a macro level as well as on individual communities.”

David Pritchard suggests the charities forging ahead are finding that having evidence to wave under the noses of funders is only one advantage of good impact measurement. To stretch a metaphor, they’re finding that going to the gym makes them look lean and fit to potential suitors, but it’s also giving them an unexpected burst of energy.

“The policy and funding climate has changed and that’s clearly a factor,” says Dr Malin Arvidson, senior research assistant at the Third Sector Research Centre (TSRC). “But it’s not just about proving what you do but also improving what you do. Doing impact evaluation can be a way of understanding what you want to achieve, and where you fail and where you can do better.”

Big benefits
There are huge internal benefits that go with good impact reporting. Charities need to have an understanding of what they are doing, why they are doing it and how it affects change. Impact reporting can help to tell them how effective their interventions, information campaigns or lobbying efforts really are.

“When you do impact measurement and you ask people what has been the benefit of it, the most common response is that they’ve been able to improve services,” says NPC’s David Pritchard. “And in some cases it can be revolutionary. I think probably the most important change is the impact on staff.”

Charity staff choose the sector because they have a passion to see change. Pritchard believes that, whether evaluation data shows that an organisation is making less of an impact than staff hoped or more, it tends to have the same effect. “In both cases, as long as it’s handled well, it can lead to an increase in motivation, and even a sense of, ‘can we get more out of this – can we get even better?’”

That’s reflected in the experience of charities like the NSPCC, who carry out regular evaluations of the success of their services and the impact they make on children and families. Indeed, according to Gerry Tissier, the charity’s head of strategic communications, the NSPCC decided a couples of years ago to make sure their
services were designed around much more demonstrable measures of impact. “Our programmes are designed specifically to enable us to get the best information that we can to assess the impact of the work that we’re doing,” he says.

The impact on staff has been noticeable. “Research has found that most highly engaged public sector staff believe they can have an impact on public service delivery, and one of the things that we’ve found important to staff is the knowledge that they’re having a personal impact on helping to end child abuse,” says Tissier.

That comes from knowing just what sort of impact an information campaign or lobbying effort has. Good impact measurement allows charity staff to tell a story about their own role in change for the better, and to feel good about their jobs.

Charity courage
But while staff motivation can benefit from impact measurement, in organisations where impact measurement is done badly or not done at all, it’s sometimes staff – and particularly management – that are to blame. Some charity heads just don’t want to test their organisation’s effectiveness, for fear of hearing some unpalatable truths.

“It takes courage to step up and say you want to know how effective you are, and be open to the fact that you may not be as effective as you hope,” says Pritchard.

Malin Arvidson says this fear stops some charities from evaluating their impact at all, and many more from evaluating it in a useful way. When she recently interviewed consultants brought in to carry out impact evaluations, they told her that several large organisations – some boasting large Government contracts – wanted to use impact measurement as little more than a marketing tool.

“They were interested in headline figures and glossy reports, but what about all the narrative – the case studies, the qualitative material? They didn’t want to broadcast that because it showed a much more complex picture. And yes, some don’t want to find out that they’re not working quite as efficiently as they ought to be. There’s a lot of : “We’ve done this for 25 years, it’s always been great, so why should we change?”

There are other, perhaps more justifiable, internal obstacles to good impact
reporting. The most obvious is a lack of resources. “I think that there are some in the
sector that may not be on board with this whole thing partly because they see it as a diversion of resource and this is a valid perspective, but one I would challenge,” says Caron Bradshaw, chief executive of the Charity Finance Group (CFG).

“However, we can’t run before we can walk. We have to remember that measuring impact is not straightforward. Social research methods are constantly evolving, and looking at the impact of your organisation takes commitment and investment. For some, some of this is investment better put somewhere else.”

Proper impact measurement takes money and manpower. It also takes understanding. Some charities are understandably unsure as to what to measure and how. As social research tools proliferate and become more sophisticated, many charities are also confused about which is the best for them.

“I think charities know the information is useful. The struggle now is how they can do it in a way that enables them to gather useful information given very limited resources and very limited time constraints,” says CES’ Sam Matthews.

“There’s a lot of confusion, and a lot of finger wagging about whether charities are doing enough, but if they’re not doing it, it’s often because they’re struggling with how to do it rather than not appreciating its value.”

Matthews says that the way charities gather and analyse data must take into account the context they’re working in. There can be no one size fits all approach to impact measurement in the third sector.

Monetary value
That much will be obvious to anyone working in it. In many cases measuring output is relatively easy – how many people contacted the helpline, how many attended the group, and so on. Measuring outcomes – how useful those interventions were – is far more difficult.

For some charities, a carefully worded questionnaire to recipients of services may be enough. Many others are turning to SROI (Social Return On Investment), a method that attempts to assign monetary value to social outcomes. Where appropriate, some are undertaking large scale academic studies to measure their impact.

But even with an increasing number of tools and theories at their disposal, many charities find impact measurement hard because of the nature of their work. “Take charities who work with vulnerable people, for instance,” says Malin
Arvidson of the TSRC. “The recipients may find it difficult to answer questionnaires
with rather abstract questions, some may be difficult to trace, some may have trouble with literacy skills. There can be lots of issues relating to beneficiary groups that make even a basic before and after survey difficult.”

But she says the situation is improving. Tools for measuring change are getting better. She cites one interactive tool, the Outcome Star, which allows organisations to plot how an individual is changing over time. Malin says it lets organisations frame impact questions in a conversational manner, and information can be aggregated to show how an organisation is performing and where its strengths lie.

Garbage data
But even when the right tools are available charities don’t always choose them. Arvidson says that often, decisions on impact measurement are taken with in-house expertise or previously gathered data in mind, rather than getting the best information from the most appropriate method.

Caroline Fiennes, founder of charity finance consultancy Giving Evidence, thinks that, for all these reasons and more, a lot of impact measurement data is “complete garbage”. For a start, she says, it’s being done for the wrong reasons, as NPC’s data suggests.

Charities are measuring impact to impress funders rather than to try and improve, and they’re not even doing that very well. “Hardly any of that data allows for comparisons between organisations, so as a funder you’re basically none the wiser. And second, most of this data is rubbish because charities are only required to measure it themselves,” she adds.

“We know in medicine that’s there’s a massive difference between trials funded by drug companies and trials of the same drug funded independently. Too often with charities your impact measurement is the same as your marketing.”

Fiennes believes that, because impact measurement is not done independently, charities can choose to measure what they want, using whatever timescale, to give
the most impressive results. And because rigorous social research is tricky, charities
tend to measure the things that are easiest. They might not be the things that will tell either a funder or the charity itself how well it is working.

Others take the point about independent auditing but wonder if it is possible to find outside auditors who understand the way many charities work. But most experts agree with another of Fiennes’ assertions: that it’s too tempting for charities to measure what they think funders want to see, rather than what is most useful. And doing that can lead to funders taking increasing control of a charity’s direction.

The impact journey
Everyone believes charities need to do more impact measurement, and that done well it can be useful at worst and revolutionary at best. What the debate suggests is that some charities have a way to go when it comes to measuring outcomes accurately, while others are quite far along the road but others – because of resources, confusion over what to measure and how, or occasionally internal reluctance – have barely begun.

Within this, Ned Wills cements his view of the sector and impact measurement. “These are challenging times for the sector. Our stakeholders are ever more sophisticated and they want to know their funds are well invested. Our response is to ensure that those we support are able to provide us with the data we need to continue the flow of funds to important grass root community projects.

“Proving the value of charitable investment is the only way this sector will continue to grow. And I have no doubts that efficient and effective measurement is an essential part of the way to do it,” he says.

What is clear is that impact measurement is here to stay. More decisions by funders and charities themselves will be made with impact measurement
data as a guide.

Hugh Wilson is a freelance journalist

Share Story: