Evidence is evidence, right?

20 November 2017

Good intentions for social interventions* are not always enough. Decision-makers need quality evidence to know whether the products or services they develop, invest in or deliver make a positive difference. Then we can avoid interventions such as Scared Straight, the crime prevention programme which had no evidence base and caused harm to the young people it was trying to influence.

The topic of what works (and what does not) is not widely discussed in New Zealand. There is a need to build a learning system by strengthening the quality, use and sharing of evidence about social services, policy formation and evaluation.

Where do we begin to tackle these system-wide needs for quality evidence? How do we know which interventions are effective, promising or harmful? How can we make better evidence-based investments?

International jurisdictions have grappled with these issues and developed standards of evidence to assess whether interventions can be shown to be effective. Standards of evidence are tools that help decision-makers know how confident they can be that an intervention is responsible for its claimed outcomes. Standards help to directly feed evidence into the system in a rigorous and systematic way. They show people how to gather better evidence, increase accountability and share information on what works.

Recent research examines various international and national standards of evidence.

We found key differences in the purpose and application of different standards of evidence. Some have a developmental approach where building evidence capability is a priority, while others have stricter criteria for demonstrating effectiveness. Most international standards take a Western perspective on the strength of evidence, but a few have been specifically developed to show what works from an indigenous perspective.

This research provides a high-level synthesis of the different approaches to assessing intervention effectiveness and recommends a standard of evidence for New Zealand which incorporates both Māori and Western approaches to evidence from conception to implementation.

Programme and service providers, who are new to outcomes evaluation and like a little extra reading, will find this research valuable. It provides a good insight into how Superu developed An Evidence Rating Scale for understanding the effectiveness of interventions in the social sector.

Looking for more information about evaluation and the use of evidence? The Using evidence for impact section of our website is a great place to start.

Another project underway follows on from a round table discussion held in mid-2017 on the use of evidence in public policy, co-hosted by Superu and the School of Government at Victoria University of Wellington.  The event was recorded in this conversation tracker on the Department of Prime Minister and Cabinet’s website.  Following up on a key theme of capability development and improving the policy process, Superu has been working with other agencies to develop a new good practice guide for the use of evidence, and we expect to be able to circulate this early in the New Year. 

 

* We use the word ‘intervention’ to cover policies, programmes, and practices.

Last update: 20 Nov 2017