Note from Managing Editor
Editorial
The elusive quest for policy impact – and the best way of measuring it
Enrique Mendizabal
Founder of On Think Tanks
It is rather difficult to get through a conversation about Think Tanks, with thinktankers, without considering the issue of policy impact. If Think Tanks differentiate themselves from other research-based organisations in their desire to, at the very least, inform policy, how then can they tell if they are being successful?
Over the years, On Think Tanks has addressed this issue from a number of angles and perspectives.
We’ve argued that measuring influence, and quantifying it, is not possible. And that if it were, doing it properly would be so expensive that it would be prohibitive for most Think Tanks –and that they should rather use those funds to do more research or to invest more in their communications.
It would be prohibitively expensive because it would demand tracking down (and somehow reporting and measuring) all possible paths of influence, including (but not only) influence through consultancy, influence through the public agenda, influence through the developing of skills of future cadres of policymakers, influence through the work of other researchers, etc. Many of these pathways are impossible to foresee.
It would also be immodest to do so, since the think tank (and the researchers) would have to assume they and only they were important and influential to merit an account of their impact. They’d also have to all-but-forget about those who influenced them: their funders, other Think Tanks, their researchers’ teachers and mentors, etc. They would have to forget, too, that, as in cases of failure to influence, luck plays a significant role in any “story of success”.
We’ve argued, too, that claiming influence is a political act –and one that Think Tanks should be careful of. It is, after all, an exercise to claim power for us, from others. By claiming to have, through their “excellent research” come up with the idea that a ministry adopted, they are, in essence, claiming that the ministry didn’t. By claiming that they convinced them to adopt it through their “excellent communications”, they are, also, claiming that they influenced them. Policymakers become passive actors in the policy process; whereas the Think Tanks take the leading role in the story.
The quest for measuring influence turns into an effort to present Think Tanks as far more powerful than they are; it turns into an effort to award Think Tanks a role they do not have: policymaking. This, by the way, goes against any theory of Think Tanks role in society. They are invariably presented as a pawn in someone else’s game: corporate power, political parties, the state, philanthropy, the media, etc. These are all far more powerful and influential than Think Tanks.
We have argued that this claim and the strategies adopted to influence policy can undermine good policymaking. After all, we do not want policymakers who will be easily influenced. Just as they listened to us today, they could listen to someone else tomorrow; and change their minds. And, let’s not forget, we could be wrong; we do not want them to take our word for it. We want them to be capable of making up their own minds based on inputs such experience, values and a range of different sources of evidence and advice.
But, also, we want our policymakers to be accountable for their decisions. That is why we elect them or appoint them. When claiming influence, however, Think Tanks are happy to take the credit but never the responsibility. In the end, the policymaker will be accountable for whatever decision was made; while the think tank will be long gone to claim influence on another policy decision.
When thinking about influence, Think Tanks must recognise that there is a difference between their goals and what they should be measuring. It is possible to recognise a long-term goal as a direction of travel or as a rallying call for an organisation or a network, while being perfectly comfortable measuring changes closer to home.
Policy change, “success” cannot be interpreted as a sign that everything the think tank did was correct: the desired change may have occurred by chance. “Failure”, on the other hand, may have happened even though the think tank did everything right.
When it comes to policy influence, Think Tanks should worry more about the things they can control and for which they should be accountable: the quality of their research; the quality of their communications; their reputation in the media, the policy community, and the general public; the quality of their staff; and their organisational health.
None of these can guarantee policy impact but can increase the chances that, if there is, the outcomes will be positive.
Articles
Three Questions that Guide How to Assess Your Policy Impact
by Julia Coffman, Center for Evaluation Innovation
The Southern Voice Meeting in Dar es Salaam and STIPRO Positioning in Policy Influence
Discourse
by Constatine Deus - Researcher and Policy Advocacy Officer at STIPRO and
Laurian Pima - Communications Officer at STIPRO
Post-Rana Plaza Monitoring: A Civil Society Initiative
by Team CPD
Tackling Information Asymmetry in the Adoption of Energy Efficiency in the Indian Buildings Sector- The Case of NITI Aayog
by Astha Ummat, Young Professional, NITI Aayog
Experience in Policy Impact
by Vanesa Weyrauch, Independent consultant and Associate Researcher at CIPPEC
An Introduction Note to Alex Gywther’s Article: Want an Impact? Tell a Good Story
by Alex Gywther, Communications Manager at UKCDS
Approaches and Challenges to Assess Policy Impact
by Andriy Andrusevych, Senior Policy Expert at Resource & Analysis Center Society and Environment and
Zoryana Kozak, Senior Policy Expert at Resource & Analysis Center Society and Environment
Down with Tobacco: A West African Tale
by Nafissatou Balde Sow, Coordinator/Regional Tobacco Taxation Project, CRES
Citizen Monitoring of Infrastructure – Experiences from a project on PMGSY Roads
by Sebin B Nidhiri, Programme Officer, Public Affairs Centre
Interviews
Dr. Samar Verma - Senior Program Officer IDRC, New Delhi
Anil Kumar Jain Adviser (Energy, Climate Change, and Overseas Engagements) - NITI Aayog
Andrew Hurst - Program Leader, TTI, IDRC Canada
Carlos Gustavo Machicado - Executive Director, INESAD
Valerie Traore - Executive Director, NIYEL
S.V. Ranganath, IAS (Retd.) Former Chief Secretary, Government of Karnataka Board Member, CSTEP
Humour
Concept - Dr. Jai Asundi, Principal Research Scientist, CSTEP
Illustration - Sandeep Khasnavis, Graphic Designer, CSTEP
Tool Review
Review of Monitoring and evaluation of policy influence and advocacy
Neeta Krishna - Associate Professor - HR Father C Rodrigues Institute of Management Mumbai
Interesting Readings
1. Policy Impact, Evaluation and Change
Summary: Measuring the impact of public policies is a crucial part of any government’s work. It is important that policies formulated and implemented by various government levels ensure that they assess and evaluate the impact of policies implemented by them. Policy impact has two components – outputs and outcomes. This presentation provides methods of assessing policy impact.
URL: http://www.slideshare.net/nida19/policy-impactecaluation-and-change
2. 15 Ways Of Measuring Think Tank Policy Outcomes-Forbes
Summary: Think Tanks can face impact evaluation challenges that are similar to those faced by the bureaucracy. Often, the actual impact of a Think tanks’ research occurs far in the future. The author provides 15 ways of measuring more immediate impact including output and outcome assessments.
URL: http://www.forbes.com/sites/alejandrochafuen/2013/04/24/15-ways-of-measuring-think-tank-policy-outcomes/#2715e4857a0b6426b81f7555
3. Communications and Impact Metrics for Think Tanks
Summary: A comprehensive article written by Fred Kuntz, former Vice President of Public Affairs at the Centre for International Governance Innovation (CIGI) based on his experiences as a communicator in various industries. The author sets the context by explaining how communication is achieved in a Think Tank, and its importance. He then highlights the problems faced by a Think Tank in measuring policy impact, specifically referring to challenges pertaining to attribution. Finally, he provides a list of 15 metrics that a Think Tank can use to measure its performance and impact.
URL: https://www.cigionline.org/blogs/tank-treads/communications-and-impact-metrics-think-tanks
4. How do you measure a think tank’s impact?
Summary: This article presents the planning, methodology and results of an impact evaluation carried out by the Mowat Centre, Canada on their own activities. While planning this evaluation they realised the main challenges faced by most Think Tanks while trying to assess policy impact – the problem of direct attribution. The article then describes a three-pronged approach used by the organisation to gauge their policy impact and the lessons learned from this impact evaluation exercise.
URL: http://mowatcentre.ca/how-do-you-measure-a-think-tanks-impact/
5. What is the evidence on evidence-informed policy making?
Summary: In February 2012, at the International Conference on Evidence-Informed Policy Making in Nigeria, over 50 delegates from 18 countries discussed, at length, the challenges faced by research organisations in informing policy making. One of the key issues that emerged from the discussions was that ‘there is a shortage of evidence on policy makers’ actual capacity to use research evidence and there is even less evidence on effective strategies to build policy makers’ capacity.’
URL: http://www.researchtoaction.org/2013/02/what-is-the-evidence-on-evidence%E2%80%90informed-policy-making/
6. Demonstrating impact: planning, partners and telling stories
Summary: Research organisations and Think Tanks are ‘increasingly being asked to demonstrate the impact of research’ by their stakeholders, especially funders. In September 2015, the London School of Hygiene & Tropical Medicine organised a two-day discussion session on the importance of demonstrating impact and the tools the School could use to demonstrate impact. This article showcases the perspectives highlighted by the participants on how they understand ‘non-academic’ impact and a set of recommendations to demonstrate this kind of impact better.
URL: http://www.researchtoaction.org/2016/01/demonstrating-impact-planning-partners-and-telling-stories/
7. Building impact over time: experiences from Zimbabwe
Summary: In this article, the Director of ESRC STEPS Centre, Institute of Development Studies, University of Sussex narrates his organisation’s experience in building long-term policy impact. He focuses on the importance of creating and measuring ‘sustained impact’, which cannot be achieved over 3-5 years. Using Zimbabwe’s Land Reform policy as case study, the author highlights that ‘[the] key to sustaining impact is engaging others in new research, and building the capacity to do this. Only when a wider body of research is developed that confirms, extends and sometimes challenges new findings will debate shift.’
URL: http://www.researchtoaction.org/2015/07/building-impact-over-time-experiences-from-zimbabwe/
Compiled by: Arushi Sen - Senior Communications Officer, CSTEP
This issue of Aditi focusses on Policy impact. What does it mean? How is it measured? This has been one of the most beautiful experiences. It was very useful to read a lot, and I would like to place on record that almost all the people/organisations that we contacted came forth with excellent outputs and articles. They were honest in sharing their thoughts. The Editorial Board as always, were enthusiastic in reviewing, providing feedback and encouraging. One feather in our cap - a French version of Aditi will also be published in the coming week. We are exploring the possibilities of a Spanish version.
Regards,
Managing Editor, Aditi
Regards,
Managing Editor, Aditi
Editorial
The elusive quest for policy impact – and the best way of measuring it
Enrique Mendizabal
Founder of On Think Tanks
It is rather difficult to get through a conversation about Think Tanks, with thinktankers, without considering the issue of policy impact. If Think Tanks differentiate themselves from other research-based organisations in their desire to, at the very least, inform policy, how then can they tell if they are being successful?
Over the years, On Think Tanks has addressed this issue from a number of angles and perspectives.
We’ve argued that measuring influence, and quantifying it, is not possible. And that if it were, doing it properly would be so expensive that it would be prohibitive for most Think Tanks –and that they should rather use those funds to do more research or to invest more in their communications.
It would be prohibitively expensive because it would demand tracking down (and somehow reporting and measuring) all possible paths of influence, including (but not only) influence through consultancy, influence through the public agenda, influence through the developing of skills of future cadres of policymakers, influence through the work of other researchers, etc. Many of these pathways are impossible to foresee.
It would also be immodest to do so, since the think tank (and the researchers) would have to assume they and only they were important and influential to merit an account of their impact. They’d also have to all-but-forget about those who influenced them: their funders, other Think Tanks, their researchers’ teachers and mentors, etc. They would have to forget, too, that, as in cases of failure to influence, luck plays a significant role in any “story of success”.
We’ve argued, too, that claiming influence is a political act –and one that Think Tanks should be careful of. It is, after all, an exercise to claim power for us, from others. By claiming to have, through their “excellent research” come up with the idea that a ministry adopted, they are, in essence, claiming that the ministry didn’t. By claiming that they convinced them to adopt it through their “excellent communications”, they are, also, claiming that they influenced them. Policymakers become passive actors in the policy process; whereas the Think Tanks take the leading role in the story.
The quest for measuring influence turns into an effort to present Think Tanks as far more powerful than they are; it turns into an effort to award Think Tanks a role they do not have: policymaking. This, by the way, goes against any theory of Think Tanks role in society. They are invariably presented as a pawn in someone else’s game: corporate power, political parties, the state, philanthropy, the media, etc. These are all far more powerful and influential than Think Tanks.
We have argued that this claim and the strategies adopted to influence policy can undermine good policymaking. After all, we do not want policymakers who will be easily influenced. Just as they listened to us today, they could listen to someone else tomorrow; and change their minds. And, let’s not forget, we could be wrong; we do not want them to take our word for it. We want them to be capable of making up their own minds based on inputs such experience, values and a range of different sources of evidence and advice.
But, also, we want our policymakers to be accountable for their decisions. That is why we elect them or appoint them. When claiming influence, however, Think Tanks are happy to take the credit but never the responsibility. In the end, the policymaker will be accountable for whatever decision was made; while the think tank will be long gone to claim influence on another policy decision.
When thinking about influence, Think Tanks must recognise that there is a difference between their goals and what they should be measuring. It is possible to recognise a long-term goal as a direction of travel or as a rallying call for an organisation or a network, while being perfectly comfortable measuring changes closer to home.
Policy change, “success” cannot be interpreted as a sign that everything the think tank did was correct: the desired change may have occurred by chance. “Failure”, on the other hand, may have happened even though the think tank did everything right.
When it comes to policy influence, Think Tanks should worry more about the things they can control and for which they should be accountable: the quality of their research; the quality of their communications; their reputation in the media, the policy community, and the general public; the quality of their staff; and their organisational health.
None of these can guarantee policy impact but can increase the chances that, if there is, the outcomes will be positive.
Articles
Three Questions that Guide How to Assess Your Policy Impact
by Julia Coffman, Center for Evaluation Innovation
The Southern Voice Meeting in Dar es Salaam and STIPRO Positioning in Policy Influence
Discourse
by Constatine Deus - Researcher and Policy Advocacy Officer at STIPRO and
Laurian Pima - Communications Officer at STIPRO
Post-Rana Plaza Monitoring: A Civil Society Initiative
by Team CPD
Tackling Information Asymmetry in the Adoption of Energy Efficiency in the Indian Buildings Sector- The Case of NITI Aayog
by Astha Ummat, Young Professional, NITI Aayog
Experience in Policy Impact
by Vanesa Weyrauch, Independent consultant and Associate Researcher at CIPPEC
An Introduction Note to Alex Gywther’s Article: Want an Impact? Tell a Good Story
by Alex Gywther, Communications Manager at UKCDS
Approaches and Challenges to Assess Policy Impact
by Andriy Andrusevych, Senior Policy Expert at Resource & Analysis Center Society and Environment and
Zoryana Kozak, Senior Policy Expert at Resource & Analysis Center Society and Environment
Down with Tobacco: A West African Tale
by Nafissatou Balde Sow, Coordinator/Regional Tobacco Taxation Project, CRES
Citizen Monitoring of Infrastructure – Experiences from a project on PMGSY Roads
by Sebin B Nidhiri, Programme Officer, Public Affairs Centre
Policy influence – Meaning and Measurement
by Stephen Yeo, Chief Executive Officer, Centre for Economic Policy Research (CEPR), London
by Stephen Yeo, Chief Executive Officer, Centre for Economic Policy Research (CEPR), London
Interviews
Dr. Samar Verma - Senior Program Officer IDRC, New Delhi
Anil Kumar Jain Adviser (Energy, Climate Change, and Overseas Engagements) - NITI Aayog
Andrew Hurst - Program Leader, TTI, IDRC Canada
Carlos Gustavo Machicado - Executive Director, INESAD
Valerie Traore - Executive Director, NIYEL
S.V. Ranganath, IAS (Retd.) Former Chief Secretary, Government of Karnataka Board Member, CSTEP
Humour
Illustration - Sandeep Khasnavis, Graphic Designer, CSTEP
Tool Review
Review of Monitoring and evaluation of policy influence and advocacy
Neeta Krishna - Associate Professor - HR Father C Rodrigues Institute of Management Mumbai
Interesting Readings
1. Policy Impact, Evaluation and Change
Summary: Measuring the impact of public policies is a crucial part of any government’s work. It is important that policies formulated and implemented by various government levels ensure that they assess and evaluate the impact of policies implemented by them. Policy impact has two components – outputs and outcomes. This presentation provides methods of assessing policy impact.
URL: http://www.slideshare.net/nida19/policy-impactecaluation-and-change
2. 15 Ways Of Measuring Think Tank Policy Outcomes-Forbes
Summary: Think Tanks can face impact evaluation challenges that are similar to those faced by the bureaucracy. Often, the actual impact of a Think tanks’ research occurs far in the future. The author provides 15 ways of measuring more immediate impact including output and outcome assessments.
URL: http://www.forbes.com/sites/alejandrochafuen/2013/04/24/15-ways-of-measuring-think-tank-policy-outcomes/#2715e4857a0b6426b81f7555
3. Communications and Impact Metrics for Think Tanks
Summary: A comprehensive article written by Fred Kuntz, former Vice President of Public Affairs at the Centre for International Governance Innovation (CIGI) based on his experiences as a communicator in various industries. The author sets the context by explaining how communication is achieved in a Think Tank, and its importance. He then highlights the problems faced by a Think Tank in measuring policy impact, specifically referring to challenges pertaining to attribution. Finally, he provides a list of 15 metrics that a Think Tank can use to measure its performance and impact.
URL: https://www.cigionline.org/blogs/tank-treads/communications-and-impact-metrics-think-tanks
4. How do you measure a think tank’s impact?
Summary: This article presents the planning, methodology and results of an impact evaluation carried out by the Mowat Centre, Canada on their own activities. While planning this evaluation they realised the main challenges faced by most Think Tanks while trying to assess policy impact – the problem of direct attribution. The article then describes a three-pronged approach used by the organisation to gauge their policy impact and the lessons learned from this impact evaluation exercise.
URL: http://mowatcentre.ca/how-do-you-measure-a-think-tanks-impact/
5. What is the evidence on evidence-informed policy making?
Summary: In February 2012, at the International Conference on Evidence-Informed Policy Making in Nigeria, over 50 delegates from 18 countries discussed, at length, the challenges faced by research organisations in informing policy making. One of the key issues that emerged from the discussions was that ‘there is a shortage of evidence on policy makers’ actual capacity to use research evidence and there is even less evidence on effective strategies to build policy makers’ capacity.’
URL: http://www.researchtoaction.org/2013/02/what-is-the-evidence-on-evidence%E2%80%90informed-policy-making/
6. Demonstrating impact: planning, partners and telling stories
Summary: Research organisations and Think Tanks are ‘increasingly being asked to demonstrate the impact of research’ by their stakeholders, especially funders. In September 2015, the London School of Hygiene & Tropical Medicine organised a two-day discussion session on the importance of demonstrating impact and the tools the School could use to demonstrate impact. This article showcases the perspectives highlighted by the participants on how they understand ‘non-academic’ impact and a set of recommendations to demonstrate this kind of impact better.
URL: http://www.researchtoaction.org/2016/01/demonstrating-impact-planning-partners-and-telling-stories/
7. Building impact over time: experiences from Zimbabwe
Summary: In this article, the Director of ESRC STEPS Centre, Institute of Development Studies, University of Sussex narrates his organisation’s experience in building long-term policy impact. He focuses on the importance of creating and measuring ‘sustained impact’, which cannot be achieved over 3-5 years. Using Zimbabwe’s Land Reform policy as case study, the author highlights that ‘[the] key to sustaining impact is engaging others in new research, and building the capacity to do this. Only when a wider body of research is developed that confirms, extends and sometimes challenges new findings will debate shift.’
URL: http://www.researchtoaction.org/2015/07/building-impact-over-time-experiences-from-zimbabwe/
Compiled by: Arushi Sen - Senior Communications Officer, CSTEP
Bang on target!
ReplyDeleteDear Sharad, thanks for the comment.
ReplyDelete