Policy influence – Meaning and Measurement
Stephen Yeo
Board of Directors
African Centre for Economic Transformation (ACET)
“Policy impact” and “policy influence” are terms that are used often, but seldom clearly defined. This is not because their meaning is not clear to all concerned: on the contrary, the meaning is either very simplistic or entirely unclear.
First, the simplistic view: “I wrote a research paper that was published in a top journal. The Minister read the paper and immediately implemented the policy I advocated in the paper.” You don’t have to know much about how politics and governments work to know that this is pretty silly. Ministers don't read research papers in scientific journals. And even if they did, they wouldn't base their policies on the latest journal article they have read. The simplistic view is attractive though, and not only because it is simple to understand. If policy worked in this way it would be easy to measure the policy influence of research: we just have to look at journal articles and link them to a policy change, or look at a change in policy and try to figure out what journal articles the Minister was reading at the time.
Fortunately (or unfortunately from the measurement perspective) the simplistic view went out of fashion quite a while ago – back in the 1960s, in fact. Since then it has been replaced by something more realistic and more nuanced, but more difficult to measure.
Where did this more nuanced view come from and what does it consist of? Its origins owe much to Harvard Professor Carol Weiss, who studied US education policy in the 1970s.She emphasised the “enlightenment” role of research, by which she meant the ability of research to create the framework within which policy-makers analysed and discussed policy choices. Researchers might have an indirect but enormous policy influence merely by shaping the terms on which the public debates about a policy were conducted and how policy-makers thought about the issue.
There's a lot to this idea – one has only to think of how development debates have shifted between the 1950s and the 1990s, from the discussions of national plans and central planning that were in fashion during the 1950s to the market oriented thinking of the 1990s, when every problem and policy issue was framed in economic terms and all the answers seemed to come from economists. The power to frame an issue can be very powerful indeed.
Fred Carden took this a bit further in his landmark study of policy influence in 2009. He identifiedthree different channels through which research can affect policy-expand policy capacities, broaden policy horizonsand affect decision regimes. Expanding policy capacity involves strengthening policy-makers’ ability to analyse policy-relevant research and assimilate the ideas it contains. Broadening policy horizons involves introducingnew ideas and options into policy debates, a notion that is similar to, but not quite the same as Weiss’s idea of research providing enlightenment or frameworks for discussion. Affecting decision regimes is also about capacity, like the first channel, but is more about the processes by which policies are debated and chosen and less about the people involved.
This is a very different way of thinking about policy influence. It goes “well beyond changing particular policies”, Carden argues that, “The most meaningful and lasting influence is less about specific policy change than about building capacity – among researchers and policy people – to produce and apply knowledge”. But in this more nuanced version, influence and impact emerge only in the long run. As Carden notes “This kind of influence can take years, or even decades, to take effect or become apparent. But it is no less important for that.”
So this leaves us with a much more complicated notion of policy influence. But the complications don’t end there. A more recent development is the emergence of more systematic and rigorous methods for evaluating the impact of policies. The best known of these methods involves Randomised Control Trials (RCTs), which have become much more common and influential over the past fifteen or twenty years. Impact evaluation – whether via RCTs or other methods – typically involves looking at existing policies rather than proposing new policies. So research can have a significant influence on the thinking of policymakers by providing evidence that an existing policy doesn’t work, rather than coming up with an idea for a brand new policy. In some ways this makes measuring research to policy linkages a little simpler, because impact evaluations are usually highly visible and often can be clearly linked to policy discourse.
All this gives us a more sensible framework for thinking how research may influence policy that goes well beyond whether a particular piece of research was responsible for a specific change in policy (be that a new law, a change in regulatory practice, or a decision to change a variable that is under the control of a policy maker).But at the same time these notions create a serious challenge for researchers and Think Tanks who need to demonstrate that their research has had an impact, or evaluators who have to assess a research programme or project.
Even in the apparently simple case of a policy change that involves a new law, it is almost never possible to “prove” that a particular piece of research brought about a change in policy. There are many factors that influence a policy decision, and research is only one factor - and in many cases research may be less important than other factors, such as the political environment. Why not just ask the politician or civil servant (who made the decision to change the policy)as to why they took the decision they did? But even this is far from straightforward - it may be far from obvious that an individual actually made the decision or indeed whether any single individual was responsible at all. So you may not know who to ask.
This means that you will be forced to rely on other sorts of evidence to measure impact. An analysis of the content of documents (white papers, green papers, speeches by politicians and civil servants) to detect the influence of a piece of research; or interviewing a broader range of people who were close to the decision making process, and trying to “triangulate” their answers. This begins to look more like the work of a good journalist or a detective. So even in the “simple” case of a discrete policy change, tracing this back to a research paper or project is a demanding task.
The challenge is even greater for the more complex forms of policy influence - for example the role of research in creating the framework for policy analysis and debate. How would you measure a change in a framework? That would require an even more elaborate document analysis and interviews with a much wider range of stakeholders.
The difficulties involved in measuring policy influence of research have been known and acknowledged for some time, but there have been relatively few systematic attempts to tackle the issue. The most ambitious example is probably the work by Carden, which examines the policy influence of 23 research projects funded by IDRC, using a common framework. But this has only scratched the surface of the problem, which is one of the most challenging and fascinating puzzles currently facing researchers – and those who evaluate them.
Stephen Yeo
Board of Directors
African Centre for Economic Transformation (ACET)
“Policy impact” and “policy influence” are terms that are used often, but seldom clearly defined. This is not because their meaning is not clear to all concerned: on the contrary, the meaning is either very simplistic or entirely unclear.
First, the simplistic view: “I wrote a research paper that was published in a top journal. The Minister read the paper and immediately implemented the policy I advocated in the paper.” You don’t have to know much about how politics and governments work to know that this is pretty silly. Ministers don't read research papers in scientific journals. And even if they did, they wouldn't base their policies on the latest journal article they have read. The simplistic view is attractive though, and not only because it is simple to understand. If policy worked in this way it would be easy to measure the policy influence of research: we just have to look at journal articles and link them to a policy change, or look at a change in policy and try to figure out what journal articles the Minister was reading at the time.
Fortunately (or unfortunately from the measurement perspective) the simplistic view went out of fashion quite a while ago – back in the 1960s, in fact. Since then it has been replaced by something more realistic and more nuanced, but more difficult to measure.
Where did this more nuanced view come from and what does it consist of? Its origins owe much to Harvard Professor Carol Weiss, who studied US education policy in the 1970s.She emphasised the “enlightenment” role of research, by which she meant the ability of research to create the framework within which policy-makers analysed and discussed policy choices. Researchers might have an indirect but enormous policy influence merely by shaping the terms on which the public debates about a policy were conducted and how policy-makers thought about the issue.
There's a lot to this idea – one has only to think of how development debates have shifted between the 1950s and the 1990s, from the discussions of national plans and central planning that were in fashion during the 1950s to the market oriented thinking of the 1990s, when every problem and policy issue was framed in economic terms and all the answers seemed to come from economists. The power to frame an issue can be very powerful indeed.
Fred Carden took this a bit further in his landmark study of policy influence in 2009. He identifiedthree different channels through which research can affect policy-expand policy capacities, broaden policy horizonsand affect decision regimes. Expanding policy capacity involves strengthening policy-makers’ ability to analyse policy-relevant research and assimilate the ideas it contains. Broadening policy horizons involves introducingnew ideas and options into policy debates, a notion that is similar to, but not quite the same as Weiss’s idea of research providing enlightenment or frameworks for discussion. Affecting decision regimes is also about capacity, like the first channel, but is more about the processes by which policies are debated and chosen and less about the people involved.
This is a very different way of thinking about policy influence. It goes “well beyond changing particular policies”, Carden argues that, “The most meaningful and lasting influence is less about specific policy change than about building capacity – among researchers and policy people – to produce and apply knowledge”. But in this more nuanced version, influence and impact emerge only in the long run. As Carden notes “This kind of influence can take years, or even decades, to take effect or become apparent. But it is no less important for that.”
So this leaves us with a much more complicated notion of policy influence. But the complications don’t end there. A more recent development is the emergence of more systematic and rigorous methods for evaluating the impact of policies. The best known of these methods involves Randomised Control Trials (RCTs), which have become much more common and influential over the past fifteen or twenty years. Impact evaluation – whether via RCTs or other methods – typically involves looking at existing policies rather than proposing new policies. So research can have a significant influence on the thinking of policymakers by providing evidence that an existing policy doesn’t work, rather than coming up with an idea for a brand new policy. In some ways this makes measuring research to policy linkages a little simpler, because impact evaluations are usually highly visible and often can be clearly linked to policy discourse.
All this gives us a more sensible framework for thinking how research may influence policy that goes well beyond whether a particular piece of research was responsible for a specific change in policy (be that a new law, a change in regulatory practice, or a decision to change a variable that is under the control of a policy maker).But at the same time these notions create a serious challenge for researchers and Think Tanks who need to demonstrate that their research has had an impact, or evaluators who have to assess a research programme or project.
Even in the apparently simple case of a policy change that involves a new law, it is almost never possible to “prove” that a particular piece of research brought about a change in policy. There are many factors that influence a policy decision, and research is only one factor - and in many cases research may be less important than other factors, such as the political environment. Why not just ask the politician or civil servant (who made the decision to change the policy)as to why they took the decision they did? But even this is far from straightforward - it may be far from obvious that an individual actually made the decision or indeed whether any single individual was responsible at all. So you may not know who to ask.
This means that you will be forced to rely on other sorts of evidence to measure impact. An analysis of the content of documents (white papers, green papers, speeches by politicians and civil servants) to detect the influence of a piece of research; or interviewing a broader range of people who were close to the decision making process, and trying to “triangulate” their answers. This begins to look more like the work of a good journalist or a detective. So even in the “simple” case of a discrete policy change, tracing this back to a research paper or project is a demanding task.
The challenge is even greater for the more complex forms of policy influence - for example the role of research in creating the framework for policy analysis and debate. How would you measure a change in a framework? That would require an even more elaborate document analysis and interviews with a much wider range of stakeholders.
The difficulties involved in measuring policy influence of research have been known and acknowledged for some time, but there have been relatively few systematic attempts to tackle the issue. The most ambitious example is probably the work by Carden, which examines the policy influence of 23 research projects funded by IDRC, using a common framework. But this has only scratched the surface of the problem, which is one of the most challenging and fascinating puzzles currently facing researchers – and those who evaluate them.
No comments:
Post a Comment