Three Questions that Guide How to Assess Your Policy Impact
Julia Coffman
Center for Evaluation Innovation
For Think Tanks and others acting to influence public policy, assessing one’s impact in a complex and crowded political environment can be challenging. When trying to understand if one’s policy work is making a difference, we recommend answering three key questions.
Question 1: Who are your audiences?
Influencing efforts ultimately is about communicating effectively to individuals or groups so that they learn, think about, or do something differently regarding a policy issue. While the research and content that Think Tanks deliver is a critical element of the influence process, “so is engaging with journalists, advocates, policymakers, and others who might interpret and use the findings”, according to Ruth Levine.
Audiences are the groups and individuals that Think Tanks or other influencers target and attempt to affect or persuade. They represent the main actors in the policy process and fall into three categories:
Within these categories, it is important to identify specifically who is being targeted. “Those who aspire to inform decision making with research and evidence should avoid talking generically about ‘policy makers’”, for example. While there are three broad categories of audiences that might be engaged, who is being targeted within those categories?
Think Tanks with theories of change and related communication strategies will already have clear answers to the audience question.
Question 2: How do you want audiences to change?
Changes are the outcomes in audiences that Think Tanks and other influencers aim for in order to progress toward a policy goal. There are three broad categories of changes, and they fall along a continuum based on how much an audience is expected to engage on a policy issue in order to achieve the influence that Think Tanks are after.
The continuum starts with basic awareness or knowledge. Here the goal is to make the audience aware that a problem or potential policy solution exists. The next point is will. The goal here is to raise an audience’s willingness to take action on an issue. It goes beyond awareness and tries to convince the audience that the issue is important enough to warrant action, and that any actions taken will in fact make a difference. The third point is action. Here, influence efforts actually support or facilitate audience action on an issue. Again, influence strategies may pursue one change with an audience or more than one simultaneously.
Keep in mind that in order for policy change to occur, somebody ultimately needs to do something differently than they are doing right now. Influence efforts need to move somebody toward action. Decades of research have shown that just making people more aware of an issue or problem generally is not enough to mobilise them to act. Education by itself is not equivalent to motivation, and new knowledge does not automatically result in attitude or behaviour change.
Question 3: How will you capture audience changes?
Once the first two questions have been answered, assessments of policy impact can focus on measuring whether changes in the identified audience have occurred. Definitions of each outcome and the measures that might indicate whether those changes have occurred can help to operationalise audience change into measurable indicators.
Capturing audience changes can involve a familiar list of traditional data collection methods, such as surveys, interviews, focus groups, or polling. But because the influence process in a political environment can be complex, and audience outcomes can be hard to measure (e.g., public will or political will), innovative methods have been developed specifically for assessing policy influence efforts. Think Tanks may find bellwether interviews, champion tracking, and policymaker ratings particularly useful.
Finally, isolating an organisation’s impact is difficult in a complex policy context that involves multiple actors. To increase the chance that any audience changes detected can be plausibly linked back to Think Tank efforts, it is critical to be as specific and precise as possible when answering the first two questions and to be sure that they link closely to the communications strategies that Think Tanks are using.
Julia Coffman
Center for Evaluation Innovation
For Think Tanks and others acting to influence public policy, assessing one’s impact in a complex and crowded political environment can be challenging. When trying to understand if one’s policy work is making a difference, we recommend answering three key questions.
Question 1: Who are your audiences?
Influencing efforts ultimately is about communicating effectively to individuals or groups so that they learn, think about, or do something differently regarding a policy issue. While the research and content that Think Tanks deliver is a critical element of the influence process, “so is engaging with journalists, advocates, policymakers, and others who might interpret and use the findings”, according to Ruth Levine.
Audiences are the groups and individuals that Think Tanks or other influencers target and attempt to affect or persuade. They represent the main actors in the policy process and fall into three categories:
- The public (or specific segments of it)
- Policy influencers (e.g., media, community leaders, the business community, thought leaders, political advisors, other advocacy organizations, etc.)
- Decision makers (e.g., elected officials, administrators, judges, etc.).
Within these categories, it is important to identify specifically who is being targeted. “Those who aspire to inform decision making with research and evidence should avoid talking generically about ‘policy makers’”, for example. While there are three broad categories of audiences that might be engaged, who is being targeted within those categories?
Think Tanks with theories of change and related communication strategies will already have clear answers to the audience question.
Example Audiences
Question 2: How do you want audiences to change?
Changes are the outcomes in audiences that Think Tanks and other influencers aim for in order to progress toward a policy goal. There are three broad categories of changes, and they fall along a continuum based on how much an audience is expected to engage on a policy issue in order to achieve the influence that Think Tanks are after.
The continuum starts with basic awareness or knowledge. Here the goal is to make the audience aware that a problem or potential policy solution exists. The next point is will. The goal here is to raise an audience’s willingness to take action on an issue. It goes beyond awareness and tries to convince the audience that the issue is important enough to warrant action, and that any actions taken will in fact make a difference. The third point is action. Here, influence efforts actually support or facilitate audience action on an issue. Again, influence strategies may pursue one change with an audience or more than one simultaneously.
Example Audience Changes
Question 3: How will you capture audience changes?
Once the first two questions have been answered, assessments of policy impact can focus on measuring whether changes in the identified audience have occurred. Definitions of each outcome and the measures that might indicate whether those changes have occurred can help to operationalise audience change into measurable indicators.
Capturing audience changes can involve a familiar list of traditional data collection methods, such as surveys, interviews, focus groups, or polling. But because the influence process in a political environment can be complex, and audience outcomes can be hard to measure (e.g., public will or political will), innovative methods have been developed specifically for assessing policy influence efforts. Think Tanks may find bellwether interviews, champion tracking, and policymaker ratings particularly useful.
Finally, isolating an organisation’s impact is difficult in a complex policy context that involves multiple actors. To increase the chance that any audience changes detected can be plausibly linked back to Think Tank efforts, it is critical to be as specific and precise as possible when answering the first two questions and to be sure that they link closely to the communications strategies that Think Tanks are using.
No comments:
Post a Comment