Approaches and Challenges to Assess Policy Impact
Andriy Andrusevych
Senior Policy Expert
Resource & Analysis Center Society and Environment
Background
Our experience in using various methods for assessing policy impact is limited to environmental policy, our Center’s major operating field.All methods used in our Center were applied within the so called M&E&L or MEL (monitoring, evaluation and learning) framework.
One of the key issues to think about when starting a policy impact assessment scheme is to set a threshold between assessing policy impact, on the one hand, and effectiveness of the organisation activities (or its projects), on the other hand.
MEL as a Pilot
We took MEL as a comprehensive system for monitoring and evaluating the impact of a specific project (related to monitoring green growth in Ukraine). This significantly simplified the application of this instrument in the organisation. The key elements of MEL included: development of indicators for expected outcomes, sources and data gathering, periodic analysis and discussion (learning). You can read more about challenges we met here.
Yet, there are some specific experiences we see worth describing in more details here.
Mechanical (or statistical) indicators are quite popular these days among analysts (e.g. Facebook likes, or web-page statistics). Yet, they have limited capacity for assessing Think Tank impact. Our target audience often is narrow (sometimes – ultra-narrow). For this reason, the fact that our policy paper was downloaded by 10,000 users may not indicate that our target persons are familiar with it. In this context, we see the need to substitute mechanical indicators, no matter how attractive diagrams and other visualisations may seem to your planning/supervisory team. For instance, we consider brief interviews with representatives of the target groups to be an effective alternative to statistical indicators for measuring specific outcomes (impact). Passive involvement of the target group may also be useful.
We found it difficult, if possible at all, to “lay” MEL like an X-ray scanner on the existing organisation planning system, especially strategic planning. MEL requires full integration into planning on the level which you aim to assess. If you wish to assess policy impact of your work – MEL needs to be integrated into strategic planning of an organisation.
We also learnt that choice of indicators to measure impact requires in-depth analysis of the context of policy.. This also applies to an initial snap-assessment of the relevant policy area using specific indicators, and to the peculiarities of a policy process. In particular, your partners, competitors or target groups may deliberately hide your impact on the policy sector (for example, by using your ideas or following specific recommendations without referring to you as a source). There could also be reasons where you may not be willing to visualise your impact; this would inevitably lead to choosing special assessment methods, in particular to avoid subjective factor when using insider information (data).
Another example from our experience: resources and pragmatism. Policy impact assessment process will require involvement of the majority of staff (involved in relevant policy work). In turn, this will require significant resources. The conclusion is clear: when designing MEL - a system for assessing your impact or effectiveness of your activities – you may want to be extremely pragmatic, if not conservative by selecting only key objects and methods for assessment. Too much of a good thing can devastate your efforts.
Challenges
Based on our experience we are now designing MEL system for the whole organisation. Form the start, assessment of policy impact of the organisation (even in specific policy area) poses some challenges. In particular, how to separate your impact from the impact of other players? How to separate assessment of policy impact from activities assessment? Regarding the latter, it is much easier to design indicators for a project or a programme, and select relevant assessment methods.
Key Considerations to Design Policy Impact Assessment System
Selection of indicators is a key element when designing MEL. We believe an effective policy impact assessment system should involve a small number of indicators, which are rooted in specific initiatives (projects). The indicators should be “tangible” and avoid generalisation, in this way they will serve as an important motivating factor for the personnel, in addition to serving policy impact assessment purpose. Lastly, while indicators should be stable, they need to be able to reflect changes in the policy environment.
We anticipate a subjective element in the assessment system, and that’s good. Subjective element is a consequence of choosing indicators, which are not mechanical (statistical).
Assessing an organisation’s policy impact requires establishing methods for measuring your own input. Without framing your own contribution it would be hard to measure a policy impact. Framing your contribution may be done through assessing separate segments: initiatives and projects implemented by the organisation. For this reason, all indicators for policy impact system should be rooted in specific project and initiatives. This would also make data gathering easier and effective.
We believe that it is necessary to clearly define the objectives for an assessment in the context of strategic planning of an organisation. It could be one of those “either-or” decisions at the very beginning. Traditionally, you may start from setting clear expected outcomes of your work in a specific policy area. Yet, the policy impact assessment process may be tuned to one or more policy process elements: key player, policy process and policy substance (subject). This could enable designing a MEL system which is not organization-centric (i.e. based on its vision, structure, priorities and tasks), but also fully reflects the structure of the relevant policy area (process). For example, at some point we might be more interested what impact we make on specific policy-makers group framing birds protection policy in Ukraine (it could be a parliamentary committee), rather than the policy substance itself or policy-making process. In contrast, tor some think-tanks, transparency of the policy-making process could be a priority.
Andriy Andrusevych
Senior Policy Expert
Resource & Analysis Center Society and Environment
Background
Our experience in using various methods for assessing policy impact is limited to environmental policy, our Center’s major operating field.All methods used in our Center were applied within the so called M&E&L or MEL (monitoring, evaluation and learning) framework.
One of the key issues to think about when starting a policy impact assessment scheme is to set a threshold between assessing policy impact, on the one hand, and effectiveness of the organisation activities (or its projects), on the other hand.
MEL as a Pilot
We took MEL as a comprehensive system for monitoring and evaluating the impact of a specific project (related to monitoring green growth in Ukraine). This significantly simplified the application of this instrument in the organisation. The key elements of MEL included: development of indicators for expected outcomes, sources and data gathering, periodic analysis and discussion (learning). You can read more about challenges we met here.
Yet, there are some specific experiences we see worth describing in more details here.
Mechanical (or statistical) indicators are quite popular these days among analysts (e.g. Facebook likes, or web-page statistics). Yet, they have limited capacity for assessing Think Tank impact. Our target audience often is narrow (sometimes – ultra-narrow). For this reason, the fact that our policy paper was downloaded by 10,000 users may not indicate that our target persons are familiar with it. In this context, we see the need to substitute mechanical indicators, no matter how attractive diagrams and other visualisations may seem to your planning/supervisory team. For instance, we consider brief interviews with representatives of the target groups to be an effective alternative to statistical indicators for measuring specific outcomes (impact). Passive involvement of the target group may also be useful.
We found it difficult, if possible at all, to “lay” MEL like an X-ray scanner on the existing organisation planning system, especially strategic planning. MEL requires full integration into planning on the level which you aim to assess. If you wish to assess policy impact of your work – MEL needs to be integrated into strategic planning of an organisation.
We also learnt that choice of indicators to measure impact requires in-depth analysis of the context of policy.. This also applies to an initial snap-assessment of the relevant policy area using specific indicators, and to the peculiarities of a policy process. In particular, your partners, competitors or target groups may deliberately hide your impact on the policy sector (for example, by using your ideas or following specific recommendations without referring to you as a source). There could also be reasons where you may not be willing to visualise your impact; this would inevitably lead to choosing special assessment methods, in particular to avoid subjective factor when using insider information (data).
Another example from our experience: resources and pragmatism. Policy impact assessment process will require involvement of the majority of staff (involved in relevant policy work). In turn, this will require significant resources. The conclusion is clear: when designing MEL - a system for assessing your impact or effectiveness of your activities – you may want to be extremely pragmatic, if not conservative by selecting only key objects and methods for assessment. Too much of a good thing can devastate your efforts.
Challenges
Based on our experience we are now designing MEL system for the whole organisation. Form the start, assessment of policy impact of the organisation (even in specific policy area) poses some challenges. In particular, how to separate your impact from the impact of other players? How to separate assessment of policy impact from activities assessment? Regarding the latter, it is much easier to design indicators for a project or a programme, and select relevant assessment methods.
Key Considerations to Design Policy Impact Assessment System
Selection of indicators is a key element when designing MEL. We believe an effective policy impact assessment system should involve a small number of indicators, which are rooted in specific initiatives (projects). The indicators should be “tangible” and avoid generalisation, in this way they will serve as an important motivating factor for the personnel, in addition to serving policy impact assessment purpose. Lastly, while indicators should be stable, they need to be able to reflect changes in the policy environment.
We anticipate a subjective element in the assessment system, and that’s good. Subjective element is a consequence of choosing indicators, which are not mechanical (statistical).
Assessing an organisation’s policy impact requires establishing methods for measuring your own input. Without framing your own contribution it would be hard to measure a policy impact. Framing your contribution may be done through assessing separate segments: initiatives and projects implemented by the organisation. For this reason, all indicators for policy impact system should be rooted in specific project and initiatives. This would also make data gathering easier and effective.
We believe that it is necessary to clearly define the objectives for an assessment in the context of strategic planning of an organisation. It could be one of those “either-or” decisions at the very beginning. Traditionally, you may start from setting clear expected outcomes of your work in a specific policy area. Yet, the policy impact assessment process may be tuned to one or more policy process elements: key player, policy process and policy substance (subject). This could enable designing a MEL system which is not organization-centric (i.e. based on its vision, structure, priorities and tasks), but also fully reflects the structure of the relevant policy area (process). For example, at some point we might be more interested what impact we make on specific policy-makers group framing birds protection policy in Ukraine (it could be a parliamentary committee), rather than the policy substance itself or policy-making process. In contrast, tor some think-tanks, transparency of the policy-making process could be a priority.
No comments:
Post a Comment