The search for the end of the rainbow – is impact assessment possible?

Juliet Pierce
March 2004

The growing importance of impact assessment

The world of international performance assessment has always aspired to monitor and evaluate development interventions along the following continuum:

Monitoring activity has tended to focus on the first part of this activity and evaluation on the latter.

The Working Group on Evaluation of the OECD’s Development Assistance Committee has set the standards for evaluation as an assessment of the relevance, effectiveness, efficiency, impact and sustainability of development interventions. Impact is defined as the impact of all changes, positive or negative, attributable to the intervention.

Now that the international community has for the first time agreed common Millennium Development Goals, the international public will be impatient to see progress. Measuring impact on poor people’s lives will be a crucial measure of the international development community’s effectiveness. As a result, interest in impact assessment in terms of impact on the MDGs, rather than an outcome assessment of a particular intervention has sharpened. However, examples of impact at that level that can be directly attributable to development interventions are proving elusive.

What is impact?

The word impact itself is a curiously brutal word. It brings up an image of a hammer hitting a surface. The dent produced by the hammer provides a visible representation of the hammer blow. The depth and shape of the dent is affected by the size and shape of the hammer’s head, the force with which the blow is struck and the malleability of the surface it hits. Where the dent is made is the result of a choice about where the hammer’s head should fall, and whether the blow is executed in an accurate fashion.

All of these factors can be applied to development interventions. However, the metaphor of the hammer and the surface is inaccurate in one major detail when applied to development contexts. It implies that the hammer is active and the surface is passive. In reality the context in which a development intervention works is never entirely passive. The result of the intervention and the longterm impact are always the result of interplay between the “hammer” and the “surface”.

Another commonly applied image is the one of the ripples caused by a pebble thrown into the pond. If the water is still, the ripples from the pebble’s entry into the water will continue to expand until they reach the edge of the pond. If the ripple meets turbulence or ripples from other pebbles being cast, then the force propelling the original ripples will be diverted or stopped.

We could represent this as a formula:

  • Originally perceived context +methodology = outcome

But

  • The effect of the development intervention + the interplay of the context with the intervention over time = impact (positive and negative).

In their assessments, proponents of the intervention tend to hunt for the positive effects, opponents may look for the negative, and of course evaluators are expected to be even-handed. To derive lessons that may be applicable in different contexts, evaluators need to unpack the context, methodology and the positive and negative impact.

It is in the context of understanding changes in the context and the effects of the development intervention on poor people’s lives that participatory monitoring is so useful. In an ideal scenario the intended beneficiaries are involved from the start, in the rationale for the intervention, the design of the development activity, the monitoring of progress and the assessment of longterm impact. This ideal scenario is rare in practice, except on a very small scale because of the time and financial costs. For poor people their time for these additional activities is usually extremely limited as the business of daily living is so arduous and time-consuming in itself.

Why is impact assessment so elusive?

The main reason that examples of impact are difficult to pin down is because the development context is complex, volatile and dynamic (an open system) and the intervention and assessment of the intervention is cross-cultural or even counter-cultural. In an increasingly globalised economy, spurred by rapid and penetrating communications systems, the interplay of the multiple factors affecting change on the ground are at least better recognised, if not fully understood. How can the impact of even relatively simple interventions be assessed in such a context? What can be properly attributed to the particular intervention? Who will be making this assessment? What vested interests are at stake in making the assessment? Over what time period will impact be judged?

When should we look for impact?

A further complicating factor is time. Immediate impact is closer to the outcome assessment. The height of the first ripple will be more that than those further way or occurring longer after the initial impact of the pebble on the water. We mostly try to measure impact early before the effect of the original ripples is lost, as applied in the concept of real-time evaluation developed by UNHCR for emergency situations. Real sustainable impact takes time to achieve. In different sectors, impact may take place on very different time scales, compare the timescales for the impact of new forestry practices with the impact of mobile phone technology.

Are there any examples of impact assessment?

Impact in international development can be measured fairly accurately when the intended intervention is precise, well managed and the monitoring and evaluation arrangements well planned. This implies a relatively scientific approach, with a proper assessment of the baseline context into which the development intervention is supposed to making an impact. In such conditions the intended impact has to be clearly articulated at the start and then carefully measured at the end. This can happen in relatively controlled circumstances such as drug trials and crop research. There is excellent positive impact assessment on the effect say of research into the application of the vaccine to prevent Rinderpest disease. Others have documented research, undertaken 60 years ago, which is now having wide-scale impact on cotton pest control. Participatory monitoring involving NGOs and community based organisations programmes such as water and sanitation programmes can be valuable in measuring the development of infrastructure, attitudes and behaviour and actual health impact if the baseline and monitoring systems are well-designed and managed. Finding and agreeing proxy indicators for success needs to be locally -owned from the start, to ensure interest in and commitment to measurement.

Other impact assessment exercises undertaken in less controlled circumstances will be similar to DAC-style ex-post evaluations which seek to clarify the positive and negative impacts of development interventions from multiple stakeholder perspectives. If sufficiently comprehensive to do justice in terms of the scale and complexity, these exercises are expensive and challenging to manage. This may deter funders especially where the findings may be unsatisfactory yet produced too late to affect any improvement in the original programme. It is often claimed that these exercises are not as valuable as interim assessment, since the lessons -emerge in a timescale that is too slow to influence new programme design.

Attribution

Does it matter whether one intervention can claim to have had a greater impact on poverty reduction in a particular context when any reduction is likely to be due to a combination of many different factors that are specific to a particular context? Most authorities agree that the problem of attribution may simply be too expensive to unravel conclusively. However there is pressure to try to prove attribution in order to account to funders for the use of their money in experimental circumstances.

Impact assessment on a larger scale than research trials

The simple truth is that we have yet to devise an affordable and reliable form of impact assessment that can work well in the broader context of international development. By coordinating the global development effort around the Millennium Development Goals, the international development community is moving its effort upstream and trying to harmonise its approach behind country level effort.

The World Bank and other lenders and funders are encouraging the use of Poverty Social Impact Analyses, as effective baseline studies to inform Poverty Reduction Strategies. It is not clear who will pay for these and whether a common instrument can be developed. The current inadequacy and inaccuracy of data and the lack of baseline studies to fill the data gap continues to be a major block to any attempt at impact assessment, especially in poor countries.

Current thinking is encouraging the strengthening of country-led approaches to strategy, and monitoring progress and impact through national indicators. This implies strengthening official systems for data collection so that the impact of new policy instruments and targeted funding on poverty indicators can be measured over time.

Official data needs to be backed by appropriate surveys conducted by national bureaux of statistics and supplemented from feedback from poor people themselves filtered through civil society organisations. Addressing this will require coordinated effort by governments (using data from ministries of planning and finance and service delivery departments and parliamentary questions), bureaux of statistics, the media, NGOs and academic communities within poor countries themselves, backed by internationals donors and lending bodies, UNDP, international NGOs and the global academic community.

Bringing this level of impact assessment effort together will be a big task, but a concerted effort in this direction may be more meaningful than the disparate effort involved in trying to prove more accurately the impact, rather than the outcomes, of many smaller scale activities.