Impact Assessment: Five lessons I have learnt

The first time I had to contend with impact assessment over 10 years ago, as the operations lead of an education services NGO, it was accompanied with a lot of apprehension and confusion. An independent, third party organization had been assigned by the donor to evaluate one of the programs we had been implementing for a year. The evaluation team would visit for three days, observe the program being delivered in schools, peruse documents, analyze any data we were able to provide on outcomes and then give the donor a report on the impact of our intervention on the children.

As soon as the stage of agreeing objectives has been completed, it is crucial to get field teams to internalize these as well. The staff who are in touch with the primary stakeholders and delivering the program interventions should always be guided by these objectives.

None of us knew what to expect or how to prepare or what data to share. But it was this experience that sparked my interest in the area of impact evaluations and in exploring how practitioners and implementing organizations can integrate this process into their operations and derive the maximum value from it. As I have learnt and grown, my understanding both from the perspective of the evaluated and the evaluator, I share five key lessons that I have learnt.

Lesson 1: Think outcomes and measurement of impact from the design stage of the program

In the broadest sense, impact assessment intends to answer the question “Did the project meet its objective?” And the first step towards answering this question is to define the objective of the project as clearly as possible. The objective may also be thought of as the problem that one is trying to solve.

One of the challenges that I encountered repeatedly is that as well-intentioned practitioners who care about making a difference, we want to do too much in one program. I cannot stress how important it is to brainstorm with project teams and arrive at one or two objectives that are central to the program and put them into words clearly and concisely.

Once the objectives have been defined, we then need to nail down the project activities that help achieve these objectives and dedicate our time and attention to ensuring that these activities are delivered as planned.

A third important step in preparing for impact measurement is to articulate ‘who’ the program is intended for and make sure that the activities in the program are directed towards this group. Thinking about and taking these three aspects into account from the design stage not only prepares the organization for an evaluation, but also ensures that the goal is always in sight during implementation.

Lesson 2: Get your field teams oriented towards impact assessment in the early stages of implementation

As soon as the stage of collectively defining objectives has been completed, it is crucial to get field teams to internalize these as well. The staff who are in touch with the primary stakeholders, and delivering the program interventions, should always be guided by these objectives. Having a team at the core of the implementation who know what the expected outcomes of the project are, is invaluable both for the project’s success and for the stakeholder experience. This is also a much more sustainable way of building impact thinking into the organization as a whole. A clued-in field team can provide timely information about the aspects of the program that are working well and those that don’t seem to be.

Lesson 3: Monitor, measure and make changes

While impact assessment is often seen as an annual or end of program term event, what supports a good evaluation process is regular periodic monitoring of the implementation. Evaluation is typically concerned with the outcomes, but as practitioners we are well aware that if processes are not functioning optimally during all phases of implementation, the expected outcomes will not be achieved. In order to understand if the program is causing the stakeholders to move towards the intended objectives, periodic measurement of the primary outcome of interest is critical.

Program staff must have short term milestones that provide insights on the effectiveness of the intervention. Measurement is a critical part of impact assessment and I cannot stress enough how important it is to design the right tools to measure the specific outcomes of each project. Finally, measurement is only useful if we are learning from it, analyzing the data we have collected, and making course corrections to ensure that we are moving towards the intended outcomes.

Lesson 4: Be open to programs not working as intended

When I was a staff member of an implementing organization, I often felt the need to prove to the external evaluators that the program is indeed achieving its objectives. Given that decisions of continuing funding, expanding or replicating a program are made on the basis of the impact the program is able to demonstrate, it is understandable that there is a great deal of pressure on practitioners during an evaluation.

While impact assessment is often seen as an annual or end of program term event, what supports a good evaluation process is regular periodic monitoring of the implementation.

Time and again, there is also a tendency to highlight isolated success stories from the program, while not paying attention to how the program has impacted the entire group of stakeholders. However, a good partnership with evaluators can assist in streamlining project activities, in identifying the areas where resources need to be allocated, and in sharpening the focus of the program to support the achievement of its objectives.

Lesson 5: The ‘Why’ is as important as the ‘How much’ in estimating impact

Increasingly in recent times, evaluations focus only on measuring whether or not the project had the intended impact on its stakeholders and quantifying the extent of change the program has been able to produce. The criticality of this data in an impact assessment cannot be denied. But answers to the ‘why’ questions could help understand if the program can be chosen for a scale up, and what contextual factors have contributed or hindered the improvement of the outcome of interest.

Considering that impact assessments aim to establish causal relationships between intervention and the outcome of interest, we need to be aware of and record the other influences on the stakeholders of the project. While a strong quantitative measurement determines the extent to which the program has impacted the outcomes, qualitative data collected from stakeholders complete the puzzle by showing us the mechanisms that are operating to achieve this impact and also to get a sense of the unintended consequences of the program that we may not be measuring.

As I tried to consolidate my learnings from a decade long tryst with impact assessment, I became more convinced of its value for implementing organizations. With organizations wanting to solve problems with minimal resources at our disposal, impact orientation has the potential to streamline activities, sharpen the focus of our programs and help us work towards making the difference we wish to see in the world.

Considering that impact assessments aim to establish causal relationships between intervention and the outcome of interest, we need to be aware of and record the other influences on the stakeholders of the project.