Evaluation Practices: A Continuing Evolution

Social interventions are designed to change existing social conditions. It is therefore natural that those involved would want to calibrate inputs, outputs and outcomes. For example, the current policy discourse focuses on measuring teacher engagement (as inputs) and children’s learning (as outcomes). The underlying assumption is that the two are linked causally. And yet interventions are sometimes designed and implemented without being attentive to how and under what conditions teacher engagement actually changes children’s learning. In this context, evaluation can be understood as a set of practices examining how a social intervention unfolds and placing ‘values’ on different aspects of the intervention. This essay explores some of the assumptions underlying social interventions and surveys the important challenges to evaluation practices in Indian contexts and across the globe.

Evaluating design, implementation, and social consequences of a program

In its full scope, evaluation encompasses three interrelated aspects of a social intervention or program: its design, its implementation, and its consequences for society. Below, we briefly discuss each of these aspects.

The evaluation of design examines the justifications for intervening (‘needs assessment’) and the specific form of the program (‘program choice’) as well as the expected processes through which the program was meant to create changes in society (‘theory of change’). Unfortunately, many real-world evaluations do not systematically attend to program design. Programs are often off-the-shelf and guided by designers’ skillsets, untested beliefs and (implicit) ideological orientations, and political compulsions of donors and others – and yet evaluations often ignore these matters. Instead, arguably, evaluations ought to connect program design with contextual relevance and evidence-based expectations of how and why the specifics of the program are likely to create positive changes in society.

In its full scope, evaluation encompasses three interrelated aspects of a social intervention or program: its design, its implementation, and its consequences for society.

Although evaluations often do not examine program design in depth, they typically do attend to program implementation. The evaluation of implementation has some commonalities with monitoring but in principle there is a clear distinction between them. Monitoring is in-house, continuous, focused on some key activity parameters linked to individual staff accountability, and identifies ‘implementation gaps’ so that program design can be tweaked in an effort to fix them.

By contrast, evaluation of implementation is done independently of implementers and is meant to understand more deeply why implementation gaps exist. However, in reality the two are often not as distinct as the literature on ‘best practices’ would demand. In fact, often implementation evaluators only perform secondary analysis of monitoring data (which are increasingly routinized and digitalized through MIS, especially for larger programs).

This limits evaluation to implementation parameters that were set for a different purpose (in-house monitoring) and data forms that were created for a different purpose and limited by in-house capacities (simple quantitative indicators that can be gauged on a continuous basis). Importantly, this cannot get at the crucial ‘why’ questions around inevitable implementation gaps, which call for independent, qualitative and interpretive methods of inquiry.

At best, mainstream impact evaluation approaches merely take recourse to the theory of change to explain the ‘how’…

Another unfortunate consequence of implementation evaluation piggybacking on monitoring is that the quality of program activities stands relatively ignored. For instance, it is far easier to collect information on the number of community members who attend a gram sabha than to collect information on the quality of engagement in the gram sabha, although often the latter is crucial for program success.

Similarly, typically we collect information about the number of teachers who attend a professional development program but we may ignore information on the quality of the training or its ability to nudge teachers to make changes in classroom practices.

The third aspect of evaluation, namely understanding social consequences of interventions, has gained prominence in recent years through the consolidation of statistical techniques to estimate ‘causal’ relationships between program outputs and social outcomes (‘impact evaluation’). Its importance lies in the clear focus accorded to the question “How effective is the program?” and its draw lies in the adoption of techniques that purportedly deliver clear answers to this question.

However, the method-centric nature of mainstream impact evaluation limits the focus to measuring impact rather than reflecting on what ‘being effective’ means. For instance, a program to encourage reading in schools may naturally measure consequent changes in language skills, but it is important to not close off inquiry into other consequences, especially longer term and relatively intangible ones. The reading program may influence students’ aspirations, their understanding of social relations, and their sense of personhood and identity.

Openness to interpreting impact becomes key in such situations. Further, by focusing on the question “How effective is the program?”, mainstream impact evaluation downplays the question “How is the program effective?”. The absent ‘how’ is particularly odd for an approach centred on causality. Mixed methods techniques such as ‘process tracing’ can address this by taking careful account of context and mechanisms (‘realist evaluation’) and the complex ways through which program activities may generate outcomes or fail to do so.

At best, mainstream impact evaluation approaches merely take recourse to the theory of change to explain the ‘how’, rather than problematize it by independently inquiring into the processes of empirical change. Indeed, it would be useful to compare the two, and to even superimpose an account of actual empirical processes over the corresponding theory of change. Such exercises could also improve decisions regarding program continuance and/or scaling up or down.

Orientations to evaluation: method, use, value

The above discussion takes us to the consideration of orientations to evaluation. It is useful to distinguish between broad orientations towards method, use and value. The first (‘method’) emphasizes research methodology, for instance experimental or quasi-experimental approaches to impact evaluation using statistical techniques.

The second orientation (‘use’) emphasizes how evaluation findings can loop back to making the program more effective. This often centres on actionable learnings from implementation evaluation, since in real world policymaking, decisions regarding continuance and/or scaling programs up or down are seldom based primarily on evaluation findings.

The third orientation (‘value’) emphasizes values in the process of evaluation itself.

Values such as improved gender relations, environmental sensitivity or health equity can be built into evaluation even if the program being evaluated is not directly focused on gender, environment or health issues. Matters of culture, indigeneity and power relations can also be given special value in the process of evaluation.

Further, evaluators may focus on creating relationships of trust and reciprocity with program personnel and local populations, thereby acknowledging and proactively addressing (even if only partially) their relatively privileged positions and countering asymmetric accountability relations (discussed below).

An overarching value is participation in evaluation. The need for participatory approaches that are effective in understanding design, implementation and impact, and are attentive to contextual specificities, has become increasingly evident in India and across the globe. Much of participatory evaluation derives from principles and methods of Participatory Rural Appraisal (PRA) and related techniques such as Participatory Learning and Action (PLA).

In the case of implementation and impact, participatory approaches can address problems of existing approaches through cogeneration of output and impact indicators with relevant communities (including indicators of quality that are often otherwise missing), strengthening reliability of baseline information, and deepening emerging understanding of both program procedures (that is, activities connecting inputs with outputs) and impact processes (that is, translation of program outputs to social outcomes).

However, typically participatory evaluation in India has focused on data collection and triangulation and seldom on seriously involving communities in program design or analysis, either of implementation or impact. Just as importantly, in practice, participatory approaches typically do not include the feedback-loop stage, which is very much in the hands of donors, program designers and policymakers.

Despite all this, it is heartening that in recent years the jan sunwai has been used innovatively for participatory implementation, monitoring and evaluation, although it still falls short of implementation analysis. We emphasize the scope for expanding participation in different aspects of evaluation. Explicit procedures can generate impact indicators identified by intended program beneficiaries. With expanding mobile phone usage, participants can use digital photography to articulate and explain the changes they have experienced from the intervention, and also expected changes that they believe have not been met.

It is important to note the demographic details based on which intervention experiences and responses could vary (for instance, gender, caste, class), and to account for them and to incorporate them into the evaluation plan. At the same time, participatory evaluation should be attentive to ‘structural violence’ in contexts with historical forms of embedded inequalities that have been reconfigured to take new forms today.

Evaluation should not ignore the implications of these inequalities and their changing nature, or how diverse institutional forms – including that of state and non-state actors – interact with these inequalities. Participatory approaches can also influence and democratize accountability relations around a program.

Sharing, dissemination and discussion of findings with communities could help to disrupt current chains of accountability routed back mainly to policymakers and donors. While development organizations are aware of this, the mind-sets of donors, consultants and evaluation agencies – who typically feel that accountability to donor investment supersedes community voices and accountability to communities – continue to dominate.

There is no gainsaying that the ecosystem of evaluation is driven by requirements of donor agencies hewing to the (quantitative) ‘methods orientation’ and to procrustean design features such as mechanical use of the logical framework (‘logframe’) template. Therefore, it is not surprising that evaluation findings are often used to justify ‘effectiveness’ of programs without a critical inquiry into what worked and how, what did not work and why, and negative cases.

For field-level CSOs, in practice, evaluation is often limited to being another organizational procedure to fulfil or another hoop to jump through, rather than offering an opportunity to critically examine objectives, strategies, and program choices vis-à-vis objectives.

There are two other issues that also deserve attention. One, the often-absent final stage of the evaluation process where findings are shared with and validated by communities. Two, the need to share findings at different levels of the program hierarchy, in diverse ways that convey the findings in an appropriate manner to stakeholders at different levels.

That is, there could be more innovative efforts to present evaluation findings in accessible and useful ways to communities, program beneficiaries, field staff, regional offices, supra-regional offices, and donors. Such a balancing across the three orientations – method, use and value – is a difficult but worthy aspiration for all evaluators.