“Uniformity of Design, Comparability and Economies of Scale” – Felt Need for Standardized Templates in Impact Assessment

Samuhik Pahal: Initially it might help our readers if you could please share how CSF is addressing the issue of improving learning outcomes through its programs, especially in the context of the learning crisis in India that has now been aggravated by the COVID-19 pandemic.

Bikkrama Daulet Singh: Central Square Foundation has been working on improving quality of learning for ten years now. We have iterated a lot on our approach, based on learnings, feedbacks and mistakes. Nearly four years ago we sharpened our work towards improving foundational learning. For the first 5-6 years, we were making many grants; we were doing a lot of ecosystem building and communications work; we were working with governments. We were doing all this around thematic areas. We had taken up assessment reforms, school leadership, early childhood education, and many other pillars.

Looking at systemic data, we saw that learning levels were falling. One interesting metric we looked at is that of ‘Learning Poverty’, which the World Bank came up with, in 2019, based on government data for many countries. They estimated learning poverty in India at 55%. Out of hundred children in India 55 at the age of 10 could not read a basic text with comprehension, and this is even before the COVID-19 pandemic hit. The learning crisis in India is stark. How does it matter? Looking at longitudinal data, talking to experts in pedagogy and teachers, it is very clear that if children do not acquire critical, foundational learning skills early enough, their learning trajectory just flattens.

We went back to the drawing board and asked ourselves about the nature of this learning crisis. When we started to look at much of the data, a clear insight was that the learning crisis starts early. Therefore, we sharpened all our work by saying, “We will focus everything on foundational learning skills.”

How we have done that is by doing 3-4 things. One, we think we need to build more awareness around the issue of foundational learning. We need to show this as an opportunity for policy makers. We want to focus on learning outcomes and not in a vague way; so, that involves choosing a metric, say ‘Learning Poverty,’ and making it a key priority.

Subsequent to this, the central government launched a national mission, Nipun Bharat, for achieving foundational literacy and numeracy. If there is no model for showing that things can improve, then it is just another program, just another scheme. Therefore, we are working now across twelve states where we have set up PMUs.

As a consortium, we sign an MoU with the state government to implement Nipun Bharat in a robust way. We bring in a project management partner. We then bring in pedagogical organizations who know the science of language or math learning, who have demonstrated programs at reasonable scale. They join as technical support organizations. We bring the consortium together, provide coordination, and undertake stakeholder management with the government. Over the next 3-4 years we hope to show some improvements.

Looking at longitudinal data, talking to experts in pedagogy and teachers, it is very clear that if children do not acquire critical, foundational learning skills early enough, their learning trajectory just flattens.

The other thing we are looking at is Ed-Tech that can provide evidence-based solutions to improve children’s foundational learning. ASER has been tracking digital adoption in rural India. Data shows that two-thirds of parents have a dedicated smart phone for learning. COVID has been a huge fillip.

We have seeded certain Ed-Techs. We have also provided funding support to a few EdTechs and are working with them in a system oriented way. These organizations do not target ‘India-1’, but Bharat, i.e., India-2 and India-3, which for example, use low-tech, WhatsApp-based models.

Samuhik Pahal: How do you see the evolution of CSF’s impact assessment strategy over the last decade?

Bikkrama Daulet Singh: I will look at it as impact assessment of CSF’s work. Impact assessment is such an important aspect of the work of the nonprofit ecosystem. There has not been enough awareness and investment around impact assessment in this space. If you look at today’s CSR spend, there are estimates that there is nine to ten thousand crores of CSR spend and forty percent of that goes into education.

However, if you look at the evidence base of school education reforms in India, the only rigorous evidence that we have is the Teaching at the Right Level (TaRL) Program from PRATHAM that was evaluated by J-PAL (Abdul Latif Jameel Poverty Action Lab). Of course, there are a few others. There is US-AID, a multilateral funder, that funded a series of organizations including Room to Read, CARE and others, where they also tracked some evaluations. However, if you see the evidence base available today, it is not very robust.

What many funders require is just some pre and post, some kind of a report. The impact assessment landscape is very sparse. When you look at CSF’s own journey, for the initial 4-5 years, we were not running programs, but were supporting partners. We used to do a lot of grant making to early stage non-profits. What we found was that the questions around impact assessment were very much a zero-sum game. You are doing no impact assessment. Alternatively, you are doing a very rigorous RCT. People would have aspirations of doing these. Actually, they are not doing anything.

When we look at our own work, over time we also built capacity in M&E, brought in experts and built the function within CSF. The way we evolved was that we started locating monitoring and impact assessment as a continuum, as a journey.

More important than even getting third party impact assessment and evaluation is to start looking at the data of your own programs, of your partners’ programs, with a very close lens.

One needs to systematically think what the theory of change of this program is, what are some of the key indicators that you should be tracking, what is the review mechanism of tracking, how do we track that data, and start to look at it. Moreover, you have to use that data to have conversations with your own program teams, with your partners, and use it to fine-tune the intervention model.

Many funders do not want to fund evaluations. They want to fund programs.

When you feel that the theory of change, the intervention, seems to be working well, then one can go on and do an impact assessment. Many organizations in the ecosystem do not do the monitoring part of the process very systematically, and then jump to do impact assessment.

Because everyone ultimately wants a pat in the back, that yes, this is working or not working. I think it is not being fair to your partners and your own programs if you are not using data to iterate and build that sort of muscle. That is the first point I am making.

The other point is that, when you come to impact assessment, there are many challenges in India. There is a dearth of organizations that can work with you to undertake the assessment. There are many data collection organizations. Then there are some organizations that do evaluations.

However, in my opinion, they are over-priced. Many of them focus on multilateral programs. For Indian foundations, for non-profits, there are very few partners to go to who can put a strong research lens on their work and design an evaluation with a reasonable level of rigor. Not every organization can go to a J-PAL and do an RCT (Randomized Controlled Trial).

I really feel that there is a gap in the ecosystem. There are many evaluations happening, especially of the CSR programs. These are often of a very low quality and have poor research design. Then we have a few RCTs done by known researchers. What we need then is to lower the barriers for organizations to access and undertake impact assessment.

We have been increasingly trying to enable comparisons between our different impact assessments. Because we started doing so many things around foundational learning, we said, “Look. Why do we not invest in a common assessment tool? Ideally, such a tool should lend itself for use even if the sites differ. Even if one of them has an offline program model, whereas another is more of an Ed-Tech model, let us try to assess all of these through a common assessment tool.”

We customized a version of EGRA-EGMA (Early Grade Math Assessment-Early Grade Reading Assessment). This is an assessment tool available as public goods and used across a large number of countries. So we adapted it and linked it to the NCERT curriculum and our context. We used that across our portfolio.

However, we still struggle with many issues. Let us take one example. We have Ed-Tech partners. All of them have their journeys around evidence building. For each of these we have found different research partners. Each of these researchers will have a different research design.

If we could create a research lab… We have been conceptualizing this. Many of these products could go through that funnel. In addition, there would be a research advisory group. It would lower barriers and costs for undertaking evaluations.

Because in many cases if the evaluation cost is such a significant investment, many organizations struggle to raise money. Many funders do not want to fund evaluations. They want to fund programs. The second goal is to ensure uniformity of design, comparability and economies of scale. These are a few of the things we have been thinking across.

Samuhik Pahal: What are the values do you think we need to focus on with respect to impact assessment?

Bikkrama Daulet Singh: A lot of the impact assessment space is very compliance driven. Many programs need impact assessment for a donor, or for a CSR partner. Therefore, they just go in and do that. They design a custom evaluation for that partner and undertake it. The challenge is that, it is not comparable. It is not relevant from a systems perspective.

When I talk about monitoring, I am speaking of systemically monitoring. In program monitoring, one has to identify key metric, look at that data, analyze it, reflect on it. This has to become an important priority…

We have been working on foundational learning in India. We have looked at the entire evidence base in this field. I can count on my fingers the number of actual reports of quality and rigor that are available that you can show to a policy maker and make a case, that this problem can be solved. If everyone started to invest in evidence of a certain caliber, then we will build the field. Here some norms need to be established.

We may have to work on some kind of an ecosystem platform where we can invest in public goods like common assessment tools, some sample designs that people can take off the shelves. Otherwise, there is so much custom work. It is not very rigorous and it is not very useful. Those are the kind of things we need to push for.

Samuhik Pahal: How can impact assessment contribute to learning processes of organizations across the ecosystem?

Bikkrama Daulet Singh: This is a cultural thing. How does one look at data and use it as a tool for improvement and feedback? When you work with governments and different organizations, you realize that there is not this culture of drawing insights from the program data that you already have. Forget even colleting new data.

Here I think the point I made earlier on really looking at monitoring as a key focus… You will often see that one does not get any feedback loop from programs, till the final end of the program or some impact assessment report.

When I talk about monitoring, I am speaking of systemically monitoring. In program monitoring, one has to identify key metric, look at that data, analyze it, reflect on it.

This has to become an important priority, if you want impact assessments or evaluations to feed back into programs. Moreover, we must create some public datasets. One can look at housing these in research arms of universities. Otherwise, there is so much duplication of work.