Evaluation is like GPS

We caught up with Margarida Pimenta at Impact in Focus to chat about all things evaluation. Read on for insights and resources.

Photo by Shopify

Can you briefly introduce yourself and your role on the Community Tech project?

My name is Margarida Pimenta, I’m the project manager for evaluation for the Community Tech project’s Makers & Maintainers programme and Discovery programme - which, taken together, fund 30 organisations to experiment with community tech projects. inFocus is acting as the “learning partner” for Power to Change which means our goal is to capture the impact of the investments being made and the work delivered in strengthening and promoting community tech as well as in generating and sharing learning about the processes.

What does ‘evaluation’ mean to you and what is its main purpose within a project?

I like to think of evaluation as a GPS (I’m someone who is very hopeless without Google Maps). Evaluation is a powerful tool that helps you understand where you are, where you want to go and whether you’re using the best route to get there (or going in the complete opposite direction). By having a systematic assessment in place, to understand how effective and relevant activities, interventions or projects are, we have a much clearer picture of our current trajectory. 

Its main purpose is trickier to define as evaluations can serve a variety of purposes such as to enhance accountability, support decision-making, stakeholder engagement or learning. 

We pay special attention to the learning aspect. One of our main goals is to provide insights that will allow the project team and grantees to adjust strategies, refine methodologies and improve outcomes in real-time or in future projects.

Can you tell us about your process for the Community Tech project and how it incorporates co-design? 

During the initial stages of our involvement, when starting to design the evaluation we took both research and recommendations at the core of our approach. We started by developing a “Theory of Change” which is a diagram that models how the activities may lead to outcomes and impact and gives us a basis to test how the changes we envision play out in practice. We’ve developed this model in consultation with key stakeholders from Power to Change, Promising Trouble and some grantees who had an opportunity to brainstorm about outcomes and reflect about which challenges the project is trying to tackle thus feeding into the model. 

We’ve then moved into a data collection stage where we gathered information through interviews with those accessing the fund but also those involved in the fund design, delivery, and support.

We’re now heading into a “learning stage”. This year we’ll be supporting Promising Trouble by providing insights that emerge from our data collection efforts and on the following year we will be leading a few group learning sessions that will bring stakeholders together to look at impact, explore expected and unexpected outcomes and identify moments, processes and results where the fund has been most influential for them.

How do you navigate different goals and expectations when working in partnership and with a variety of different organisations and people?

This is a challenge we encounter very frequently; I don’t have a secret formula but I feel that remaining flexible is one of the most important things. Evaluators are usually people who hate going without a plan but being prepared to adapt to new circumstances or to new information is just as important as having a solid plan. Another way I try to manage diverse perspectives and expectations from people is to aim for “common ground”. By trying to understand what the underlying concerns and motivations are I can aim to build shared goals and objectives. This often means I must find creative ways of catering to people’s different needs.

Can you share any learnings about how to demonstrate impact with a project like Community Tech? 

Demonstrating impact is challenging for almost any project. I see a lot of impact reports out there who are solely focused on showcasing the positives of projects or programmes and fail to provide a genuine account of the project journey.

Demonstrating impact requires a combination of quantitative metrics, qualitative insights, and a touch of storytelling.  In a project like Community Tech which is very “short-lived” (from the funding and Evaluation timeline perspective) tracking quantitative metrics like engagement, number of participants and active contributors is important but gathering qualitative insights from participants about their experiences, learnings and transformations is key to understanding the nuances of impact. That is one of the reasons why we’ve decided to invest more effort in interviews, learning-sessions and in gathering materials for case studies as opposed to focusing on surveys and capturing statistics. Some of the things we are looking for to “paint a picture of impact” are collaborations and partnerships as evidence of broader reach of the initiatives, evidence of sustainability and growth, sharing and engaging in the wider tech community, challenges and learnings and strategies to overcome them.

Can you suggest any readings or resources about evaluation for those interested in finding out more?

For those really interested in getting more in-depth knowledge Michael Quinn-Patton’s "Principles of Focused Evaluation" is a kind of Evaluation bible, although it is not free. 

These are very helpful free guides and resources that I often refer back to:

The World Bank's "Impact Evaluation in Practice": A practical guide that covers the fundamentals of impact evaluation for practitioners. 

The M&E Universe by INTRAC: The M&E Universe is a free, online resource developed by INTRAC to support development practitioners involved in monitoring and evaluation (M&E). 

Better Evaluation: An international collaboration to improve evaluation practice and theory by sharing and generating information about options (methods or tools) and approaches. 

Finally, where can people find out more about you and your work?

Go to the inFocus website to find out about some of the work we do. Particularly, our Case studies page.