Going beyond reporting requirements to gain insights
I was recently paying a credit card bill when I saw the reports on my spending categories – some questionable splurges had happened – and I decided to keep a close eye on my spendings from then on. My phone too guilt-trips me daily with screen time data, eventually pushing me to reduce my screen time. So if data can help us improve our performance, why aren't we using data/metrics to assess our effectiveness when it comes to impact incubation? During one of our training sessions in the ‘Incubating Incubators program’, when I brought up the topic of assessing the performance of incubators themselves, few had a clear plan for collecting and measuring data on their own performance, indicating a lack of clarity on why this is crucial.
Interestingly, in this month’s Wednesday Wisdom Session with Abby Davidson of Aspen Network of Development Entrepreneurs (ANDE) and Nicholas Colloff of Argidius Foundation, one key learning was that a lot could be learnt about incubator performance by intentionally designing and monitoring metrics with a clear learning objective in mind.
I decided to dig further into the topic, and in this interview, Harry Devonshire, the evaluation & learning manager at Argidius Foundation, offers more insights on how well-designed metrics can help improve impact. Here is an excerpt from the interview:
Q: Metrics are often put in as an afterthought or to satisfy funder reporting requirements, or both. Why should ESOs/incubators consider ‘conscious design’ of metrics? What are the benefits? Any examples of where this has really helped?
Data-informed learning is a powerful way to improve impact. For example, a number of years ago, Technoserve decided to prioritize cost-effectiveness across their programs using a metric called Return on Total Investment. The average ROTI, an indicator of the cost-effectiveness at which impact is achieved, has grown from 3.4 in 2017 to 5.2 in 2021. This is in spite of the discount COVID has wrought. This means that for every $1 spent on program cost, supported enterprises increase their revenues by $5. An enterprise development program that costs $5,000 per enterprise, helps that enterprise grow their revenues by $25,000. Seventy-five percent of the latest generation of programs are achieving ROTIs higher than 6, with some achieving ROTIs over 30.
To not just generate but to strategically use this metric, Technoserve collected the underlying impact and cost data across their programs, regardless of whether funders required it or not.
In this case, metrics were approached from the angle of R&D rather than reporting.
Q: What do funders look for in addition to impact metrics? What are they learning about an ESO based on the maturity of their metrics design?
It depends. If they are accountable to the public, such as a sovereign donor or national government, then accountability is often the priority. A certain level of capacity is necessary to meet their requirements.
Private funders can be more flexible. One community of funders is on a journey from "proving impact"’ to "improving impact." At Argidius, we are interested in impact and learning in equal parts. If we are not going to learn something that will help ourselves and the sector become increasingly effective, impactful, and sustainable, then we don’t do it.
We are particularly impressed by ESOs that are driving their own learning agendas. For example, we first met Balloon Ventures when they were seeking research funding for a randomized control trial to test if and how the Lean Start-Up Methodology could cost-effectively grow small businesses in developing countries. A commendable commitment, especially given results are soon to be published come success or failure.
Q: What are 3 principles that incubators can use in designing metrics for monitoring and evaluation?
The three principal questions to ask oneself are:
What does success look like?
How would you know?
And what do you not know, that if you did, would enable you to be increasingly impactful?
At one level this can look very similar across different incubators. All entrepreneurs seeking support want to grow their businesses. All incubators, implicitly or explicitly, have growing enterprises in their theory of change.
There are three simple indicators that are decent proxies of growing businesses:
revenue generation, indicating traction in the market and scale;
full-time employment – the easiest type of employment to measure – at the early stages of enterprise formation, indicating capacity and commitment of key persons; and
finance mobilized, indicating the extent that others are buying into the enterprise’s potential.
Then there might be more specific interests from a wide range of topics that an incubator wants to focus on, whether related to inclusion, sectors, or themes such as particular SDGs.
The final principle is to consistently collect the underlying data over time. We always see a wide variation in the performance of incubatees. Underlying data enables you to evaluate what is and isn’t working, and for whom.
Q: Learning (and metrics to support that) are a key aspect of the SCALE framework that Argidius has developed. How does the ‘learning’ help incubators / ESO’s?
A growing body of evidence shows that ESOs, including incubators, can represent excellent value for money in stimulating growth and job creation. The same body of evidence shows that whether this is true depends on a few critical characteristics which have been acronymized as SCALE.
The L is our topic of conversation today- Learn by evaluating enterprise performance. Be curious to learn what works for who; find out the extent to which incubatees are growing over time; have conversations with entrepreneurs to find out what changed, what has been valuable and what hasn't, and use that information to make continuous improvements.
All of this requires you to
set expectations with your incubatees accordingly, that you will be following up, that you are very interested in their performance as a means to improve your own;
build relationships with your incubatees;
And work to make data collection a valuable interaction rather than an extractive demand. One way of doing this is by working with incubatees to help them better understand, collect and use their own data for improved decision making.
The SCALE report and toolkit brings together evidence, case studies and tools to help incubators, ESOs and their funders integrate these performance driving characteristics. You can access it here: How to Fulfill the Potential of Business Development Services using SCALE (argidius.com)
To learn how you can use metrics to communicate your value to funders, watch this month's Wednesday Wisdom Session “Moving from Metrics to Insights: On the path to better impact funding”.
Co-founder and CEO at Villgro USA
Arun is passionate about scaling impact by helping impact incubators succeed. He led Villgro India's health sector before his current role at Villgro USA
Evaluation & Learning Manager, Argidius Foundation
Experienced in for-profit and non-profit sectors, in East African, Latin American, British and global contexts, Harry is experienced in developing and managing impact and learning frameworks, and supports Argidius's leadership in the global enterprise development sector.