How to measure impact (LabWorks 2015)

We were invited to speak on the Impact Panel at LabWorks 2015, hosted by Nesta in London UK. Labworks, now in it’s fourth year, is a roving international gathering for practitioners working in innovation units in and alongside governments & non-profits (side note: I had the pleasure of co-hosting last year’s gathering, Labs for Systems Change, in Toronto with Joeri of Solutions Lab!). The gathering is a place to learn from peers, make new connections, and share what is working and not working in the world of Labs, in order to further the practice of social innovation.

The timing of the gathering was a challenge. Our team was in the final two weeks of the Burnaby prototype, which meant anticipation was high and there was a tonne to do on the ground. For example, it meant I would miss the Kudoz Badging Ceremony, our phase one wrap celebration where hosts, Ku-doers, family members, and agency staff were recognized for their achievements and we shared where we got to with the prototype. It also meant my return flight would touch down in Vancouver just-in-time for the Fifth Space demo day, meaning I could help out with set up and production on the day-of but would be less helpful in the lead up event management. Thus, I was torn about joining the gathering. But we decided it was worth it. So I hopped on a plane to meet up with the International lab community to share our work on “The Impact Imperative” Panel.

The gathering itself was well organized and executed; however, I found myself anxious that the public conversation seemed to have not progressed much since this time last year. With so many inspiring practitioners from around the world in attendance, I was looking forward to moving beyond the standard meet & greet, in order to share and swap the actual work we’re all hard at work doing… the nitty gritty granular stuff. 

Grateful for the insightful side chats, I was left wondering, how can we make the sides the main event at future gatherings? How can we use current work – like Kudoz and Fifth Space – as live case studies? And how can we have many more live case studies to sink our teeth into and spark critical debate? Given that this work is still so unproven, the more we ground it in tangible examples, the more we can understand where our philosophies diverge and converge and whether we are chasing the same outcomes. For, getting a handle on the wicked challenges we face requires a coordinated full court press.

Below is the write up from my presentation, with thanks to Sarah, Jonas, and Yani for help in pulling it together in record time!

At InWithForward, our goal is to transition our welfare systems from social safety nets to trampolines. To enable individuals and families to flourish, rather than simply be protected from harm. Over the past 10 years, the Directors of IWF have worked in 5 countries on issues ranging from domestic violence to aged care – including designing the Family by Family prototype.

We try to shift our welfare systems by doing deep, immersive ethnographic work to reset what constitutes a good outcome and then prototyping new theories of change. That means prototyping the interactions we think will shift the factors that determine those outcomes, and testing ways to measure change in outcomes over time.

For example, after 3 months of living with folks with a disability in Apartment #303 of a social housing complex, we redefined what constituted a good outcome. The disability system defines a good outcome as people not being socially isolated. And this is measured by the # of hours of support a person had in their life and the # of connections they had in the community.

Fay received 16 hours of support a week from a service provider, and had over 11 connections in her life (a high number!) and yet she still felt bored, lonely, without much to talk about; far from what we might consider a flourishing life. Indeed, what Fay lacked was a source of novelty and learning. Something else to talk about and look forward to other than her latest medical surgery.

With Fay and 50 other folks, we began to see that connections alone weren’t sufficient to lead to a good outcome like human flourishing. But rather, the literature on flourishing requires that connections to people and places also help build ones sense of self and sense of future possibility. Making these deeper connections would mean having access and exposure to novel learning experiences. Yet, adults with developmental disabilities don’t have a lot of options to continue education after high school. In British Columbia, courses on offer for adult special education are limited to things like: Electronic & General Assembly (ie. factory assembly line work), Food Services (ie. cleaning tables and dishes), Retail & Business Services (ie. data entry and sorting mail). 

To get to these outcomes, we’d need to co-design a whole set of new interactions and a whole new measurement system.

This has become Kudoz. 

Kudoz is a platform for learning and exchange. Small business owners, freelancers, students, family members sign-up to host a one-hour learning experience. On everything from botany in the forest to building props at a theatre. Individuals with a developmental disability can log onto the online catalogue, choose a badge to work on, and sign-up for experiences that widen and deepen their interests & networks. 

Along the way, we’re tracking the change in Ku-doers (people who participate in Kudoz) via their online profile & dashboard and an App that’s in development.

We’re tracking whether there is change in their

1. voice – are they changing how they perceive themselves?
2. choices – are they changing how they spend their time?
3. purpose – are they changing their future orientation and goal specificity?
4. independence – are they increasing their level of initiative, responsiveness, communication?
5. skills – are they earning badges and reflecting on what they have learned?

We know from social cognitive career theory (see diagram below) that if we can increase people’s set of experiences, shape their interests and outcome expectations and build interest-based networks, we have a good shot of increasing their employability and financial independence: two things the disability system cares very much about.

Getting to this theory of change—and to the measurement tools—has taken over 15 iterations. And we’re still not there. We have much more design and prototyping to do. That’s because for us, measurement needs to uphold three important principles:

1. data as a learning tool, not an accountability tool.
   
2. data owned and accessed by individuals themselves.
   
3. data collection as a meaningful and reflective experience. 

We see how much of the social impact conversation is dominated by a focus on large-scale studies, randomized control trials, and using open data sets. But, we believe before you can get to all of this, you first need good data.

And in the disability space—and every other space we’ve worked from child protection in Australia to homelessness in Toronto—the existing data tells us very little about what matters to people. 

It tells us how many hours of professional care is offered, and how satisfied people are. It doesn’t tell us what is changing, for whom, when, and why. And the poor data that does exist is collected in a way that is obtrusive and often antithetical to behaviour change. 

That’s why prototyping both the front-end measurement experience and the back-end systems for capturing and visualizing data in real-time is so important, and hard to do!

For more about the Kudoz prototype and where we are at, check out our roundup blog post about phase one. And, don’t miss our Prospectus, highlighting the methodology and innovation principles underpinning Fifth Space & Kudoz.