We were joined by special guest Margaret (Meg) Hargreaves at our August 2019 Systems Change Community of Practice gathering. Meg is a Senior Fellow at NORC a research institute at the University of Chicago who specializes in systems change evaluation. Tori interviewed Meg as a follow up to Meg’s webinar with Mark Cabaj, Planning an Evaluation of Systems Change.

Tori: As we experiment with evaluating systems change, we’ve modified our traditional evaluation requirements and are using storytelling to notice signals of change. Our Systems Change Accelerator grantee partners appoint a Learning Steward who is familiar with their system. Learning Stewards observe the system and look out for any shifts, then reflect stories about what they learn back to the team. What are your suggestions do you have for the Learning Stewards?

Meg: This is an approach of evaluation called Developmental Evaluation, created by Michael Quinn Patton. It's more about the process of walking with an initiative than it is about pushing particular methods.

I want to expand our idea about what could go inside those stories. Stories aren’t only made up of information that comes from observation, though that is important. Numbers can be ok too. You can talk about how things have changed - percent change, difference between this place and another. Track it, keep it.

Feel free to use all kinds of evidence; not only what you do but what others do. Weave narratives together. Don't worry much about attribution, which means you claim to have caused a thing all by yourself. We heard so many examples of contribution today, where multiple organizations contributed to some change that might not have happened if it were just up to individual organizations. That is the core of systemic change - that there is an aligned, collaborative, synergistic, intersectional working together that is moving the needle beyond what a single organization is capable of.

It's important to develop a strategic theory of change, deciding where you should focus. You should be looking for evidence to test your theory: does this actually lead to that? We're constantly making assumptions. We can test those. Put what you are learning into your narrative, lifting up the small changes that are expected and unexpected.

Tori: In the webinar  you mentioned that as we shift from programmatic evaluation to systemic evaluation, we can still use some of those traditional methods that we are familiar with. I love this recommendation because it validates the experience that we bring with us even as we step into a new paradigm. You suggest that what needs to change is the kinds of questions we're asking.

Meg: There are robust methods for how to interview people well or to do surveys. Continue to follow those guidelines. They don't go away just because we're working with systems. Be stewards of robust data and reliable measures. Tell me what you didn't expect, as well as what you did expect. How is this behavior at the margins changing what you're doing, which may not have been one of your original goals? It's a healthy combination. Where you start and where you've been really effect where you're going. In that way, historical data is really important to integrate. There is almost no data off limits, unless it's bad data. Newspapers, county budgets, annual reports. Everything tells you what's going on in a systems context.

Instead of a program or individual being what you measure, you measure ecosystems, networks, communities. Instead of studying how a program is doing, you ask, how is the network doing?

Having a collective understanding of the problem you're addressing is key to your theory of change. You're working in a group that shares a connection to your theory. Ask that group questions about what they're doing toward this shared goal. Then talk about the individual and collective efforts you see. Don't assume that you know what the outcomes are going to be; instead, ask what they are; this is called outcome harvesting.

Tori: Systems change theory teaches us that things tend to get worse before they get better. When we're evaluating the impact of our work, how do we distinguish a moment to be patient from a signal that we need to redirect because something is getting worse or not changing?

Meg: Bob Williams has a great book where he defines the difference between acting systematically (in a nice, orderly fashion) versus acting systemically. Take a huge pause and act systemically, which means, recognize you're in a very dynamic situation. Many others who like order may be freaking out, and you're not allowed to join the freak out.

Recognize panarchy, that there is a moment of chaos that is actually creative destruction that generates new order. When everything has to change, help others recognize what has to be let go. Don't make it forced march, but help people notice early adopters or pilot projects or easy wins to show that it's not so bad when you shift. "Wow, I didn't die, my superiors didn't kill me." People have to experience change positively even though it creates chaos in their lives. Help people as they are experimenting to stay calm as they try new things. Then, create supports to help people through this fairly traumatic change process: leadership, resources, facilities, coaching, consultants. Don't give them the false hope that you have to wait until it's over, because change is never really over. People will get to a new normal.

Meg, on zooming out to study change across our region: It's important that Rotary Charities focus not only on what your grantees are doing, but what they're doing with others and each other to actually grow a movement beyond what individual grant dollars can pay for. This learning community is a perfect example of this. Even today, I took down notes, thinking "that is brilliant" and I'm sure everyone here is borrowing from each other thinking, "I didn't know that was a cause of change. I didn't think I could count that [as shift in our system]." There is no shame in that. It's called learning, it's not called stealing.

It's not only important to understand at the organizational level, the micro level. It's really important to ask funders to look at the larger pattern shifts. It's about shifting how philanthropy works to be able to look at pattern shifts systemically. What you're doing here is such a wonderful example. In addition to supporting individual learning stewards, you need to have your own learning stewards to think about how you're going to evaluate this overall.

What's so lovely about this is that after so many years of systems change being too weird, it's now being seen by even large foundations and governments as part of the solution. For example, human service departments are now being combined with health departments to form Health and Human Service Departments. Institutions are moving from Child Protection to Child Wellbeing, which accounts not only for services provided but also for prevention activities, integrating multiple spheres of activity. This is freeing people from the silos and funding restrictions and regulations that have been so focused on attribution and control, and are now being blended, braided, allowing people to fill the gaps that haven't been addressed before.