by Freya Bradford, Director of Systems Change and Learning

This is the last in a four-part series that shares our best learning about using a systems change approach to address complex community-level problems. The first blogs in the series are about bringing people together around a common purpose, moving upstream of a problem to find leverage points for change, and designing interventions that change the way a system behaves over time. This installment explores how systems change initiatives can learn together to continuously improve and align their work.   

It is no secret that working towards upstream, durable change on complex problems is challenging work. Systems change work is long-term, includes many actors and actions, progress is non-linear and is intertwined with a context that is always changing. These factors make it difficult to predict and measure progress using methods that are typically used in the social sector. Despite these challenges, it is imperative that we learn together as we go. This final installment in the four-part systems change series explores two systemic learning questions, potential indicators of change to look for, and methods we might use to capture signs of change all along our journeys.  

Learning in Systems Change 

“Strategic learning is even more important once you realize that it is possibly the only outcome in systems change we can control.” – Mark Cabaj, Tamarack Institute 

Most of us who have been working in the social sector are familiar with learning and evaluation as something that we might do at annual intervals, or the end of program, especially if a funder requires it. We measure the extent to which we reached our intended goals and targets, if the people (or places) we reached were better off after our program. A report is written and hopefully some insights are woven back into the next iteration of the program.  

In a programmatic approach, learning and evaluation assess strategy. In systems change approach, learning and evaluation are part of strategy.  

In complex environments learning is a means of change; systems change through learning. Learning is part of the everyday culture of the work. Regular reflection and sensemaking are part of every meeting, hypotheses and experiments frame actions, and insights turn into new wiser actions.    

Learning accelerates change by helping people learn from others, see their work within a greater context, notice patterns emerging and amplify what is showing promising signs of change, and correct what looks like unintended negative consequences.   

Systemic Learning Questions  

Learning for systems change is intentional and includes lines of inquiry that evolve with the initiative over time. The ultimate learning question is whether or not the complex problem is improving. However, that is a long-term question that can take years to be realized. Underpinning that are two critical systemic learning questions I have found helpful when working collaboratively on upstream change. All three questions are presented in the diagram below.    

As with systems action, systems learning will likely include a constellation of interrelated learning questions and methods. Given the complexity of systems change work and the multiple actors and potential users and uses for the results of learning, the universe of potential inquiry questions and approaches can quickly become untenable.  

Mark Cabaj of Tamarack Institute in Canada suggests refining learning questions and approaches by considering a set of principles. The list that follows is inspired by his original list.   

• Consider the end user’s needs. What might participants need to know to adapt their approaches, what might funders want to know, what do your ultimate beneficiaries need to know?  
• What decisions or prototypes are ahead? What might you need to know to inform what’s next?  
• How might you involve stakeholders and spread evaluation across the initiative with a regular opportunities to share what they learn?  
• How might you explore changes at nested levels – individual, organization, and systems – and changes that show up in different systems? 
• How might you combine qualitative and quantitative approaches  – “No numbers without narrative, no narrative without numbers” (Mark Cabaj) 
• How might you capture unintended outcomes? Failures?  

The next sections unpack the two foundational learning questions that can be explored while keeping an eye on the ultimate needle your initiative seeks to move.  

Are we working more systemically?  

The first two blogs in this series are about how systems change work requires us to work differently together than we typically do as individual organizations focused on discrete missions. This fundamental systems learning question is about regularly taking stock of how, and in what ways, actors in a system are working more like an ecosystem and moving from isolated responses to a common problem to a more orchestrated and attuned constellation of actions. More than a means to an end, I am noticing that this transformation is being increasingly talked about as an end itself, an embodiment of the ultimate change a group seeks in a broader system.  

The following are some potential high-level outcomes related to this shift. Specific indicators will depend on the context of an individual initiative.  

Potential Outcomes of Working more Systemically    

• Increases in the number and diversity of people and organizations working together around the issue 
• Greater alignment around a shared purpose, goals and roles 
• Inclusion of diverse perspectives of what is contributing to the current problem, and where leverage points for change may be 
• Increases in individual understanding of their own role in the problem and potential solutions  
• Improved information flow across the system, especially around actions, results, and learning 
• More sharing of resources within the system (across organizational boundaries), and an improved ability to connect to and leverage resources from outside the system 
• Inclusion of those with lived experience with the issue, and increases in their decision-making power 
• Greater co-creation and experimentation to change system structures and patterns 
• Improvements in ability to learn and reflect with others in the system, notice early signs of change, and use learning to improve action of partners (both within the collaboration and at home organizations) 
• Mindset shifts among partners – both around the issue you are working on, and around ways of working together (e.g. comfort with uncertainty, embracing failure, etc.) 

Are we nudging the system towards better results for all?  

When we’re aiming to tell the story of what is changing in a system, it is important to keep some practical things in mind. Plainly, systems change work will never “prove itself” by typical standards we have judged programmatic work, and the change will likely not look like what we’d expect initially.    

Systems Change Evaluator, Meg Hargreaves, reminded us at a 2019 Community of Practice meeting, “Don’t worry much about attribution… (in) systems change there is an aligned, collaborative, synergistic, intersectional working together that is moving the needle beyond what a single organization is capable of.” Systems change usually involves many actions, contributed by many stakeholders, that shift and adapt over time as the context around the problem changes. Trying to disentangle these parts is neither practical nor worthwhile. Evaluation and learning in these complex and interdependent environments is not about proving causality or attributing change to any one actor or set of actions, but rather to illuminate any change that is happening and to learn from the actions that may have contributed to a change.  

The ultimate change in a systems approach often takes many years to see reflected in population level numbers (e.g. changes in graduation rate, sufficient new housing to meet demand, reduced rates of poverty). However, there are things that we might look for that can indicate a system is moving in a new direction. These show up at nested levels – individual, organizations, and systems – and may not look like the things we’re used to celebrating. In Evaluating Ecosystem Investments, FSG outlines several helpful interim outcomes of systems change.   

Intermediate Outcomes of a Systems Change Approach   

• Shifts in flow of funding or other resources – more resources available for an issue, changes in use of resources 
• Shifts in decision-making power 
• Shifts in media or public messages or political discourse 
• Increases in number of people and organizations who feel equipped to take action 
• Increases in number of people and organizations taking action on an issue 
• Increases in people and organizations adopting new behaviors, standards, principles, or processes 
• Increases in talent pipeline 
• Strengthened evidence base (new data, publications, etc.) 
• Changes in policy and its precursors (policymakers willing to take action, items on agenda, etc.) 

Turn Actions into Explicit Experiments 

To find indicators of change for your specific initiative, it can be helpful to turn planned actions into experiments by creating hypotheses that make your theory of change more explicit.  

“If we [do this action], then [this will occur]. If [that occurs], then [this system factor/pattern will change.”      

Using the housing example from the previous blog in this series, one of Housing North’s hypotheses might look like: 

“If we deliver a collective regional voice to policymakers at the state level, then policy makers will understand our needs more accurately. If policymakers understand our needs more accurately, then enabling legislation will be passed giving local governments more tools and subsidies to expand housing options.”  

When explicit like this, hypotheses can quickly suggest the indicators you might look for to measure progress. Ask yourself: 

What signs of change might we be able to observe that may indicate the effects we     hypothesized are beginning to occur?  

Using the example above, we could outline and track potential indicators around three things: the delivery of the regional voice to legislators, the increase in policymaker understanding of our needs, and the passing of enabling legislation. 

Capturing Signs of Change – Systemic Learning Methods 

Signs of systemic change can be found using regular group sensemaking processes, more traditional evaluative techniques, and some newer approaches that are well-suited to complex work. Meg Hargreaves reminded our Community of Practice that tried and true evaluation practices need not be abandoned in systems change work. “There are robust methods for how to interview people well or to do surveys. Continue to follow those guidelines. They don't go away just because we're working with systems. Be stewards of robust data and reliable measures. Tell me what you didn't expect, as well as what you did expect.” 

Methods that might be particularly helpful to measure working systemically include: 

• Social network analysis and mapping can be a robust way to see changes in the breadth and depth of relationships across a system. It can be used to visualize who is connected, the strength of the relationship, and even what resources are flowing between individuals.   
• Increased alignment and commitment of partners can be investigated using surveys like Network Impact’s Network Health Scorecard, the Wilder Collaboration Factors Inventory or the Center for Creative Leadership’s Direction, Alignment and Commitment survey. 
• Our Systems Change Community of Practice created this Habits of Systemic Mindset tool to track the behaviors that may be indicative of a shift towards a more systemic mindset.  

Collecting signs of shifts in systems will be more context dependent, but may include things like: 

• Ripple Effects Mapping, Most Significant Change, and Outcomes Harvesting can help to collect stories of the changes people working within, or experiencing, the system have observed. We’ve adapted this underlying concept into a recurring activity at our Systems Change Community of Practice. Periodically we do a whole group share we call “Signs of Change” where each initiative shares one or two signs that something may be changing in the system they are working within. These are typically shared as signs of change in either policy, practice, resource flows, relationship, power or mindsets. Rotary Charities staff collect these to help us collectively see what may be changing across initiatives and the region. This could be replicated at team, organization, and initiative levels. 
• Setting aside time for regular group sensemaking can help to elicit signs of change and actionable insights on a regular basis. Hallie Preskill (et al) recommend 21 intentional group learning activities that range from 15 minute activities that can be added to any agenda to more robust exercises that take an hour or more. These activities are structured to ensure everyone in the room contributes what they are seeing or sensing about a particular focusing topic. What, So What, Now What and After Action Reviews are particularly powerful examples from this list of exercises that can easily become a regular practice to help move a group from insight to adaptive action.  
• Systems change work often includes advocacy and communications work where typical methods remain helpful like audience surveys, media tracking, content analysis, champion scorecards and bellwether interviews.  

Putting it all together 

If you’ve stayed with me through this whole series you have covered a lot of terrain - from bringing folks together with an invitation to co-create, to making treks “upstream” to find the source of complex problems, to designing a constellation of actions that is aimed at leverage points for change in a system, to finally how to use learning as a strategy to accelerate adaptation and change. This summer and early fall, we will engage in some developmental learning ourselves and will use this framework to create case studies that show the work of systems change in action through the stories of 3 – 5 of the systems change initiatives we have funded. We look forward to being able to share these stories in the fall.  

Freya Bradford is the Director of Systems Change & Learning at Rotary Charities of Traverse City. She loves to talk about systems change so please feel free to contact her with any feedback or questions: fbradford@rotarycharities.org. 

Resources 

Evaluating Complexity Propositions for Improving Practice, Preskill, H. (et al) 

Evaluating Ecosystem Investments, Mack, K. (et al)  

Set of Principles for Evaluating Systems Change Efforts, Cabaj, M. Tamarack Institute 

Facilitating Intentional Group Learning. A practical guide to 21 learning activities, Preskill H. (et al)  

Systems Change Evaluation – An Interview with Meg Hargreaves 

Converge Toolkit