Measuring Change while Changing Measures: Learning in, and from, the Evaluation of Making Connections

Teaching cases are factual stories of one foundation’s in-depth experiences related to evaluation and learning. Stories highlight important challenges that confront foundations in their evaluation work, and put readers in the role of decision makers who are confronted with problems and options for solutions as the story unfolds. This teaching case was produced for the Evaluation Roundtable, a network of evaluation and learning leaders in foundations.

Making Connections was an ambitious, multi-site, decade-long community change effort supported by the Annie E. Casey Foundation. It started in 1999 and aimed to improve outcomes for vulnerable children by transforming their neighborhoods and helping their parents achieve economic stability, connect with better  services and supports, and forge strong social networks.

Making Connections achieved some of its intended goals. Along the way, it tested the potential and the limitations of tools and strategies for evaluating community change efforts.

Making Connections’ evaluation, which would span eight years and cost almost $60 million, was complex and multidimensional, with many moving parts. This case study focuses on just one slice of the evaluation: measurement choices and challenges. It emphasizes three challenges, among many, that are particularly relevant to evaluations of community change initiatives:

1. How to measure population-level change for an initiative that seeks community-wide results but often operates at a smaller scale at the program level, with aspirations for change that shift over time.
2. How to frame an evaluation when it has multiple, evolving, and sometimes competing purposes that include measuring outcomes, building local capacity, empowering neighborhood residents, enabling implementers to “manage to results,” and enabling the funder to “learn while doing.”
3. How multiple data needs and uses, at differing levels of implementation and management and different phases of the work, drive evaluation options and choices.

Making Connections’ evaluation struggled with these challenges as the initiative’s evaluators, implementers, and managers strove to simultaneously satisfy the need for real-time learning, results-based accountability, and genuine improvements in outcomes for residents of deeply troubled neighborhoods.