Looking for Shadows: Evaluating Community Change in the Plain Talk Initiative

Teaching cases are factual stories of one foundation’s in-depth experiences related to evaluation and learning. Stories highlight important challenges that confront foundations in their evaluation work, and put readers in the role of decision makers who are confronted with problems and options for solutions as the story unfolds. This teaching case was produced for the Evaluation Roundtable, a network of evaluation and learning leaders in foundations.

The 1990s marked the beginning of a shift by large national foundations from mainly programmatic investments to a deeper engagement in long-term, complex community change. Comprehensive community initiatives (CCIs) emerged as an exciting new approach. They were known for:

• Working across multiple sectors
• Seeking results at the individual, family, community, organizational, and systems level
• Focusing on building community capacity, social capital, and neighborhood empowerment along with more tangible outcomes
• Recognizing the contexts and conditions that affect outcomes
• Responding to community dynamics and evolving over time
• Attempting to saturate communities with an intervention.

Social policy evaluators in the 1990s were, for the most part, good at measuring the impacts of single programmatic interventions and less good at measuring community change—something that often is abstract and intangible—and poorer still at measuring how initiatives influence systemic change. CCIs are challenging to evaluate because outcomes are difficult to quantify, and they often don’t show up for many years.

The Plain Talk Initiative’s evaluation design included within-site and cross-site analysis and featured multiple data collection strategies, including a baseline and follow-up survey, a heavy emphasis on qualitative research, and review of administrative data. While the case describes the many challenges encountered with the complex evaluation of a complex initiative, the Plain Talk evaluation ultimately showed it was possible to create local learning partnerships, involve residents in meaningful data activities, measure elusive effects, and use data to advance the work in communities.