Measuring critical service-learning and its long-term impacts: a methodological discussion for community-engaged capstone programs

Abstract: Service-learning has been criticized for naming a commitment to justice that is unmatched by practices that facilitate positive social change. “Critical” service-learning has an explicit focus on social justice that requires 1) working to redistribute power among participants in the service-learning partnership; 2) developing authentic relationships rooted in connection; 3) and operating from a social change perspective. However, the literature lacks descriptions of how to operationalize and measure the elements of critical service-learning, particularly within graduate-level capstone courses. To address this gap, MPH Capstone Experiences: Promising Practices and Lessons Learned and a forthcoming paper, Using a Critical Service-Learning Approach to Prepare Public Health Practitioners, describe how we apply and assess critical service-learning constructs in a community-led, group-based capstone program. Our next objective is to conduct an impact evaluation to measure lasting effects of capstone programing on students and partners; however, there are few examples of how to assess elements of critical service-learning over the long term. This roundtable is an opportunity to discuss how to better measure critical-service learning constructs and document long-term impacts of critical service-learning programming. We will share current and proposed measures as a foundation for discussion and invite others to bring their measurement instruments for review and feedback. Our goal is to collectively move critical service-learning measurement and evaluation efforts forward through facilitated conversation and critique.

Narrative: Capstone is a yearlong, community-led, critical service-learning graduate-level course that serves as a master’s thesis substitute for students in the Health Behavior and Health Equity, Social Justice, and Human Rights MPH concentrations at The University of North Carolina at Chapel Hill Gillings School of Global Public Health (Gillings). Created in 2009, Capstone’s aims are to: 1) increase capacity among students and partner organizations to address public health issues and promote equity; 2) create new and or/improved public health resources, programs, services, and policies that advance health equity; 3) enhance student preparedness and marketability for public health careers; and 4) strengthen campus/community partnerships. Our community-led approach prioritizes community partners’ specific interests and gives students an opportunity to work on a range of approaches to social change (e.g., community organizing, policy advocacy, education) with a variety of organization types (e.g., nonprofit, government, health care, social services, academic) on a wide array of public health issues (e.g., harm reduction, food access, tobacco control, affordable housing, aging, substance use, etc.). Over an entire academic year, each team of four to five MPH students works with a partner organization and its constituents to produce a set of four to six deliverables (e.g., literature reviews, data collection instruments, program and evaluation plans, needs assessments, policy briefs, training materials, presentations, manuscripts, etc.) that are responsive to the community partner organization’s self-identified needs.

We describe Capstone’s staffing model; project recruitment, selection, and matching processes; course format; and assignments in our recently published paper, MPH Capstone Experiences: Promising Practices and Lessons Learned. Our forthcoming publication, Using a Critical Service-Learning Approach to Prepare Public Health Practitioners, details how we operationalize and assess each of Mitchell’s (2008) three elements of critical service-learning: 1) working to redistribute power among participants in the service-learning partnership; 2) developing authentic relationships rooted in connection; 3) and operating from a social change perspective. Our research for these papers was approved by UNC Chapel Hill's Institutional Review Board (IRB 21-0510). The data for our papers came from mid- and end-of-semester evaluations administered to students and community partners as well as course evaluations administered to students by Gillings. To ensure our findings reflect our current programming, we analyzed data from academic years 2020 and 2021, during which time 98 students and 22 community partners participated in Capstone. The teaching team received a 100% response rate from students and community partners on the eight internal evaluations (four per year) and an overall response rate of 72% from students on the four Gillings course evaluations (two per year). Mutual benefit was used as a proxy measure for power sharing. Skill development emerged as the primary benefit from the student data. Our evaluation found that useful deliverables, enhanced capacity, impacts beyond the scope of the Capstone project, and increased ability to implement more inclusive and equitable processes were the primary benefits of Capstone programming for community partners. To assess authentic relationships, we analyzed community partners’ satisfaction with their teams’ adherence to agreed upon group values and processes as well as students’ and community partners’ qualitative responses about their teams’ greatest achievements. Effective collaboration emerged as a theme within both data sets. Due to the varied nature of the Capstone projects and the wealth of potential indicators, it was most difficult to assess working from a social change orientation. Given these limitations, we analyzed project proposals for evidence of an explicit focus on social justice or equity-related work and reviewed all qualitative data for organic mentions of a shift in social change perspective. Fifty-one percent of project proposals met our criteria for evidence of a social change orientation. A handful of testimonials demonstrated how our programming can develop orientations toward social change that can have ripple effects for our partner organizations’ work and students’ career trajectories. By documenting our practices and pedagogy dedicated to critical service-learning, our papers fill a gap in the literature and directly answer the call from Stith et. al. (2021) for more critical service-learning programs to document and focus on how their values inform their practices within and beyond the classroom.

More research is needed to understand the long-term impacts of our critical service-learning programming on students and community partners. However, the literature lacks consensus on how to assess critical service-learning constructs. Therefore, a participatory roundtable at IARSLCE offers a unique opportunity to move critical service-learning measurement and evaluation efforts forward through facilitated conversation and critique. We will start our round table with a brief overview of our lessons learned from the research that informed our two papers before turning our discussion toward our future research and evaluation plans. Specifically, we will:
• Introduce the Capstone program model and give participants insights into our promising practices and lessons learned.
• Share our current continuous quality improvement assessments and how we use these tools to measure processes and outcomes associated with critical service-learning.
• Share our draft long-term impact measures for students and community partners, inviting discussion and refinement.
• Invite roundtable participants to share their critical service-learning measures.
• Discuss methodological considerations for evaluating critical service-learning programming, particularly long-term impacts.
Our roundtable will leave participants with new resources, tools, questions, and methodological ideas that will contribute to strengthening the implementation and evaluation of critical service-learning programming.