The Service-Learning Quality Assessment Tool (SLQAT): A Standardized, Quantitative Instrument for Measuring Service-Learning Quality

Abstract: Our field lacks systematic mechanisms with which to capture differences or track the presence of key best practices and the nuances of implementation quality across service-learning (SL) experiences. To address this issue, we engaged in a five-year, multi-institutional initiative focused on developing a standardized, quantitative, and scorable rating instrument designed to measure SL quality in academic courses – the Service-Learning Quality Assessment Tool (SLQAT). We conceptualized, developed, piloted, redesigned, and weighted an instrument, which was then finalized using expert raters from across the US and internationally, resulting in a version with numerical base weights and three different levels and scores of implementation quality for 28 "essential elements" of SL. This instrument can guide researchers in evaluating more consistently the impacts of different essential elements of SL on student outcomes. This presentation shares the process and outcomes of the instrument development as well as uses, next steps, and implications for the field.

Narrative: The diverse and contextualized nature of service-learning (SL) practice poses challenges to conducting institutional and multi-site research (Bailis & Melchoir, 2003; Furco, 2003). SL varies across a range of differences in course design, partnerships, student experience, and instructor and institutional characteristics (Bringle, Clayton, & Hatcher, 2013; Furco, 2003; Heffernan, 2001; Roldan et al., 2004; Waterman, 2003). Research studies have identified key practices fundamental to the integrity and quality of SL courses, and elements impacting student outcomes (e.g., Botelho et al., 2020; Eyler & Giles; Heffernan, 2001; Jacoby, 2015; Steinke et al., 2002; Waterman, 2003). In addition, implementation quality matters (Eyler & Giles, 1999; Kuh, 2008; Mabry, 1998), and SL courses that implement more essential practices are also more likely to result in positive outcomes for students (Celio et al., 2011).

No quantitative instrument currently assesses the extent to which a course incorporates these key elements of high-quality practice and at what quality level. Existing tools are designed either for specific disciplines, incorporate only a few key components, or are qualitative in nature. A quantitative instrument for consistent measurement of the presence and quality of best practices would allow for more accurate and nuanced SL research. This was the goal of our five-year project, part of a larger US DoE grant-funded initiative. The research team consisted of administrative faculty, staff, and graduate students from two US research universities.

Instrument Development:
Development of the Service-Learning Quality Assessment Tool (SLQAT) was an iterative process from 2016 to 2021 (Matthews et al., 2023). Beginning with an intentionally large set of SL best practices, multiple rounds of discussions and a systematic elimination process clarified and reduced the number of elements, which were categorized and rated based on their hypothesized importance to student learning outcomes. Quality level statements operationalized implementation levels, with weighted point values for each element and level.

Methods of Verification:
The instrument and each element were subjected to sustained, deep discussion over several years on wording, relevance, and importance, and elements were further clarified and winnowed down through piloting with a series of courses, critical consideration by the research team, and formalized feedback at community engagement conferences. In 2020 we invited service-learning scholars and practitioners in the US and internationally to review and quantify the element weights. This process invited raters to read each of the 28 elements and its short description, then to assign a weight ranging from 1 to 9 to represent the influence of the given element on student learning outcomes. Raters also provided feedback on the validity, comprehensiveness, and wording of the instrument, and their own level of expertise in SL.
The finalized pool of 58 respondents represented instructional faculty, administrators, and other roles, primarily in higher education settings in the US and seven other countries. This research centered on instrument development and was confirmed by our institutional review board as not entailing human subjects research, although we still received respondents' consent for research use of responses.

Outcomes/Impacts:
Respondent baseline weight ratings showed that all the proposed "essential elements" were considered highly impactful on student learning outcomes (overall mean 7.42 (SD, 1.48) and individual mean element weights from 5.83 to 8.55 on the 9-point scale). Comparison of the ratings by respondents who self-identified as having “advanced” vs. “intermediate” levels of service-learning experience found that the more-expert raters identified elements as more impactful on average. Because these differences were statistically significant (t(54)=2.72, p<.01), we decided to use the ratings by the “advanced” group only (n=36), in order to maximize respondent expertise.
The current version of the instrument was finalized using these expert ratings as the basis for quantitative scoring. Mean element weight ratings were rounded to the nearest 0.5, resulting in final weights from 6.0 to 9.0 with an approximately bell-curved distribution. Each element was intended to be scorable on a range of levels of implementation quality (“below baseline,” “baseline,” “above baseline”), with corresponding differences in value based on hypothesized contributions to student learning outcomes. The research team conducted extensive discussions, with consideration given to potential benefits and challenges of different spread (e.g., +/- 10% vs. 50%), whether ratings should include a true "zero" (i.e., scale vs. ordinal), and the practical interpretive implications for scores. The research team opted for an intermediate level of impact, assigning +/- 25% as the variation from baseline for the quality levels, resulting in a range of possible ratings (4.5 to 11.3) reflective of differences in both importance and implementation quality of the elements.

This revised SLQAT (Furco et al., 2023) thus consists of 28 “essential elements” (grouped into five conceptual dimensions: course design, learning, student, instructor, and community partner/partnership). Each element has a title, short description, question to guide determination of evidence of the element’s presence or absence in the data sources, base weight, and three levels of descriptive text with corresponding implementation quality categories and modified weights.

Significance:
This process has resulted in a quantitative, scorable instrument publicly available to the field for a multitude of uses. The revised SLQAT may be used as an outcome variable or a predictor variable, depending on the analysis desired, and also opens a line of research for confirming or challenging the instrument's underlying assumptions (e.g., relative "weights" of each element; impact of missing elements on course quality; equivalence of overall course quality across different contexts; etc.). Beyond research purposes, the SLQAT may also serve as a tool for designing coursework, for reflective self-assessment of practice, and for campus classifications of SL. Currently, the instrument is migrating to an online technology-based platform that can produce dashboards and other user-friendly functionality.

Keywords: instrument development, service-learning quality, assessment

References:
List of references (word count = 351) included in supplemental materials