The role of design for causal analysis to achieve the access, inclusion and equity in SLCE research

Abstract: The purpose of this roundtable is to generate discussion of the application of research methods that yield causal conclusions about topics germane to SLCE and can achieve inclusive and equitable outcomes for students and communities. Participants will engage in discussions regarding research designs applied to SLCE scholarship and explore opportunities for future work using emerging techniques for causal inference. The goal of this session is to engage participants in discussions that lead to a co-authored manuscript that reviews existing research methods applied to SLCE scholarship and proposes future directions for causal designs.

Narrative: Since the early 1990’s, the credibility revolution in economics has generated strategies for research designs that have explored a variety of topics and strongly influenced policy decisions related to education and society (Angrist & Pischke, 2010). However, research on service-learning and community engagement less frequently applies methods that yield causal conclusions about research whether about students, practitioners, or communities.
In a forthcoming review, the presenter and a collaborator present a review of the literature applying causal designs for understanding the role of education that develop civic dispositions that last a lifetime, including some limited research on the role of service-learning and other experiential education in preparing youth for future civic engagement. This contribution reconciles the occasionally contradictory evidence regarding the role of education in developing civic dispositions by critically examining the designs and investigating the role that research design plays in drawing conclusions from the data examined the research.
Traditional causal designs rely heavily on experimental methods featuring randomization to assignment, which are rare in service-learning research (Steinberg et al., 2013). Randomization, when successful, enables results to be interpreted as causal because differences between treated and control groups are purely chance. After the treatment is applied, the differences in the groups can be attributed to the treatment, which is a key feature of internally valid research.
The seminal work by Markus, Howard, and King (1993) was notable in the early research on service-learning because of its use of a strong quasi-experimental design including “blind assignment” to treatments. Osborne, Hammerich, and Hensley (1998) replicated the approach. More recently, Brown (Brown, 2011) compared social dominance outcomes among students randomly assigned to be engaged in service-learning and those who were assigned research on out-groups that are often the focus of service-learning activities. Pong and Lam (2023) used experimental random assignment to examine service-learning’s effects on emotional intelligence quotient.
But experimental methods are not the only designs that yield causal interpretations. For example, natural experiments can offer “as good as random” assignment to treatment conditions. A variety of methods such as fixed effects longitudinal designs (Dahan, 2020) and difference in differences (Walcott et al., 2018) can be applied to natural experiments to yield defensible causal interpretations. Beyond the natural experiment, an increasingly popular method of estimating causal effects in service-learning research is through the propensity score analysis approach. Exemplars include Song, Furco, Maruyama, and Lopez (2017); Soria, Hufnagle, Lopez-Hurtado, and Do (2019); and Schulzetenberg, Wang, Hufnagle, Soria, Maruyama, and Johnson (2020). Other methods that can generate causal conclusion include the regression discontinuity designs and the complier’s average causal effect (Mo et al., 2022) and the synthetic control method (Pearl et al., 2013).
The aim of this roundtable is to engage researchers interested in discussions around methods for causal inference, to discuss important exemplars of causal designs applied to SLCE research, and to engender a collaborative space for those interested in pursuing causal conclusions through well-designed research to push the boundaries of inquiry into SLCE, its processes and outcomes.
The plan for the session will be:
10 minutes: review various designs with antecedents in the SLCE literature to frame the need for this discussion. These include experimental designs; fixed effects panel designs and difference in differences designs; and propensity score designs.
10 minutes: discuss emerging and innovative strategies for causal inference that could be applied to SLCE program design and development to yield a potentially causal conclusion from future research. Such designs include the regression discontinuity design; synthetic control design; and instrumental variables and treatment non-compliance designs.
25 minutes: open discussion regarding the emerging methods for causal inference including question and answer regarding these strategies for those unfamiliar with the methods.
15 minutes: open discussion of existing / emerging research applying various designs that yield causal conclusions, to discuss potential collaborations leading to a coauthored review article for publication in an appropriate venue.