Abstract
This research illustrates the James Madison University (JMU) assessment process
through an examination of the University’s Alternative Spring Break (ASB) program.
The two-hour a week training program for ASB trip leaders emphasizes leadership,
interpersonal skills, problem-solving, and civic responsibility. Trip leaders are
administered assessments on three separate occasions. Comparisons are made between
the Pre-test, Pre-trip, and Post-trip results for each of the subscales. In addition, trip
leaders are compared to trip participants. Findings and implications for program
improvement are discussed.
The Assessment Process
Program Outcomes
1. To improve leaders’ personal competence:
A. Leaders will develop confidence in their leadership skills.
B. Leaders will improve interpersonal skills.
C. Leaders will improve problem-solving skills.
2. To increase leaders’ civic responsibility:
A. Leaders will develop an appreciation for diverse perspectives.
B. Leaders will develop a commitment to future service.
C. Leaders will increase in their awareness of social justice issues.
Methods
Participants and Procedure
• Leaders are defined as those students who volunteer to lead an ASB trip and
participate in the leadership training course.
• Trip participants are those who volunteer to participate in an ASB trip led by an
ASB leader.
• In 2006-2007, 35 leaders and 69 trip participants completed the assessment.
• In 2008-2009, 32 leaders and 39 trip participants completed the assessment.
• Data was collected at three time points for leaders and two time points for trip
participants.
• Leaders were administered the assessment within 2 weeks of beginning training,
the week before spring break and immediately after returning from their spring
break trip.
• Trip participants were administered the assessment 3 weeks prior to the trip and
immediately following the trip.
Instrument
In 2006-2007, the ASBQ was a multiple-choice instrument with 42 items. Two of
the interpersonal relationships subscale items were removed, resulting in 40 items on the
2008-2009 assessment. The test consists of five subscales, two of which were adapted
from the Civic Attitudes and Skills Questionnaire (Moely, Mercer, Ilustre, Miron &
McFarland, 2002). The Civic Action subscale and the Interpersonal and Problem-Solving
Skills subscale. The items on these two subscales were answered using a 5-point Likertlike scale. That is, the respondents could choose from five options, ranging from
“Strongly Disagree” to “Strongly Agree.” The remaining three subscales, addressing
Interpersonal Relationships, Personal Competency, and Social Justice, include items
adapted from the Student Service-Learning Course Survey (Wang, Ye, Jackson, Rodgers
& Jones, 2005). Students respond to the Interpersonal Relationships subscale using a 7point Likert-like scale with choices ranging from “Strongly Disagree” to “Strongly
Agree”. The Personal Competency and Social Justice subscales were presented on a 7point Likert-like scale, with choices ranging from “Completely Disagree” to “Completely
Agree.” In the 2008-2009 assessment, all items were on a 7-point Likert-like scale and all
items used the “Strongly Disagree” to “Strongly Agree” response choices.
Results
In 2006-2007 and 2008-2009, leaders showed significant change on the Personal
Competency and the Social Justice subscales.
2006-2007: Personal Competency subscale: significant increase from pre-test to pre-trip
(t (34) = -5.38, p < .001), pre-test to post-test (t (34) = -7.69, p = .003), and pre-trip to
post-test (t (34) = -3.25, p <.001). Social Justice subscale: significant increase from pretest to pre-trip (t (34) = -4.47, p <.001) and pre-test to post-trip (t (34) = -3.89, p <.001).
2008-2009: Personal Competency subscale: significant increase from pre-test to post-trip
(t (31) = -6.176, p <.001) and pre-trip to post-trip (t (31) = -6.134, p <.001). Social Justice
subscale: significant increase from pre-test to post-trip (t (31) = -6.016, p <.001) and pretrip to post-trip (t (31) = -5.690, p <.001).
Other findings: In 2006-2007, leaders showed significant change on all subscales
except the Interpersonal Relationships subscale. There was also a significant interaction
of time by group (leaders, trip participants) for the Personal Competency subscale (F (1,
102) = 9.18, p = .003) and for the Social Justice subscale (F (1, 102) = 13.804, p < .001).
In 2008-2009, leaders and participants did not change at different rates across the two
times points (pre-trip and post-trip) for any of the subscales.
Discussion
• In 2008-2009, the ASB program moved to a 7-point scale to provide more
opportunity for variance in leader responses. This change makes it difficult to
compare across the two cohorts (2006-2007, 2008-2009) on the Civic Action
subscale and the Interpersonal and Problem-Solving Skills subscale, which were
previously on 5-point scales.
• In 2008-2009, the ASB program moved to using “Strongly Disagree” and
“Strongly Agree” response choices. This change makes it difficult to compare
across the two cohorts (2006-2007, 2008-2009) on Personal Competency and
Social Justice scales, which previously used “Completely Disagree” and
“Completely Agree”.
• Leaders in both the 2006-2007 and 2008-2009 cohorts report significant increases
in both personal competency and social justice, supporting the effectiveness of
program leadership training in meeting two of its primary objectives.
Limitations
• In 2008-2009, the ASB program moved from a 5-point scale to a 7-point scale
and standardized the response choices across the various subscales.
• In 2008-2009 the percentage of ASB leaders and trip participants who completed
the ASB assessments at all relevant time points dropped, possibly impacting the
representativeness of the sample.
Future Research
• JMU’s ASB program will continue to collect data using the 7-point scale and
standard anchors, allowing for future comparisons across cohorts.
• JMU’s ASB program is embedding assessments within mandatory leader training
activities to ensure leaders complete the assessments at all three time points.