Improvements in Student Learning as a Result of ARC’s SLO Assessment Process:
Brief Report on Cohort I (2007-2008 vs. 2010-2011) and Cohort II (2008-2009 vs. 2011-2012)
Yuj Shimizu, PhD, Department of Planning, Research, Technology, and Professional Development, American River College, 08/01/2012
The following report reveals clear robust evidence of improvements in student learning produced as part of
ARC’s SLO assessment process.
SLO student self-assessment surveys have been part of ARC’s college-wide SLO assessment process since
2007-2008. To enable a structured and sustainable SLO assessment process, the college was divided into three
cohorts, with only a 3rd of the college conducting surveys in any given year 1. At present, two of the college’s
three cohorts have now undergone both an assessment and re-assessment cycle. In this report, these first two
cohorts were examined at an aggregate level for evidence of improvements in student learning resulting from
within ARC’s SLO assessment process.
These survey results are one form of evidence regarding achievement of student learning outcomes for a given
course. These results are analyzed using a statistical procedure to determine if one or more SLOs are rated
more negatively compared to the majority of SLOs for a given course (i.e., if a statistically significant negative
deviation exists 2). Such data indicates that action may be warranted to improve achievement for the lower rated
SLO(s) (see Figure 1).
Figure 1.
Example of a negative
deviation.
The percentage of surveyed courses that contain a significant negative deviation taken at each assessment
period reflects a broad aggregate measure of student learning. To elaborate, a high percentage would indicate
that there may be many courses in need of departmental action in order to improve student learning. By
contrast, a low percentage would indicate that there may only be a few courses in need of such action.
Importantly, a reduction in the percentage of surveyed courses that contain a significant negative deviation from
an initial assessment to a re-assessment would represent evidence of improvements in student learning produced
within ARC’s SLO process. The following report examined the assessment and re-assessment of Cohort I
(2007-2008 vs. 2010-2011) and Cohort II (2008-2009 vs. 2011-2012) to determine if such a reduction had taken
place.
RESULTS
Statistical analyses were conducted on SLO student self-assessment survey data for 450 different courses
(aggregated over sections and instructors) for all 18 departments in Cohort I and for 428 different courses for all
1
2
A full listing of departments by cohort can be found at http://inside.arc.losrios.edu/~slo/
Analyses are conducted using the Repeated-Measures Analysis of Variance (ANOVA) Statistical procedure and a standard of p < .01.
[1]
22 departments in Cohort II to determine whether improvements in student learning had occurred as a result of
ARC’s SLO assessment process. Across both cohorts, clear evidence of improvements in student learning was
obtained.
Cohort I
An analysis of the initial assessment of Cohort I (2007-2008) revealed that 26% of surveyed courses contained a
significant negative deviation. By the time of re-assessment (2010-2011), this percentage had dropped to 17%
(see Figure 2, left panel), a 9% difference. This amounts to a 35% relative improvement (reduction) from the
initial assessment to the re-assessment cycle.
At the department level, the vast majority (70%) experienced improvements across assessment cycles.
Figure 2.
Percentage of Courses Containing Signficant Negative Deviations by Assessment Year and Cohort
30
25
20
15
10
Assessment
Re-assessment
Assessment
Cohort I
Re-assessment
Cohort II
Cohort II
Similar results were obtained for Cohort II. The initial assessment of Cohort II (2008-2009) revealed that 21%
of surveyed courses contained a significant negative deviation. By re-assessment (2011-2012), this percentage
had dropped to 13% (see Figure 2, right panel), an 8% difference. This amounts to a 38% relative improvement
(reduction) from the initial assessment to the re-assessment cycle.
At the department level, of those showing changes, the majority (65%) experienced improvements across
assessment cycles (50% improved vs. 27% declined vs. 23% showed no change).
In summary, together across Cohorts I and II, solid and substantial evidence of improvements in student
learning were observed, both broadly across courses and departments as a whole. These results serve as
validation for the student self-assessment stream of ARC’s two-pronged approach to SLO assessment 3.
3
Evidence of improvements in student learning resulting from Faculty Designed Assessment will be discussed elsewhere.
[2]
Improvements in Student Learning as a Result of ARC’s SLO Assessment Process:
Report on Cohort I, 2007-2008 vs. 2010-2011
Yuj Shimizu, PhD, Department of Planning, Research, Technology, and Professional Development, American River College, 08/22/2011
ANALYSIS I
Statistical analyses were conducted on student self-assessment SLO data for 450 different
courses (aggregated over sections and instructors) for all 18 departments in Cohort I to
determine whether improvements in student learning had occurred as a result of ARC’s SLO
assessment process. Both the first (2007-2008) and second (2010-2011) cycles were analyzed.
Analysis I was conducted in two stages. First the courses were analyzed to determine whether
a statistically significant deviation(s) was present for each course. Such a deviation would
indicate that the SLOs were not evenly rated.
If the analyses returned a significant result, the data pattern was subsequently examined to
determine if, in particular, a negative deviation was present (see Figure 1). A significant
negative deviation is a clear indicator that student success on a particular student learning
outcome may need to be improved through corrective action.
Figure 1.
Example of a negative
deviation.
Importantly, a reduction in the percentage of surveyed courses that contained a significant
negative deviation from the first cycle to the second cycle would represent initial evidence of
improvements in student learning outcomes produced within ARC’s SLO process.
1
Results
In the first cycle (2007-2008), 26.32% of surveyed courses contained a significant negative
deviation. In the second cycle (2010-2011), only 16.84% of surveyed courses contained a
significant negative deviation (see Figure 2). Therefore, the absolute percentage of courses
that contained a significant negative deviation dropped by 9.48%. This amounts to a 36%
relative improvement (reduction) for the second cycle compared to the first.
Figure 2.
Cohort I: % of courses containing
significant negative deviations
30
20
10
0
2007-2008 (1st Cycle) 2010-2011 (2nd Cycle)
At the department level, the vast majority (70%) experienced improvements compared to their
first cycle.
These results represent ARC’s initial evidence of improvements in student learning outcomes at
the course level produced through the student self-assessment portion of our college’s SLO
assessment model.
ANALYSIS II
A follow up analysis was conducted (at the behest of J. Gamber) to determine whether the
specific courses to which departmental actions had been taken in an attempt to improve
student learning (as indicated by “action completed” on their implementation reports)
contained fewer significant negative deviations in 2010-2011 than was the case in 2007-2008,
thereby indicating improvement in student learning specifically for courses to which actions had
been taken.
2
Results
In 2007-2008, prior to the corrective actions, 37.5% of courses to which departments would
ultimately take action contained a significant negative deviation*. Following the corrective
actions, only 25% of these courses contained a significant negative deviation (see Figure 3).
This change represents a 12.5% absolute decrease and a 33% relative improvement (decrease)
from the first to the second cycle.
Figure 3.
Cohort I: % of courses containing sig.
neg. deviations that were acted upon
after 1st cycle
40
30
20
10
0
2007-2008 (1st Cycle)
2010-2011 (2nd Cycle)
Therefore, the results of analysis II represent ARC’s initial evidence of improvements in student
learning outcomes at the course level produced specifically through actions that were taken
toward achieving that goal. These data indicate a direct association between corrective actions
taken and subsequent improvements in student learning.
In Summary, after Cohort I went through its initial re-assessment, solid and substantial
evidence of improvements in student learning were observed, both broadly across courses and
departments as a whole, and specifically for courses in which actions were taken in an attempt
to improve student learning. These results also provide important validation for the student
self-assessment stream of ARC’s two-pronged approach to SLO assessment.
*Side note: One might wonder why only 37.5% (rather than 100%) of courses acted upon
contained a significant negative deviation to begin with when negative deviations are supposed
to be the indicator to enact corrective action. The answer is that at the outset of this process, the
college did not have the aid of a statistical test or standard by which to identify courses
containing statistically significant negative deviations. This drawback within our processes was
corrected in 2009-2010 with the adoption of the repeated-measures ANOVA statistical
procedure for identifying significant deviations and the adoption of the p < .01 standard. As
such, starting with Cohort III (2009-2010), this starting percentage is expected to be at or near
100%.
3