National Institute for Learning Outcomes Assessment
March 2014
Institutional Assessment Practices Across
Accreditation Regions
Nora Gannon-Slater, Stanley Ikenberry, Natasha Jankowski, & George Kuh
nowledge accountability connection self-reflection educate action under stand communicate listen learn access quality innovation success
ngenuity intellect curiosity challenge create achievement connection self-reflection educate action under stand communicate listen learn access
uality innovation success ingenuity intellect curiosity challenge knowledge accountability connection under stand communicate listen learn
ccess quality innovation success ingenuity self-reflection educate action under stand intellect knowledge accountability connection self-reflection
ducate action under stand communicate curiosity challenge create achievement connection self-reflection curiosity challenge create achievement
onnection self-reflection knowledge accountability connection self-reflection educate action under stand communicate listen learn access quality
nnovation success ingenuity intellect curiosity challenge educate innovation success ingenuity intellect curiosity challenge create achievement
nowledge accountability connection self-reflection educate action under stand communicate curiosity challenge create achievement connection
elf-reflection under stand communicate listen learn access quality action educate action under stand communicate listen learn action under stand
ommunicate listen learn access quality innovation success ingenuity intellect curiosity challenge knowledge accountability connection access
uality self-reflection curiosity challenge create achievement learn access quality innovation success ingenuity self-reflection educate action
nder stand intellect knowledge accountability connection self-reflection educate action under stand knowledge accountability connection selfeflection educate action under stand communicate listen learn access quality innovation success ingenuity intellect curiosity challenge connection
nowledge accountability connection self-reflection educate action under stand communicate listen learn access quality innovation success ingenuity
hallenge create achievement connection self-reflection educate action under stand connection self-reflection under stand communicate listen learn
ccess quality action create achievement connection self-reflection educate action under stand communicate listen learn access quality innovation
uccess educate action communicate listen learn access quality action educate action under stand communicate educate innovation success selfeflection knowledge accountability communicate listen learn achievement connection self-reflection educate action under stand communicate listen
earn access quality innovation success ingenuity intellect access quality innovation success self-reflection curiosity challenge create achievement
onnection self-reflection under stand educate action under stand communicate listen learn action under stand communicate listen learn access
uality innovation success ingenuity curiosity challenge create achievement connection self-reflection under stand communicate listen learn
Accreditation Report
www.learningoutcomesassessment.org
National Institute for Learning Outcomes Assessment | 1 Contents
Institutional Assessment Practices Across Accreditation Regions .....3
Executive Summary.....3
Introduction.....5
Articulation of Learning Outcomes....5
Assessment Drivers....7
Multiple Measures to Assess Learning....8
Uses of Assessment....10
Communication of Assessment Results....12
Organization and Structural Support for Assessment....14
Priorities for Advancing Assessment Work....15
Implications.....16
NILOA Mission
The National Institute for Learning Outcomes
Assessment’s (NILOA) primary objective is
to discover and disseminate ways that academic
programs and institutions can productively
use assessment data internally to inform
and strengthen undergraduate education, and
externally to communicate with policy makers,
families and other stakeholders.
Acknowledgments
We very much appreciate the provosts and their
desig­nates who set aside time from their very busy
schedules to complete the questionnaire. We are
in your debt.
The NILOA Team
References.....18
Appendix A....19
Appendix B....20
NILOA
Advisory Panel.....23
About NILOA.....24
NILOA Staff.....24
NILOA Sponsors.....24
National Institute for Learning Outcomes Assessment | 2 Institutional Assessment Practices Across Accreditation Regions
Nora Gannon-Slater, Stanley Ikenberry, Natasha Jankowski, & George Kuh
Executive Summary
How are colleges and universities gathering and using evidence of student learning and how do these practices vary across
accreditation regions? To address these and related questions, the National Institute for Learning Outcomes Assessment
(NILOA), in 2013, asked the provosts/chief academic officers at all regionally accredited undergraduate degree-granting, twoand four-year public, private, and for-profit institutions in the US (n=2,781) about the assessment activities underway on their
campuses and how these institutions were using assessment results. Of those invited, provosts (or their designates) at 1,202
institutions (43%) responded. In this report based on the survey, we explore similarities and differences in assessment practices
across the seven regional accreditation bodies. In addition, by comparing responses from this 2013 survey with those obtained
from the 2009 NILOA survey, we flesh out changes in institutions’ assessment practices over time.
Of note is that this report was specifically prepared for a meeting of the heads of the regional accreditation agencies through
the Council of Regional Accrediting Commissions (C-RAC), thus this report assumes a level of background knowledge and
awareness of regional accreditation processes and regions. Further, it is worth noting that while the findings lend themselves
to consider that correlations between institutional responses might be caused by the accreditation regions, or that differences
between the regions might be due to the regional accreditors, this causal relationship is not supported in the data or a finding
of the report. A variety of factors including institutional type, size, control, and mission impact choice of assessment processes
and measures, thus it is not the case that the findings are caused by the accreditation regions alone. Further, regions vary in the
size of the institutional grouping served and thus some of the findings may reflect differences based in part on the number of
institutions which make up a region.
Findings
The most noteworthy finding from the survey is that the similarities in assessment practices across accreditation regions
outweigh the differences. While the great majority of universities and colleges tend to have learning outcomes statements that
apply to all students regardless of major, there is some variance across regions with respect to the degree to which academic
programs have articulated learning outcomes statements and whether those statements align with institution-wide learning
outcomes.
While regional accreditation and specialized/program accreditation remain the primary drivers for assessment work at colleges
and universities across all regions, survey responses suggest that increasing impetus for assessment is being driven by internal
needs, including the use of assessment evidence to support program reviews, modify curricula, revise learning goals, and otherwise improve educational processes and effectiveness. Certain drivers of assessment practice have increased in relative importance over time—such as governing board and presidential mandates, statewide or coordinating board mandates, and faculty
or staff interest in improving student learning. Although important in 2009, national calls for accountability and institutional
membership initiatives appear less important today in prompting institutional assessment of student learning.
With the growth in articulation of learning outcomes statements and the prevalence of multiple drivers of assessment work,
institutions are now using more—and more varied—approaches to assess student learning today than in 2009. The use of
internally developed assessments (e.g., locally developed measures, classroom-based performance assessments, portfolios, etc.)
has significantly increased across all regions and institutions report they find the most valuable approaches to understanding
student learning are classroom-based assessments, national student surveys, and rubrics.
While productive use of assessment evidence remains a major challenge, reported use of assessment results has significantly
increased at institutions in all accreditation regions, especially for purposes of program review, curriculum modification, institutional improvement, academic policy development, and resource allocation. Institutions report that assessment results are
more often used to guide changes in policy and practice at the course or department/program level than at the college or institutional level. While it is heartening to see the increased use of assessment evidence for internal purposes, the use of assessment
results in response to accountability requirements has also increased, regardless of region.
National Institute for Learning Outcomes Assessment | 3 Transparency of institutions’ assessment activity—with both external audiences as well as on campus—remains a concern.
Several approaches to transparency were found to be useful in different regions for sharing assessment information internally,
such as the use of assessment committees, dean’s councils, and faculty meetings or retreats. But regardless of region, statements
of outcomes for student learning were the most likely to be shared with external audiences as contrasted with student learning
outcomes results. Most campuses across regions reported that assessment of student learning had substantial support from
their institution’s current organization and governance structures, but that more work was needed to effectively use assessment
results internally to improve student learning, and to involve more faculty in student assessment through professional development opportunities.
In open-ended responses, respondents indicated a desire to balance the work and culture of assessment with preparation for
accreditation. Respondents appeared to be seeking ways to bring these two worlds together, in closer alignment. While accreditation demands may serve to move assessment work forward, the results of assessment will need to be demonstrably useful in
enhancing student learning and advancing institutional improvement if assessment is to be credible and sustainable over the
long term.
Implications
The most important takeaway from this analysis is the relative congruence of institutional assessment practices across the seven
regional accreditation agencies. While critics of regional accreditation often focus on differences and inconsistencies between
regions, on the core question of assessing student learning, the responses from provosts from region to region were generally
quite similar. This finding suggests the survey’s most noteworthy implication: Colleges and universities are generally aware
of and responsive to the collective regional accreditation priority of assessing student learning. While institutions are not
yet where they need to be regarding assessing student learning and especially in using results to improve teaching and learning,
the past few years have brought progress. As national leaders in institutional improvement and academic quality assurance,
regional accreditation bodies can benefit from even greater coordination of policies and approaches to assessment. In his new
book, Paul Gaston (2014) acknowledges the work of C-RAC and suggests additional areas for collaboration.
The findings point to a second implication that we present as a question: If regional accreditation remains the prime driver
of student learning outcomes assessment, what does this mean for regional accreditors? Our sense is that the time is
propitious now to move quality assurance to the next level, shifting the focus from the doing of assessment to the
impactful uses of assessment evidence in ways that make a genuine difference to students and institutions. Evidence
suggests that in the last few years institutions have made significant gains in the capacity to assess student learning outcomes.
Four years ago, the typical college or university used an average of three different assessment approaches at the undergraduate
level. By 2013, the average number of approaches had increased to five; and the range of assessment measures being employed
had expanded. More institutions have established student learning outcomes at the institution level and more programs have
aligned their learning outcomes with the overall institutional goals—all of which may have prompted use of more outcomes
measures. What is surprising is the increase of the types of measures used by institutions. For example, use of rubrics, classroom-based assessments, and portfolios all jumped substantially across all regions since 2009.
In all accreditation regions, assessment of student learning has moved higher on institutional agendas. More measures are being
used to assess student learning, and even now the uses of assessment appear to be shifting from mere “compliance” toward
guiding efforts to foster student success and institutional improvement. Regional accreditors now have an opportunity (and
arguably an obligation) to urge campuses to use what they know about student attainment to improve student success and
strengthen institutional performance and to support campuses in their efforts to do so—thus, increasing the value of learning
outcomes assessment to campuses and to the assurance of quality in American higher education.
National Institute for Learning Outcomes Assessment | 4 Institutional Assessment Practices Across Accreditation Regions
Nora Gannon-Slater, Stanley Ikenberry, Natasha Jankowski, & George Kuh
Introduction
How are colleges and universities gathering and using evidence of student
learning and how do these practices vary across accreditation regions? To address
these and related questions, the National Institute for Learning Outcomes
Assessment (NILOA), in 2013, asked the provosts/chief academic officers at
all regionally accredited undergraduate degree-granting, two- and four- year
public, private, and for-profit institutions in the US (n=2,781) about the assessment activities underway on their campuses and how these institutions were
using assessment results. Of those invited, provosts (or their designates) at
1,202 institutions (43%) responded. The characteristics of institutions participating in the survey generally reflect the national profile in their institutional
sectors, Carnegie classifications, and geographic regions. The responses from
institutions document a broad range of assessment activities. Some institutions
were well advanced in assessment efforts while others were just initiating this
important work.
Learning outcomes assessment is
key to addressing both affordabilty
and access issues.
(provost at a master’s institution)
This analysis examines assessment practices across accreditation regions. It also
incorporates a comparison of responses from the 2013 survey with findings from
a comparable 2009 NILOA survey. Included is a snapshot of assessment practices
in the Western Association of Schools and Colleges (WASC), the Southern Association of Colleges and Schools Commission on Colleges (SACS), the Northwest
Commission on Colleges and Universities (Northwest), the North Central Association of Colleges and Schools–The Higher Learning Commission (HLC), the
New England Association of Schools and Colleges Commission on Institutions
of Higher Education (NEASC), and the Middle States Commission on Higher
Education (Middle States).1
Of note is that this report was specifically prepared for a meeting of the heads of
the regional accreditation agencies through the Council of Regional Accrediting
Commissions (C-RAC), thus this report assumes a level of background knowledge and awareness of regional accreditation processes and regions. Further, it is
worth noting that while the findings lend themselves to consider that correlations
between institutional responses might be caused by the accreditation regions, or
that differences between the regions might be due to the regional accreditors,
this causal relationship is not supported in the data or a finding of the report. A
variety of factors including institutional type, size, control, and mission impact
choice of assessment processes and measures, thus it is not the case that the findings are caused by the accreditation regions alone. Further, regions vary in the
size of the institutional grouping served and thus some of the findings may reflect
differences based in part on the number of institutions which make up a region.
Articulation of Learning Outcomes
Similar to the national snapshot (reported in Knowing What Students Know and
Can Do: The Current State of Student Learning Outcomes Assessment in US Colleges
And Universities), most institutions within each of the accreditation regions
reported having articulated institution-wide learning outcomes that apply to all
undergraduates, regardless of major. In every accreditation region the reported
articulation of institution-wide learning goals for undergraduate students
increased during the past four years (Figure 1).
For purposes of analysis, WASC includes two regional accrediting associations that of the Accrediting Commission for Community and Junior Colleges (ACCJC)
and that of Senior College and University Commission.
1
National Institute for Learning Outcomes Assessment | 5 Figure 1. Institution adoption of institution-wide student learning outcomes
from 2009 to 2013.
In addition to institution-wide outcomes, some 70% of participating schools also
reported having student learning outcomes defined for all academic programs,
although there was considerable variation between accreditation regions with
respect to the degree to which learning outcomes for academic programs are or
are not aligned with institutional outcomes expected of all students (Figure 2).
• Institutions within WASC, SACS, Northwest, and Middle States were
more likely to have learning outcomes defined for all academic programs
that aligned with institutional learning outcomes.
• Some 33% of institutions in Northwest reported that some departments,
schools, or programs have defined field-specific learning outcomes that
align with institutional learning outcomes.
• Institutions within NEASC were least likely to report having learning
outcomes for academic programs in alignment with institutional
learning outcomes.
Figure 2. Program adoption and alignment of learning outcomes to institutionlevel outcomes.
National Institute for Learning Outcomes Assessment | 6 Assessment Drivers
A variety of forces prompt institutions to gather information about student
learning (Figure 3). Similar to findings reported in 2009 and the 2013
national report, regional and specialized/program accreditation bodies
remain the prime drivers of assessment work at colleges and universities
across all regions. Furthermore, with few exceptions, the assessment drivers
listed here are equally important across all regions.
• State mandates for assessment were more important to institutions in
SACS than to those in Middle States or NEASC.
With so many competing demands
on faculty time, assessment needs
to be sustainable and manageable.
For that to happen it needs to be
useful.
(provost at a doctoral institution)
• Participation in consortium or multi-institution collaborative efforts
along with statewide governing or coordinating board mandates for
assessment were slightly more important in driving assessment in
member schools of HLC.
One encouraging change since 2009 is that more campuses seem to be interested in using assessment evidence to guide internal improvement efforts
through program review, curriculum modification, revision of learning goals,
and other efforts. This begs an important question: Might regional accrediting
bodies grasp this opportunity and fashion policies to reinforce the use of assessment for purposes of improvement as well as accountability in ways that enable
institutions to harvest greater impact and benefit from the investment in the
assessment of student learning?
No
Minor
Importance Importance
Moderate
Importance
High
Importance
Figure 3. Drivers of assessment, by region.
National Institute for Learning Outcomes Assessment | 7 While most of the forces that prompt student learning outcomes assessment
have remained relatively unchanged in importance from 2009 to 2013 (Table
1)1, the findings show some variation across the regions:
• Governing board and presidential mandates for assessment increased in
importance in SACS, Northwest, HLC, NEASC, and Middle States
schools.
• The influence of statewide or coordinating board mandates for assessment increased in all regions except SACS.
• Faculty or staff interest in assessment became more important for HLC
institutions
• Worthy of note is that national calls for accountability and institutional
membership initiatives were less important in 2013 than in 2009 in
prompting student learning outcomes assessment in all regions, but
especially for institutions within SACS, HLC, NEASC, and Middle
States regions.
Multiple Measures to Assess Learning
Experts in the assessment world generally agree that no single tool or approach
can fully capture the complete picture of student learning. Fortunately, there are
many more assessment tools and approaches available today than there were a
decade ago (Borden & Kernel, 2010), and American colleges and universities
across all regions are using them more frequently. Furthermore, the number of
internal assessments (e.g., locally developed measures, classroom-based performance assessments, portfolios, etc.) has also significantly increased across all
regions (Table 2).2
Appendix A contains information on methodology and Appendix B contains tabulated results for assessment drivers
by region.
2
Appendix B contains tabulated results of assessment uses by region.
1
National Institute for Learning Outcomes Assessment | 8 Institutions in accreditation regions indicated similar preferences for engaging
a variety of kinds of assessment measures (Figure 4). Regardless of accreditation
region, institutions reported that the most valuable approaches for assessing
undergraduate student learning outcomes are (in this order) classroom-based
assessment, national student surveys, and rubrics.
• Institutions within HLC and SACS were more likely to use measures
of general knowledge and skills.
• Alumni surveys were more frequently used by member schools within
Middle States and NEASC.
Figure 4. Assessment approaches used at the institution level to represent
undergraduate student learning.
National Institute for Learning Outcomes Assessment | 9 Uses of Assessment
Institutions within all regions reported not only increases in the types of assessment measures they use but also more use of assessment results for a range of
purposes. However, gathering information about student accomplishment can
be an empty exercise if the data are not used in meaningful and productive ways.
Compliance with regional and program accreditation expectations were the
most frequently reported “uses” of assessment, suggesting an opportunity for
regional accreditors to support shifting the purpose for assessing student learning
to institutional responsibility for improving student success and institutional
performance. Use of assessment for a range of other reasons (beyond accreditation) also saw substantial increases from 2009 to 2013 (Table 3)3.
As the responses shown in Table 3 indicate, institutions across all accreditation
regions reported more use of assessment results in program review, curriculum
modification, institutional improvement, academic policy development, and
resource allocation. In addition to these internal uses of assessment evidence,
external accountability requirements also increased significantly for all institutions, regardless of region.
3
Many faculty struggle with
determining how to conduct
a proper assessment and
then how to use the results,
and many of the disciplinary
meetings are very broad and
not specific in this regard.
(provost at a master’s
institution)
Appendix B contains tabulated results of assessment uses by region.
National Institute for Learning Outcomes Assessment | 10 Figure 5 shows that the uses of assessment results tend to be similar across regions.
• Across regions, institutions frequently reported using assessment
evidence for various improvement-related tasks, such as curriculum
modification, strategic planning, policy development, and faculty
development.
• Institutions were least likely to use assessment results to communicate
with prospective students, families, and alumni.
• Member schools of WASC were more likely to use assessment results for
program review than schools in other regions.
• Institutions within HLC, Northwest and SACS were more likely to use
assessment results for benchmarking than NEASC.
Initiative overload is a very
real problem. Shrinking state
funding compounds this by
reducing staff and increasing
administrative requirements at
the same time.
(provost from a public
institution)
• Institutions in WASC and SACS were more likely to use assessment
results for resource allocation and budgeting.
Not at all
Some
Quite a bit
Very Much
Figure 5. Extent to which assessment results are used for each purpose, by region.
Assessment results were more often used to guide changes in policy and practice at the course or department/program level than at the college or institution
level (Figure 6). Institutions within regions were equally likely to use assessment results for making changes at the institution level or in specific curricular
requirements or courses. At the school/college levels or department/program
levels, only minor differences existed in how assessment results are being used.
National Institute for Learning Outcomes Assessment | 11 • Middle States member institutions reported more use of assessment
results at the school or college level and department/program level than
did schools within NEASC.
Very Much
Quite a bit
Some
Not at all
Figure 6. Extent to which changes are made based on assessment results, by
region.
Communication of Assessment Results
In all accreditation regions, institutions appear to be beginning to communicate
assessment results, both within the institution and to the general public. The strategies employed by institutions across regions were quite similar (Figure 7).
• For all regions except Middle States, the most effective means of communicating assessment results within the institution were presentations of
assessment findings at faculty meetings or retreats.
• Assessment committees were also reported as an effective way to communicate assessment results, especially for institutions within Middle States
and equally important to faculty meetings or retreats for member schools
of HLC.
• Institutions within SACS found the dean’s council more effective for
communicating assessment results than did other accreditation regions.
• Institutions within Middle States and Northwest regions reported
communicating assessment results through websites to be more effective
than did other regions.
• Member schools of SACS and NEASC indicated that email updates were
more effective means of communicating assessment results than did other
regions.
National Institute for Learning Outcomes Assessment | 12 Figure 7. Percentage of institutions effective internal communication, by region.
How are assessment activity and evidence communicated beyond the campus?
Across all accreditation regions, the most commonly shared information with
external audiences was institutionally endorsed learning outcomes for all students
(Figure 8).
• SACS member schools were less likely to make public student learning
outcomes statements than campuses in other regions.
• Institutions within Middle States, HLC, and WASC were more likely to
make assessment resources, current assessment activities, and assessment
plans publicly available.
• NEASC schools were least likely to make assessment information public.
Not at all
Some
Figure 8. Extent to which assessment information is publicly available, by region.
Quite a bit
Very Much
National Institute for Learning Outcomes Assessment | 13 Organization and Structural Support for Assessment
Most institutions reported that student learning outcomes assessment work was
supported by their institution’s current organization and governance structures.
Such support was strongest for institutions in SACS, HLC, and Middle States
regions.
Figure 9. Percentage of institutions rating supportiveness of organization and
governance structure.
The value of assessment lies
not in the program or an
individual course that is
assessed, but in understanding
that the real benefit of
outcomes mastery is adequate
preparation for success at
the next level. This means
changing how we work - how
classes are scheduled, how
we advise, how we develop
programs, and revise courses
- everything is different for us
with learning in mind. That’s
the value [of the assessment]
conversation we need to share
internally and externally.
Overall, institutions reported a wide range of supports for assessment activities
on campus. The most important elements in all regions included
(provost at an associate’s
institution)
• institutional research office and personnel;
• institutional policy/statements related to assessing undergraduate
learning;
• existence of an institutional assessment committee;
• faculty engagement and involvement in assessment; and
• availability of professional staff dedicated to assessment.
Minor differences existed across the accreditation regions with regard to the
kinds of structures and conditions respondents considered supportive of assessment (Figure 10).
• SACS member schools were more likely to indicate institutional policies
and statements about assessing undergraduate learning.
• HLC schools were more likely to report assessment committees.
• SACS schools were more likely to note an office of institutional research
and the availability of assessment personnel, data management systems,
and software.
• WASC schools were more likely to say funds targeted for outcomes
assessment.
• Middle States, SACS, and WASC were more likely than HLC, NEASC,
and Northwest institutions to indicate professional assessment staff and
significant involvement of student affairs staff.
National Institute for Learning Outcomes Assessment | 14 Not at all
Some
Quite a bit
Very Much
Figure 10. Extent to which various items support assessment activity, by region.
Priorities for Advancing Assessment Work
A list of the top ten areas of institutional need to advance assessment work
was identified by accreditation region, with significant commonality among
eight priorities. At the top of each region’s list were providing professional
development for faculty and staff and furthering faculty use of results. Less
important but noteworthy was increasing faculty involvement in assessment
in general. Also, frequently selected by institutions were the needs for additional
financial or staff resources, greater institutional assessment staff capacity, better
technologies and analytics, greater sharing and access to assessment results, and
more valid and reliable assessment measures. The relative emphasis on each of
these priorities tended to vary across regions (Figure 11).
• NEASC member institutions emphasized needing more valid and reliable assessment measures of student learning and the need for more
faculty involved in assessment.
• Institutions in the Northwest accreditation region tended to stress the
need for greater institutional assessment staff capacity and additional
financial or staff resources.
• SACS institutions emphasized needing more professional development
for faculty and stronger administrative and leadership support.
• WASC campuses focused on the need for better assessment technology
and analytics.
National Institute for Learning Outcomes Assessment | 15 Figure 11. Percentage of institutions indicating areas of need to advance assessment, by region.
Implications
While critics of regional accreditation often focus on differences and inconsistencies between regions, on the core question of assessing student learning,
evidence reported here suggests an overall consistency. Institutions are aware of
and responding to the collective regional accreditation priority of assessing student
learning. While institutions are not yet where they need to be regarding assessing
student learning and using results, advances have been made since 2009 in the
process of assessment, from stating desired institutional outcomes for all students
to using more and a wider variety of methods to obtain assessment results to
reporting and using assessment information.
Four years ago, the typical college or university used an average of three different
assessment approaches at the undergraduate level. By 2013, the average number
of approaches had increased to five; and the range of assessment measures being
employed had expanded. That schools are using more measures is not surprising
as more institutions have established student learning outcomes at the institution
level and more programs have aligned their learning outcomes with the overall
institutional goals—all of which may have prompted use of more outcomes
National Institute for Learning Outcomes Assessment | 16 measures. What is surprising is the increase of the types of measures used by
institutions. For example, use of rubrics, classroom-based assessments, and portfolios all jumped substantially across all regions since 2009. It is also important
to note that the motivation for assessing student learning outcomes appears to
be shifting from mere “compliance” toward a focus on using the information to
foster student success and guide institutional improvement.
Yes, some regional differences exist, but in absolute terms these variances do not
appear to be major. For example, with respect to the communication of assessment results within the institution—at faculty meetings or retreats, in assessment
committees, and as part of the work of dean’s councils—these modes of sharing
assessment information are happening, but they were identified as effective means
of communicating assessment results to different degrees across regions. Institutions within some regions are more likely to make assessment plans, resources,
and activities publicly available. There are regional differences with respect to
both structural and organizational support for assessment as well as priorities for
advancing assessment work. Still, the similarities across regions far outweigh the
contrasts.
Colleges and universities
must evolve from a culture
of compliance to a culture
of evidence-based decision
making.
As national leaders in institutional improvement and academic quality assurance,
regional accreditation bodies can benefit from sharing views and coordinating
policies and approaches to assessment. More than 1,000 invited responses to
open-ended questions were received, and 600 of them related to regional accreditation. Respondents expressed concern that the work done on assessment, driven
by preparation for a regional accreditation visit, often lost momentum following
reaffirmation of accreditation status. Respondents said they hoped to get through
their accreditation process with positive reviews but also expressed a very real
fear that they would not meet expectations. Furthermore, respondents expressed
concerns for balancing the creation of a culture of assessment within their institutions and “doing assessment because it matters for teaching and learning,” with
the perception that assessment is “something we do because it is a requirement
for accreditation.” The perception of assessment as something done to comply
with regional accreditation mandates was viewed as a barrier to faculty engagement and involvement with assessment processes and as limiting the potential of
assessment to improve student learning. Overall, the respondents were asking for
ways to bring the purposes of accountability and improvement into closer alignment so that while accreditation moves assessment work forward, institutions
retain it as a valued tool for its usefulness to student learning and institutional
improvement.
Evidence suggests that over the last four years institutions have made significant
gains in their capacity to assess student learning outcomes. Even now, apart from
compliance with external demands, institutions appear to be shifting in the direction of institutional improvement—toward needs and interests of faculty and
staff, concerns about academic effectiveness and value, and interests of presidents
and governing boards. Regional accreditors should seize the moment and find
ways to use the momentum of this shift to enhance the role of learning outcomes
assessment in improvement efforts on campuses and in the assurance of quality
in American higher education.
National Institute for Learning Outcomes Assessment | 17 References
Borden, V., & Kernel, B. (2010). Measuring quality in higher education: An inventory of instruments, tools, and resources. Retrieved
from Association for Institutional Research website at http://apps.airweb.org/surveys/default.aspx
Gaston, P. L. (2014). Higher education accreditation: How it’s changing, why it must. Sterling, VA: Stylus.
Kuh, G. D., Jankowski, N., Ikenberry, S. O., & Kinzie, J. (2014). Knowing what students know and can do: The current state of
student learning outcomes assessment in U.S. colleges and universities. Urbana, IL: University of Illinois and Indiana University,
National Institute for Learning Outcomes Assessment (NILOA).
National Institute for Learning Outcomes Assessment | 18 Appendix A
Data Collection and Analysis
The 2013 NILOA national survey of chief academic officers was conducted by the Center for Survey Research at Indiana University
between April and September 2013. The sample included provosts or chief academic officers at the 2,781 regionally accredited
undergraduate degree-granting institutions listed in the Higher Education Directory, published by Higher Education Publications,
Inc. A total of 1,202 institutions completed the survey for a response rate of 43%.
The survey was administered primarily online, with the initial invitation followed by three email reminders; a paper copy of the
questionnaire was mailed to those who had not completed the survey after the third email reminder. Many of the questions were
used previously in the NILOA 2009 questionnaire. Additional questions were revised or added, informed by changing practices in
the field and input from NILOA’s National Advisory Panel, a select group of assessment experts, and a focus group of chief academic
officers convened during the January 2013 Association of American Colleges and Universities (AAC&U) meeting. A copy of the
final 2013 survey may be found here: http://www.learningoutcomeassessment.org/knowingwhatstudentsknowandcando.html
The characteristics of participating colleges and universities in terms of accreditation region (and other variables) were generally
similar to the national profile.
Table A1
Accreditation region: 2013 participating institutions compared with national profile
Accreditation Region
2013
Current National
Middle States
16%
17%
NEASC
8%
7%
HLC
38%
35%
Northwest
6%
5%
SACS
24%
27%
WASC
8%
9%
As with the 2009 survey, we asked respondents to identify their position within the institution if they were not the provost
who received the initial invitation to complete the survey. Table A2 outlines that among about three quarters of the responding
institutions the provost or someone in the provost’s office completed the questionnaire.
Table A2
Survey 2013 respondents by position
Position
Provost/CAO (including 136 assistant/associate provost)
Director of assessment (or person responsible for assessment)
Dean (or assistant/associate dean)
%
74%
18%
8%
N
N = 883
N = 223
N = 96
Frequency tables were produced for accreditation region; questionnaire items 1, 2, 3, 5, 13, 14, 15, and 16 were analyzed using the
cross tabs procedure in SPSS (21), which yielded chi-square tables that identified statistically significant differences. These results
were further analyzed to determine whether selected responses differed across institutions within regions. Items 6, 8, 10, 11, and
12 have interval scales and one-way analysis of variance (ANOVA) was used to identify statistically significant differences between
various groupings of institutions. A post-hoc test with Bonferroni correction was applied to control for an inflated type-I error
rate. Statistically significant results were those at the .05 level or below. In addition, items that were paired from the 2009 and
2013 survey iterations were examined for change over time. Finally, responses to items 4 and 17–20 (the open-ended questions)
were reviewed by two NILOA researchers, specifically with a focus on accreditation region. Broad codes were then developed in
conversation about the general reading of the responses and thematic codes were applied to identify accreditation-related comments.
National Institute for Learning Outcomes Assessment | 19 Appendix B
Survey Results by Region
Table B1
Relative emphasis of assessment drivers from 2009 to 2013, by region.
Assessment Drivers
Faculty or staff
interest
Institutional
improvement
Governing board/
president
Statewide/
coordinating
mandate
Regional
accreditation
Program
accreditation
National calls for
accountability
Institutional
membership
initiatives
WASC
2009 2013
3.0
3.3
SACS
2009 2013
3.1
3.2
Northwest
2009 2013
3.1
3.3
HLC
2009 2013
3.1
3.2
NEASC
Middle States
2009 2013 2009 2013
2.9
3.0
3.1
3.2
3.6
3.6
3.6
3.6
3.5
3.4
3.5
3.5
3.6
3.4
3.7
3.6
2.6
3.0
2.5
3.1
2.1
2.9
2.4
2.8
2.4
2.6
2.5
3.1
2.0
2.7
2.1
2.7
1.9
2.6
2.0
2.4
1.8
2.4
1.9
2.4
4.0
4.0
3.9
3.9
3.8
3.8
3.7
3.8
3.7
3.7
3.9
3.9
3.6
3.7
3.7
3.8
3.4
3.5
3.6
3.7
3.7
3.6
3.6
3.8
3.0
2.8
3.0
2.6
2.8
2.6
3.0
2.7
2.9
2.4
3.1
2.8
2.2
1.9
2.3
2.0
1.9
1.8
2.7
1.9
2.2
1.6
2.2
1.8
Response options include No Importance, Minor Importance, Moderate Importance, High Importance
National Institute for Learning Outcomes Assessment | 20 Table B2
Percentage of each type of assessment administered from 2009 to 2013, by region.
Assessment Type
National student
surveys
Locally developed
surveys
General knowledge
and skills measures
Classroom-based
performance
assessments
Externally situated
performance
assessments
Portfolios
Rubrics
Alumni surveys
Employer surveys
WASC
2009 2013
60%
77%
SACS
2009 2013
84%
88%
Northwest
2009 2013
83%
92%
HLC
2009 2013
87%
89%
NEASC
Middle States
2009 2013 2009 2013
83% 92% 81% 89%
44%
56%
51%
62%
34%
57%
50%
60%
23%
58%
44%
60%
27%
41%
56%
59%
32%
39%
47%
53%
23%
28%
26%
36%
5%
60%
27%
69%
15%
68%
18%
62%
9%
60%
21%
65%
3%
33%
13%
36%
3%
55%
6%
39%
0%
59%
12%
39%
5%
25%
24%
5%
48%
66%
57%
28%
7%
26%
37%
18%
39%
75%
61%
47%
6%
21%
29%
16%
44%
67%
59%
38%
6%
29%
40%
19%
39%
68%
65%
45%
5%
16%
21%
2%
47%
73%
77%
50%
7%
26%
41%
15%
43%
65%
75%
41%
National Institute for Learning Outcomes Assessment | 21 Table B3
Extent to which student learning assessment results are used for various purposes, by region.
WASC
SACS
Northwest
HLC
NEASC
Middle States
Assessment Use
2009 2013 2009 2013 2009 2013 2009 2013 2009 2013 2009 2013
Regional
2.9
4.0
3.5
3.8
3.2
3.8
3.2
3.7
3.1
3.4
3.3
3.8
accreditation
Program
3.1
3.7
3.4
3.7
3.1
3.4
3.2
3.6
3.1
3.4
3.4
3.7
accreditation
Program review
2.3
3.6
2.3
3.2
2.2
3.0
2.2
3.1
1.9
3.0
2.3
3.3
Curriculum
modification
External
accountability
requirements
Learning goals
revisions
Institutional
improvement
Academic policy
development or
modification
Strategic planning
Institutional
benchmarking
Resource allocation
and budgeting
Trustee/governing
board deliberations
2.3
3.2
2.5
3.0
2.4
2.9
2.5
2.9
2.3
2.9
2.6
3.1
2.2
3.1
2.3
3.2
2.3
2.9
2.4
3.1
2.1
3.1
2.3
3.4
2.5
3.0
2.8
3.0
2.6
3.0
1.6
2.6
2.4
2.8
2.8
3.1
2.0
2.8
2.2
2.9
2.2
2.7
2.2
2.7
2.0
2.5
2.3
2.9
1.5
2.6
1.7
2.5
1.6
2.5
1.6
2.6
1.6
2.5
1.7
2.7
2.3
1.8
2.6
2.5
2.7
2.1
2.7
2.6
2.4
2.1
2.8
2.6
2.5
2.1
2.6
2.5
2.3
2.0
2.3
2.0
2.6
2.1
2.8
2.5
1.5
2.3
1.6
2.2
1.4
2.2
1.5
2.0
1.3
1.8
1.7
2.4
2.2
2.2
2.4
2.3
2.2
2.3
2.3
2.2
2.1
2.0
2.3
2.2
Response options include: N/A (not shown), Not at All, Some, Quite a Bit, Very Much.
Written-in responses for the “other” category included new program development or program specific benchmarking.
NILOA National Advisory Panel
Joseph Alutto
Provost
The Ohio State University
Trudy W. Banta
Professor
Indiana University-Purdue University
Indianapolis
Wallace Boston
President and CEO
American Public University System
Molly Corbett Broad
President
American Council on Education
Judith Eaton
President
Council for Higher Education Accreditation
Richard Ekman
President
Council of Independent Colleges
Mildred Garcia
President
California State University Fullerton
Susan Johnston
Executive Vice President
Association of Governing Boards
Stephen Jordan
President
Metropolitan State University - Denver
Mary Kalantzis
Dean, College of Education
University of Illinois Urbana-Champaign
Paul Lingenfelter
President
State Higher Education Executive Officers
George Mehaffy
Vice President
Academic Leadership and Change
American Association of State Colleges and
Universities
Charlene Nunley
Program Director
Doctoral Program in Community College
Policy and Administration
University of Maryland University College
Kent Phillippe
Associate Vice President, Research and
Student Success
American Association of Community Colleges
Randy Swing
Executive Director
Association for Institutional Research
Carol Geary Schneider
President
Association of American Colleges and
Universities
Michael Tanner
Chief Academic Officer/Vice President
Association of Public and Land-grant
Universities
Belle Wheelan
President
Southern Association of Colleges and Schools
Ralph Wolff
President
Western Association of Schools and Colleges
NILOA Mission
NILOA’s primary objective is to
discover and disseminate ways that
academic programs and institutions
can productively use assessment data
internally to inform and strengthen
undergraduate education, and externally to communicate with policy
makers, families and other stakeholders.
Comments and questions about this
paper should be sent to
[email protected].
Ex-Officio Members
Timothy Reese Cain
Associate Professor
University of Georgia
Peter Ewell
Vice President
National Center for Higher Education
Management Systems
Stanley Ikenberry
President Emeritus and Regent Professor
University of Illinois
George Kuh
Director, National Institute for Learning
Outcomes Assessment
Adjunct Professor, University of Illinois
Urbana-Champaign
Chancellor’s Professor Emeritus, Indiana
University
Jillian Kinzie
Senior Scholar, NILOA; Associate Director,
Indiana University
National Institute for Learning Outcomes Assessment | 23 About NILOA
•
The National Institute for Learning Outcomes Assessment (NILOA) was established in December 2008.
•
NILOA is co-located at the University of Illinois and Indiana
University.
•
The NILOA website contains free assessment resources and can be found at http://
www.learningoutcomesassessment.org/.
•
The NILOA research team has scanned institutional websites, surveyed chief
academic officers, and commissioned a series of occasional papers.
•
One of the co-principal NILOA investigators, George Kuh, founded the National
Survey for Student Engagement (NSSE).
•
The other co-principal investigator for NILOA, Stanley Ikenberry, was president
of the University of Illinois from 1979 to 1995 and of the American Council of
Education from 1996 to 2001.
NILOA Staff
NATIONAL INSTITUTE FOR LEARNING OUTCOMES ASSESSMENT
Stanley Ikenberry, Co-Principal Investigator
George Kuh, Co-Principal Investigator and Director
Peter Ewell, Senior Scholar
Jillian Kinzie, Senior Scholar
Pat Hutchings, Senior Scholar
Timothy Reese Cain, Senior Scholar
Paul Lingenfelter, Senior Scholar
Natasha Jankowski, Assistant Director and Research Analyst
Robert Dumas, Research Analyst
Katie Schultz, Research Analyst
Carrie Allen, Research Analyst
Jelena Pokimica, Research Analyst
Nora Gannon-Slater, Research Analyst
NILOA Sponsors
Lumina Foundation for Education
The Teagle Foundation
University of Illinois, College of Education
Produced by Creative Services | Public Affairs at the University of Illinois for NILOA. 10.032
National Institute for Learning Outcomes Assessment | 24 tellect curiosity challenge educate innovation success ingenuity intellect curiosity challenge create achievement knowledge accountability connection
lf-reflection educate action under stand communicate curiosity challenge create achievement connection self-reflection under stand communicate
ten learn access quality action educate action under stand communicate listen learn action under stand communicate listen learn access quality
novation success ingenuity intellect curiosity challenge knowledge accountability connection access quality self-reflection curiosity challenge create
hievement learn access quality innovation success ingenuity self-reflection educate action under stand intellect knowledge accountability connection
lf-reflection educate action under stand knowledge accountability connection self-reflection educate action under stand communicate listen learn
cess quality innovation success ingenuity intellect curiosity challenge connection knowledge accountability connection self-reflection educate action
der stand communicate listen learn access quality innovation success ingenuity challenge create achievement connection self-reflection educate
tion under stand connection self-reflection under stand communicate listen learn access quality action create achievement connection self-reflection
ucate action under stand communicate listen learn access quality innovation success educate action communicate listen learn access quality action
ucate action under stand communicate educate innovation success self-reflection knowledge accountability communicate listen learn achievement
nnection self-reflection educate action under stand communicate listen learn access quality innovation success ingenuity intellect access quality
novation success self-reflection curiosity challenge create achievement connection self-reflection under stand educate action under stand communicate
ten learn action under stand communicate listen learn access quality innovation success ingenuity curiosity challenge create achievement connection
lf-reflection under stand communicate listen learn access quality action create achievement connection self-reflection educate action under stand
mmunicate listen learn access quality innovation success educate action communicate listen learn access quality action educate action under stand
eate achievement connection self-reflection under stand communicate listen learn access quality action create achievement connection self-reflection
ucate action under stand communicate listen communicate educate innovation success self-reflection knowledge accountability connection self
flection educate action under stand communicate listen learn access quality innovation ingenuity intellect connection self-reflection under stand
mmunicate listen learn access quality action create achievement connection self-reflection educate action under stand communicate listen learn
National Institute for Learning Outcomes Assessment
For more information, please contact:
National Institute for Learning Outcomes Assessment (NILOA)
University of Illinois at Urbana-Champaign
340 Education Building
Champaign, IL 61820
learningoutcomesassessment.org
[email protected]
Phone: 217.244.2155
Fax: 217.244.5632