As seen in the March/April 2009 issue of Trusteeship magazine.
Reprinted with permission of AGB.
Quality
Ensuring
Educational
Means Assessing Learning
Here’s an advertisement you’ll never see:
“attend our university, where we’re not
sure what—or even if—our students
are learning. we have dedicated faculty
members and great facilities, so we think
we’re doing a good job, but the truth is that
we have no evidence to support that.”
S
S
Take"8":4
Without assessing student learning
outcomes, there is no reliable way to
measure and demonstrate an institution’s
educational quality.
Because accrediting bodies increasingly
are seeking evidence of student-learning
outcomes, more governing boards and
top administrators will need to take
assessment seriously in the future.
The process of creating a “culture of
evidence” encourages colleges and universities to reflect on what aspirations
they have for their students and then to
generate evidence concerning how well
they are meeting those goals.
30
TrusTeeship
By C aT h r a e l K a z i n a n d
d av i d G . pay n e
Promotional coPy like that might draw
some double takes, but not many applicants.
yet, allowing for some hyperbole, this is the
situation at many well-regarded institutions of
higher education.
As many experienced board members
know, given the national debate in recent
years over institutional accountability,
learning—the heart of the educational
enterprise—is often treated as a by-product of other more measurable processes.
If the instructors are well-qualified, we
assume that students are learning. Or we
measure the quality of an institution by
inputs, such as the admissions-test scores
of entering students. But the measure
of educational quality should be determined by outputs or outcomes—by what
students have learned as a result of their
educational experience.
The fact is, no matter how excellent the
curricula, no matter how stellar the faculty, no matter how talented the student
body, without assessing student learning
outcomes, there’s no reliable way to measure and demonstrate educational quality.
That’s a shame, especially for all of
those colleges and universities with
excellent curricula, stellar faculty, and
talented students. More unsettling still
is that without a way to demonstrate that
learning actually occurs in college classrooms, the entire postsecondary enterprise rests on a shaky foundation.
Numerous assessment measures
have been developed in recent years,
and several higher-education associations have developed plans to help their
member institutions demonstrate their
accountability. Yet evidence of student
outcomes is still less commonly gathered
and publicized than “input data” such
as endowments and students’ scores on
admissions tests. More governing boards
and top administrators will be facing the
need to take assessment seriously in the
future, however, given that accrediting
bodies increasingly are seeking evidence
of student-learning outcomes. But as
many institutions have found out, obtaining sound evidence of student learning—documenting what students know
and can do—is not easy and requires an
institution-wide commitment.
What About Grades?
At this point, some people will respond,
“We have a way to assess student learning: It’s called grades.” And it’s true
that grades make it possible to draw
some inferences about relative student
performance in a specific class. But does
­Beverly’s B- in Introduction to Psychology mean that she learned more than
Curt, who got a C+? What if Beverly and
Curt took the course from different professors? Or at different institutions?
Student outcomes are integral to
institutional quality, and that quality is
fundamental to fulfilling the missions
of the institutions that governing boards
oversee. Without a reliable measure of
what students have learned, however,
it is impossible to determine if the
institution is doing a good job. It is also
impossible to improve—or at least to
determine whether improvements that
have been made are making a difference.
And without standardized measures, it
is impossible for stakeholders to make
conclusions about the relative strengths
and weaknesses of different programs
or institutions.
In other words, providing and improving quality demands measurement. This
is not a new idea in business, which is
accustomed to using metrics to target,
measure, and improve quality. But in
higher education, the notion of using
metrics to define targets for student
learning and determine whether they
have been met is sometimes considered
antithetical to academic freedom and
institutional autonomy.
In part, this attitude derives from the
belief that learning is too complex, too
ineffable to be measured by a single
test. This is true, which is why institutions should create a balanced assessment portfolio that reflects the multiple
goals for learning at the institution. But
too often the problem is that faculty
members and administrators are simply
uncertain about how to describe their
goals for student learning and transform
them into measurable outcomes.
Creating a
‘Culture of Evidence’
Over the past two years, the Educational Testing Service has developed a
framework for helping colleges and universities define aspirations for student
learning and translate them into measurable outcomes. Assisted by a panel of
M a rc h /A p r i l 20 0 9
31
32
educators, researchers, and educational
policy advocates, we published a series
of white papers under the collective title
A Culture of Evidence (www.ets.org/
cultureofevidence).
In the first paper, we looked at the
consequences for prospective students,
employers, and institutions of not
­having a systematic approach to collecting ­evidence about student learning. ­Without such evidence, we noted,
it is difficult to see where changes are
needed, to examine the effects of such
changes, and to engage in a process of
continuous improvement.
In particular, we looked at the need for
evidence about student learning in four
main areas:
• Workplace readiness and
general skills;
• Discipline-specific knowledge
and skills;
In the third paper, we proposed a
method for helping an institution create
a culture of evidence on its campus. This
method is based on ETS’s own approach to
test development, which begins by asking
what claims we want to be able to make
on the basis of a test. Similarly, the first
step in the process of building a culture of
evidence is to determine what the institution wishes to be able to say about what its
graduates know and can do. This step is
critically important and must precede any
attempt to find or build the “right test.”
Rather than impose an answer, we
encourage institutional stakeholders to
examine their institution’s mission and
work together to define the claims that
they want to make about their graduates.
For example, do they want to be able to say
that their students are prepared to succeed
in the global economy? If so, what does
that mean? That they can communicate
• Are the major employers of our graduates satisfied with new hires’ knowledge
and skills? If not, in what areas are they
lacking?
• How do graduates of our institution
compare with those of peer institutions?
Once the claims that the institution
currently wants to make about its graduates have been defined and agreed upon,
the next steps are to conduct an “assessment audit” to determine what types
of assessments the college is currently
conducting and whether they, in fact,
support the claims that the institution
has identified as important. If additional
evidence is needed—if there are gaps
between what the institution would like
to be able to claim about its students’
learning and the claims the available
data can support—then the institution
needs to consider other assessments that
might be useful.
• “Soft” skills, such as teamwork, communication, and creativity; and
• Students’ engagement in their
own learning.
In the second Culture of Evidence
report, we looked at what is currently
available to assess these areas. We
focused on the 12 most widely used standardized assessments, such the National
Survey of Student Engagement (NSSE)
and ETS’s Major Field Tests (MFTs), and
provided information about the instruments that would be useful to colleges
beginning or already engaging in developing a culture of evidence.
effectively to a variety of audiences? That
they have an understanding and appreciation of diverse cultures? If so, what would
be the evidence for those claims? Such
evidence could come in many forms, from
standardized assessments to presentations
and portfolios.
Governing boards can be invaluable
in this process by asking key questions
of administrators and faculty members.
For example:
• How will the administration and faculty members use the information gained
from assessments to actually improve
teaching and learning?
Once assessments have been administered and results are available, the
crucial next step is for the institution to
examine the results in light of the aspirations articulated in the first step. To what
degree have the aspirations for student
learning been realized? Rigorous honesty
is important for the process to be useful.
At the same time, it is essential to create
an atmosphere in which the results can
be examined without fear of blame or
retribution. In such an environment, it
is possible to ask what changes need to
be made to address learning shortfalls
and to ensure that current successes
Trusteeship
Student outcomes are integral
to institutional quality, and that
quality is fundamental to fulfilling
the missions of the institutions
that governing boards oversee.
Without a reliable measure of
what students have learned,
however, it is impossible to
determine if the institution is
doing a good job.
for teaching and other professions, and
scores on graduate- and professionalschool admissions tests.
Whether the United States remains
economically competitive will depend in
part on the ability of our postsecondary
institutions to prepare high-performing
graduates equipped for the global workplace. America’s colleges and universities
are still the world standard in higher
education. The challenge they face now
is to build on their strengths while focusing more on educational outcomes and
demonstrating to students the value of a
higher education—that is, on creating a
culture of evidence of learning.
Once an institution has established
and sustained a culture of evidence,
it will be able to claim truthfully and
with pride:
“Attend our university, where we create
graduates who are prepared to succeed
continue. The results of the data analysis
need to be communicated to all stakeholders, including board members.
As this effort continues over time, the
institution creates a culture of evidence.
It is a not linear process, but rather cyclical and iterative.
in the global economy. And we have the
evidence to prove it.” n
Aspirations and Autonomy
The “culture of evidence” process places
a premium on the autonomy of the institution and faculty members, the institution’s mission, and the populations and
stakeholders the institution serves. The
process also encourages colleges and
universities to reflect on what aspirations
they have for their students and then to
generate evidence that indicates how well
they are meeting those goals. At the same
time, using reliable and valid standardized assessments as one component of the
assessment portfolio makes it possible to
benchmark the institution’s performance
against national and regional peers.
To round out the picture of institutional effectiveness, trustees might periodically ask the administration for other
types of information on how graduates
are faring, such as job-placement rates,
pass rates on certification examinations
Authors: Cathrael Kazin is director of strategic
relations and David G. Payne is associate vice
president of the Educational Testing Service
(ETS).
T’SHIP LINKS: William F. Massy, Steven W.
Graham, and Paula Myrick Short, “Getting
a Handle on Academic Quality.” September/
October 2007. Stephen Klein and Steve
Uhlfelder, “Should College Students Be Tested
to Hold Institutions Accountable for Student
Learning?” May/June 2006. Dary T. Erwin,
“The ABCs of Assessment.” March/April 2003.
M a rc h /A p r i l 20 0 9
Copyright © 2009 by Educational Testing Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS). 11707
33