The Assessment of Higher Education
Learning Outcomes (AHELO)
OECD’s Study of the Scientific and Practical
Feasibility of Assessing Baccalaureate-Level
Student Learning Outcomes Across Nations
and Institutions
Lock Haven University, Pennsylvania
February 2, 2012
Charlie Lenth, SHEEO
National Project Manager
[email protected]
Introduction and Orientation
Lots of Topics—Limited Time
1. AHELO background, design and development
2. U.S., SHEEO, state and institutional
involvement
3. Current status, immediate tasks and
timelines
4. Field manuals—institutional coordination,
sampling, test administration
5. Discussion of next steps
AHELO Background, Context
and Development
• OECD background
• OECD’s education studies, surveys and analyses
– Programme for International Student Assessment (PISA)
– Institutional Management in Higher Education (IMHE)
• Changing Education Needs, Methods and Objectives
• 2006 Council of Education Ministers’ call for
postsecondary assessment, commissioned papers
and “expert” discussion to develop project design
Basic Project Objectives and Design
• Feasibility study—to determine scientific and
practical feasibility of assessing college learning
across nations
• Assess learning across and within the disciplines
or program areas at degree completion
• Minimum five diverse nations with non-scientific
sample of 10 institutions each
• Small, random sample of students with
institutions as primary unit of analysis
• Student, faculty, institutional surveys and data to
allow meaningful interpretation and analysis
U.S. Involvement in AHELO
• Research, design and development (CAE,
Peter Ewell, ETS, CHEA and ACE, others)
• No federal, state or institutional
involvement until 2009
• Invitation from OECD in 2010, and from
SHEEO to state members and institutions
• Meeting participation: Washington, New
York with CAE for Generic Skills, OECD for
GNE and NPM
U.S. Financial Support
for AHELO Participation
• Lumina Foundation is largest foundation donor
• Hewlett Foundation for initial U.S. participation;
other smaller grants later
• U.S. Department of Education
– Payment of national participation fees
– Encouragement for FIPSE competitive program
application in 2010
– Discretionary grant provided September 2011
– OECD Education Policy Committee
Country Participation in AHELO
Generic Skills
Economics
Engineering
Columbia
Egypt
Finland
Korea
Kuwait
Mexico
Norway
Slovakia
United States
Belgium
Egypt
Italy
Mexico
Netherlands
Russia
Slovakia
Australia
Canada
Columbia
Egypt
Japan
Mexico
Russia
Slovakia
Total countries: 16 plus observers and others asking to join
Overview--U.S. Participants’ Roles
• National level
– US Dept of Ed provides financial support and member of OECD
Education Governing Board
– SHEEO as National Project Manager (NPM) and national representative
in Group of National Experts (GNE)
– Foundations, stakeholders (CHEA), other interested parties
• State level
– SHEEO agency provides project leadership, coordination, budget
oversight in Connecticut, Missouri and Pennsylvania
• Institutional level
–
–
–
–
–
AHELO Institutional Coordinator and implementation “team”
Sample 200 students and administer test and surveys
Sample 40 faculty to complete 20-minute survey
Complete institutional survey by IR and other relevant offices
Access to data and determine its institutional uses
U.S. Higher Education Institutions
Participating in AHELO Feasibility Study
•
•
•
Connecticut Board of Regents for Higher Education
– Southern Connecticut State University (public regional university;
9,000 undergraduates)
Missouri Department of Higher Education
– Central Methodist University (private university; 3,500 students)
– Missouri State University (public institution; 16,000 undergraduates)
– Truman State University (public institution; 6,000 undergraduates)
– University of Central Missouri (public regional; 12,000 students)
– Webster University (private, not-for-profit; 4,000 undergraduates)
Pennsylvania State System of Higher Education (PASSHE)
– Cheyney University of Pennsylvania (public historically black
university; 1,300 undergraduates)
– Clarion University (public institution; 5,100 undergraduates)
– Lock Haven University (public institution; 5,000 undergraduates)
– Millersville University (public institution; 7,200 undergraduates)
Institutional Sampling Plan
http://www.sheeo.org/misc/ahelo/27samplingmanual.pdf
• AHELO Sampling Manual provides international
guidelines for both student and faculty samples
• Basic components:
1. Institutions provide census of “in scope” students and
faculty to NPM (SPSS.sav or Excel .xls/.xlsx preferred)
2. NPM will draw equal probability samples using
prescribed routines and submit for ACER review
3. Flexibility for stratified or “augmented” sample design
if justified and documented by institution (e.g., strata
for discipline/program areas, student characteristics,
special accommodations, etc.)
4. Review and approval by ACER Sampling Team
(led by Statistic Canada—Jean Dumais)
Sampling Steps and Timelines
1. Institutions use Sampling Manual to prepare
student and faculty sampling plans/census files
2. Coding guidelines and CIP crosswalk available at
http://www.sheeo.org/misc/ahelo/CIP-2010-to-ISCEDcrosswalk.xlsx
3. State Coordinators and NPM review and submit
all U.S. plans to ACER/Statistic Canada
4. Review by and negotiations with ACER Sampling
Team (allow at least one week)
5. Approved (or clarification) of institutional
sampling frameworks by Statistics Canada (allow
two prior to initial test administration)
Institutional Coordination Manual
http://www.sheeo.org/misc/ahelo/29TestAdministrationManual.pdf
• Manual outlines institutional guidelines applicable
to diverse international contexts
• Institutional Coordinators (ICs) undertake
training, sampling, scheduling, system testing,
beginning 1-2 months before testing dates
• ICs, State Coordinators and NPM set and approve
test dates and other scheduling parameters
• ICs recruit and train Test Administrators (TAs)
(Minimum one TA per room of 40 students)
• ICs monitor test sessions, response rates,
address needs as they emerge, work with State
Coordinators and NPM to meet project objectives
Test Administration Procedures
http://www.sheeo.org/misc/ahelo/29TestAdministrationManual.pdf
• International TA Manual is non-secure document
for distribution as necessary
• Outlines testing conditions and restrictions (e.g.,
no electronic devices other than computers
provided, access to other websites blocked)
• Access to testing platform, complete instructions,
student identifiers and testing passwords, etc.
• Testing session parameters:
– Welcome and necessary instructions – 10 minutes
– Performance tasks – 90 minutes
– Multiple choice questions – 30 minutes
– Student contextual survey – 10 minutes
Questions and Challenges
• NPM and state coordinators will seek and share
consistent answers to technical questions (e.g.,
file layout, IDs, fields, international codes)
• Institutions can work together to plan student
incentives and follow-up procedures (e.g.,
invitation, incentives, rescheduling missed testing
dates)
• Institutional and state coordinators set schedules
and provide project timelines consistent with
international guidelines and deadlines
Subsequent Steps
• U.S. lead scorer will be trained on international
guidelines, then recruit and train 5-7 U.S scorers
• National scoring done in May on secure platform
using prescribed procedures and common rubrics
• Scored assessment results analyzed and reported
by ACER collaborators and researchers
• Review by Technical Advisory Group and OECD
• Preparation, review and release of Feasibility
Study findings and recommendations
• Final project conference 10-11 December 2012
Please Take Note
• OECD and ACER are acutely aware that the entire
feasibility study rests on voluntary participation
by nations, institutions, students and faculty
• Motivation may be a challenge for students,
faculty and even institutions, but potential and
international importance are high
• Resulting data, analyses and uses are expected
to include institutional access and some
“comparative benchmarks,” depending upon
reliability, validity, and other criteria
Context and Consequences
• AHELO is part of a larger postsecondary
education agenda with implications for students,
institutions, nations and the global environment.
• Much has already been learned about both
scientific and practical feasibility for assessment
within an international setting.
• OECD and other major organizations will shape
future decisions and financial commitments on
the results of this study.
Thank you.
Charles S. Lenth
State Higher Education
Executive Officers
[email protected]
www.sheeo.org