Retreat on Student Learning
and Assessment, Level I
September 24-25, 2009
The Westin Long Beach
Long Beach, CA
Retreat Handouts
Retreat on Student Learning and Assessment, Level I
Table of Contents
Agenda / Schedule ……………………………………………………………………………………………….
2
Westin Floor Plan ………………………………………………………………………………………………… 4
Mentor Biographies ………………………………………………………………………………………………
5
Attendee Directory …………………………………………………………………………...
10
Plenary: Assessment in Learning-centered Institutions (M. Allen) …………………………………………...
16
Lecture/Discussion 1: An Outcomes-based Assessment Model for General Education (A. Driscoll) ……..
22
Lecture/Discussion 2: Assessment for Student Affairs Staff and Other Campus Professionals (M. Allen) ... 33
Lecture/Discussion 3: Direct and Indirect Assessment Methods (B. Wright) …….………………………...
59
Direct Assessment Methods – A Close-Up Look ………………………………………………...
66
Indirect Assessment Methods – A Close-Up Look ………………………………………………
77
Lecture/Discussion 4: Unique Issues in Assessment for Community Colleges (F. Trapp) ………………...
83
Handout for Unique Issues in Assessment for Community Colleges …………………………..
100
Lecture/Discussion 5: Developing and Applying Rubrics (M. Allen) ………………………………………..
119
Lecture/Discussion 6: Analyzing Evidence of Student Learning to Improve Our Practice (A. Driscoll) ….
136
Lecture/Discussion 7: The Administrators’ Role in Assessment of Student Learning (B. Wright) ………...
140
Assessment Tips for the Top ………………………………………………………………………
141
Top Ten Ways to Kill Your Assessment Program ………………………………………………...
144
The Administrators’ Role in Assessment of Student Learning: Setting the Context for Success
145
L/D 8: Assessment for Community College Career and Technical Educational Programs (F. Trapp) …….
148
Handout for Assessment for Community College Career and Technical Education …………..
163
Plenary: The Learning-centered Institution: Curriculum, Pedagogy and Assessment for Student Success
(A. Driscoll)
Resources …………………………………………………………………………………………………………
185
Bibliography ………………………………………………………………………………………..
206
Rubrics - ACSCU……………………………………………………………………………………
211
205
Program Learning Outcomes Rubric ……………………………………………………
212
General Education Assessment Rubric ………………………………………………….
214
Portfolio Rubric ……………………………………………………………………………
218
Capstone Rubric …………………………………………………………………………..
220
Program Review Rubric …………………………………………………………………..
222
Educational Effectiveness Framework …………………………………………………..
224
Rubrics – ACCJC …………………………………………………………………………………...
226
Rubric Table September 2007 ……………………………………………………………..
227
Online Resources; SLOs, Rubrics, & Assessment ………………………………………………
230
Assessment Quickies: Student Learning Outcomes Assessment in Ten Easy Steps …………..
233
Notes …….. …………………………………………………………………………………………
235
2010 WASC Academic Resource Conference – Save the Date…………………………………….
Back
Cover
Retreat on Student Learning and Assessment, Level I
SCHEDULE/PROGRAM
Thursday, September 24, 2009
9:00 – 10:00 am
Arrival, check-in, registration
10:00 - 11:30 am
Welcoming remarks Salon B
Plenary: Assessment in Learning Centered Institutions (M. Allen) Salon B
Introduction of mentors
11:30 –12:15pm
Meet in mentor groups and schedule appointments
12:15 – 1:15 pm
Lunch in teams Salon A
1:15 – 2:45 pm
Lecture/Discussions:
1. An Outcomes-based Assessment Model for General Education (A. Driscoll) Salon C
2. Assessment for Student Affairs Staff & Other Campus Professionals (M. Allen) Salon D
3. Direct and Indirect Approaches to Assessment (B. Wright) Barcelona/Casablanca
4. Unique Issues in Assessment for Community Colleges (F. Trapp) Salon B
2:45-3:00 pm
Snack Break Salon B
3:00-4:30 pm
Lecture/Discussions:
5. Developing and Applying Rubrics (M. Allen) Salon D
6. Analyzing Student Learning to Improve Our Practice (A. Driscoll) Salon C
7. The Administrator’s Role in Assessment of Student Learning (B. Wright)
Barcelona/Casablanca
8. Assessment for Community College Career and Technical Educational Programs
(F. Trapp) Salon B
4:30 – 6:00 pm
Work Session: Team planning / Appointments with mentors
6:00 pm
Dinner on your own
2
Friday, September 25, 2009
7:00-8:30am
Breakfast and appointments with mentors Salon A
8:30-10:00 am
Plenary: The Learning-centered Institution: Curriculum,
Pedagogy and Assessment for Student Success (A. Driscoll) Salon B
10:00 – Noon
Work Session: Team planning / Appointments with mentors
Noon – 1:00 pm
Lunch; submit questions for Q&A Salon A
1:00 – 1:45 pm
Q & A (Mentor Panel) Salon B
1:45 – 2:00 pm
Snack Break Salon B
2:00 – 4:00 pm
Mentor Group Final Session –Teams present progress on their projects
4:00 pm
Retreat ends
* * * * * * * * * * * * * * * * * * Lecture/Discussions * * * * * * * * * * * * * * * * * * * * *
1. An Outcomes-Based Assessment Model for General Education This workshop will provide a practical guided
opportunity for participants to design outcomes-based assessment components of a model for general education,
tailored to their institutional purposes and mission. (A. Driscoll) Salon C
2. Assessment for Student Affairs Staff and Other Campus Professionals This session is designed for campus
professionals who want to learn more about assessment. (M. Allen) Salon D
3. Direct and Indirect Approaches to Assessment Participants will review criteria for selecting among assessment
methods and the strengths and limitations associated with a variety of commonly used techniques.
(B. Wright) Barcelona/Casablanca
4. Unique Issues in Assessment for Community Colleges Participants will discuss several aspects of the
community college in higher education, the nature of the students served by those institutions, and the implications for
assessment of learning outcomes that flow from those considerations. Practical suggestions and illustrations for
assessment work will be offered. (F. Trapp) Salon B
5. Developing and Applying Rubrics Participants will review rubric examples, consider strategies for developing
rubrics, and learn how to use rubrics for teaching, grading, and assessment. (M. Allen) Salon D
6. Analyzing Student Learning to Improve Our Practice This workshop will engage participants
in an analysis process for reviewing student work as evidence of achieving learning outcomes. Participants will discuss
implications of the process for improving teaching and learning. (A. Driscoll) Salon C
7. Administrators and Assessment of Student Learning It’s true that faculty need to take primary responsibility for
assessment, but administrators also have a critical role to play in creating an environment where good assessment is
possible and leads to real improvement. (B. Wright) Barcelona/Casablanca
8. Assessment for Community College Career and Technical Educational Programs Participants will consider
ways to articulate program-level learning outcomes, analyze the curricular design of the program and explore several
common ways to assess behavioral and performance learning outcomes for competency-based education situations in
community colleges. (F. Trapp) Salon B
3
4
MENTOR
BIOGRAPHIES
5
WASC Retreat on Student Learning and Assessment
Mentor Biographies
Mary Allen
Dr. Mary Allen is a consultant in higher education, specializing in assessment and
accreditation. She is the former Director of the California State University Institute for
Teaching and Learning and a Professor Emerita of Psychology from California State
University, Bakersfield. Mary has published books on the assessment of academic
programs and general education, and she has offered assessment presentations and
workshops at AAHE, AAC&U, and WASC conferences. She is a sought after speaker,
consultant, and workshop presenter and has worked with over 120 colleges, universities,
and college systems.
Contact info:
Email: [email protected]
Jan Connal
Dr. Jan Connal has engaged in a variety of educational research activities over a long
career at Cerritos College. In her current faculty capacity, she co-coordinates the campus
SLO assessment activities, facilitates faculty classroom research and inquiry projects,
serves as the internal evaluator for the campus Title V grant, and chairs the
Developmental Education Committee (which is strongly promoting assessment, faculty
inquiry and evidence-based effective practices). Jan is also active statewide currently as
a Faculty Inquiry Network coach, with special emphasis on assessment and evaluation for
18 campus-based faculty inquiry projects, and previously as a member of the Steering
Committee for the Basic Skills Initiative (BSI) in California’s community colleges.
Other professional positions held include Dean for Institutional Advancement and
Planning, Dean of Educational Support Services, Director of Research, Development and
Planning. Jan holds a PhD in Educational Psychology (Research Methods and
Evaluation) from UCLA, MA in Experimental Psychology from CSU Fullerton, and BA
in Psychology from Chapman (College) University.
Contact info:
Email: [email protected]
Amy Driscoll
Dr. Amy Driscoll retired as Director of Teaching, Learning, and Assessment at
California State University, Monterey Bay, and was most recently a Consulting Scholar
with the Carnegie Foundation for the Advancement of Teaching. Previously, Amy was
the Director of Community/University Partnerships at Portland State University, where
she initiated the community-based learning and capstone aspects of the university’s
innovative curriculum. Dr. Driscoll has presented at AAHE, AAC&U, WASC, and
National Assessment Institute conferences. She has also mentored more than 60
institutions in their development and implementation of institutional assessment and/or
community engagement. Her most recent book is Developing Outcomes-based
Assessment for Learner-centered Education, A Faculty Introduction co-authored with
Swarup Wood, a chemistry professor, and published by Stylus (2007).
Contact info:
Email: [email protected]
6
WASC Retreat on Student Learning and Assessment
Mentor Biographies
Lynda Harding
Ethelynda Harding is Professor Emerita of Biology at California State University,
Fresno. Formerly director of of Teaching, Learning, and Technology, she had primary
responsibility for faculty professional development and academic technology. She helped
implement Fresno State's outcomes assessment program, coordinated program review,
and, as Accreditation Liaison Officer, led the most recent accreditation process. She has
presented regionally and nationally on topics including outcomes assessment, faculty
development, faculty personnel policies, instructional multimedia, and microbial ecology.
Contact info:
Email: [email protected]
John A. Hughes
Dr. John Hughes began his teaching career at The Master’s College (TMC) in the fall 1981
as the director of the College’s Teacher Education department. Since 1995, he has served as
the Vice President for Academic Affairs. Throughout his tenure at TMC he has been
involved in various aspects of the institution’s self-study and accreditation processes. He led
the faculty through the development and implementation of the College’s institutional
assessment plan. He worked with TMC’s information technology (IT) staff in the design and
development of the Assessment Information Management System (AIMS) which is used to
collect, store, summarize, and analyze assessment data relating to program-level student
learning outcomes. For three years he was a member of the WASC Proposal Review
Committee, and has also had the opportunity to serve on a number of CPR and EER visit
teams.
Contact Info:
The Master’s College
(661) 259-3540
Email: [email protected]
Robert Pacheco
Robert Pacheco, Director of Research, Development and Planning and Accreditation
Liaison Officer at Barstow College. He is member of the Executive Board of the RP
Group of California. Previously, Bob was a tenured faculty and a member of the
California State Academic Senate Committee on SLOs and Accreditation. He was an AtLarge Representative in the college's Academic Senate and is a member of Barstow
College’s Strategic Planning Committee, SLO Assessment Committee and the
Matriculation Committee.
Contact info:
Director of Research, Development and Planning
Barstow College
Email: [email protected]
7
WASC Retreat on Student Learning and Assessment
Mentor Biographies
Frederick P. Trapp
Fred Trapp is well known in California having served on the steering committee for the
California Assessment Initiative and provided numerous workshops on learning
outcomes and assessment on behalf of the Research and Planning Group. He has also
presented at numerous regional and national conferences, including the Assessment
Forum sponsored by the American Association for High Education. Until his retirement
in December 2007 Fred was employed as the Administrative Dean for Institutional
Research and Academic Services at the Long Beach Community College District for 26
years. In that role he provided campus leadership for he assessment of student learning
outcomes from the curriculum design, instructional delivery and research perspectives.
He currently serves as an evaluator for the Accrediting Commission for Community and
Junior Colleges (ACCJC) within the Western Association of Schools and Colleges and is
the Senior Associate with the Cambridge West Partnership, LLC.
Contact info:
Fred Trapp
5739 E. Hanbury Street, Long Beach, CA 90808-2049
562 429-6996 (home)
Email: [email protected]
Gary Williams
Dr. Gary J. Williams is the Instructional Assessment Specialist at Crafton Hills College in
Yucaipa, California. He has 20 years of experience in higher education at a variety of institutions,
large and small, public and private. His range of experience spans student services, residence life,
international student advising, first year experience, college success, student engagement student
learning outcomes, and instructional assessment. He currently serves on the RP Group &
Academic Senate for California Community Colleges (ASCCC) SLO Assessment Cooperative
Committee, and has served on ASCCC Accreditation and Student Learning Outcomes
Committee, as well as the Basic Skills Initiative Program Committee for the California
Community Colleges.
Dr. Willams earned his Doctorate in Education (Ed.D.) from UCLA with a focus on assessment,
organizational change, and organizational culture. He also holds a Master's degree in Educational
Psychology from Marist College in New York, and a Bachelor of Arts in English Literature from
Bates College in Maine. He lives in Riverside, California with his wife Kyoko and 2 children,
Harrison and Mako. His hobbies include Taekwondo and playing the Okinawan Shamisen.
Contact info:
Gary J. Williams, Ed.D.
Instructional Assessment Specialist
Crafton Hills College
11711 Sand Canyon Road
Yucaipa, CA 92399
Tel: (909) 389-3567
Email: [email protected]
8
WASC Retreat on Student Learning and Assessment
Mentor Biographies
Swarup Wood
Swarup Wood is an Associate Professor of Chemistry for an economics and policy infused
environmental science degree program at California State University at Monterey Bay. He is
chair of CSUMB’s general education faculty learning communities and for the past eight years
Swarup has served as CSUMB Assessment Fellow, working closely with CSUMB’s Center for
Teaching, Learning, and Assessment. He has given many workshops and oral presentations
nationally (AAHE and AAC&U) on how to develop learning outcomes, criteria, and standards,
and on his research of what faculty learn from their collaborative review of student work. His
recent book with Amy Driscoll, Developing Outcomes-based Assessment for Learner-centered
Education shares much of CSUMB’s experience as the campus developed and refined learning
outcomes, developed criteria and standards for learning outcomes, and conducted collaborative
faculty assessment of student work. Swarup is currently very involved in CSUMB’s work
towards reaccreditation.
Contact info:
Swarup Wood Ph.D. , Professor of Chemistry
California State University at Monterey Bay
Office Phone 831-582-3926
Email: [email protected]
Barbara Wright
Barbara Wright is an Associate Director at the Senior Commission of the Western
Association of Schools and Colleges, where she leads educational programming efforts in
addition to working with individual campuses. She served for over 25 years as a faculty
member in German at the University of Connecticut, before retiring in 2001. Although
her graduate training was in German language and literature, her interests expanded over
the years to include language acquisition, women's studies, curricular reform, general
education, and assessment. From 1988 to 1990 she directed a FIPSE-funded project to
assess a new general education curriculum at UConn, and from 1990 to 1992 she served
as director of the American Association for Higher Education's Assessment Forum. From
1995 to 2001 she was a member of the New England Association of Schools and
Colleges' Commission on Institutions of Higher Education, and she has participated in
team visits for several regional accreditors. Barbara would be pleased to consult with
campus teams on all aspects of assessment, from basic principles and key questions
through choice of methods, evidence gathering, interpretation, and use of results for
program improvement. She is especially interested in qualitative approaches to the
assessment of general education's more challenging, seemingly ineffable goals. She
believes that faculty are more willing to engage in assessment when assessment practices
correspond to their highest ambitions for students’ intellectual and personal development.
She received her PhD from Berkeley.
Contact Info:
Barbara D. Wright, Associate Director
Western Association of Schools and Colleges
Email: [email protected]
9
ATTENDEE
DIRECTORY
10
Report Name: Workshop Attendee Directory
16-Sep-2009
Report Date:
Record Count: 176
Event# : 757797 (24-Sep-09) - Status: Active
Event Title : WASC Workshop: Retreat on Student Learning and
Assessment, Level I
full name (first middle last)
job title
company/organization
email
Nicola Acutt
Associate Dean of programs
Presidio Graduate School
[email protected]
Eddie Afana
Acting Dean, Research and Planning
Los Angeles Trade Technical College
[email protected]
Mohammad Amin
Professor
National University
[email protected]
Kim Anderson
SLO Coordinator
Long Beach City College
[email protected]
Eileen Apperson
SLO Coordinator/Program Review Chair
Reedley College
[email protected]
Scott Ashmon
Assistant Professor of Biblical Languages
Concordia University
[email protected]
Elizabeth Atkinson
Associate Dean of Faculty
Linfield College
[email protected]
Michael Barber
Prof. of Theology
John Paul the Great Catholic University
[email protected]
William Barrick
Professor
The Master's Seminary
[email protected]
Debrea Bean
Associate Provost
National University
[email protected]
Ron Benton
Asst. VP, Administrative Services
Colorado Christian University
[email protected]
Janna Bersi
Associate VP, Academic Resource Mngt &
Planning
Bookstore Manager
CSU Dominguez Hills
[email protected]
Camilla Betwell
College of the Marshall Islands
[email protected]
Roberta A. Bigelow
Associate Dean, CLA
Willamette University
[email protected]
Andreea Boboc
Assistant Professor
University of the Pacific
[email protected]
Joan Bouillon
Dean of Academic Affairs
The Art Institute of California - Sacramento
[email protected]
Marie Boyd
Curriculum/SLO Coordinator
Chaffey College
[email protected]
La Shawn Brinson
Professor
LA Southwest College
[email protected]
Devon Brooks
Associate Dean for Faculty Affairs
USC School of Social Work
[email protected]
Ronald W. Brown
Interim, Assoc. Vice Chancellor
Lone Star College System
[email protected]
Robert Burns
Department Chair
Diablo Valley College
[email protected]
Irv Busenitz
Academic Vice President
The Master's Seminary
[email protected]
Ana Luisa Bustamante
Dept. Chair
Phillips Graduate Institute
[email protected]
Deborah Buttitta
Dept. Chair
Phillips Graduate Institute
[email protected]
Angela Caballero de Cordero
Noncredit Matriculation Coordinator/Counselor
Allan Hancock College
[email protected]
Anthony Cadavid
SLO Facilitator
East Los Angeles College
[email protected]
Stacey Caillier
Director, Teacher Leadership Program
High Tech High Graduate School of Education
[email protected]
Thomas Camacho
Administrative Coordinator
USC School of Social Work
[email protected]
Jomi Monica Capelle
Student Services Support
College of the Marshall Islands
[email protected]
Daniel Cardenas
President
The Art Institute of California Sunnyvale
[email protected]
Moya Carter
Associate Dean, Student Affairs
Pitzer College
[email protected]
Fred Chapel
Core Faculty
Antioch University Los Angeles
[email protected]
Byron En-pei Chung
President
The Art Institute of California-San Francisco
[email protected]
Catherine Collins
Professor of Rhetoric & Media Studies
Willamette University
[email protected]
Maria Dolores Costa
Director, Faculty Development
California State University, Los Angeles
[email protected]
West Coast University
[email protected]
Debi Gerger Debi Gerger
David Debrum
Safety & Security Director
College of the Marshall Islands
[email protected]
Sarah M. Dennison
AVP Assessment/Institutional Effectiveness
Education Management Corporation
[email protected]
Nancy Deutsch
Staff Development Coordinator
Cypress College
[email protected]
Jose Dial
Dean of Academic Affairs
College of the Marshall Islands
[email protected]
Dwight Doering
Professor of Education
Concordia University
[email protected]
Qingwen Dong
Professor of Communication
University of the Pacific
[email protected]
Charles Dunn
Associate Professor of Mathematics
Linfield College
[email protected]
RegOnline (Copyright 1996-2009 - All rights reserved)
Page :
1 of 5
11
Report Name: Workshop Attendee Directory
16-Sep-2009
Report Date:
Record Count: 176
Event# : 757797 (24-Sep-09) - Status: Active
Event Title : WASC Workshop: Retreat on Student Learning and
Assessment, Level I
full name (first middle last)
job title
company/organization
email
Helen Easterling Williams
Dean
Azusa Pacific University
[email protected]
Paul Eastup
Media & Arts Div Chair
Marymount College
[email protected]
Antonia Ecung
Dean, Academic Affairs
Porterville College
[email protected]
Siobhan Fleming
Associate Vice Chancellor-Research &
Institutional Effectiveness
SLO Coordinator
Lone Star College System
[email protected]
Patricia Flood
Los Angeles Mission College
[email protected]
Judy Foster
Institutional Effectiveness Coordinator
Diablo Valley College
[email protected]
Michelle Fowles
Dean, Research and Planning
Los Angeles Valley College
[email protected]
WILLIAM FRANKLIN
AVP STUDENT SUCCESS
CSU Dominguez Hills
[email protected]
Barbara Fuller
Dept. Chair
Phillips Graduate Institute
[email protected]
Jose Alfredo Gallegos
Assistant Research Analyst
East Los Angeles College
[email protected]
Leslie Gargiulo
Academic Dean
Allied American University
[email protected]
Joanne Gartner
Director of Curriculum
LA College International
[email protected]
Irene Girton
Associate Dean for Arts & Humanities
Occidental College
[email protected]
Shelley Glickstein
Dean of Academic Affairs
The Art Institute of California--LA
[email protected]
Roger Gomez
President
The Art Institute of California - Sacramento
[email protected]
George F Gonzalez
San Jacinto College
[email protected]
Randy Grant
Director of Research & Institutional
Effectiveness
Professor of Economics
Linfield College
[email protected]
Allison Guerin
Academic Operations Manager
Presidio Graduate School
[email protected]
Bradley Hale
Associate Professor
Azusa Pacific University
[email protected]
La Vonne Hamilton
Institutional Research
LA Southwest College
[email protected]
Brian Timothy Harlan
Senior Director, Institutional Research &
Assessment Group
Director of Admissions
Occidental College
[email protected]
John Paul the Great Catholic University
[email protected]
Faculty Director or Service Learning and
Learning Communities
California State University, Los Angeles
[email protected]
West Coast University
[email protected]
Cherron R Hoppes
Dean, Undergraduate Programs
Golden Gate University
[email protected]
Heather Hubbert
Assistant Dean of Students
California Baptist University
[email protected]
Behzad Izadi
Professor
Cypress College
[email protected]
Isabel Izquierdo
Instructor
Diablo Valley College
[email protected]
Herschel Jackson
Director, Student Life and Leadership
Estrella Mountian Community College
[email protected]
Jeremiah Jackson
Prof. of Business
John Paul the Great Catholic University
[email protected]
Karen Jackson
VP Administration
Phillips Graduate Institute
[email protected]
Veronica Jaramillo
SLO Coordinator
East Los Angeles College
[email protected]
Patrick Jefferson
Dean
LA Southwest College
[email protected]
Young-Jun Jeon
Dean of Business Affaris
Shepherd University
[email protected]
Michelle R. Johnson
Institutional Research Coordinator
Reedley College
[email protected]
Angela Jones
Associate Dean of Academic Affairs
The Art Institute of California-San Francisco
[email protected]
Phillip Jones-Thomas
Professor
LA Southwest College
[email protected]
K. Jimmy Jimmy Juge
Assistant Professor
University of the Pacific
[email protected]
Martin Joseph Harold
Michelle Hawley
Errin Heyman
Miriam Kahan
West Coast University
[email protected]
Susan Keller
Professor
Western State University
[email protected]
Shalom Kim
Dean of Planning and Assessment
Shepherd University
[email protected]
Erin King-West
Faculty
Phillips Graduate Institute
[email protected]
Lisa Anne Kramer
Director of Assessment and Evaluation
Golden Gate University
[email protected]
RegOnline (Copyright 1996-2009 - All rights reserved)
Page :
2 of 5
12
Report Name: Workshop Attendee Directory
16-Sep-2009
Report Date:
Record Count: 176
Event# : 757797 (24-Sep-09) - Status: Active
Event Title : WASC Workshop: Retreat on Student Learning and
Assessment, Level I
full name (first middle last)
job title
company/organization
email
Lucinda Kramer
Associate Professor
National University
[email protected]
Deonne Kunkel
Instructor, Eng. Dept.
Chabot College
[email protected]
Susan Lamb
Vice President of Office of Instruction
Diablo Valley College
[email protected].
Bryan Lamkin
Professor
Azusa Pacific University
[email protected]
Lora Lane
SLO Assessment Coordinator
Los Angeles Harbor College
[email protected]
Melinda R Lester
Dean of Academic Affairs
The Art Institute of California - Orange County
[email protected]
Muriel Lopez Wagner
Director of Institutional Research
Pitzer College
[email protected]
Marilyn K. Maine
Department Chair
Los Angeles Trade Technical College
[email protected]
Gary B. Martin
SLO Coordinator
Cosumnes River College
[email protected]
Randy Martinez
Professor
Cypress College
[email protected]
Leslyn J. McCallum
Professor/SLO Coordinator
San Jose City College
[email protected]
Cathie McClellan
Associate Professor
University of the Pacific
[email protected]
Scott McClintock
Assistant Professor
National University
[email protected]
J. Cynthia McDermott
Chair
Antioch University Los Angeles
[email protected]
Ryan McIlhenny
Professor
Providence Christian College
[email protected]
Caren Meghreblian
Dean of Academic Affairs
The Art Institute of California-San Francisco
[email protected]
Holly Menzies
Professor
California State University, Los Angeles
[email protected]
Patty Meyer
Faculty
American Film Institute
[email protected]
Dianne Moore
VP of Nursing Operations
West Coast University
[email protected]
Jason Moore
Executive Vice President/Provost
Pioneer University
[email protected]
Kim P Moore
President
Pioneer University
[email protected]
Susan Mun
Institutional Researcher
San Diego Mesa College
[email protected]
Janice Novak
Business Instructor
Chabot College
[email protected]
Dawn Nowacki
Professor of Political Science
Linfield College
[email protected]
Karen Nowak
Dean of Academic Affairs
The Art Institute of California - Hollywood
[email protected]
Allison Ohle
Chief of Staff
High Tech High Graduate School of Education
[email protected]
Charles E. Osiris
Dean,Student Services/Counseling &
Matriculation
SLO Facilitator
Allan Hancock College
[email protected]
Mona Panchal
East Los Angeles College
[email protected]
Stephen Parmelee
Assistant Professor of English
Pepperdine University
[email protected]
Mark David Parsons
Assistant Professor
Claremont School of Theology
[email protected]
Parviz Partow-Navid
Professor and Director of Student Services
California State University, Los Angeles
[email protected]
Deborah Rena Paulsen
Chair, Arts/Media/Humanities
Los Angeles Mission College
[email protected]
Linda Perez
San Jacinto College
[email protected]
Joe Petricca
Information Management Liaison Business
Analyst
Executive Vice Dean
American Film Institute
[email protected]
Yohan Pyeon
Dean of Academic Affairs
Shepherd University
[email protected]
Susan Regier
Div. Chair Language Arts
Porterville College
[email protected]
Elena Reigadas
Psychology Professor
Los Angeles Harbor College
[email protected]
Gregory J. Riley
Professor
Claremont School of Theology
[email protected]
Jelena N. Ristic
Assistant Dean, Undergraduate Programs
Golden Gate University
[email protected]
Lori Roberts
Professor
Western State University
[email protected]
Troy Roland
President
National Polytechnic College of Science
[email protected]
Constance Rothmund
Dean of Academics
National Polytechnic College of Science
[email protected]
Carlos Royal
Social Sciences Dept
Marymount College
[email protected]
RegOnline (Copyright 1996-2009 - All rights reserved)
Page :
3 of 5
13
Report Name: Workshop Attendee Directory
16-Sep-2009
Report Date:
Record Count: 176
Event# : 757797 (24-Sep-09) - Status: Active
Event Title : WASC Workshop: Retreat on Student Learning and
Assessment, Level I
full name (first middle last)
job title
company/organization
email
Amanda Ryan-Romo
SLO Facilitator
East Los Angeles College
[email protected]
Joe Safdie
SLO Coordinator
San Diego Mesa College
[email protected]
Ramona Santiesteban
Director, Disability Resources
Estrella Mountian Community College
[email protected]
Vasemaca Savu
Instructor
College of the Marshall Islands
[email protected]
Jon F. Schamber
University of the Pacific
[email protected]
Alan Scher
Director of Educational Effectiveness &
Assessment
Faculty
Phillips Graduate Institute
[email protected]
Anita Schutz
Instructor
College of the Marshall Islands
[email protected]
Fred Scott
Solutions Consultant
LiveText
[email protected]
Michael Semenoff
Institutional Research, Director
Marymount College
[email protected]
Austin Shepard
Director, Academic Enrichment Programs
Estrella Mountian Community College
[email protected]
Brian Simpson
Associate Professor
National University
[email protected]
Katrina Sitar
Assistant to the Dean of Faculty
Pitzer College
[email protected]
Joe Slowensky
[email protected]
Keith Smith
Director of Assessment and Strategic Curricular Chapman University
Initiatives.
Assistant Professor
University of the Pacific
Laura Soloff
President
The Art Institute of California--LA
[email protected]
Carole Splendore
Learning Assessment Coordinator
Chabot College
[email protected]
Rebecca Stein
SLO Coordinator
Los Angeles Valley College
[email protected]
Rosalinda Sumaoang
Instructor-Developmental Ed
College of the Marshall Islands
[email protected]
Duncan Sutton
Dir. Music/Worsip Ministries
The Salvation Army, CFOT
[email protected]
Wayne Tikkanen
Faculty Director of General Education
California State University, Los Angeles
[email protected]
Pogban Toure
Professor
LA Southwest College
[email protected]
Elizabeth Trebow
Dept. Chair and ALO
Phillips Graduate Institute
[email protected]
Bill Tsatsoulis
GVP - Western Region
Education Management Corporation
[email protected]
Paulina Van
Assessment Director
Samuel Merritt University
[email protected]
Dan Van Voorhis
Assistant Professor of History
Concordia University
[email protected]
Obed Vazquez
Instructor
Diablo Valley College
[email protected]
Reuben Veliz
Business/Economics Dept
Marymount College
[email protected]
Tom Vessella
SLO Cordinator
Los Angeles Trade-Technical College
[email protected]
Tom Vitzelio
SLO Coordinator/Instructional Specialist
Chaffey College
[email protected]
Michael Vlach
Professor
The Master's Seminary
[email protected]
Nancy Wada-McKee
Asst. Vice President, Student Affairs
California State University, Los Angeles
[email protected]
Dan Walden
Dean
LA Southwest College
[email protected]
Kerry Walk
Pitzer College
[email protected]
Tracy Ward
Associate Dean of Faculty for Academic
Administration
Dean of Academic Services
California Baptist University
[email protected]
Rachel Westlake
Division Dean of Math & Computer Science
Diablo Valley College
[email protected]
Lisa Wheland
Director of Institutional Effectiveness
the Art Institute of California - Orange County
[email protected]
Ted Wieden
Senior Dean of Curriculum and Instruction
Diablo Valley College
[email protected]
June Wiley
Vice President of Academic Affairs
LA College International
[email protected]
David Willoughby
Faculty
Phillips Graduate Institute
[email protected]
Joan Woosley
University Registrar
California State University, Los Angeles
[email protected]
Kristine Wright
Professor
LA Southwest College
[email protected]
Sally Wu
Behavioral Sciences Dept
Marymount College
[email protected]
Mina Yavari
Professor
Allan Hancock College
[email protected]
RegOnline (Copyright 1996-2009 - All rights reserved)
[email protected]
Page :
4 of 5
14
Report Name: Workshop Attendee Directory
16-Sep-2009
Report Date:
Record Count: 176
Event# : 757797 (24-Sep-09) - Status: Active
Event Title : WASC Workshop: Retreat on Student Learning and
Assessment, Level I
full name (first middle last)
job title
company/organization
email
Glenn Yoshida
SLO Coordinator
LA Southwest College
[email protected]
Randy Zarn
AVP Student LIfe
CSU Dominguez Hills
[email protected]
George Zottos
Outcomes Assessment Specialist
[email protected]
Heather Zuber
Professor
Riverside Community College/Moreno Valley
Campus
Western State University
RegOnline (Copyright 1996-2009 - All rights reserved)
[email protected]
Page :
5 of 5
15
Plenary
Assessment in Learningcentered Institutions
Mary Allen
16
Assessment in Learning-Centered Institutions
Mary J. Allen, [email protected], September 2009
Why so much emphasis on assessment?
•
Academic
Program Goals
Curriculum
How Students
Learn
Course
Structure
Pedagogy
Course Delivery
Faculty
Instructional
Role
Assessment
Campus
Accreditation Expectations
•
A Learning-Centered Focus
Students learn:
• The concepts, theories, research findings, techniques, and values of the discipline
• How to integrate what they learn to solve complex, real-world problems
• An array of core learning outcomes, such as collaboration, communication, critical
thinking, information literacy, and leadership skills
• Cohesive program with systematically-created opportunities to synthesize, practice,
and develop increasingly complex ideas, skills, and values—to develop deep and
lasting learning
• Students construct knowledge by integrating new learning into what they already
know.
• Feedback guides student improvement.
• Students can learn, clarify ideas, and develop alternative perspectives through
reflection and interpersonal interactions.
• Students engage in learning experiences to master course learning outcomes.
• Grades indicate mastery of course learning outcomes.
• Based on engagement of students
• Help students be “intentional learners” (AAC&U; greaterexpectations.org)
Faculty use a repertoire of teaching techniques to meet the needs of diverse students and
to promote different types of learning outcomes, such as
• Active learning
• Collaborative and cooperative learning
• Community-service learning
• Homework and laboratory assignments
• Lectures and discussion
• Online learning
• Problem-based learning
• Design learning environments to meet student and program needs
• Share interests and enthusiasm with students
• Provide students formative feedback on their progress; grade student work
• Mentor student development in and out of the classroom
• Assess class sessions, courses, and programs to improve their effectiveness
• Faculty use classroom assessment to improve day-to-day learning in courses (Angelo
& Cross, Classroom Assessment, Jossey-Bass, 1993).
• Faculty use program assessment to improve learning throughout the curriculum.
• Faculty and others assess their impact to improve institutional effectiveness.
• Co-curriculum and support services are aligned to support learning.
• Program reviews and campus decision-making are conducted within a “culture of
evidence.”
• Recognition and reward systems value contributions to learning and encourage
flexibility to uncover new ways to encourage/support learning.
• Routine campus conversations on learning
17
The Cohesive Curriculum
•
•
•
•
Coherence
Synthesizing Experiences
Ongoing Practice of Learned Skills
Systematically Created Opportunities to Develop Increasing Sophistication and Apply What
Is Learned
Curriculum Map
Course
100
101
102
103
200
229
230
280
290
Outcome 1
I, D
Outcome 2
Outcome 3
I
D
D
D
D
M
Outcome 4
Outcome 5
I
D
D
D
D, M
D
M
D, M
M
I = Introduced, D = Developed & Practiced with Feedback, M = Demonstrated at the Mastery
Level Appropriate for Graduation
Academic Program Assessment
Program assessment is an on-going process designed to monitor and improve student learning.
Faculty:
• develop explicit statements of what students should learn.
• verify that the program is designed to foster this learning.
• develop a meaningful, manageable, sustainable assessment plan.
• collect empirical data that indicate student attainment.
• assess the data and reach a conclusion (faculty are satisfied or disappointed with the extent of
student learning).
• use these data to improve student learning.
Elements of an Assessment Plan
•
•
•
•
What assessment evidence will be collected?
When and how often will it be done?
Who will assess and reflect on the results?
How will results, implications, and related changes be documented?
18
Quotations from the Wise and Experienced
1. “Assessment is an on-going process. We don’t ‘get it done’; we ‘get on with it.’”
Outcomes Assessment, Miami of Ohio
2. “Three cardinal rules for evaluation or assessment: ‘Nobody wants to be evaluated, nobody
wants to be evaluated, and finally, nobody wants to be evaluated.’”
Frank Newman
3. “Much of the literature on assessment suggests, and the Task Force agrees, that an institution
will benefit from assessment only if faculty and cocurricular professionals see a use for the
results and if they take the lead in formulating questions which assessment can help answer.”
Willamette Task Force on Outcomes Assessment
4. “Self-assessment is not the goal. Self-adjustment is the goal. That’s what makes Tiger Woods
and Michael Jordan great. That’s what makes Socrates so impressive. That’s what our best
students and teachers do. They self-adjust, with minimal effort and optimal effect.”
Grant Wiggins
5. “Assessment per se guarantees nothing by way of improvement, no more than a thermometer
cures a fever.”
T. J. Marchese
6. “While in the process of developing new outcomes/objectives, the department or
administrative unit can easily identify assessment procedures that will be so time- and
resource-consuming that they will become an end in themselves and not a means of
determining whether a specific outcome/objective has been achieved. If this occurs, the longterm result is likely to be abandonment of the process.”
James O. Nichols
7. “. . . institutional evaluation should use objective data where available and purposeful but
make no apologies for using subjective data. Or, it is better to be generally right than
precisely wrong.”
R. L. Miller
8. “The most important thing about assessment is that it promotes dialogue among faculty.”
Mary Senter
Most of us don’t have to assess every outcome in every student every year!
19
Some Basic Vocabulary
•
Direct vs. Indirect Assessment
•
Value-Added vs. Absolute Learning Outcomes
•
Embedded Assessment
•
Formative vs. Summative Assessment
•
Authentic Assessment
•
Triangulation
If you have absolute outcomes, your assessment plan should emphasize direct, authentic,
summative assessment, with triangulation.
20
Sample Assessment Plan
Outcome 3, dealing with the understanding of research methods in the discipline, will be assessed
every fourth year starting in 2010/11 by assessing the quality of research skills demonstrated in
capstone (taken right before graduation) research projects and by embedding relevant questions in
final exams in this course. An ad hoc faculty committee will develop and score the test items, and
they will develop and apply a rubric to analyze the capstone projects. Focus groups on student
perceptions concerning their understanding of research methods will be conducted by Assessment
Center staff, and they will work in consultation with the ad hoc committee.
Does this plan involve:
• Direct assessment?
• Indirect assessment?
• Authentic assessment?
• Formative assessment?
• Summative assessment?
• Triangulation?
Closing the Loop
Sometimes results support the status quo. Celebrate!
If results suggest the need for change, you might consider these four types of change:
• Pedagogy—e.g., changing course assignments; providing better formative feedback to
students; use of more active learning strategies to motivate and engage students
• Curriculum—e.g., adding a second required speech course, designating writing-intensive
courses, changing prerequisites
• Student support—e.g., improving tutoring services; adding on-line, self-study materials;
developing specialized support by library or writing center staff; improving advising to
ensure the courses are better sequenced
• Faculty support—e.g., providing a writing-across-the-curriculum workshop; campus support
for TAs or specialized tutors
Assessment in Learning-Centered Institutions- 5
21
Lecture / Discussion 1:
An Outcomes-based
Assessment Model for
General Education
Amy Driscoll
22
An Outcomes-based Assessment
Model for General Education
Amy Driscoll
WASC EDUCATIONAL SEMINAR
Level I
September 24, 2009
Definitions
„
„
Assessment is the process of gathering
information/data on student learning.
General education generally describes
basic or foundational knowledge/skills
and attitudes that all undergraduates
are required to have for graduation.
Possibilities:Purpose/Definition
„
„
“The purpose of assessment is to
improve learning” (Angelo, 2000)
“Assessment is a dynamic pedagogy
that extends, expands, enhances, and
strengthens learning” (Driscoll, 2001)
23
Thinking about Assessment
„
„
„
„
Does assessment flow from the institution’s
mission and reflect the educational values?
Does assessment address questions that
people really care about?
Does assessment help faculty fulfill their
responsibilities to students, to the public?
Does assessment of general education
describe students’ readiness for other
curriculum?
Aligning Mission with Goals for
General Education:
Our central mission is to develop life-long
learning skills, impart society’s cultural
heritage, and educate and prepare for both
the professions and advanced study.
Goals: life-long learning skills
„
cultural heritage
Aligning Institutional Values With
Goals for General Education:
„
ESU has a commitment to academic
and personal integrity.
GOALS: Academic Integrity
Personal Integrity
24
Aligning Institutional Vision with
Goals of General Education
”to enhance recognition of the value of
higher education”
„ “…to enhance the intellectual, social,
cultural, and economic qualities of
urban life”
Goals: Valuing Higher Education
Urban Citizenship
„
Assessment Protocols
for Learning-centered Assessment
OUTCOMES
GOAL
Evidence
Criteria
Standards:
a) Exemplary Achievement
b) Satisfactory Achievement
c) Unsatisfactory Achievement
Goals
„
Broad descriptions
„
Categories of learning outcomes
„
End toward which efforts are
directed
25
Examples of Goals
„
„
„
„
Critical Thinking
Citizenship in a Democracy (Grad.
School of Education)
Team work and Collaboration (School of
Community Health
Ethics
Student Learning Outcomes
„
Refer to Results in Terms of Specific Student
Learning, Development, and Performance (Braskamp
and Braskamp, 1997)
„
„
Answer the Question – “What Do We Expect of
Our Students?” (CSU Report 1989)
Describe Actual Skills, Understandings, Behaviors,
Attitudes, Values Expected of Students
Examples of Outcomes
Math: Use arithmetical, algebraic, geometric and statistical
methods to solve problems.
Ethics: Identify and analyze real world ethical problems or
dilemmas and identify those affected by the dilemma.
Culture and Equity: Analyze and describe the concepts of
power relations, equity, and social justice and find examples
of each concept in the U.S. society and other societies.
Team work: Listens to, acknowledges, and builds on the ideas of
others.
26
Evidence
„
„
„
Student Work that Demonstrates
Achievement of Outcomes
(Assignments, Projects, Presentations,
Papers, Responses to Questions, Etc.)
Designed for appropriate level of
learning expectations (outcomes)
Opportunity for Different Ways of
Demonstrating Learning
Examples of Evidence
Teamwork
Role play or case study
Project or problem solving assignment
Math
Mathematical and statistical projects and papers
Ethics
A written account
A multi-media presentation or display board
An audio tape
Criteria
„
„
Qualities Desired in Student Work (Evidence)
Represent Powerful Professional
Judgment of Faculty
„
Guide Student Learning Efforts
„
Promote Lifelong Learning
„
Support Faculty in Making Objective
Evaluations
27
Examples of Criteria
Math
„
„
„
Accuracy
Complexity
Clarity and Coherence
Ethics
„
„
Complexity (broad, multifaceted, interconnected)
Conscious Awareness
Culture and Equity
„
„
Range of Cultures
Reflectivity and Integration
Teamwork
ƒ Respect
ƒ Flexibility
Standards
„
Describe Different Levels of Criteria
„
Describe Specific Indications of Criteria
„
Promote Understanding of Criteria
„
Support Faculty in Making Objective
Evaluations
Examples of Standards
Math (Accuracy)
„
„
Satisfactory: Contains few errors and those errors do not significantly
undermine the quality of the work.
Considers and uses data, models, tools or processes that
reasonably and effectively address issues or problems.
Unsatisfactory: One or more errors that significantly undermine the quality
of the work.
Uses data, models, tools or processes in inappropriate or
ineffective ways.
Ethics (Complexity)
„
Standard for Excellent: Consistently views sophisticated and significant
„
Standard for Satisfactory: Usually views sophisticated and significant
„
dilemmas and issues with a broad focus and from multiple perspectives.
dilemmas and issues with a broad focus, but may sometimes use a more narrow focus
and may use fewer perspectives.
Standard for Unsatisfactory: Mainly views issues and dilemmas in simple
terms and usually does so with a limited focus and minimal perspectives.
28
Assessment Protocols
OUTCOMES
GOAL
Evidence
Criteria
Standards:
a) Exemplary Achievement
b) Satisfactory Achievement
c) Unsatisfactory Achievement
Assessment Sample
„
Educational Goal
-Personal integrity
„
Outcomes
-Students articulate an individual code of ethics and
apply it to personal decisions of integrity
-Student describe and assume personal responsibility
in collaborative endeavors, and respect and support
the responsibilities of others
Personal Integrity
„
Evidence
-Written code with discussion of two different life
decisions based on the code
-Multimedia presentation
-Letter of application for professional position
-Dramatization of ethical issues
„
Criteria
-Reflection
-Multiple perspectives
29
Personal Integrity
Standards
-Excellence in Reflection: Consistently raises
questions, checks assumptions, connects with
previous experiences, acknowledges biases
and values and engages in self-assessment
-Excellence in Multiple Perspectives: Examines
thinking and experiences of others, considers
those affected by decisions, and considers
diverse courses of action
Assessing Student Learning:
Course, Program and
Institutional Levels
7.
Revise outcomes
and criteria,
Improve pedagogy
and curriculum for
learner success
6.
Review and
analyze student
evidence
5. Collect evidence of
student achievement
4.
Make outcomes, evidence, criteria, and
standards “public and visible” (syllabi,
programs, brochures)
1.
Preparation: Determine
purpose(s) and
definition of
assessment; Examine
mission and values
2.
Design
assessment:
Articulate goals,
Develop clear
outcomes,
evidence, criteria,
and standards
3. Alignment of
curriculum and
pedagogy with learning
outcomes
Step 3: Aligning Curriculum
and Pedagogy with Learning
Outcomes
„
„
„
Outcomes and Criteria as Planning Focus
Faculty Alignment Grids
Learner Grids
30
Step 4: Making Learning
Outcomes --„
Public and Visible
ƒ Relevant and Meaningful
ƒ Motivating and Supportive of Learning
Step 5: Collect Evidence of
Student Achievement
„
Collect representative samples
(3 Exemplary, 3 Satisfactory, 3
Unsatisfactory)
Step 5: Review and Analyze
Evidence
„
„
„
Read holistically to determine whether
outcomes are achieved (reliability).
Several readings to identify examples of
criteria (validity).
Final reading for insights about
pedagogy, class structure and
environment, and learning supports.
31
Step 6: Process Results:
Improving Learning
„
„
„
„
Documentation of student achievement of
outcomes
Identification of curricular gaps/foci and
pedagogical weaknesses/strengths
Clarification of outcomes, criteria &
standards
Redesign of evidence
Expanding Assessment of
General Education
„
Graduate exit surveys
„
Alumnae surveys
„
Employer surveys
SUMMARY
Outcomes-based assessment for general
education can provide a foundation for
integrating institutional goals in the major
programs of study. The assessment
protocols provide a foundation for students to
become successful learners. Faculty who
collaborate to develop general education
assessment protocols become more
intentional with their teaching and learning
plans.
32
Lecture / Discussion 2:
Assessment for Student
Affairs Staff and Other
Campus Professionals
Mary Allen
33
Assessment for Student Affairs Staff
and Other Campus Professionals
Mary J. Allen, [email protected], September 2009
Assessment
The assessment of student affairs and administrative units is an on-going process designed to
monitor and improve the effectiveness of the unit being assessed. Professionals in each unit:
• Develop explicit statements of the unit’s mission and objectives.
• Verify that the unit’s operations are organized to foster the objectives.
• Collect empirical data that indicate how well objectives are being met.
• Use these data to improve the unit’s effectiveness (“close the loop”).
•
•
ƒ
ƒ
ƒ
Why so much emphasis on assessment?
Accreditation expectations for institutional effectiveness
Assessment establishes a culture of evidence for supporting effective, reflective, selfmonitoring, self-correcting institutions.
Articulating Objectives (Nichols & Nichols)
Processes (e.g., travel claims or applications are processed efficiently and equitably)
Learning Outcomes (e.g., students who receive training can write an effective resume or can
use the campus email system; staff who receive training can effectively use campus
accounting procedures; students who are served by the Counseling Center report fewer plans
to withdraw from campus; students who participate in this event can describe the effects of
alcohol on drivers; employees are aware of campus health and safety procedures)
Satisfaction Indicators (people supported by the unit report satisfaction with the service, e.g.,
students report satisfaction with Health Center services)
Mission and Objectives
Mission: a holistic vision of the values and philosophy of the unit.
Objectives: desired processes, learning outcomes, and satisfaction ratings. Each should be tied
to an assessment technique (e.g., a survey) with an associated standard (e.g., an
average rating of at least 3.5 on a 5-point rating scale).
The unit’s mission should (Nichols & Nichols, p. 35):
• Describe the purpose of the unit. What services are provided? To whom?
• Be brief (less than one page).
• Be aligned with the campus mission.
34
•
•
Be known by the staff.
Be used by the staff to make decisions and set priorities.
Examples of Mission/Vision/Goals Statements
Example 1: Student Affairs Division (Oregon State University. Retrieved 9/21/07 from
http://oregonstate.edu/studentaffairs/missionvision.html)
Our Mission. The Division of Student Affairs contributes to and facilitates the success of
students and Oregon State University.
Our Vision. Faculty and staff provide leadership for the positive development of community at
Oregon State University.
We collaborate with others to enhance the educational environment and support the teaching and
learning process.
We value and respect the individual and believe that sharing knowledge changes lives.
Example 2: Library Mission Statement (Nichols & Nichols, p. 36)
“The university educates students to assume leadership roles in the state, nation, and world
through its nationally recognized programs of undergraduate, graduate, and professional study.
Its fundamental purpose is the creation and dissemination of knowledge. The university libraries
support this mission. Specifically, the university libraries strive to meet the information needs of
the academy, its students, faculty and staff, by employing contemporary knowledge management
techniques to develop collections, provide access to information sources, and instruct individuals
in contemporary bibliographic methodologies.”
Example 3: Accounting Office Mission Statement (Nichols & Nichols, p. 36)
“The Accounting Office seeks (1) to provide administrators with accurate and timely financial
data to assist them in the management of the institution’s resources, and (2) to ensure that
financial records are maintained in accordance with generally accepted accounting principles and
guidelines as established by State and Federal Agencies.”
Example 4. Student Affairs Goals: Ferris State University (2003; Retrieved 9/21/07 from
http://www.ferris.edu/htmls/administration/studentaffairs/assessment/03SAReport.pdf)
“The primary goal of Student Affairs is to provide activities, programs, and facilities that support
the personal development, educational progress and career goals of all students.
• Create and foster an environment in which diverse talents and backgrounds are recognized
while providing unifying common experiences.
• Encourage understanding and appreciation for others.
• Establish an environment that is safe, secure, and helps students to maximize their mental
and physical health.
• Support and advance institutional values by developing and enforcing behavioral standards
for students.
• Foster a sense of responsibility for personal and community safety through education which
reinforces personal accountability for one’s actions.
35
•
•
•
•
•
•
•
•
•
Help students become informed decision-makers in order to reduce alcohol and other drug
abuse.
Build respect for the value of community and positive group affiliation.
Serve as educational resource personnel to others in the University community.
Continue communication and collaboration with faculty, staff, and administrators campuswide to meet the educational goals of the University.
Engage in assessment activities that evaluate the effectiveness of all our programs,
departments, and the Division as a whole on an ongoing basis.
Provide quality service, which includes personal, timely attention to our customers.
Effectively recruit and retain students.
Assist students in securing financial resources to help pay for their educational costs.
Provide accurate and timely institutional, State, and Federal reports as required.”
Effective outcomes/objectives should be:
• Consistent with the unit and campus mission.
• Realistic.
• Few in number.
• Assessable.
• Used by staff to set priorities and make decisions.
Examples:
1. Accurate, real-time class enrollment data are continuously available to faculty and
administrators.
2. Students who attend a Career Orientation Workshop can prepare a resume and use our online bulletin board to monitor potential employment opportunities.
3. All students attending orientation will receive email accounts and will know how to use the
email system to communicate with students, faculty, and staff.
4. Interlibrary loan materials will be delivered within eight working days.
5. Students report satisfaction with Health Center Services; ratings will average at least 3.80 on
a 5-point rating scale.
6. On average, at least 100 students will attend each cultural event sponsored by the ASI.
7. Faculty who attend Blackboard workshops will be able to create and update online course
materials.
8. Student government meetings follow procedures defined in the Handbook.
9. Staff who are certified to use the enrollment management system can independently add and
delete courses, place enrollment restrictions on courses, and monitor course enrollments.
10. Students who use the Writing Center at least three times in a semester improve writing skills.
11. Students who participate in the diversity retreat will report increased understanding of people
of racial and ethnic backgrounds different from their own.
12. Students who attend New Student Orientation can describe General Education requirements
and an array of services available to them on campus.
36
Student Affairs Student Learning Outcomes at Southern Illinois University Edwardsville
(Retrieved 9/21/07 from http://www.siue.edu/AQIP/goal1/AssessmentStudentAffairs.ppt#256)
1. Integrate classroom and out-of-classroom learning experiences.
2. Integrate values learned from prior experience with values learned at the University.
3. Attend activities, programs, and events not previously experienced prior to attending the
University.
4. Demonstrate the value of diversity and community.
5. Contribute to at least one group for the purpose of developing projects, programs,
relationships or performing volunteer service.
6. Seek the advice and counsel of peers, faculty, staff, and others.
7. Demonstrate the value of their own health and wellness and that of others.
8. Make decisions based on values and ethical principles.
9. Articulate personal and career goals.
10. Demonstrate communication skills and behaviors necessary for the work place.
11. Demonstrate a sense of curiosity and appreciation for lifelong learning.
First-Year Experience Courses
Frequently Staffed by Faculty, Student Affairs Professionals, or Both
The First-Year Initiative Survey
The First-Year Initiative (FYI; http://www.webebi.com/University/FYI) benchmarking
survey was piloted in 2001 and is designed to assess ten types of learning outcomes typically
fostered in first-year experience seminars:
● Study strategies
● Academic/cognitive skills
● Critical thinking
● Connections with faculty
● Connections with peers
● Out-of-class engagement
● Knowledge of campus policies
● Knowledge of wellness/spirituality
● Management of time/priorities
● Knowledge of wellness (Swing, 2004, p. 119)
In addition, it collects demographic information (e.g., gender, age, living arrangements, alcohol
use) and assesses campus satisfaction and some aspects of course delivery (e.g., effective
readings, engaging pedagogy).
Examples of First-Year Experience Course Outcomes, Objectives, and Goals*
Bryant University. Bryant University’s required first-year seminar, Foundations for Learning,
is designed to “help students take responsibility for their education” by:
● Understanding the importance of being actively involved in the educational process
● Developing cognitive and metacognitive abilities
37
●
●
●
●
●
Developing a fuller understanding of a range of learning and study strategies
Learning how planning and prioritizing impact academic success
Developing self-concept including an awareness of health and wellness issues
Developing communication skills including those related to collaboration and leadership
Engaging in scholarly activates such as group discussion, conducting research, and
synthesizing materials
● Understanding the importance of respecting diversity as a member of the Bryant
community and a citizen of the world (Hazard, 2005, p. 24)
Mount Mary College. The first-year seminar at Mount Mary College, Leadership for Social
Justice, is strongly recommended to all new, traditional-aged students, and it has six primary
objectives:
● To introduce students to Mount Mary’s mission and the Mount Mary Women’s
Leadership Model
● To increase self-knowledge leading to an understanding of personal leadership styles
● To develop and increase skills and strategies for dealing with difficult issues and
conflict
● To expand knowledge of local and global social justice issues
● To experience service-learning as a means of growing in leadership, selfunderstanding, and knowledge of social justice issues
● To develop reading, writing, and oral communication skills (End, 2005, pp. 97-98)
Northern Illinois University. Northern Illinois University’s University 101, University
Experience, course is an elective for first-semester freshmen, and it is designed to help students:
● Understand the challenges and expectations of college
● Develop strategies for academic success
● Adjust to the university community and become involved
● Communicate with faculty
● Learn to manage time and money
● Learn how to use technology and NIU’s resources
● Live in a diverse community
● Prepare for a career (House, 2005, p. 104)
Olympic College. Olympic College offers General Studies 100, Strategies for Academic
Success, a requirement for students requiring developmental English courses and an elective for
other students. Students in this course learn:
● To demonstrate knowledge of the purposes, values, and expectations of higher education
● To demonstrate basic self-awareness and self-management
● To demonstrate academic skills of learning how to learn
● To write an educational/career plan
● To demonstrate knowledge of physical, social, and emotional wellness (Huston, 2005, p.
123)
Temple University. Faculty at Temple University teach a one-credit, elective course, Learning
for the New Century, with four major goals:
● Enhance students’ intellectual development and improve their study behaviors and skills
● Enhance students’ social development and engagement in the campus community
38
● Promote collaborative learning and group work
● Allow students to practice technology applications and retrieval of information.
(Laufgraben, 2005, p. 152).
*All FYE outcomes are taken verbatim from B. F. Tobolowsky, B. E. Cox, & M. T. Wagner
(Eds.). (2005). Exploring the Evidence: Reporting Research on First-Year Seminars, Volume III
(Monograph No. 42). Columbia, SC: University of South Carolina, National Resource Center for
The First-Year Experience and Students in Transition.
Methods to Assess Objectives
Properties of Good Assessment Techniques
•
•
•
•
•
•
Valid—directly reflects the objective being assessed
Reliable—especially inter-rater reliability when subjective judgments are made
Actionable—results help reviewers identify what’s working well and what needs more
attention
Efficient and cost-effective in time and money
Interesting—staff care about results and are willing to act on them
Triangulation—multiple lines of evidence point to the same conclusion
Frequently-Used Strategies (Nichols & Nichols)
1. Counts (e.g., number of students who eat in the cafeteria or the number of days to process an
invoice)
2. Client satisfaction measures (e.g., ratings from surveys, interviews, and focus groups; broadbased and point-of-contact data may be collected).
3. External evaluation reports (e.g., Health Department review of the food service unit)
4. Learning Outcomes (e.g., quality of student resumes after a workshop at the Career Center).
Try to concentrate on direct, authentic assessment—to verify that learners can demonstrate
what you want them to learn. Rubrics are useful for making subjective judgments about
students’ learning. If you do indirect assessment of the achievement of learning outcomes
(based on perceptions of learning), consider gap analysis (comparing importance and
achievement ratings).
Some Ways to Collect Assessment Data
•
•
•
•
•
•
Co-curricular portfolio
Essays in required composition courses
Information literacy assignments (to assess library services)
Assignments and exams in first-year seminars (homework, reflective papers, exam questions)
Quick checks during programs (clickers or hand signals)
Quizzes at the end of sessions (perhaps linked to discussion or used as pre/post quizzes)
39
•
•
•
Community service learning reflective papers
Systematic observations (e.g., observe Student Senate meetings to see if procedures are
followed)
Focus groups, interviews, surveys
Sometimes data are analyzed separately for subgroups of respondents, such as international
students, athletes, evening students, or recently-hired employees to verify that all campus
segments have benefited from the unit’s services.
Resources
ACPA. You might find the work of the ACPA Commission on Assessment for Student
Development (http://www.myacpa.org/) useful. Only members can access details.
Learning Reconsidered (http://www.learningreconsidered.org/tools/project.cfm?sid=15). See the
SLO Identification Rubric for many ideas concerning possible learning outcomes.
Some Resources/Examples That Might Be Useful For Assessment of the Library
ACRL Standards: http://www.ala.org/ala/acrl/acrlstandards/standardslibraries.htm
Boston College: http://www.bc.edu/libraries/about/assessment/
Kapoun: http://www.webpages.uidaho.edu/~mbolin/kapoun2.htm
Yale: http://www.library.yale.edu/assessment/toolkit/
References
Allen, M. J. (2004). Assessing Academic Programs in Higher Education. Bolton, MA: Anker.
Nichols, K. W., & Nichols, J. O. (2000). The Department Head’s Guide to Assessment
Implementation in Administrative and Educational Support Units. New York: Agathon Press.
[http://www.agathonpress.com]
Swing, R. L. (Ed.), Proving and Improving, Volume II: Tools and Techniques for Assessing the
First College Year (Monograph No. 37). Columbia, SC: University of South Carolina, National
Resource Center for The First-Year Experience and Students in Transition.
Upcraft, M. L., & Schuh, J. H. (2000). Assessment Practice in Student Affairs: An Applications
Manual. San Francisco: Jossey Bass.
Upcraft, M. L., & Schuh, J. H. (1996). Assessment in Student Affairs: A Guide for Practitioners.
San Francisco: Jossey Bass.
Schuh, J. H., & Associates, (2008). Assessment Methods for Student Affairs. San Francisco:
Jossey Bass. [expected release is October 24, 2008]
40
San Jose State University Counseling Services Feedback Form*
The Counseling Services’ staff would appreciate your feedback on our services so we can better
understand what is working well and what could be improved. We would very much appreciate your
honest feedback. This survey is voluntary and non-identifying, so please do not put your name on this
form.
Please indicate your degree of agreement with each of the following statements, using this scale:
N/A=This question does not apply to me.
1=Strongly Disagree
2=Disagree
3=Neutral
4=Agree
5=Strongly Agree
_____ 1. My counselor helped me learn about myself.
_____ 2. After receiving counseling services, I am more confident that I can succeed in my
studies.
_____ 3. My counselor helped me learn about support services and resources at SJSU.
_____ 4. My experience with Counseling Services helped me learn skills that I can develop to
maximize my potential for academic success.
_____ 5. My experience with Counseling Services helped me learn skills that I can develop to
maximize my potential for personal success.
_____ 6. I would recommend this service to a friend.
_____ 7. Overall, Counseling Service staff are caring professionals.
_____ 8. Overall, Counseling Service staff are effective professionals.
9. Please describe one or two things Counseling Services staff did that you found particularly
helpful.
10. Please describe one or two things that Counseling Services staff could do to provide you
better support.
11. Please provide some background information about yourself.
a. Age ____
b. Major: ______________
c. Gender: ______________
d. Class Level: ___________
e. Ethnicity________________
f. Reason for Visit: ___ Personal Concerns ___ Educational Concerns
g. Number of times you have visited the Counseling Center: ________
Thanks for your feedback!
*Please send comments or suggestions to Wiggsy Siversten, San Jose State University. 2/06
41
Yale Library Focus Group Study
Retrieved September 21, 2007 from
http://www.library.yale.edu/assessment/toolkit/DIGquestions.doc
Delivery Improvement Group
Focus Group Report
January 2002
Background:
As part of its charge, the Shipping and Delivery Service Improvement Group (known as the
Delivery Improvement Group or DIG) was asked to “clarify the vision for quality shipping and
delivery services in the Yale University Library,” and to “understand the current ‘as is’
processes.” The charge also asked the group to “solicit and review internal assessment data,
including feedback from customers and providers of the services.” Focus Groups seemed a
logical way to address the charge.
Yale University Library delivery service managers (Manager, Shipping and Receiving and
Manager, Library Shelving Facility) were asked to identify their key customer groups. Email
invitations were sent to the managers of these departments asking them to identify participants
for the focus groups.
In the end, thirteen customers were invited to provide their perspective on the current state of
delivery of materials within the library system, trends they anticipate in the future, and their
ideals of delivery services. Two focus groups were held in August 2001 during work hours.
Nine participants attended the first session; four the second.
Preparation:
A number of books in the Yale University Library collections were useful in preparing to lead
the focus group. Especially helpful was:
Focus groups for libraries and librarians / Beryl Glitz.
New York, NY : Forbes, c1998.
SML, Stacks: Z678.88 G65 1998 (LC)
Key to a successful focus group is a targeted statement of purpose. For this focus group, the
purpose was:
A) To assess “as is” state of current delivery services—Eli Express, US mail, UPS, LSF, etc.
B) To learn more about the shipping and delivery needs of various units of the library.
C) To learn more about trends in the use of shipping and delivery.
We used a small handheld tape recorder with a tape that lasted 45 minutes and was flipped once.
Two members of DIG participated in each session: Holly Grossetta Nardini as facilitator at both
sessions, John Gallagher as observer at the first session, Carol Jones as observer at the second
session.
42
We arrived early to set the chairs up in a comfortable arrangement and to test the equipment.
Moderator’s Introduction:
I’m Holly Grossetta Nardini, Service Quality Support Director, and chair of the new Delivery
Improvement Group. I’d like to also introduce John Gallagher, Head of Circulation at the
Medical Library, who’s a member of the task force and will be helping today.
We have asked you here today because the Library is exploring possible improvements to its
internal and external delivery services, both through Shipping & Receiving and LSF. You’ll see
a yulib-l message later this week outlining our charge.
You all have some knowledge about delivery. We asked LSF and S&R to identify their key
customer groups and your units were identified. Your individual input is important and will be
used to improve service. Thank you for taking the time to come to this discussion.
This is a focus group. I am going to be asking you some questions about your experiences and
the information we learn from you will be used to help decide what direction we pursue for
improvements.
We’ll start with general questions about the current state of delivery services—Eli Express, US
mail, UPS, LSF, etc. Then we’ll talk a bit about trends in your use of shipping and delivery
services and finally we’ll try to learn more about the shipping and delivery needs of various units
of the library.
Note that when we talk about LSF, we will only be discussing its delivery operations, not the
selection process for material transferred to LSF.
Goal is not to suggest solutions but to identify issues and problems.
Overview
We’ll be here about an hour.
I’ll ask you a series of open-ended questions. Feel free to express your ideas. Feel free to ask
for clarification. An interesting discussion, not a test, a debate or a lecture.
Please feel free to say what’s on your mind. There are no right or wrong answers. I know that
you won’t all agree with each other and, in fact, the more views we hear the better, since only a
small number of people can be reached in a focus group.
Data collected at this session will be aggregated. Your names will not be linked to your
comments. We are tape recording session to be sure we capture your comments, but not to
identify individuals. For that reason, I will ask you to speak clearly, loudly and one at a time.
Interaction is encouraged. I am here to ask questions and facilitate, but the focus is on your
opinions and experiences. Each person’s input is important. I ask you to jump in if you want to
affirm or disagree with any opinion.
43
Questions
Name, department, position? One or two sentences about what you do.
What is use do you currently make of shipping and delivery services?
Prompts: UPS packages, computers & furniture, LSF deliveries and returns, Eli Express
How have your demands for delivery changed over time and what other changes do you foresee?
Can you describe one experience with sending or receiving an item on- or off-campus?
Imagine you are shipping something for the first time, what would expect from the service?
Imagine you are waiting for delivery of material for a third party, what would you expect from
the delivery service?
What other shipping services would you like to see the Library provide?
Conclusion
At the end of each session, we briefly reviewed the purpose of the discussion. We asked each
member of the group to sum up their feelings about the topic and add anything they may have
wanted to say earlier.
Finally, I asked a general question: Did we miss anything?
All participants were thanked and reassured about the anonymity of the session.
44
Muhlenberg College Dining Services Catering Survey
In an effort to continue to provide the best service possible, we would appreciate a few minutes of your time to provide us with your
thoughts and input on how we can best serve Muhlenberg College. Please complete the brief survey below.
1.
How well do you feel Sodexho Catering is meeting your present needs? (Please rate your satisfaction by
circling one number on the scale below.)
VERY WELL
10
9
NOT WELL
8
7
6
5
4
3
2
1
Please comment:
2.
Listed below are key elements of a quality catering program. For each element, please rate Sodexho’s
performance.
FOOD
EXCELLENT
POOR
a.
Taste of food
10
9
8
7
6
5
4
3
2
1
b.
Nutritional value of food
10
9
8
7
6
5
4
3
2
1
c.
Appearance of food
10
9
8
7
6
5
4
3
2
1
d.
Variety of food
10
9
8
7
6
5
4
3
2
1
e.
Temperature of food
10
9
8
7
6
5
4
3
2
1
f.
Vegetarian offerings
10
9
8
7
6
5
4
3
2
1
8
7
6
5
4
3
2
1
SERVICE
g.
Timing of service
10
9
h.
Courteous service
10
9
8
7
6
5
4
3
2
1
i.
Attentive service
10
9
8
7
6
5
4
3
2
1
j.
Appearance of personnel
10
9
8
7
6
5
4
3
2
1
k.
Professionalism and etiquette
10
9
8
7
6
5
4
3
2
1
ATMOSPHERE
l.
Appearance of dining area
10
9
8
7
6
5
4
3
2
1
m. Cleanliness of china, flatware, glass
10
9
8
7
6
5
4
3
2
1
n.
Table settings
10
9
8
7
6
5
4
3
2
1
o.
Cleanliness of equipment
10
9
8
7
6
5
4
3
2
1
45
MEETING YOUR NEEDS
p.
Understanding your food
service requirements
10
9
8
7
6
5
4
3
2
1
q.
Receptiveness to new ideas
and suggestions
10
9
8
7
6
5
4
3
2
1
r.
Efficiency in planning of your event
10
9
8
7
6
5
4
3
2
1
s.
Creativity and imagination in
presenting new menu ideas
10
9
8
7
6
5
4
3
2
1
Consistency of services provided
10
9
8
7
6
5
4
3
2
1
t.
3.
When working with the sales associate are we following up and communicating effectively with you?
4.
Do you feel that when you give Sodexho your pricing needs that we are able to develop a menu to fit?
5.
Have you experienced any significant problems with Sodexho Catering during the past year?
YES
NO
If YES, please explain:
6.
Does the event sheet and/or confirmation letter provide you with enough information for your event?
YES
NO
If NO, please explain:
7.
In your association with our Catering Sales Coordinator, how would you rate his/her performance: (Circle on
number for each.)
EXCELLENT
a.
Effectiveness
10
9
b.
Responsiveness to challenges
10
9
8
7
6
5
4
3
2
1
c.
Creativity in providing imaginative menu ideas 10
9
8
7
6
5
4
3
2
1
d.
8.
8
POOR
7
6
5
4
3
2
Follow up once event is booked
10
9
8
7
6
5
4
3
2
to ensure correct information
On average, how often each month do you use Sodexho Catering for events you are hosting?
1
1
46
1-2 times a month
3-5 times a month
More than 5 times a month
9.
On average, how often each month do you participate in a campus catering event that is organized or
arranged by another campus organization or department?
Annually
Once every 4 months
1-2 times a month
More than 3 times a month
Comments:
Thank you for your valuable input and cooperation.
Name
Date
Department
Please send your completed survey to:
John Pasquarello
General Manager
Muhlenberg College Dining
Seegers Union Building
47
Posted Feedback from Muhlenberg College Napkin Survey
General's Quarters Napkin Board Activity
Date: March 6, 2007
Comment
Why is everything so expensive this year? The food hasn't gotten
ANY better! Why should I have to pay more for stupid checkered
paper & fancy names.
Response
Some prices did go up this year, but so did Meal Equivalency. The value
of a swipe is now $4.50. There is no charge and no change in prices
because we now use checkered paper.
Hello! G.Q.. peeps! I wanted to know if you could get Fiji water? It is
my native land & I miss it! Thanks! Oh yeah, can you get balsamic
dressing for the salads? Thanks!
Due to the exclusive agreement with Pepsi, Fiji Water is not an available
product. We do have balsamic dressing at the deli, just bring your salad
there and we'll be happy to put some on for you.
Egg Whites would be great!
Just ask at the grill, we do have them in both a pasteurized product as
well as being able to use the whites only from a whole egg.
The grapes were part of our summer sizzlers promotion, we're glad you
liked them. We do offer grapes and cheese at our grab and go cooler,
just not in cups.
I want purple grapes in the little cups!
Please get plain Yogurt.
Dear GO, It is ridiculous that a wrap, ONE wrap, that's it, without a
soda or chips, goes over 1 swipe. Please make your increased prices
correspond with the equivalency of a swipe.
We will look into that right away!
Our regular Turkey, Ham, and Tuna wraps with a soda and chips still
are one swipe. We have added some Premium sandwiches to our line-up,
without eliminating last years options, to enhance your dining, including
wheat wraps too.
Who can I actually talk to about my problems with GO instead of this
piece of paper?
You can talk to me, Joe McCloud. I am the new GO manager and I am
here 5 (sometimes more) days a week. My office number is: x-3476.
Dear GO, Why do you enjoy scamming Muhlenberg Students? I
don't approve
It sounds like you may have had an unpleasant dining experience. Please
stop by and see me so we can discuss it. Joe McCloud, GQ manager
(list was truncated to save paper)
48
Writing Rubric
Johnson Community College, downloaded 12/22/04 from
http://www.jccc.net/home/depts/6111/site/assmnt/cogout/comwrite
6 = Essay demonstrates excellent composition skills including a clear and thought-provoking thesis,
appropriate and effective organization, lively and convincing supporting materials, effective diction
and sentence skills, and perfect or near perfect mechanics including spelling and punctuation. The
writing perfectly accomplishes the objectives of the assignment.
5 = Essay contains strong composition skills including a clear and thought-provoking thesis, although
development, diction, and sentence style may suffer minor flaws. Shows careful and acceptable use of
mechanics. The writing effectively accomplishes the goals of the assignment.
4 = Essay contains above average composition skills, including a clear, insightful thesis, although
development may be insufficient in one area and diction and style may not be consistently clear and
effective. Shows competence in the use of mechanics. Accomplishes the goals of the assignment with
an overall effective approach.
3 = Essay demonstrates competent composition skills including adequate development and organization,
although the development of ideas may be trite, assumptions may be unsupported in more than one
area, the thesis may not be original, and the diction and syntax may not be clear and effective.
Minimally accomplishes the goals of the assignment.
2 = Composition skills may be flawed in either the clarity of the thesis, the development, or organization.
Diction, syntax, and mechanics may seriously affect clarity. Minimally accomplishes the majority of
the goals of the assignment.
1 = Composition skills may be flawed in two or more areas. Diction, syntax, and mechanics are
excessively flawed. Fails to accomplish the goals of the assignment.
Revised October 2003
49
California State University, Fresno Scoring Guide for Writing
Scoring Level
4
- Accomplished
3
- Competent
2
- Developing
1
- Beginning
Knowledge of Conventions
In addition to meeting the requirements
for a “3,” the writing is essentially errorfree in terms of mechanics. Models the
style and format appropriate to the
assignment.
While there may be minor errors, the
paper follows normal conventions of
spelling and grammar throughout and has
been carefully proofread. Appropriate
conventions for style and format are used
consistently throughout the writing
sample. Demonstrates thoroughness and
competence in documenting sources; the
reader would have little difficulty
referring back to cited sources.
Frequent errors in spelling, grammar
(such as subject/verb agreements and
tense), sentence structure and/or other
writing conventions distract the reader.
Writing does not consistently follow
appropriate style and/or format. Source
documentation is incomplete. It may be
unclear which references are direct quotes
and which are paraphrased.
Writing contains numerous errors in
spelling, grammar, and/or sentence
structure which interfere with
comprehension. Style and/or format are
inappropriate for the assignment. Fails to
demonstrate thoroughness and
competence in documentation.
Clarity and Coherence
Rhetorical Choices
In addition to meeting the
requirements for a “3,” writing flows
smoothly from one idea to another.
The writer has taken pains to assist
the reader in following the logic of
the ideas expressed.
Sentences are structured and word are
chosen to communicate ideas clearly.
Sequencing of ideas within
paragraphs and transitions between
paragraphs make the writer’s points
easy to follow.
In addition to meeting the requirements
for a “3,” the writer’s decisions about
focus, organization, style/tone, and
content made reading a pleasurable
experience. Writing could be used as a
model of how to fulfill the assignment.
The writer has made good decisions
about focus, organization, style/tone,
and content to communicate clearly and
effectively. The purpose and focus of
the writing are clear to the reader and
the organization and content achieve the
purpose well. Writing follows all
requirements for the assignment.
Sentence structure and/or word choice
sometimes interfere with clarity.
Needs to improve sequencing of ideas
within paragraphs and transitions
between paragraphs to make the
writing easy to follow.
The writer’s decisions about focus,
organization, style/tone, and/or content
sometimes interfere with clear, effective
communication. The purpose of the
writing is not fully achieved. All
requirements of the assignment may not
be fulfilled.
Sentence structure, word choice, lack
of transitions and/or sequencing of
ideas make reading and understanding
difficult.
The writer’s decisions about focus,
organization, style/tone, and/or content
interfere with communication. The
purpose of the writing is not achieved.
Requirements of the assignment have
not been fulfilled.
Retrieved June 6, 2002 from http://www.csufresno.edu/cetl/assessment/ (click on WritingScoring.doc)
50
Scoring Rubric for Reflection Papers
(Compiled by California Polytechnic State University Service-Learning Program)
Retrieved March 14, 2007 from http://www.ccccd.edu/servicelearning/faculty-ref-paperrubric.html
- Excellent Paper Civic Awareness and Responsibility
The paper demonstrates that the student:
• understands the complex nature of social problems and has identified several of the
causes leading to the social problem addressed by the agency;
• understands that there are forces in action which may cause misfortune over which
individuals have no control. (i.e. realizes that individuals are not always solely to blame
when they are faced with misfortunes; that it's not just a matter of "pulling yourself up by
the bootstraps;")
• sees a relationship between the work of grass roots service agencies and local, state and
national government;
• can explain in great detail the programs and services provided by the agency;
• is committed to continued involvement in the community and/or in political processes
while in school or after graduation (OR makes a thoughtful argument against or
questioning such involvement);
• has identified ways in which he/she can contribute to the community, including both
skills and knowledge;
• grasps the concept of social justice;
• made commitments to the agency that exceeded those required by the class and fulfilled
all of them.
Critical Thinking
The paper shows that the author:
• views situations from multiple perspectives; able to observe multiple aspects of the
situation and place them in context;
• perceives conflicting goals within and among the individuals involved in a situation and
recognizes that the differences can be evaluated;
• recognizes that actions must be situationally dependent and understands many of the
factors which affect their choice;
• makes appropriate judgements based on reasoning and evidence;
• has reasonable assessment of the importance of the decisions facing clients and his or her
responsibility as a part of the clients' lives;
• began to think in new ways; about the clients served, society and social problems in
general, him/herself as a person;
• not only understands the purpose(s) and programs of the agency selected but uses critical
thinking skills to evaluate its effectiveness and to develop recommendations for
improvement;
• realizes that he/she can learn outside the classroom because he/she has accessed
information from a variety of sources in the field (i.e. observation, interview, reading
51
materials, etc.) thereby demonstrating capacity for self-guided, life-long learning
activities;
• able to use many sources of information within a social environment;
• sees how and where skills and information gained through service involvement can be
applied to other situations;
• reflects on and can articulate the meaning of a "real life" experience.
Personal Development
The paper indicates that the student:
• realizes how much he or she can learn from others, including those considered to be
"underprivileged;"
• appreciates people whose values, lifestyles or cultures are different from his or her own;
• has examined his own beliefs in light of the experience;
• sees evidence that the author continues in the process of developing a philosophy of life;
• sees how service involvement could impact his personal career development;
• understands some of the factors that make the people who are served and/or agency staff
different from him/herself.
- Proficient Paper Civic Awareness and Responsibility
The paper demonstrates that the student:
• is likely to continue his interest in his issue area;
• appreciates the complex nature of the social issue addressed by the agency and names at
least two causes;
• understands that there are forces in action which may cause misfortune over which
individuals have no control. (i.e. realizes that individuals are not always solely to blame
when they are faced with misfortunes; that it's not just a matter of "pulling yourself up by
the bootstraps");
• has fulfilled all commitments made to the agency including eight hours of service;
• has a sense of the contributions that he/she can make in terms of his/her skills and
knowledge;
• is committed to working with the same or a similar agency at some point in his or her
future (OR provides a well thought out argument against or questioning such
involvement).
Critical Thinking
The paper shows that the author:
• not only understands the purpose(s) and programs of the agency selected but uses critical
thinking skills to evaluate its effectiveness and to develop at least two recommendations
for improvement;
• sees how and where skills and information gained through service involvement can be
applied to other situations;
• has accessed information from a variety of sources in the field (e.g. observation,
interview, reading related materials, discussion groups), thereby demonstrating a capacity
for applying "learn by doing" in the community as a method for life-long learning;
• observations are fairly thorough and nuanced although they tend not to be placed in a
broader context;
52
provides a cogent critique from one perspective, but fails to see the broader system in
which the aspect is embedded and other factors which may change;
• uses both unsupported, personal belief and evidence but is beginning to be able to
differentiate between them;
• perceives legitimate differences of view point;
• demonstrates a beginning ability to interpret.
Personal Development
The paper indicates that the student:
• realizes that he/she can learn from people whose values, lifestyles or cultures are different
from his/her own;
• understands some of the factors that make the people served and/or agency staff different
from him/herself;
• sees how service involvement could impact his/her personal and career development.
•
- Acceptable Paper Civic Awareness and Responsibility
The paper demonstrates that the student:
• is aware at a general level of social problems and their complex nature;
• recognizes a need for people to get involved;
• demonstrates some idea of how and where his/her skills and knowledge can be used for
community betterment.
Critical Thinking
The paper shows that the author:
• understands the purpose(s) and programs of the agency selected and provides at least one
idea of how its services might be improved;
• has accessed information from a variety of sources in the field (i.e. observation,
interview, reading related materials, discussion groups);
• gives examples of observed behaviors or characteristics of the client or setting, but
provides no insight into reasons behind the observation;
• observations tend to be one-dimensional and conventional or unassimilated repetitions of
what has been heard;
• tends to focus on just one aspect of the situation;
• uses unsupported personal beliefs frequently as "hard" evidence;
• may acknowledge differences of perspective but does not discriminate effectively among
them.
Personal Development
The paper indicates that the student:
• realizes that he or she can learn from others, including those considered to be
"underprivileged;"
• is tolerant of people whose values, lifestyles or cultures are different from his or her own.
- Unacceptable Paper Civic Awareness and Responsibility
The paper demonstrates that the student:
• lacks information about social problems and/or interest in addressing them;
53
demonstrates no personal commitment to helping find a solution for community
problems;
• has not fulfilled his/her commitments to the agency.
Critical Thinking
The paper shows that the author:
• does not see how skills and information gained through service involvement can be
applied to other situations.
Personal Development
The paper indicates that the student:
• believes he or she has little to learn from others, including those considered to be
"underprivileged;"
• is not tolerant of individual differences and continues to rely on traditional stereotypes to
describe and deal with people who are different from him/herself;
• has undergone no examination of his/her own beliefs in light of the service experience.
•
54
Intentional Learning Scoring Rubric*
Learning
Below Basic
Outcome
BB
Self-Aware and Self-Directed
1. Articulate their
Does not provide
reasons for study
reasons for study
within the context
or provides
of a liberal arts
irrelevant or
education
inappropriate
reasons for study
within a context of
liberal arts
education.
2. Describe,
Does not address
evaluate, and
all three aspects of
improve their own this outcome
learning processes (describe,
evaluate, and
improve) or
focuses only on
memorization of
isolated facts.
Basic
B
Proficient
P
Advanced
A
Provides one or more
valid reasons that
focus on positive
impact on one of the
following broad
areas: the student’s
personal,
professional, or civic
life.
Identifies more than
one learning strategy
and goes beyond
memorization of
isolated facts, but
concentrates on
learning within
specific courses
and/or provides
minimal discussion
related to evaluation
and improvement.
Provides valid
reasons that focus on
positive impact on at
least two of the
following broad
areas: the student’s
personal,
professional, and
civic life.
Identifies a variety of
learning strategies
and when they are
most effective.
Describes strategies
for improving
learning. The
response goes beyond
specific courses,
suggesting awareness
that learning is a lifelong activity and/or
that learning involves
making connections
across contexts.
Discusses a variety of
valid reasons that
focus on positive
impact on all of the
following broad areas:
the student’s personal,
professional, and civic
life.
3. Develop plans
for pursuing
learning goals
Does not provide a
plan to pursue
learning goals or
describes a plan
that focuses on
memorization of
isolated facts.
Provides a plan that
goes beyond
memorization of
isolated facts, but the
plan lacks sufficient
detail to make
effective learning
likely.
Provides a plan that
is likely to result in
effective learning.
The plan addresses at
least one major issue,
such as:
• time management
• use of learning
skills refined
through personal
experience
• need to monitor
learning and
possibly adapt the
plan
• need to make
connections across
contexts
4. Set, pursue, and
reflect upon their
learning goals
Does not address
all three aspects of
this outcome:
setting, pursuing,
and reflecting on
Addresses setting,
pursuing, and
reflecting on learning
goals, but the
response suggests
Addresses setting,
pursuing, and
reflecting on learning
goals in sufficient
detail to suggest self-
Response has all the
characteristics
indicating proficiency,
plus demonstrates
sophisticated
development of
learning skills that are
broadly applicable in
and out of the
classroom and that
involve making
connections across
contexts, such as
connecting academic
learning to personal or
professional
experiences.
Provides a plan that is
likely to result in
effective learning, as
well as sophisticated
discussion of at least
two major issues, such
as:
• time management
• use of learning
skills refined
through personal
experience
• need to monitor
learning and
possibly adapt the
plan
• need to make
connections across
contexts
Addresses setting,
pursuing, and
reflecting on important
learning goals and
indicates routine, on-
55
learning goals.
need for external
support from family
members, friends,
teachers, or others to
initiate and/or
complete at least one
of these processes.
reliant learning.
going reflection and
flexibility in revising
short- and long-term
goals and/or learning
strategies.
Identifies at least two
diverse or conflicting
concepts, viewpoints,
or priorities in the
situation being
addressed, but does
not elaborate in
sufficient detail to
demonstrate clear
understanding and/or
does not identify
obvious conflicts.
Recognizes that
others’ opinions and
viewpoints have
value, but shows lack
of discrimination or
analysis, as if all
perspectives are
always equally valid
or as if one’s own
perspective is always
superior.
Examines at least two
perspectives.
Identifies major
diverse or conflicting
concepts, viewpoints,
or priorities present
in the situation being
addressed.
Identifies major
diverse or conflicting
concepts, viewpoints,
or priorities present in
the situation being
addressed, as well as
subtle nuances and
complexities.
Demonstrates the
value of multiple
perspectives and
recognizes that one’s
own perspective is
not always superior
and that all
perspectives may not
be equally valid.
Response has all the
characteristics
indicating proficiency,
plus explores the
processes of evaluating
conflicting
perspectives and/or
demonstrates a
commitment to seek
out dissenting
viewpoints.
Examines the
phenomenon from
multiple viewpoints
and explores subtle
nuances and
complexities among
the viewpoints and/or
provides sophisticated
discussion evaluating
their relative merit.
Does not identify
connections or
focuses on invalid
connections.
Identifies valid
connections, but tends
to focus on the
obvious, such as
connecting related
disciplines.
Identifies valid
connections that go
beyond the obvious.
Does not identify
links or identifies
invalid links
among topics and
concepts presented
in different
Identifies valid links
among topics and
concepts in different
courses, but tends to
focus on the obvious
or does not fully
Identifies valid links
among topics and
concepts presented in
different courses,
goes beyond the
obvious, and explains
Multiple Perspectives
5. Identify diverse Does not identify
or conflicting
diverse or
concepts,
conflicting
viewpoints, and/or concepts,
priorities
viewpoints, or
priorities or
(revised May
identifies conflicts
2008)
that are irrelevant
to the situation
being addressed.
6. Articulate the
value of
considering
multiple
perspectives
Does not articulate
the value of
considering
multiple
perspectives.
7. Examine
phenomena from
multiple
viewpoints.
Considers the
phenomenon from
one perspective or
consistently favors
a single
perspective
(revised May
2008)
Make Connections
8. See connections
in seemingly
disparate
information
9. Recognize links
among topics and
concepts presented
in different
courses
Examines multiple
perspectives and
identifies some
relevant
commonalities and
conflicts.
Identifies valid
connections that are
subtle, sophisticated,
and/or creative and
discusses insights or
implications based on
these observations.
Identifies valid links
that are subtle,
sophisticated, and/or
creative and discusses
insights or
implications associated
56
courses.
10. Synthesizes
disparate facts,
theories, and
concepts
Does not
synthesize
disparate facts,
theories, and
concepts or
provides an invalid
synthesis.
explain the nature of
the links.
Provides a valid
synthesis, but does
not explicitly address
major relevant aspects
of the disparate
information.
the nature of the
links.
Provides a valid
synthesis that
explicitly addresses
major aspects of the
disparate
information.
11. Work within a Does not propose
Proposes simplistic or
context of diverse
a strategy, or
undeveloped
and conflicting
proposes
strategy(ies) for
concepts,
irrelevant or
working within this
viewpoints, and/or unreasonable
situation.
priorities (revised
strategy(ies) for
May 2008)
this situation.
Apply Skills and Knowledge to Different Contexts
12. Adapt what is
Does not adapt
Describes a valid
learned in one
what is learned in
adaptation, but the
situation to
one situation to
solution relies on
problems
problems in
concrete similarities
encountered in
another situation
between the two
another
or describes an
contexts.
invalid adaptation.
Describes reasonable
strategy(ies) for
working within this
situation.
13. Connect
intellectual study
to personal life
Does not connect
intellectual study
to personal life or
describes invalid
connections.
Describes valid
connections between
intellectual study and
personal life that go
beyond concrete
similarity between
the two contexts.
14. Draw on a
wide range of
knowledge to
make decisions
Does not present a
decision, does not
provide the
rationale for a
decision, or relies
on one line of
information to
make a decision.
Describes valid
connections between
intellectual study and
personal life, but the
connections rely on
concrete similarities
between the two
contexts.
Makes a decision
based on a narrow
range of knowledge,
perhaps applying
ideas from a single
course or discipline or
from closelyconnected disciplines.
Describes a valid
adaptation that goes
beyond concrete
similarity between
the two contexts.
Makes a reasonable
decision based on
more than a narrow
range of knowledge.
with the links.
Provides a valid
synthesis that
explicitly identifies
sophisticated or
creative connections
involving subtle
nuances and
complexities in the
disparate information.
Describes creative,
sophisticated
strategy(ies) for
working within this
situation.
Describes a creative
and/or sophisticated
adaptation that has the
potential for
developing more
effective solutions or
new insights about the
problem being
addressed.
Describes creative
and/or sophisticated
connections between
intellectual study and
personal life that lead
to new insights or
behaviors.
Makes a creative or
particularly effective
decision based on
sophisticated
integration of ideas
from a wide range of
knowledge.
*Developed with support from a Teagle Foundation grant. Retrieved January 4, 2008 from
Report on First Year at http://www.roanoke.edu/teagle
57
Student Conduct Essay Writing Rubric
Created by Vicki Castillon and Leonard Valdez at Sacramento State University; shared at a session at the 2008 WASC conference by
Lori Varlotta and Beth Merritt Miller. This rubric is used to assess required essays for students who violated campus policies.
Retrieved July 13, 2008 from http://4255856193304126708-a-wascsenior-org-s-sites.googlegroups.com/a/wascsenior.org/assessingstudent-learning-outside-the-classroom/Home/Sample_Instrumentspg1.doc?attredirects=0.
1. Student completely
lacks understanding/
knowledge. Significant
improvement needed.
2. Student
understanding/
knowledge is limited.
Improvement needed.
Demonstrates
knowledge of
policy violation
The student
demonstrates no
understanding of the
policy and how his/her
behavior violates policy.
The student
demonstrates limited
understanding of the
policy and how his/her
behavior violates policy.
Accepts
responsibility for
actions
The student
demonstrates no
acceptance of
responsibility for his/her
actions.
The student
demonstrates no increase
in awareness and
maturity.
The student
demonstrates no
commitment to refrain
from future policy
violations.
The student
demonstrates no
understanding of the
value of academic
integrity.
Demonstrates
increased
awareness and
maturity.
Demonstrates
commitment not
to violate
University
policy in the
future.
Demonstrates
understanding of
the value of
academic
integrity.
3. Student
understanding/
knowledge is adequate.
There is room for
improvement.
The student adequately
demonstrates
understanding of the
policy and how his/her
behavior violates policy.
4. Student
understanding/
knowledge is very good.
Little need for
improvement.
The student very clearly
demonstrates
understanding of the
policy and how his/her
behavior violates policy.
The student
demonstrates limited
acceptance of
responsibility for his/her
actions.
The student
demonstrates limited
increase in awareness
and maturity.
The student
demonstrates limited
commitment to refrain
from future policy
violations.
The student adequately
demonstrates acceptance
of responsibility for
his/her actions.
The student very clearly
demonstrates acceptance
of responsibility for
his/her actions.
The student adequately
demonstrates increase in
awareness and maturity.
The student very clearly
demonstrates increase in
awareness and maturity.
The student adequately
demonstrates
commitment to refrain
from future policy
violations.
The student very clearly
demonstrates
commitment to refrain
from future policy
violations.
The student
demonstrates limited
understanding of the
value of academic
integrity.
The student adequately
demonstrates
understanding of the
value of academic
integrity.
The student very clearly
demonstrates
understanding of the
value of academic
integrity.
5. Student
understanding/
knowledge is
exceptional. No
improvement needed.
The student
demonstrates thorough
and complete
understanding of the
policy and how his/her
behavior violates policy.
The student
demonstrates thorough
and complete acceptance
of responsibility for
his/her actions.
The student
demonstrates thorough
and complete increase in
awareness and maturity.
The student
demonstrates thorough
and complete
commitment to refrain
from future policy
violations.
The student
demonstrates thorough
and complete
understanding of the
value of academic
integrity.
SCORE
58
Lecture / Discussion 3:
Direct and Indirect
Assessment Methods
Barbara Wright
59
Using Direct and Indirect
Assessment Methods
Barbara D. Wright
Associate Director, ACSCU/WASC
[email protected]
The Assessment Loop
1. Outcomes,
questions
4. Use
3. Interpretation
September, 2009
2. Gathering
evidence
Level I Assessment Retreat, Long
Beach, CA
2
What is an “assessment method”?
† It’s how you collect the evidence,
direct or indirect, that will tell you
about the quality of your students’
learning (step 2 on the loop) and how
to improve it.
September, 2009
Level I Assessment Retreat, Long
Beach, CA
3
60
Direct? Indirect?
† Direct evidence demonstrates your
students’ learning directly, in an
unfiltered way.
† Indirect evidence is mediated by the
person responding to a questionnaire,
interviewer, etc. It is influenced by
perceptions, experiences, etc.
September, 2009
Level I Assessment Retreat, Long
Beach, CA
4
What do you use when?
† Direct evidence tells you what your
students know and can do in relation
to your learning outcomes
† Indirect evidence can reveal why and
how students learned what they
learned – or didn’t – if you ask the
right questions.
September, 2009
Level I Assessment Retreat, Long
Beach, CA
5
So what should we choose? It
depends on your question.
Choose the method(s) most likely to
provide evidence that will answer your
question.
†
†
†
†
†
Best practice: multiple methods
Direct evidence is the gold standard
Indirect evidence fills out the picture
Both are useful at step 3: interpretation
Descriptive data are the third major source
of evidence and also useful when combined
w/ other methods
September, 2009
Level I Assessment Retreat, Long
Beach, CA
6
61
Shifts in our understanding of
assessment
from . . .
to
† Isolated facts,
skills
† Memorization,
reproduction
† Comparing
performance
against other
students
September, 2009
† A full range of
knowledge, skills,
dispositions
† Problem solving,
investigating,
reasoning,
applying,
communicating
† Comparing
performance to
established criteria
Level I Assessment Retreat, Long
Beach, CA
7
Shifts in assessment, cont.
† Scoring right,
wrong answers
† a single way to
demonstrate
knowledge, e.g.
m/c or shortanswer test
† Simplified evidence
September, 2009
† Looking at the
whole reasoning
process
† Multiple methods &
opportunities, e.g.,
open-ended tasks,
projects,
observations
† Complex evidence
Level I Assessment Retreat, Long
Beach, CA
8
Shifts in assessment, cont.
† A secret,
exclusive & fixed
process
† Reporting only
group means,
normed scores
† Scientific
† A filter
† An add-on
September, 2009
† open, public &
participatory
† Disaggregation,
analysis,
feedback
† Educative
† A pump
† Embedded
Level I Assessment Retreat, Long
Beach, CA
9
62
Shifts in assessment, cont.
† “teacher-proof”
assessment
† Students as objects
of measurement
† episodic, conclusive
† Reliability
September, 2009
† Respect, support for
faculty & their
judgments
† Students as
participants,
beneficiaries of
feedback
† continual,
integrative,
developmental
† Validity
Level I Assessment Retreat, Long
Beach, CA
10
Choice of assessment method matters.
† Students value and learn what we
teach and test.
† How we teach and test matters as
much as what
† What and how we assess also
matters.
† We get more of what we test or
assess, less of what we don’t.
September, 2009
Level I Assessment Retreat, Long
Beach, CA
11
Higher-order thinking …
( adapted from L. Resnick, 1987)
† It’s nonalgorithmic, i.e., the path of
action is not fully specified in advance.
† It’s complex, i.e., the total path is not
“visible” from any single vantage point.
† It often yields multiple solutions, each
with costs and benefits.
† It requires nuanced judgment and
interpretation
† It involves application of multiple
criteria, which may conflict with one
another
September, 2009
Level I Assessment Retreat, Long
Beach, CA
12
63
Higher order thinking, cont …
† It often involves uncertainty; not
everything about the task is known or
can be.
† It requires self-regulation; someone
else is not giving directions.
† It involves making meaning, discerning
patterns in apparent disorder.
† It is effortful: the elaborations and
judgments required entail considerable
mental work and are likely to take time.
September, 2009
Level I Assessment Retreat, Long
Beach, CA
13
Other approaches to higher-order
learning:
† Bloom’s taxonomy
† Perry Scheme of Intellectual
Development
† Biggs’ and Entwistle’s work on
surface and deep learning
†?
September, 2009
Level I Assessment Retreat, Long
Beach, CA
14
The hierarchy of specificity
Institutional outcomes
College outcomes
Department & program
outcomes
Course-level
outcomes
September, 2009
Level I Assessment Retreat, Long
Beach, CA
15
64
The hierarchy of specificity
Oral and written communication
Professional communication
Ability to write for business
Ability to write a business plan
September, 2009
Level I Assessment Retreat, Long
Beach, CA
16
Where are skills practiced? Where can evidence be
gathered? Think horizontally as well as vertically.
Oral & written communication
Professional communication
Ability to write for business
Ability to write a business plan
Internship * Student government * Business courses * Gen Ed
September, 2009
Level I Assessment Retreat, Long
Beach, CA
17
The bottom line . . .
† Choose methods that are consistent
with shifts, trends in higher education
and assessment practice.
† Choose methods that support the
educational outcomes you value, e.g.
higher-order intellectual skills and
dispositions
September, 2009
Level I Assessment Retreat, Long
Beach, CA
18
65
Direct Assessment Methods -A Close-up Look
by
Barbara D. Wright
Associate Director,
Accrediting Commission for Senior Colleges and Universities
Western Association of Schools and Colleges
Alameda, CA 94501
[email protected]
September 10, 2009
66
Portfolios: collections of student work (and sometimes other material such as
transcripts, test scores, or performance reviews) intended to illustrate achievement of
learning outcomes. The mantra is “collect, select, reflect, connect.”
Advantages:
•
•
•
•
•
•
•
•
•
•
Are adaptable to different
ƒ levels of assessment (i.e. individual student, program, institution)
ƒ purposes (i.e. cross-sectional snapshot; change/progress over time)
ƒ kinds of materials (i.e. written work, tapes of performances,
student selfassessments)
Can tell us where student are and how they got there
Emphasize human judgment, meaning-making
Provide information likely to be used
Have become extremely popular, hence an easy sell
Engage students, faculty
Are educational for both students and faculty
Reduce fears of misuse
Can be managed by students – to some extent
Are supported by many different software programs
Disadvantages:
•
•
•
•
•
•
Can be labor-intensive
Can be cumbersome to store, navigate through
Must relate contents to articulated outcomes
Require carefully defined criteria for review, e.g. rubrics
Require training for reviewers
Require distinguishing between usefulness of the portfolio for students (e.g., to
showcase work, impress prospective employers, inform advisors) and for
assessment of learning
Solutions/responses:
•
•
•
•
•
•
Collect samples of work, not everything from everybody
Use electronic storage and retrieval
Give students responsibility for maintaining the portfolio
Invest in outcomes, because they’re the basis for everything anyway
Invest in good criteria for education’s sake
Invest in training for faculty development’s sake
67
Capstones: a wide variety of culminating projects, assignments, performances, or
even experiences, e.g., faculty-supervised community service, internships
Advantages:
•
•
•
•
•
•
•
•
Are cumulative
Are integrative
Are adaptable to demonstration of
ƒ skills
ƒ general education
ƒ professional field or major
ƒ dispositions
ƒ institutional outcomes
ƒ combinations
Are motivating for students
Set standards for degree completion, graduation
Provide an occasion for department-level discussion, interpretation
Invite external evaluation
Help students make the transition to
ƒ self-assessment
ƒ professional assessment
ƒ life-long learning
Disadvantages:
•
•
•
•
•
•
•
•
Pose challenge of capturing all students in their final year/semester
Differences within/among majors demand flexibility plus commonality
May mean an additional course requirement
Require coordinating multiple dimensions of learning & assessment
Can be labor-intensive
Must relate to carefully articulated outcomes
Require carefully defined criteria for review, e.g. rubrics
Require distinguishing between purpose of the capstone for students and for
program assessment
Solutions/responses:
•
•
•
•
•
Require the capstone for graduation
Introduce as widely as possible across the institution
Include capstone experiences within existing courses
Provide resources, staff support
View resources, labor, as worthwhile investment
68
Performances: activities, live or recorded, designed to demonstrate specific
outcomes, e.g. a poster presentation, conduct of a class, a musical or theatrical
performance, client counseling, facilitation of a group discussion, “think aloud” analysis
of a text.
Advantages:
•
•
•
•
•
•
•
•
•
Have face validity in terms of preparation for student’s real-life goals
Put emphasis on what the student can do (as opposed to knowing about):
ƒ require application
ƒ may require spontaneous adaptation, problem-solving
ƒ are integrative
ƒ provide a reality check
Give students with practical intelligence, skills, a chance to shine
Can elicit affective outcomes, e.g. poise, grace under pressure
Are motivating, encourage practice, rehearsing
Put the emphasis on active learning
Promote coaching relationship between students and faculty, especially when
there are external reviewers
Promote self-assessment, internalization of standards
Are highly adaptable, even to liberal arts
Disadvantages:
•
•
•
•
•
•
Can be labor-intensive, time-consuming, expensive
Must relate to articulated outcomes
Require careful definition of criteria, e.g. rubrics
Require careful training of reviewers, including external reviewers
Require coordination, scheduling, esp. of external reviewers
May frighten off insecure students
Solutions/responses:
•
•
•
•
•
Review a sample of students
Embed in routine, non-threatening situations (e.g., internship, clinical setting)
Use digital means to make performances accessible to reviewers
Regard outcomes, criteria, and training as an educational investment
Remind students they must demonstrate employability
69
Common assignments, template assignments, secondary
readings, and other embedded assessments: student work produced in
response to a course assignment is examined for multiple purposes, e.g., to determine
command of course material but also to assess writing skill, information literacy, critical
thinking, etc.
• “Common assignments”: the same assignment across multiple courses;
• “template assignments”: the same format but not identical assignment across
multiple courses
• “Secondary readings”: student work is examined “secondarily” for other qualities
beyond command of course material.
Advantages:
•
•
•
•
•
•
•
Use work produced by students as a normal part of their course work
Solve the problem of quality of student effort
Are efficient, low-cost
Have face validity
Provide maximally useful information for improvement with minimum slippage
Encourage discussion, collaboration among faculty & support staff
Can create campus-wide interest
Disadvantages:
•
•
•
•
•
•
Require considerable coordination
Can be time-consuming to create, implement
Can be time-consuming, labor-intensive to score
Must be designed in relation to specific outcomes
Require careful definition of criteria for review, e.g., rubrics
Require careful training of reviewers
Solutions/responses:
•
•
•
•
•
•
Focus on what’s important
Use “common questions” if an entire common assignment is impractical
Regard outcomes, criteria, and training as an educational investment
Provide support, “teaching circles’ to discuss implementation, findings
Remember the efficiencies, benefits
Make the investment
70
Course management programs: Software that allows faculty to set up chat
rooms, threaded discussions, etc., and capture student responses
Advantages:
•
•
•
•
•
•
•
•
•
•
•
Are adaptable to wide range of learning goals, disciplines, environments
Use work produced electronically by students as a normal part of course
participation
Record threaded discussions, chat, ephemera that are impossible or cumbersome
to capture face to face
Give quiet students an opportunity to shine
Can preserve a large volume of material, allow sorting, retrieval, data analysis
Are efficient, low-cost
Are unintrusive
Solve the problem of quality of student effort
Allow prompt feedback
Develop students’ metacognition when assessment results are shared
Often include tests, quizzes, tasks as part of package, supporting multiple-method
approach, convenience
Disadvantages:
•
•
•
•
•
•
•
•
•
Rely heavily on student writing skill, comfort with technology
Pose challenges to higher levels of aggregation beyond individual course or
student
May discourage collaboration among faculty, staff, programs
Managing large volume of material can be difficult, intimidating
“No significant difference” bias may short circuit improvement
Tests, quizzes may promote recall, surface rather than deep learning
Built-in survey tools encourage collection of indirect rather than direct evidence
Direct observation of student performances is difficult or impossible
Software may drive the assessment effort, instead of assessment goals and values
driving choice, use of the software
Solutions/responses:
•
•
•
•
•
•
•
Develop good, focused outcomes, criteria, rubrics
Use built-in data management tools
Supplement if necessary, e.g. with “The Rubric Processor”
Invest in training of faculty, external reviewers
Use tests, quizzes with caution, supplement with authentic tasks
Negotiate with the maker, customize the software
Aim for program-level, not just individual or course-level improvement
71
Classroom Assessment/Research: an approach to assessment pioneered by
K. Patricia Cross and Thomas A. Angelo; provides a large collection of techniques
individual instructors can use in their classrooms to discover what students are learning –
or not – and to make rapid adjustments.
Advantages:
•
•
•
•
•
•
•
•
Takes place at ground zero of learning process for:
ƒ maximum relevance, usefulness
ƒ minimum slippage
Offers maximum privacy, minimum risk, anxiety
Is conducted continuously, has formative benefit
Can provide feedback on both
ƒ what students know and can do
ƒ and how they got there, what helps or hinders
Motivates students to become more active, reflective learners
Can also be used by faculty collectively for the bigger picture
Is faculty-friendly, respectful of privacy, autonomy
Offers significant resources (e.g., T. Angelo and K. P. Cross, Classroom
Assessment Techniques,1992) and support networks, especially for community
college educators
Disadvantages:
•
•
Is unstructured, highly dependent on individuals’ cooperation for
ƒ administration of CATs (classroom assessment techniques)
ƒ reporting of results
Presents challenge of generalizing to program or institution level
Solutions/responses:
•
•
•
•
•
•
Provide consistent, careful leadership, oversight
Get buy-in from faculty, others
Start with agreement on shared outcomes, goals
Provide training
Make assessment a campus-wide conversation
Emphasize the potential for truly useful information for improvement
72
Student self-assessment: The student demonstrates the ability to accurately
self-assess a piece of work or performance, usually in relation to one or more outcomes
and a set of criteria, e.g. rubrics
Advantages:
•
•
•
•
•
•
•
The ultimate in active learning, engagement, ownership of one’s learning
Highly adaptable
Extremely educational for students
Promotes internalization of intellectual, personal, professional standards
Is an essential component of ongoing professional, personal development
Is an essential component of life-long learning
Faculty can aggregate individual results to identify general findings, trends
Disadvantages:
•
•
•
•
•
•
•
Challenging, especially at outset, for both students and faculty
Requires clear outcomes, criteria (e.g., rubrics), expectations for level of
proficiency
Requires student to assess with candor, not spin
May cause anxiety, avoidance
Long-standing habits, personality traits may need to be overcome (e.g., selfconsciousness, excessive modesty, unrealistically high self-appraisal)
Requires tact and true coaching attitude from instructor, ability to critique the
work or performance, not the person
Requires careful management of others who may be present
Solutions/responses:
•
•
•
•
•
•
•
Experienced instructors guide, mentor novice instructors
Students receive orientation, training
Outcomes, criteria, expectations are clear, widely distributed and understood
Examples of self-assessment are available
Process is presented as primarily developmental, formative
Examples of progress over extended time provide encouragement
Self-assessment is risk-free
73
Local tests: tests designed in relation to the specific course, program, or institution’s
curriculum and learning outcomes, as opposed to generic, commercially available tests.
Can be cumulative (e.g. comprehensives in the major) or less encompassing but still
cross-cutting. Format may vary; need not be multiple choice, as in most commercial tests.
Advantages:
•
•
•
•
•
•
•
•
•
Tests are traditional, widely accepted academic practice
Testing across courses or programs requires active faculty participation
Can stimulate discussion about alignment of goals, curriculum, pedagogy, etc.
Can be designed to have content validity
Can adapt readily to institutional changes in curriculum, outcomes
Can be open-ended, integrative, highly creative in format
Can provide good quality of student effort if course-embedded
Provide directly relevant, useful information
Forestall comparison with other institutions
Disadvantages:
•
•
•
•
•
•
•
•
•
Run risk of focusing more on surface than deep learning
Provide no norms for reference
May contain ambiguous, poorly constructed items
May offer questionable reliability and validity
May be expensive if test construction is contracted out
Will not elicit good quality of student effort if seen as add-on
Will create misunderstanding of assessment if seen as a threat
May become a missed opportunity to use more innovative approaches
May invite finger-pointing
Solutions/responses:
•
•
•
•
•
•
•
If norms, benchmarks are important, supplement with purchased test
Use on-campus expertise
Be careful, pilot any test before large-scale administration
Provide a “gripe sheet”
Accept that assessment is ultimately human judgment, not psychometric science
Keep the focus on useful information & improvement, not test scores per se
Depersonalize issues, avoid finger-pointing
74
Commercially available, standardized tests:
Advantages:
•
•
•
•
•
•
•
•
Are a traditional, widely recognized & accepted means of assessment
Require little on-campus time or labor
Prepare students for licensure, other high-stakes testing
Are norm-referenced
Offer longitudinal data, benchmarks
Are technically high-quality
May reflect recent, important trends in the field (e.g., ETS Major Field Tests)
Can be useful as part of a multiple-method approach
Disadvantages:
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
May offer poor content validity
Generally do not provide criterion-referenced scores
Test students’ ability to recognize “right” answers
Reflect students’ test-taking ability
Often elicit poor quality of student effort, particularly as add-on
Reinforce faculty bias toward “empty vessel” theory of education
Reinforce student bias toward education as memorizing, regurgitating “right”
answers (i.e. “surface” rather than “deep” learning)
Reinforce everybody’s bias toward assessment as testing
Carry risk of misuse of scores, invidious comparisons
Provide little insight into students’ problem-solving & thinking skills or ability to
discriminate among “good” and “better” answers
Offer no opportunity for test takers to construct their own answers verbally,
numerically, graphically, or in other ways
Give students no opportunity to demonstrate important affective traits, e.g.,
persistence, meticulousness, creativity, open-mindedness.
Are less likely than local methods to stimulate productive discussion
Are more likely to elicit finger-pointing, anxiety, resistance
Can be very expensive ($10-$30/student, plus administration costs)
Generally do not provide good value (i.e., useful information for cost)
Solutions/responses:
•
•
•
•
Test samples of students, use matrix sampling
Negotiate with test maker
Supplement with other methods
Use with caution
75
Direct or indirect? Some methods can work both ways . . .
Classroom research: Classroom research is included here as a direct method but
it can function as either a direct or an indirect method. Of the dozens of classroom
assessment techniques (or CATs) developed by Cross and Angelo, some demonstrate
what students know and can do, while others elicit reflection, perceptions, and other
forms of indirect evidence.
Course management programs: Course management programs make it
possible for faculty to capture discussions and other evidence that would be ephemeral in
the classroom; hence they are classified here as a direct method. Such programs often
include a survey or questionnaire template, however, that makes it easy to construct and
administer surveys online. See discussion of surveys in handout on “Indirect Methods.”
Focus groups: Focus groups are generally regarded as an indirect method of
assessment because students are encouraged to talk about their personal experiences and
perceptions. However, they can also function as a direct method, if the topic of discussion
is an issue in the major and students are guided by the protocol to demonstrate their
command of disciplinary concepts, theories and methods. In this case, students generally
do not receive a grade for their role in the discussion, but the recording is analyzed by
faculty to draw more general conclusions about the strengths and weaknesses of the
academic program.
Portfolios: Portfolios can function as both a direct and an indirect assessment
method. They are direct in the sense that student work is displayed and can be rated,
providing direct evidence of knowledge and skills. The reflective essays, in which
students look back on various pieces of their work, describe what each represented in
terms of challenges or achievements, and evaluate their personal progress as learners, are
indirect evidence of a high order.
Student self-assessment: Self-assessment is classified here as a direct method
because the performance of self-assessment demonstrates directly how skilled students
are at self-assessment. However, the process may be structured to elicit student reflection
on how learning occurred, what helped or didn’t, etc. In other words, self-assessment can
also function as an indirect method.
76
Indirect Assessment Methods -A Close-up Look
by
Barbara D. Wright
Associate Director,
Accrediting Commission for Senior Colleges and Universities
Western Association of Schools and Colleges
Alameda, CA 94501
[email protected]
September 10, 2009
77
Surveys: Common method of gathering information from people on a wide variety of
topics (personal characteristics, expectations, experience, attitudes, values, behaviors,
perceptions, satisfaction), generally in the form of a questionnaire, which may be
distributed in hard copy or online or – less often – administered by phone.
Advantages:
•
•
•
•
•
•
•
•
•
•
•
•
Are well-known, broadly accepted
Are adaptable to many different kinds of research questions
Are adaptable to different audiences (students, alums, employers, non-completers)
Items can vary in format, e.g., yes/no, rating scales, lists, open-ended questions
Can reveal the “why” and “how” behind the “what”
Results allow statistical analysis, reporting
Self-reports are generally truthful, accurate
Many surveys are commercially available, usually can be customized, e.g., NSSE,
CCSSE, CSEP, CCSEP, CIRP, Noel-Levitz
Purchased surveys provide norms, benchmarks, detailed reports
Software programs are available, e.g., Survey Monkey, Zoomerang
Software and email make surveys swift, cheap to administer
Data are easy to store and analyze
Disadvantages:
•
•
•
•
•
•
•
•
•
Construction of survey requires expertise, time, clarity about purposes
Hiring consultants and purchasing survey services can be costly
Surveys run danger of being too long, too broad
Response rate may be very low
Low response rate reduces representativeness, usefulness of results
Structured format reduces chance of unanticipated findings
Institutions often over-survey, leading to survey fatigue, wasted resources
Collected data are often not used, shared
Telephone surveys can be slow, expensive; becoming less popular
Solutions/responses:
•
•
•
•
•
•
•
•
Use on-campus talent to construct the survey, analyze results
Reward contributors
Be clear about purpose and educational outcomes to be investigated
Keep the survey as appealing, brief, easy to deal with as possible
Create “buzz” with pre-survey communications
Use reminders, incentives to increase response rate
Use “captive audiences” when appropriate, e.g., class meetings, seniors lined up
for commencement
Pool and consolidate campus survey efforts, make maximum use of existing data
78
Interviews: One-on-one conversations designed to elicit a variety of information;
may range from highly structured (much like an orally conducted survey) to open-ended
and exploratory.
Advantages:
•
•
•
•
•
More personal than the written survey
More appropriate for some audiences, e.g. high-status trustees, wealthy donors,
elusive non-completers
Allow for more probing, rephrasing, to elicit targeted information
Can reveal the “why” and “how” behind the “what”
Are useful as follow-up to survey results
Disadvantages:
•
•
•
•
Are labor-intensive at every stage: planning, scheduling, conducting, data
recording and analysis
Require skilled interviewers
Do not reach as large an audience as paper or online surveys
May elicit socially acceptable rather than accurate responses
Solutions/responses:
•
•
•
•
Be clear about purpose
Use selectively
Use in combination with other methods
Develop a basic protocol
79
Focus groups: structured, in-depth, group discussions of specific topics, guided by
a trained moderator and generally audiotaped, videotaped, or recorded by an assistant
moderator.
Advantages:
•
•
•
•
•
•
•
•
•
Allow examination of otherwise elusive perceptions, feelings, attitudes, ideas
Adaptable to wide variety of target groups, topics, issues
Offer insights into strengths, weaknesses of educational experience
Can reveal the “why” and “how” behind the “what”
Are useful in tandem with a survey project:
ƒ at the front end, as a way to identify productive topics, questions
ƒ at the back end to help interpret, clarify results
May reveal new, entirely unanticipated problems, insights
Can be implemented relatively quickly, cheaply
Rubric or matrix may be used to score multiple focus groups and arrive at
findings
Can do double duty as a direct method, too
Disadvantages:
•
•
•
•
•
•
•
•
Moderators must be identified, trained
Development of the topics, questions, and matrix requires care
Sensitive topics may not lend themselves to focus group discussion
Scheduling can be a challenge
Smaller numbers of students are reached than with surveys
Incentives for student participation may be needed
Conduct of individual focus groups will necessarily vary
Results may not led themselves to statistical analysis, generalization
Solutions/responses:
•
•
•
•
•
Use campus expertise, volunteers, to keep costs down
Train new moderators by having them observe skilled moderators
Present participation in focus group to students as privilege, opportunity
Share interesting, surprising findings broadly, but keep identities confidential
Use as an opportunity to show the institution listens carefully, takes student
seriously
80
Ethnographic research: Selected students serve as participant-observers,
gathering information about learning and/or student experience through conversations
with fellow students, observation, and reflection on their own experiences. Participantobservers meet regularly with faculty and/or staff conducting the study to refine
questions, share findings, analyze them, and plan next steps.
Advantages:
•
•
•
•
•
•
•
•
Provides an insider perspective otherwise unavailable
Allows longer-term inquiry, e.g., a semester as opposed to one-time interview
Allows in-depth study, exploration of “why” and “what to do” as well as “what”
Provides access to elusive values, attitudes
Can include non-verbal information such as body language, demeanor
Has potential to produce unanticipated, surprising findings
Has high likelihood of producing useful, actionable information
Is adaptable, e.g., to student life as well as academic issues
Disadvantages:
•
•
•
•
•
•
Requires careful definition of the topic of study
Is time-consuming
Requires training, continuing attention, regular meetings
Quality, commitment of participant-observers may vary
Attrition of participant-observers may reduce usefulness of results
Few models are available
Solutions/responses:
•
•
•
•
•
•
Choose participant-observers carefully
Provide training, incentives, support
Focus the inquiry but allow for evolution of project, adaptation to unexpected
findings
Provide incentives to participant-observers and /faculty/staff coordinating project
Create a risk-free environment
Avoid identification of individuals when reporting findings
81
Direct or indirect? Some methods can work both ways . . .
Classroom research: Classroom research is described here as a direct method but
it can function as either a direct or an indirect method. Of the dozens of classroom
assessment techniques (or CATs) developed by Cross and Angelo, some demonstrate
what students know and can do, while others elicit reflection, perceptions, and other
forms of indirect evidence.
Course management programs: Course management programs make it
possible for faculty to capture student discussions and other performances that would be
ephemeral in the classroom; hence they are classified here as a direct method. Such
programs often include a survey or questionnaire template, however, that makes it easy to
construct and administer surveys online. See discussion of surveys in this handout on
“Indirect Methods.”
Focus groups: Focus groups are generally regarded as an indirect method of
assessment because students are encouraged to talk about their personal experiences and
perceptions. However, they can also function as a direct method, if the topic of discussion
is an issue in the major and students are guided by the protocol to demonstrate their
command of disciplinary concepts, theories and methods, or other learning. In this case,
students generally do not receive a grade for their role in the discussion, but the recording
is analyzed by faculty to draw more general conclusions about the strengths and
weaknesses of the academic program.
Portfolios: Portfolios can function as both a direct and an indirect assessment
method. They are direct in the sense that student work is displayed and can be rated,
providing direct evidence of knowledge and skills. The reflective essays, in which
students look back on various pieces of their work, describe what each represented in
terms of challenges or achievements, and evaluate their personal progress as learners, are
indirect evidence of a high order.
Student self-assessment: Self-assessment is classified here as a direct method
because the performance of self-assessment demonstrates directly how skilled students
are at self-assessment. However, the process may be structured to elicit student reflection
on how learning occurred, what helped or didn’t, etc. In other words, self-assessment can
also function as an indirect method.
82
Lecture / Discussion 4:
Unique Issues in
Assessment for
Community Colleges
Fred Trapp
83
Unique Issues in Assessment
for Community Colleges
Presented by
Fred Trapp
Cambridge West Partnership, LLC
Administrative Dean, Institutional Research/Academic Services (retired)
Long Beach Community College
[email protected]; [email protected]
What Is Unique?
„
„
„
„
„
„
„
„
„
Transfer function
Developmental education function
State mandates, regulations & mandatory
curriculum documentation
What is a program?
Grading vs. assessment
Liberal arts/general education “deli”
Flex days, faculty professional development $
Commission rubrics, deadlines, reports
Career & technical programs (another show)
2
The Learning Improvement Cycle:
1. Define/Refine student learning outcomes
based on input from stakeholders.
2. Design
Assessment tools, criteria &
standards directly linked to
each outcome
6. Document results
& outline needed changes in
curriculum, instructional materials,
teaching strategies, or assessment means.
3. Implement Assessment
tool(s) to gather evidence of
student learning.
5. Identify gaps
between desired & actual
results.
4. Analyze and
evaluate the collected data
(make sense of it)
see also examples at end of Handout Packet
3
84
The Transfer Function
„
Using Administrative Data
„ How well do our students perform
upon transfer?
• GPA after one year
• GPA in senior year
• Transcript analysis
4
GPA Comparison
CC Transfers After One Year at CSU
3.00
2.50
2.00
1.50
1.00
0.50
0.00
Fall 1
Fall 2
Fall 3
CC System
Fall 4
Fall 5
Your CC
http://www.asd.calstate.edu/performance/index.shtml
5
Family Educational Rights &
Privacy Act (FERPA)
„
„
Protects student educational records
Allows disclosure for research to improve
instruction. (section 99.31 of FERPA
regs.)
„
„
When research reported, personal identity of
students must be suppressed.
Data must be destroyed when study is
completed.
http://www.ed.gov/policy/gen/guid/fpco/index.html
6
85
Senior Year
Student Performance Report
Crs Abv.
Crs No.
ACCY
301
Course Title
Cr. Hr
Admin Accounting
Grade
3
C
ECON
230
Economic Statistics I
3
C
FIN
353
Real Estate Valuation
3
C
FIN
451
Real Estate Law
3
B
MGMT
372
Operations Mgmt I
3
B
MGMT
491
Organization Behavior
3
C
7
Transcript Analysis
ECON 490: Money and Banking (CSU upper division)
Community College
(99 students)
Average Grade 2.69
Non Community College
(1,054 students)
Average Grade 2.50
http://www.CalPASS.org
8
Transferred Students
Opinion Data
Survey former students
„
–
–
What did we do well to help you?
–
What do we need to do to improve?
„
„
„
Do you feel as prepared for junior and senior
level work as other students?
Collaborate with the four-year school
Exploit holidays if you send the survey yourself
Perhaps focus groups
9
86
Developmental Ed Function
„
Programs and services that commonly
address
„
„
„
„
Academic preparedness
Development of specific learning strategies
Affective barriers to learning
Typical program components
„
„
„
„
Writing
Mathematics
Reading
Study skills
http://www.ncde.appstate.edu/
10
Global Outcomes & Skills for
Developmental Education
„
Ability to
„
„
„
„
„
Write at the college level
Perform mathematical processes
Read with comprehension
Use appropriate study skills
Basic skills initiative (BSI)*
„
Events, effective practices, resources, publications
* http://www.cccbsi.org/
11
Developmental Education
Assessment Plan
Goal/Statement of Purpose- To provide quality developmental
education for students who need basic academic skills.
Program Intended Ed. Outcomes
Program completers will …
1. Be prepared to perform successfully in college
Level composition courses.
2. Perform mathematical processes well enough
to complete freshman college mathematics
requirements.
3. Be successful in first semester college courses
that require significant reading.
12
87
Developmental Education
Means of Assessment
„
Rely on diagnostic assessment and placement
„
„
Standardized exams
„ ASSET
„ COMPASS
„ Accuplacer
„ MDTP
Locally prepared/managed exams
http://www.cccco.edu/ChancellorsOffice/Divisions/StudentServices/Matriculation/MatriculationArchives
/tabid/627/Default.aspx
act.org; ets.org
13
Developmental Education, Questions
& Means of Assessment
„
How well are they doing compared to a
standard?
„
„
How much did they gain?
„
„
Post-test overall exam score
Pre & Post test
Where are large numbers of students still
weak?
„
Analysis of sub-scores
14
Developmental Education, Questions
& Means of Assessment
„
Over several assignments, how well are they
doing compared to our expectations?
„
„
How well do they do in subsequent courses?
„
„
Portfolio of work
Success in entry-level college courses
Have their attitudes changed and how?
„
Surveys of attitudes
15
88
Developmental Education
Assessment Plan
Program Intended Ed.
Means of Assessment & Criteria for Success
Outcomes
1a. At least 70% of the students will complete college composition
Program completers will …
on the first attempt.
1. Be prepared to perform
1b. A faculty panel will use a rubric & rate at least 90% of the
successfully in college level
completers as satisfactory (4 or 5) on a 5-point scale for each
composition courses.
category of the exit writing sample.
2. Perform mathematical processes
2a. Seventy percent of the completers will score 75% or higher
well enough to complete freshman
on a locally devised test of pre-college math competencies.
college mathematics
requirements.
2b. Seventy percent of completers will score 75% or higher on each
3. Be successful in first semester
3a. Eighty percent of completes will score 85% or better on a standard
college courses that require
reading comprehension exit test.
significant reading.
3b. Eighty percent of completers will score at or above the national
sub score area of the locally devised test.
average on the post test sub score for reading & vocabulary. No completer
will score below the 40th percentile.
16
Developmental Education
Assessment Plan & Report
Means of Assessment & Criteria for Success
Summary of Data Collected
1a. At least 70% of the students will complete college composition
1a. Eight-two percent (338 or 412 students)
on the first attempt.
completed ENGL 1.
1b. A faculty panel will use a rubric & rate at least 90% of the
1b. Organization - 92%; Grammar- 78%
completers as satisfactory (4 or 5) on a 5-point scale for each
Rhetoric- 89%
category of the exit writing sample.
2a. Eight-one percent of completers scored
2a. Seventy percent of the completers will score 75% or higher
75% or better.
on a locally devised test of pre-college math competencies.
2b. Linear equations- 90%; Inequalities- 84%
2b. Seventy percent of completers will score 75% or higher on each
Quadratic equations- 88%; Graphing- 62%
sub score area of the locally devised test.
Powers & roots- 86%; Proportions- 78%
3a. Eighty percent of completes will score 85% or better on a
standard reading comprehension exit test.
3a. Eighty-nine percent (213 of 239 students)
3b. Eighty percent of completers will score at or above the national
average on the post test sub score for reading & vocabulary. No
completer will score below the 40th percentile.
Scored 85% or better.
3b. Sixty-two percent scored at or above the
national average on reading; 88% scored at or
above the national average on vocabulary.
17
Developmental Education
Logistics of Assessment*
„
„
Obtaining data for students recommended to but who do
not take developmental courses
Tracking students though basic skills migration and into
the first college-level course in the discipline
„
Obtaining feedback from college-level course faculty
„
Tracking persistence to completion
„
Basic skills initiative work on campus will dovetail
see your campus Research Office
18
89
State Mandates, Regulations &
Required Curriculum Documentation
„
State Mandates & Regulations- CCAR, Title 5,
Chapter 6
„
„
Curriculum documentation- Outline of record
(section 55002)
Integrated course outline (Academic Senate)
„
„
Learning Outcomes
Typical
„
„
„
„
Means of classroom instruction
Out of class Assignments
Means of evaluation and criteria
Equates to a plan for course-level assessment
19
Objectives vs. SLOs
„
„
„
Both can be expressed in terms of what
the student knows, believes and can do
To some, objectives are more discrete
or smaller pieces of learning
To some, SLO’s are the “big ideas”
20
Integrated Course Outline of Record
As an Assessment Plan
Statement of Purpose: Role of the course in curriculum (commonly captured on supporting
documents and not necessarily in the outline of record)
Intended Ed.
Outcome*
Means & Criteria
for Assessment*
Results of
Assessment**
Use of Results**
Student learning
Representative
assignments,
instructional methods,
kinds and criteria for
evaluation
Actual performance data
on assessments.
Action steps taken
outcomes
Instructional
Strategies*
Not part of the
assessment plan, but
recorded in the outline of
record.
Not part of the outline,
but recorded separately
(course portfolio,
program plans/reviews).
Not part of the outline,
but recorded separately
(course portfolio,
program plans/reviews).
*in the Course Outline of Record
**in the assessment report
see the Course-level SLO Assessment Plan Worksheet
21
90
Curriculum Review & Approval Process
Curriculum review and approval process (Title 5,
section 55002)
„
„
Integration and alignment questions
„ Is the course outline integrated?
•
„
„
„
Course outcomes, assignments or activities, methods of
instruction, and evaluation strategies
How does the course support program outcomes?
How does the course support GE outcomes?
Course sequences & prerequisites (T5 section 55500-30)
22
Good Practice in Course Design to
Assess Learning Outcomes
„
„
„
„
Where do we find evidence of student
learning?
How do we observe and document that this
learning is taking place?
What are some key elements of student work
to which we pay attention? (Primary traits)
Development and use of rubrics, exams, etc.
to evaluate learning evidence.
23
Good Practice in Course Design- Shaping
Outcomes and Instructional Activities
Key Questions for Faculty
1.
2.
3.
4.
Intended outcomes- What do students need to be able to do “out
there” in life roles for which this course prepares them?
Assessment task- What can students do “in here” to demonstrate the
intended outcomes?
Concepts & issues- What do students need to understand
(knowledge) in order to demonstrate the intended outcomes?
Skills- What skills do students need that are essential to the intended
outcomes?
Ruth Stiehl The Outcomes Primer 2002
24
91
Course-level Assessment Example
Themes,
Concepts,
Issues
Assessment
Tasks
Skills
Student
Learning
Outcomes
25
Curriculum Alignment
Assessments
Objectives/
Outcomes
Curriculum
Alignment
Instructional Activities/
Materials
see Bloom's Revised Taxonomy in the Handout Packet
26
What is an Instructional Program?
Regulation & Local Definition
„
State/Governing Authority Authorized Degree or
Certificate Program Award
„
„
https://misweb.cccco.edu/webproginv/prod/invmenu.htm
General Education Patterns or Certificate of
Achievement
„
Within a vocational certificate (SCANS)
„
Developmental Instruction
„
Discipline Pathways and Transfer Preparation
„
Noncredit Instruction
„
„
Locally defined programs
Locally defined programs
27
92
Good Practice in Program DesignShaping Outcomes and Instructional
Activities
Key Questions for Faculty
1.
2.
Intended outcomes- What do students need to be able to do
“out there” in life roles for which this program prepares them?
Capstone assessment tasks- What can students do in this
program to show final evidence of the intended outcomes?
If a capstone is not available…..
3.
4.
Courses- What learning experiences (courses) are necessary to
prepare the student?
Prerequisites- What must students be able to do before
engaging in this learning?
Ruth Stiehl The Outcomes Primer
28
Program Assessment Example
Prerequisites
Courses
Capstone
Assessment
Tasks
Student
Learning
Outcomes
29
Alignment Illustration
Computer Information Science
CIS 200
Course
Outcomes
College (GE)
Competency
Program
Outcome
Critical Thinking &
Problem Solving
Analyzes &
designs a
solution when
presented with
a business,
math or science
problem
according to
specifications
set by the
department
Means of
Assessment
and Std for
Success
Analyzes given
problems for
specifications in
terms of input,
output and
processing.
For each problem
the learner
correctly:
Designs a
solution to a
given problem.
Completes a
flowchart or
algorithm
Assessment
Results & Use of
Results
Determines what is
to be read,
printed, and
converted
Designs test cases
Anne Arundel College, Arnold, MD
30
93
Curriculum Map for Program Assessment
(no capstone course)
Outcome 1
Outcome2
Crs 1
I
Crs 2
R/D/P
I
Crs 3
M
R/D/P
Crs 4
Outcome 3
Outcome 4
I
R/D/P
I
R/D/P
Crs 5
R/D/P
M
Crs 6
M
I= introduced
M
M= mastery demonstrated
R/D/P= re-enforced/developed/practiced
Do your program-level assessment work in the courses where mastery is demonstrated.
31
Embedded Assessments Matrix
Linking Course to GE/Program Outcomes
„
„
Linking courses to general education/program
core competencies
For each course provide a rating score on this
outcome using these behavioral descriptors
„
„
„
„
0 Does not include instruction in this area
1 Includes some instruction or practice & assessment
2 Addresses the outcome as the focus in 20% or more of
the course.
3 Addresses the outcome as the focus in 33% or more of
the course
Bellevue College, Bellevue, WA
32
Embedded Assessments Matrix
Linking Course to GE/Program Outcomes
Outcome 1 Outcome2 Outcome 3 Outcome 4
Crs 1
1
Crs 2
2
Crs 3
3
Crs 4
1
1
2
2
1
2
Crs 5
2
3
Crs 6
3
1= very little emphasis
3
2= modest emphasis
3= major emphasis
Do your program-level assessment work in the courses where a major emphasis is placed.
33
94
Means of Assessment- Grades
„
Evaluation of individual students (grading) =
assessment
„
Focus is individual not groups of students
A summative, not formative act
Objectivity of single evaluator vs. group
Generally not accepted as direct evidence
„
Use the grading process
„
„
„
„
„
Agreed upon course exam or part of exam
Row and column model for assignments
see Handout- Using the Grading Process for Assessment
34
Embedded Assessment Strategy
(Row and Column Concept)
Criteria
Tim Jane Mary Joe Dave Average
Spelling
Grammar
3
2
4
5
1
3
2
2
3
5
2.6
3.4
Punctuation
Structure
4
3
5
2
2
3
3
5
4
3
3.6
3.8
Total
13
17
10
12
15
Student Grade
C
A
D
C
B
Total down the column for individual grading. Analyze across the row for
assessment of intended outcomes from the group.
Jim Nichols
35
The Grading Processes
Implications for Assessment
„
Using the Grading Process & Existing
Assignments (stealth assessment)
„
Build a grading rubric for an assignment
„
„
„
Determine the criteria on which you evaluate student
work (primary trait analysis)
Describe shades of performance quality
As a faculty group, sample student work and
apply the rubric
see Handouts- Rubrics & Johnson Co. CC example
36
95
Course-level Assessment
(a requirement once unique to ACCJC)
„
Strategies to select a starting point
„
„
„
„
Courses with major work (emphasis)
aligned to a GE outcome
Student equity & basic skills initiative
courses
CTE program capstone course or course
with a capstone project
Stand-alone courses via routine review
cycle on course outlines or discipline
(program) review cycle
see Handout Packet- Choosing the Right Assessment Tools
37
Liberal Arts- Assessing Outcomes
Disciplinary/Professional Societies
„
„
„
„
„
„
„
„
„
Carnegie Academy for the Scholarship of Teaching &
Learning (CASTL)
American Psychological Association (APA)
Association of College & Research Libraries (ACRL)
National Institute for Science Education (NISE)
National Communications Association (NCA)
American Sociological Association (ASA)
American Historical Association (AHA)
Association for Institutional Research (AIR)
see list of URLs in the Handout Packet
38
Liberal Arts- Assessing Outcomes
„
Consortia and Associations
„
„
„
„
„
„
Quality Undergraduate Education Project (QUE)
Association of American Colleges & Universities (AAC&U)
League for Innovation in the Community Colleges (Learning
Outcomes for the 21st century & Getting Results)
National Postsecondary Educational Cooperative- Sourcebooks
on Assessment (NPEC)
Research, Planning & Assessment Group (RP Group)
California State University, Institute for Teaching and Learning
(CSU, ITL)
see list of URLs in the Handout Packet
39
96
General Ed-Assessment Strategies
„
Embedded
„
Map the curriculum to the outcomes
„
Identify courses that qualify
„
•
Bellevue example
„
Cascade to course outcomes and activities
„
Operationally define the outcomes
•
•
„
„
Mira Costa example
Palomar example
Build rubrics
Sample assignments
Exams (standardized or locally developed)
„
„
„
Outcomes related to exam content
Sample classes
CAAP- share the burden
40
Mira Costa Example
„
Started with three programs- general ed.;
career & technical education; noncredit
„
„
„
Created mission statements for each area of
GE
Mapped courses to GE outcomes
Use on-line reporting form- 1 outcome per
course
see Handouts- Area B Mission Statement & Matrix; Assessment
Reporting Form
see http://www.miracosta.edu/Governance/Outcomes/index.htm
41
Palomar College- Communication, GE
Outcome- (1 of 6 core skills)
„
„
Students will communicate effectively in many different situations, involving
diverse people and viewpoints. (core skill)
„
1. Speaking: Students will speak in an understandable and organized fashion to
explain their ideas, express their feelings, or support a conclusion.
„
2. Listening: Students will listen actively and respectfully to analyze the substance of
others’ comments.
„
3. Reading: Students will read effectively and analytically and will comprehend at the
college level.
„
4. Writing: Students will write in an understandable and organized fashion to explain
their ideas, express their feelings, or support a conclusion.
Performance benchmarks (beginning, developing and accomplished) are
available for each outcome.
http://www.palomar.edu/alp/ (historic URL, look for core skills hot link)
Palomar College, San Marcos, CA
42
97
Speaking Core Skill Demonstrated
Competence Levels
„
Beginner (1 of 4 descriptors)
Clearly state and address assigned topic
„
„
Developing (1 of 8 descriptors)
Develop a clear thesis
„
„
Accomplished (1 of 5 descriptors)
Support a clear thesis, with supporting points, that move to
a conclusion
„
Palomar College, San Marcos, CA
43
Flex Days and Assessment
(a resource to “die for”)
„
Faculty learning communities
„
„
„
„
„
„
Reviewing sample student work
Comparing it to criteria and standards for
performance
Making sense of the data
Planning/implementing future changes
Poster sessions or sharing of experiences with
others
Department meetings
44
WASC Senior Commission Rubrics
„
„
„
„
„
Quality of academic program learning
outcomes
Use of capstone experiences for
assessment
General education assessment process
Use of portfolios for assessment
Integration of student learning
assessment into program review
see 5 WASC Rubrics in the Assessment Retreat Materials
45
98
ACCJC SLO Assessment Rubric
„
Levels of Implementation
„
„
„
„
Awareness
Development (presently expected)
Proficiency (by Fall 2012)
Sustainable continuous quality
improvement
see Handout Packet- ACCJC SLO Progress Rubric
46
Homework
(using the ACCJC SLO Progress Rubric)
Compare each bullet point in the
development vs. proficiency levels of
implementation
„
1.
2.
3.
4.
What is the difference between the levels for each point?
How is each point accomplished at your college?
What evidence do you have to support your answer to
question #2?
Where is that evidence kept?
see Handout Packet- ACCJC SLO Progress Rubric
47
The Key Ideas . . .
„
„
„
Being focused on learning evidence: “How do
I know that my students are learning?”
Engaging in dialogue with colleagues on what
is working with our courses/programs.
This is an ongoing cycle of continuous
improvement.
see Handout Packet- examples of assessment cycles
48
99
Handout Packet for
Unique Aspects of Assessment
In Community Colleges
Presentation
Fred Trapp, Ph.D.
Cambridge West Partnership, LLC
Administrative Dean,
Institutional Research/Academic Services (retired)
Long Beach City College
September 2009
100
Web References cited in Unique Issues in Assessment for Community Colleges
California State University performance index
http://www.asd.calstate.edu/performance/index.shtml
Family Educational Rights & Privacy Act (FERPA)
http://www.ed.gov/policy/gen/guid/fpco/index.html
CalPASS
http://www.CalPASS.org
National Center for Developmental Education
http://www.ncde.appstate.edu
California Community College Basic Skills Initiative
http://www.cccbsi.org/
American College Testing
http://www.act.org
Educational Testing Service
http://www.ets.org
California Community College Chancellor’s Office, Student Services, Matriculation,
Matriculation Archives
http://www.cccco.edu/ChancellorsOffice/Divisions/StudentServices/Matriculation/Matriculation
Archives/tabid/627/Default.aspx
California Community College Chancellor’s Office, Academic Affairs, Program Inventory
https://misweb.cccco.edu/webproginv/prod/invmenu.htm
Carnegie Academy for the Scholarship of Teaching and Learning (CASTL)
http://www.carnegiefoundation.org/programs/index.asp?key=21
American Psychological Association (APA)
http://www.apa.org/ed/eval_strategies.html
Association of College and Research Libraries (ACRL)
http://www.ala.org/ala/mgrps/divs/acrl/issues/infolit/index.cfm
Field Tested Learning Assessment Guide for Science, mathematics Engineering & Technology
(FLAG) Project of the National Institute for Science Education (NISE)
http://www.flaguide.org/
101
National Communications Association (NCA)
http://www.natcom.org/index.asp?bid=264
American Sociological Association (ASA)
http://www.enoah.net/ASA/ASAShopOnlineService/productslist.aspx?CategoryID=ASACDDM&selection=3
American Historical Association (AHA)
http://www.historians.org/perspectives/issues/2009/0903/0903for2.cfm
Association for Institutional Research (AIR)
http://www.airweb.org/?page=1217
Quality Undergraduate Education Project (QUE)
http://www2.gsu.edu/~wwwque/about/index.html
(Chemistry, Biology, History, English)
Association of American Colleges and Universities (AAC&U)
http://www.aacu.org/
League for Innovation in the Community Colleges (Project-Learning Outcomes) also (Getting
Results: On-line Professional Development for Faculty)
http://www.league.org/gettingresults/web/module6/assessing/index.html
National Postsecondary Education Cooperative (NPEC)
http://nces.ed.gov/NPEC/
Research, Planning & Assessment Group (RP Group) of the California Community Colleges
http://www.rpgroup.org/
California State University, Institute for Teaching and Learning (CSU, ITL)
http://www.calstate.edu/ITL/
Rubistar (free tool to help create rubrics)
http://rubistar.4teachers.org/index.php
Mira Costa Community College
http://www.miracosta.edu/governance/Outcomes/index.htm
Palomar College (historic site for outcomes, look for core skills)
http://www.palomar.edu/alp/
3
Fred Trapp, Cambridge West Partnership, LLC;
[email protected]
102
Course-Level Student Learning Outcome Assessment Plan & Report
Course:
Department/Program:
Statement of Purpose (role of the course in the curriculum), GE area________, required in ______ program, elective in
_______ program, etc.:
Intended Educational
Outcomes:
What do students demonstrate
that they know or can do in
your course? (SLO)
1.
2.
Means and Criteria for
Assessment:
What activities/assignment/
instrument/methodology will
you use to produce evidence of
student mastery of this
outcome? Describe the
approach you will take to
assess the outcome (Who, When
& What is Success?)
1a.
1b.
2a.
2b.
Results of Assessment
Use of Results:
Describe what actually
happened- when, how many &
in what way were students
assessed. How well did the
students perform- how many
accomplished your standard of
success? What sense do you
make of these results?
Comparing your expectations
to the results, what changes
have you made in pedagogy,
assessment means, or standard
of success? What are the
implications for further
assessment work?
103
Bloom's Revised Taxonomy
Bloom created a learning taxonomy in 1956. During the 1990's, a former student of Bloom's, Lorin
Anderson, updated the taxonomy, hoping to add relevance for 21st century students and teachers. This new
expanded taxonomy can help instructional designers and teachers to write and revise learning outcomes.
Bloom's six major categories were changed from noun to verb forms.
The new terms are defined as:
Remembering
Retrieving, recognizing, and recalling relevant knowledge
from long-term memory.
Understanding
Constructing meaning from oral, written, and graphic
messages through interpreting, exemplifying, classifying,
summarizing, inferring, comparing, and explaining.
Applying
Carrying out or using a procedure through executing, or
implementing.
Analyzing
Breaking material into constituent parts, determining how
the parts relate to one another and to an overall structure or
purpose through differentiating, organizing, and
attributing.
104
Evaluating
Making judgments based on criteria and standards through
checking and critiquing.
Creating
Putting elements together to form a coherent or functional
whole; reorganizing elements into a new pattern or
structure through generating, planning, or producing.
Because the purpose of writing learning outcomes is to define what the instructor wants the student to do
with the content, using learning outcomes will help students to better understand the purpose of each activity
by clarifying the student’s activity. Verbs such as "know", "appreciate", "internalizing", and "valuing" do not
define an explicit performance to be carried out by the learner. (Mager, 1997)
Unclear Outcomes
Revised Outcomes
Students will know described
cases of mental disorders.
Students will be able to review a
set of facts and will be able to
classify the appropriate type of
mental disorder.
Students will understand the
relevant and irrelevant numbers
in a mathematical word problem.
Students will distinguish between
relevant and irrelevant numbers
in a mathematical word problem.
Students will know the best way
to solve the word problem.
Students will judge which of the
two methods is the best way to
solve the word problem.
Examples of unclear and revised outcomes.
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching and assessing: A
revision of Bloom's Taxonomy of educational outcomes: Complete edition, New York : Longman.
Cruz, E. (2003). Bloom's revised taxonomy. In B. Hoffman (Ed.), Encyclopedia of Educational Technology.
Retrieved August 22, 2007, from http://coe.sdsu.edu/eet/articles/bloomrev/start.htm
Forehand, M. (2005). Bloom's taxonomy: Original and revised.. In M. Orey (Ed.), Emerging perspectives on
learning, teaching, and technology. Retrieved August 22, 2007, from http://projects.coe.uga.edu/epltt/
105
Using the Grading Process for Assessment
To be helpful to faculty who want to improve student performance as well as to serve the goals of program
and general education assessment of student learning, grading must be seen as a process that includes:
1. Identify the most valuable kinds of learning in a course and articulate those outcomes
2. Construct exams and assignments that will match and test that learning outcome
3. Set standards and criteria that is assignment, exam or performance specific
4. Use primary trait analysis to build a scoring rubric*
5. Guide student learning
6. Implement changes in teaching that are based on information from the grading process
The classroom grading process, with well-constructed rubrics, can be harnessed for program or general
education assessment. In doing so, two assumptions are being made:
1. Whatever learning you are trying to promote across the curriculum is being taught and assessed
now.
2. Learning skills such as critical thinking or problem solving is context-specific in the disciplines.
A program faculty or general education committee might want to do or know the following:
1. Assure that effective classroom assessment is taking place.
2. Find the common learning expectations among courses.
3. Check the sequence of skills taught in the program.
4. Identify what is required of graduates.
5. Isolate strengths and weaknesses in student performance at the conclusion of the
6. Track student performance over time.
program.
*see the handout on rubrics
Source:
Walvoord, Barbara and Anderson, Virginia. Effective Grading: A Tool for Learning and Assessment.
Jossey Bass, San Francisco, 1998. ISBN 0-7879-4030-5
Other good sources:
Milton, Ohmer; Pollio, Howard; and Eison, James. Making Sense of College Grades. Jossey Bass, San
Francisco, 1986. ISBN 0-87589-687-1
Wiggins, Grant. Educative Assessment: Designing Assessments to Inform and Improve Student
Performance. Jossey-Bass, San Francisco, 1998. ISBN 0-7879-0848-7
106
Rubrics Handout
A rubric is a scoring tool that divides assignments into component parts or criteria used for evaluation. It provides a
detailed description of what is acceptable vs. unacceptable qualities of performance. An analytic rubric makes clear
distinctions among the evaluation criteria while a holistic rubric merges the criteria together to stimulate a general
judgment about the quality of student work.
Questions To Ask When Constructing Rubrics
1. What criteria or essential elements must be present in the student’s work to ensure that it is high in quality?
2. How many levels of achievement (mastery) do I wish to illustrate for students?
3. For each criteria or essential element of quality, what is a clear description of performance at each
achievement level?
4. What are the consequences of performing at each level of quality?
5. What rating scheme will I use in the rubric?
6. When I use the rubric, what aspects work well and what aspects need improvement?
Additional Questions To Consider
1. What content must students master in order to complete the task well?
2. Are there any important aspects of the task that are specific to the context in which the assessment is set?
3. In the task, is the process of achieving the outcome as important as the outcome itself?
Source: Huba, Mary E. and Freed, Jann E. Learner-Centered Assessment on College Campuses. Allyn & Bacon,
Boston, MA, 2000. ISBN 0-205-28738-7.
Additional good references:
Moskal, Barbara M. (2000). Scoring Rubrics: What, When and How? Practical Assessment, Research & Evaluation,
7(3). Available online: http://ericae.net/pare/getvn.asp?v=7&n=3.
Stevens, Dannelle and Levi, Antonio. Introduction to Rubrics. Stylus Publishing, Herdon, VA 2004. ISBN 1-57922114-9. September 2004
The assessment leader at Winona State University (MN) has an excellent set of rubrics at this URL
http://www.winona.edu/AIR/ Once there click on the sample rubrics link in the left frame.
The Center for Learning and Teaching Excellence at Arizona State University has a bank of rubrics at this URL
http://clte.asu.edu/resources/instructors/ Select the Assessment Web link in the center of the page.
CSU System Office has an excellent rubrics at this URL http://www.calstate.edu/itl/sloa/index.shtml
Example in action:
Raymond Walters College has been making extensive use of rubrics and primary trait assessment, for individual
course assignments. See examples link at http://www.rwc.uc.edu/phillips/index_assess.html
107
Johnson County Community College- Writing Outcome
Outcomes Statement: Upon receipt of an associate degree from Johnson County Community College, a student
should be able to write a clear, well-organized paper using documentation and quantitative tools when appropriate.
Outcome Rubric:
6 = Essay demonstrates excellent composition skills including a clear and thought-provoking thesis, appropriate and
effective organization, lively and convincing supporting materials, effective diction and sentence skills, and perfect or
near perfect mechanics including spelling and punctuation. The writing perfectly accomplishes the objectives of the
assignment.
5 = Essay contains strong composition skills including a clear and thought-provoking thesis, although development,
diction, and sentence style may suffer minor flaws. Shows careful and acceptable use of mechanics. The writing
effectively accomplishes the goals of the assignment.
4 = Essay contains above average composition skills, including a clear, insightful thesis, although development may be
insufficient in one area and diction and style may not be consistently clear and effective. Shows competence in the use
of mechanics. Accomplishes the goals of the assignment with an overall effective approach.
3 = Essay demonstrates competent composition skills including adequate development and organization, although the
development of ideas may be trite, assumptions may be unsupported in more than one area, the thesis may not be
original, and the diction and syntax may not be clear and effective. Minimally accomplishes the goals of the
assignment.
2 = Composition skills may be flawed in either the clarity of the thesis, the development, or organization. Diction,
syntax, and mechanics may seriously affect clarity. Minimally accomplishes the majority of the goals of the
assignment.
1 = Composition skills may be flawed in two or more areas. Diction, syntax, and mechanics are excessively flawed.
Fails to accomplish the goals of the assignment.
Standards: Ten percent of students who have met the requirements for an associate degree at JCCC will earn 6
(excellent) on each of the communication rubrics. Thirty percent of students earning an associate degree will score 5
(very good) or 6 (excellent). Eighty percent will earn scores of 4 (satisfactory) or higher and the top 98 percent will earn
scores of 3 (minimal accomplishment of educational goals) or higher. The remaining 2 percent of the associate degree
recipients are expected to earn the score of 2 (unsatisfactory) on the communication rubrics. The score of 1 represents
a skill level beneath the expectation of all associate degree recipients at JCCC. Hence, no associate degree recipients
are expected to score at the level of 1 on the communications rubrics.
Suggested Assignment Guidelines
An appropriate assignment (e.g., paper, homework, project) would allow students to demonstrate composition skills by
asking them to:
•
•
•
•
•
develop a clear thesis statement;
develop main points with appropriate and convincing supporting materials;
utilize appropriate and effective organization of content;
demonstrate a clear and coherent writing style that uses effective diction and sentence skills; and
demonstrate correct mechanical skills including spelling and punctuation.
108
Choosing the Right Assessment Tools, Gary Williams, Crafton Hills College
Assessment
Tool
Data:
Direct or
Indirect
Domain:
Cognitive,
Psychomotor, or
Affective
Formative or
Summative
D
C
F, S
Bloom's Tax:
Knowledge,
Comprehension,
Application or
Analysis/
Pros
Cons
Synthesis/Eval
variable K, C, A,
ASE
Oral Speech
D
C, A
F, S
Debate
K, C, A, ASE
D
C, P, A
F, S
variable K, C, A,
ASE
Product
Creation &
Special Reports
D
Flowchart or
Diagram
C
F, S
C, A, ASE
easily graded with
rubric allows other
students to see and
learn what each
student learned
connects general
education goals with
discipline-specific
courses
difficult for ESL
students stressful for
students takes course
time must fairly grade
course content beyond
delivery
provides immediate
feedback to the
student reveals
thinking and ability
to respond based on
background
knowledge and
critical thinking
ability
requires good rubric
more than one evaluator
is helpful difficult for
ESL students stressful
for students takes course
time
students can
display skills.
knowledge, and
abilities in a way that
is suited to them
must have clearly
defined criteria
and evaluative measures
"the look" can not override the content
displays original
synthetic
thinking on the
part of the student
more difficult to
grade, requiring a
checklist or rubric
for a variety of different
answers
perhaps the
best way to display
overall high level
difficult for some
students to do on the
spot
thinking and
articulation
abilities
109
Assessment
Tool
Data:
Direct or
Indirect
Domain:
Cognitive,
Psychomotor, or
Affective
Formative or
Summative
D
C, P
S
Bloom's Tax: Knowledge,
Comprehension, Application
or Analysis/
Pros
Cons
Synthesis/Eval
Portfolios
variable
provides the
students with a
clear record of
their work and
growth
best evidence of
growth and change
over
time
students can
display skills.
knowledge, and
abilities in a
way that is
time consuming to
grade different content
in portfolio makes
evaluating difficult
and may require training
bulky to manage
depending on size
suited to them
D, I
A
S
Exit Surveys
ASE
D
C, P
F, S
Performance
variable K, C, A, ASE
D
C, P , A
F, S
promotes selfassessment
provides good
summative data
easy to manage
data if Likertscaled
responses are used
provides best
display of skills
stressful for students
and abilities
provides
may take course time
excellent
opportunity for
peer review
students can
display skills.
knowledge, and
abilities in a
way that is
suited to them
some students
may take the evaluation
very hard - evaluative
best method to
Measure
growth overtime
with regards to a
course or program
- cumulative
Capstone
project or
course
Likert scales limit
feedback, open-ended
responses are bulky to
manage,
statements must
be carefully
framed
focus and breadth
of assessment
are important
understanding all the
variables to produce
assessment
ASE
results is also
important may result in
additional course
requirements
110
Assessment
Tool
Data:
Direct or
Indirect
Domain:
Cognitive,
Psychomotor, or
Affective
Formative or
Summative
Bloom's Tax: Knowledge,
Comprehension, Application or
Analysis/
Pros
Cons
Synthesis/Eval
D
C, A
F, S
Team Project
variable K, C, A, ASE
D, I
C, A
S
Reflective
selfassessment
essay
Surveys
provides invaluable
ability to evaluate
affective growth in
students
must use evidence
to support
conclusions, not
just selfopinionated
assessment
provides good
indirect data
data can be
compared
longitudinally can be
used to
respondents may
be influenced by
factors other than
those being
considered
ASE
I
Satisfaction
and Perception
connects general
education goals with
discipline-specific
courses
C, P, A
S
C, A, ASE
requires
coordination and
agreement on
standards
must fairly grade
individuals as well
as team grading is
slightly more
complicated
student interaction
may be a
challenge
determine
outcomes over a long
period of time
validity and
reliability most be
closely watched
111
Mira Costa College
Area B (Physical Universe and its Life Forms)
Area B Mission Statement (drafted 10/20/06)
Students in Area B will be able to investigate and explain physical phenomena through the application of
empirical knowledge using mathematical and scientific processes and concepts.
Anthropology
Students completing courses in anthropology within Area B will understand what it means to be human from
a biological perspective. They will garner this understanding through integration of scientific method and
evidence, including comparisons with other animal species and development of ecological and evolutionary
paradigms.
Life Sciences
Students in the Life Sciences will become scientific thinkers who are curious and knowledgeable about
biological systems and who rely on experimentation, logic, evidence, objective reasoning and healthy
skepticism to explain natural phenomena.
112
Mira Costa College
Student Learning Outcome Status Report
Department: ____________
Discipline: ____________
Course: ____________
SLO Written (semester & year): ____________
Assessment Administered (semester & year): ____________
Evaluation of Assessment Data Completed (semester & year): ____________
GE Program-Level Outcomes: Effective Communication, Critical Thinking, Global Awareness and Responsible Citizenship,
Information Literacy, Aesthetic Literacy and Appreciation, Productive Work Habits.
CTE Program-Level Outcomes: Technical Skills, Application of Discipline Skills, Critical Thinking and Problem Solving,
Communication, Professional Behavior.
A) Student Learning Outcome
B) General Education or CTE SLO(s) to which
course SLO aligns (see above)
C) Assessment Task(s)
D) Expected Level of Achievement/Baseline
E) How Data were Gathered and Evaluated
F) Results of Evaluation
G) Use of Data/Plans
Accrediting Commission for Community and Junior Colleges (ACCJC)
113
Western Association of Schools and Colleges
Rubric for Evaluating Institutional Effectiveness – Part III: Student Learning Outcomes
(See attached instructions on how to use this rubric.)
Levels of
Implementation
Characteristics of Institutional Effectiveness in
Student Learning Outcomes
(Sample institutional behaviors)
Awareness
Development
• There is preliminary, investigative dialogue about student learning outcomes.
• There is recognition of existing practices such as course objectives and how they relate to
student learning outcomes.
• There is exploration of models, definitions, and issues taking place by a few people.
• Pilot projects and efforts may be in progress.
• The college has discussed whether to define student learning outcomes at the level of
some courses or programs or degrees; where to begin.
• College has established an institutional framework for definition of student learning
outcomes (where to start), how to extend, and timeline.
• College has established authentic assessment strategies for assessing student
learning outcomes as appropriate to intended course, program, and degree learning
outcomes.
• Existing organizational structures (e.g. Senate, Curriculum Committee) are
supporting strategies for student learning outcomes definition and assessment.
• Leadership groups (e.g. Academic Senate and administration), have accepted
responsibility for student learning outcomes implementation.
• Appropriate resources are being allocated to support student learning outcomes and
assessment.
• Faculty and staff are fully engaged in student learning outcomes development.
Proficiency
• Student learning outcomes and authentic assessment are in place for courses, programs
and degrees.
• Results of assessment are being used for improvement and further alignment of
institution-wide practices.
• There is widespread institutional dialogue about the results.
• Decision-making includes dialogue on the results of assessment and is purposefully
directed toward improving student learning.
• Appropriate resources continue to be allocated and fine-tuned.
• Comprehensive assessment reports exist and are completed on a regular basis.
• Course student learning outcomes are aligned with degree student learning outcomes.
• Students demonstrate awareness of goals and purposes of courses and programs in
which they are enrolled.
Sustainable
Continuous
Quality
Improvement
• Student learning outcomes and assessment are ongoing, systematic and used for
continuous quality improvement.
• Dialogue about student learning is ongoing, pervasive and robust.
• Evaluation and fine-tuning of organizational structures to support student learning is
ongoing.
• Student learning improvement is a visible priority in all practices and structures across the
college.
• Learning outcomes are specifically linked to program reviews.
15 Fred Trapp, Cambridge West Partnership, LLC;
[email protected]
114
Pierce College
Speech 101, Public Speaking, Course Assessment Loop
Start
Faculty created an assessment rubric
with three main criteria: delivery,
organization, research
All full-time faculty used the rubric on randomly
selected (every 5th student) speeches and rated
students 1-4 according to the rubric on each criteria
Student speeches were assessed and
the data was discussed by the Speech
faculty
End (for now)
Changed pedagogy1. Beefed up research instruction
2. Provided supplemental experiences
with research.
The faculty concluded that the
weakest area of student performance
was research
16 Fred Trapp, Cambridge West Partnership, LLC;
[email protected]
115
Cabrillo College
Critical Thinking, General Education Assessment Loop
Start
Instructors individually assessed
one critical thinking assignment in,
one class
Faculty scored the student work with a rubric
and analyzed student performance and needs
Department met to share results.
They concluded that students
needed help with reading.
End (for now)
Faculty changed pedagogy1. Revamped classes to include reading
techniques.
The department received funds for training
in integrating reading and writing.
17 Fred Trapp, Cambridge West Partnership, LLC;
[email protected]
116
Capital College (CT)
Common Writing Assignment, General Education Assessment Loop
Start
GE outcomes were articulated &
grading rubrics were developed
12 faculty teaching 15 classes in
different disciplines provided a
common writing assignment
to their students
100 sample papers were
scored twice by two
readers each using a holistic
then an analytic rubric
Students were weakest in the development
of ideas supported by evidence and in the
use of language.
Students who reported having written essays
in classes other than English demonstrated
greater levels of writing skills & were more
likely to achieve a proficiency score.
End (for now)
1. Committee on writing standards started a coordinated
dialogue & professional development activities to improve
writing across the college by supporting early and continuous student practice
in writing with emphasis on development of ideas and use of language.
2. College policy and practice were reviewed to find ways to enforce the ideal
of completing developmental English first or if placed into upper level composition
completing it within the first 15 units of college work.
18 Fred Trapp, Cambridge West Partnership, LLC;
[email protected]
117
Bakersfield College
Biology Allied Health Curriculum Pathway (Locally Defined Program) Assessment Loop
Start
Wrote the SLOs for courses
in the health-care pathway
1. Looked at success & retention
data in individual pathway courses
& the program in total.
2. Assessed the course SLOs.
Analyzed & discussed
the data
Changed the curriculum1. Rewrote course linkages & layout.
2. Reduced hours in courses.
End (for now)
Changed curriculum1. Added a supplemental instructional lab
for those who needed it.
2. Added a capstone course for students on
the waiting list to get into programs.
Assessed with embedded
exam questions
19 Fred Trapp, Cambridge West Partnership, LLC;
[email protected]
118
Lecture / Discussion 5:
Developing and Applying
Rubrics
Mary Allen
119
Developing and Applying Rubrics
Mary J. Allen, [email protected], September 2009
Rubrics provide the criteria for classifying products or behaviors into categories that vary along a
continuum. They can be used to classify virtually any product or behavior, such as essays,
research reports, portfolios, works of art, recitals, oral presentations, performances, and group
activities. Judgments can be self-assessments by students; or judgments can be made by others,
such as faculty, other students, fieldwork supervisors, and external reviewers. Rubrics can be
used to provide formative feedback to students, to grade students, and/or to assess programs.
There are two major types of scoring rubrics:
• Holistic scoring — one global, holistic score for a product or behavior
• Analytic rubrics — separate, holistic scoring of specified characteristics of a product or
behavior
Rubric Examples
•
•
•
•
•
•
Holistic Critical Thinking Scoring Rubric (Facione & Facione)
Holistic Critical Thinking Rubric (Portland State University)
Critical Thinking Rubric (Northeastern Illinois University)
Scoring Guide for Critical Thinking (California State University, Fresno)
Information Competence (CA State University)
Writing Rubric (Roanoke College)
120
Holistic Critical Thinking Scoring Rubric
Facione and Facione
4
3
2
1
Consistently does all or almost all of the following:
Accurately interprets evidence, statements, graphics, questions, etc.
Identifies the salient arguments (reasons and claims) pro and con.
Thoughtfully analyzes and evaluates major alternative points of view.
Draws warranted, judicious, non-fallacious conclusions.
Justifies key results and procedures, explains assumptions and reasons.
Fair-mindedly follows where evidence and reasons lead.
Does most or many of the following:
Accurately interprets evidence, statements, graphics, questions, etc.
Identifies relevant arguments (reasons and claims) pro and con.
Offers analyses and evaluations of obvious alternative points of view.
Draws warranted, non-fallacious conclusions.
Justifies some results or procedures, explains reasons.
Fair-mindedly follows where evidence and reasons lead.
Does most or many of the following:
Misinterprets evidence, statements, graphics, questions, etc.
Fails to identify strong, relevant counter-arguments.
Ignores or superficially evaluates obvious alternative points of view.
Draws unwarranted or fallacious conclusions.
Justifies few results or procedures, seldom explains reasons.
Regardless of the evidence or reasons, maintains or defends views based on
self-interest or preconceptions.
Consistently does all or almost all of the following:
Offers biased interpretations of evidence, statements, graphics, questions,
information, or the points of view of others.
Fails to identify or hastily dismisses strong, relevant counter-arguments.
Ignores or superficially evaluates obvious alternative points of view.
Argues using fallacious or irrelevant reasons, and unwarranted claims.
Does not justify results or procedures, nor explain reasons.
Regardless of the evidence or reasons, maintains or defends views based on
self-interest or preconceptions.
Exhibits close-mindedness or hostility to reason.
(c) 1994, Peter A. Facione, Noreen C. Facione, and The California Academic Press. 217 La Cruz
Ave., Millbrae, CA 94030.
Permission is hereby granted to students, faculty, staff, or administrators at public or nonprofit
educational institutions for unlimited duplication of the critical thinking scoring rubric, rating
form, or instructions herein for local teaching, assessment, research, or other educational and
noncommercial uses, provided that no part of the scoring rubric is altered and that "Facione and
Facione" are cited as authors.
Retrieved September 2, 2005 from http://www.insightassessment.com/pdf_files/rubric.pdf
121
Portland State University Studies Program Holistic Critical Thinking Rubric*
Inquiry and Critical Thinking Rubric
Students will learn various modes of inquiry through interdisciplinary curricula—problem
posing, investigating, conceptualizing—in order to become active, self-motivated, and
empowered learners.
6 (Highest)—Consistently does all or almost all of the following:
• Accurately interprets evidence, statements, graphics, questions, etc.
• Identifies the salient arguments (reasons and claims) pro and con.
• Thoughtfully analyzes and evaluates major alternative points of view.
• Generates alternative explanations of phenomena or event.
• Justifies key results and procedures, explains assumptions and reasons.
• Fair-mindedly follows where evidence and reasons lead.
• Makes ethical judgments.
5—Does most the following:
• Accurately interprets evidence, statements, graphics, questions, etc.
• Thinks through issues by identifying relevant arguments (reasons and claims) pro and con.
• Offers analysis and evaluation of obvious alternative points of view.
• Generates alternative explanations of phenomena or event.
• Justifies (by using) some results or procedures, explains reasons.
• Fair-mindedly follows where evidence and reasons lead.
4—Does most the following:
• Describes events, people, and places with some supporting details from the source.
• Make connections to sources, either personal or analytic.
• Demonstrates a basic ability to analyze, interpret, and formulate inferences.
• States or briefly includes more than one perspective in discussing literature, experiences, and
points of view of others.
• Takes some risks by occasionally questioning sources or by stating interpretations and
predictions.
• Demonstrates little evidence of rethinking or refinement of one’s own perspective.
3—Does most or many of the following:
• Respond by retelling or graphically showing events or facts.
• Makes personal connections or identifies connections within or between sources in a limited
way. Is beginning to use appropriate evidence to back ideas.
• Discusses literature, experiences, and points of view of others in terms of own experience.
• Responds to sources at factual or literal level.
• Includes little or no evidence of refinement of initial response or shift in dualistic thinking.
• Demonstrates difficulty with organization and thinking is uneven.
122
2—Does many or most the following:
• Misinterprets evidence, statements, graphics, questions, etc.
• Fails to identify strong, relevant counter arguments.
• Draws unwarranted or fallacious conclusions.
• Justifies few results or procedures, seldom explains reasons.
• Regardless of the evidence or reasons, maintains or defends views based on self-interest or
preconceptions.
1 (lowest)—Consistently does all or almost all of the following:
• Offers biased interpretations of evidence, statements, graphics, questions, information, or the
points of view of others.
• Fails to identify or hastily dismisses strong, relevant counterarguments.
• Ignores or superficially evaluates obvious alternative points of view. Argues using fallacious
or irrelevant reasons and unwarranted claims.
• Does not justify results or procedures, nor explain reasons.
• Exhibits close-mindedness or hostility to reason.
X—No basis for scoring. (Use only for missing or malfunctioning portfolios.)
*taken verbatim from Stevens, D. D., & Levi, A. J. (2005). Introduction to Rubrics. Sterling,
VA: Stylus, pp. 122-123
123
Northeastern Illinois University General Education Critical Thinking Rubric
Retrieved 3/2/05 from http://www.neiu.edu/~neassess/gened.htm#rubric
Quality
Macro Criteria
No/Limited Proficiency
(D&E)
Some Proficiency (C)
Proficiency (B)
High Proficiency (A)
1. Identifies & Explains Issues
Fails to identify, summarize, or
explain the main problem or
question.
Represents the issues
inaccurately or inappropriately.
Identifies main issues but does
not summarize or explain them
clearly or sufficiently
Successfully identifies and
summarizes the main issues, but
does not explain why/how they
are problems or create questions
2. Distinguishes Types of
Claims
Fails to label correctly any of the
factual, conceptual and value
dimensions of the problems and
proposed solutions.
Successfully identifies some, but
not all of the factual, conceptual,
and value aspects of the
questions and answers.
Successfully separates and labels
all the factual, conceptual, and
value claims
3. Recognizes Stakeholders and
Contexts
Fails accurately to identify and
explain any empirical or
theoretical contexts for the
issues.
Presents problems as having no
connections to other conditions
or contexts.
Shows some general
understanding of the influences
of empirical and theoretical
contexts on stakeholders, but
does not identify many specific
ones relevant to situation at
hand.
Correctly identifies all the
empirical and most of theoretical
contexts relevant to all the main
stakeholders in the situation.
4. Considers Methodology
Fails to explain how/why/which
specific methods of research are
relevant to the kind of issue at
hand.
Identifies some but not all
methods required for dealing
with the issue; does not explain
why they are relevant or
effective.
Successfully explains
how/why/which methods are
most relevant to the problem.
5. Frames Personal Responses
and Acknowledges Other
Perspectives
Fails to formulate and clearly
express own point of view, (or)
fails to anticipate objections to
his/her point of view, (or) fails to
consider other perspectives and
position.
Formulates a vague and
indecisive point of view, or
anticipates minor but not major
objections to his/her point of
view, or considers weak but not
strong alternative positions.
Formulates a clear and precise
personal point of view
concerning the issue, and
seriously discusses its
weaknesses as well as its
strengths.
Clearly identifies and
summarizes main issues and
successfully explains why/how
they are problems or questions;
and identifies embedded or
implicit issues, addressing their
relationships to each other.
Clearly and accurately labels not
only all the factual, conceptual,
and value, but also those implicit
in the assumptions and the
implications of positions and
arguments.
Not only correctly identifies all
the empirical and theoretical
contexts relevant to all the main
stakeholders, but also finds
minor stakeholders and contexts
and shows the tension or
conflicts of interests among
them.
In addition to explaining
how/why/which methods are
typically used, also describes
embedded methods and possible
alternative methods of working
on the problem.
Not only formulates a clear and
precise personal point of view,
but also acknowledges
objections and rival positions
and provides convincing replies
to these.
124
California State University, Fresno General Education Scoring Guide for Critical Thinking
Retrieved 3/2/05 from http://www.csufresno.edu/cetl/assessment/CTScoring.doc
Scoring Level
4 - Accomplished
3 - Competent
2 - Developing
1 - Beginning
Interpretation
Analysis & Evaluation
Presentation
Analyzes insightful questions
Refutes bias
Critiques content
Examines inconsistencies
Values information
Examines conclusions
Uses reasonable judgment
Discriminates rationally
Synthesizes data
Views information critically
Argues succinctly
Discusses issues thoroughly
Shows intellectual honesty
Justifies decisions
Assimilates information
Asks insightful questions
Detects bias.
Categorizes content.
Identifies inconsistencies
Recognizes context
Formulates conclusions
Recognizes arguments
Notices differences
Evaluates data
Seeks out information
Argues clearly
Identifies issues
Attributes sources naturally
Suggests solutions
Incorporates information
Identifies some questions
Notes some bias
Recognizes basic content
States some inconsistencies
Selects sources adequately
Identifies some conclusions
Sees some arguments
Identifies some differences
Paraphrases data
Assumes information valid
Misconstructs arguments
Generalizes issues
Cites sources
Presents few options
Overlooks some information
Fails to question data
Ignores bias
Misses major content areas
Detects no inconsistencies
Chooses biased sources
Fails to draw conclusions
Sees no arguments
Overlooks differences
Repeats data
Omits research
Omits argument
Misrepresents issues
Excludes data
Draws faulty conclusions
Shows intellectual dishonesty
125
Rubrics for Assessing Information Competence in the California State University
ACRL Standard
1. Determine the
Extent of the
Information
Needed
Beginning
Student is unable to effectively formulate a
research question based on an information
need.
2. Access the
Needed
Information
Effectively and
Efficiently
Student is unfocused and unclear about search
strategy.
Time is not used effectively and efficiently.
Information gathered lacks relevance, quality,
and balance.
3. Evaluate
Information and
its Sources
Critically
Student is unaware of criteria that might be
used to judge information quality. Little effort
is made to examine the information located
4. Use
Information
Effectively to
Accomplish a
Specific Purpose
Student is not
aware of the information necessary to research
a topic, and the types of data that would be
useful in formulating a convincing argument.
Information is incomplete and does not support
the intended purpose.
Student is unclear regarding proper citation
format, and/or copies and paraphrases the
information and ideas of others without giving
credit to authors. Student does not know how
to distinguish between information that is
objective and biased, and does not know the
role that free access to information plays in a
democratic society.
Proficient
Student can formulate a question that is
focused and clear. Student identifies
concepts related to the topic, and can find
a sufficient number of information
resources to meet the information need.
Student executes an appropriate search
strategy within a reasonable amount of
time. Student can solve problems by
finding a variety of relevant information
resources, and can evaluate search
effectiveness.
Student examines information using
criteria such as authority, credibility,
relevance, timeliness, and accuracy, and
is able to make judgments about
what to keep and what to discard.
Student uses appropriate information to
solve a problem, answer a question, write
a paper, or other purposes
Advanced
Question is focused, clear, and complete. Key
concepts and terms are identified. Extensive
information sources are identified in numerous
potential formats.
Student is aware and able to analyze search
results, and evaluate the appropriateness of the
variety of (or) multiple relevant sources of
information that directly fulfill an information
need for the particular discipline,
Multiple and diverse sources and viewpoints of
information are compared and evaluated
according to specific criteria appropriate for
the discipline. Student is able to match criteria
to a specific information need, and can
articulate how identified sources relate to the
context of the discipline.
Student is aware of the breadth and depth of
research on a topic, and is able to reflect on
search strategy, synthesize and integrate
information from a variety of sources, draw
appropriate conclusions, and is able to clearly
communicate ideas to others
Student understands and recognizes the concept
of intellectual property, can defend him/herself
if challenged, and can properly incorporate the
ideas/published works of others into their own
work building upon them. Student can
articulate the value of information to a free and
democratic society, and can use specific criteria
to discern objectivity/fact from
bias/propaganda.
Student gives credit for works used by
5. Understand the
quoting and listing references. Student is
Economic, Legal,
an ethical consumer and producer of
and Social Issues
information, and understands how free
surrounding the
access to information, and free
Use of
expression, contribute to a democratic
Information, and
society.
Access and Use
Information
Ethically and
Legally
*Prepared by the CSU Information Competence Initiative, October 2002, based on the 2000 ACRL Information Literacy Competency Standards For Higher
Education. For more information, see http://www.calstate.edu/LS/1_rubric.doc.
126
Draft of Writing Rubric—Retrieved August 28, 2008 from http://web.roanoke.edu/Documents/Writing%20Rubrics.July%2007.doc
Basic
Shows some engagement with
the topic without elaboration;
offers basic observations but
rarely original insight
Proficient
Demonstrates engagement
with the topic, recognizing
multiple dimensions and/or
perspectives; offers some
insight
Focus and
Thesis
Below Basic
Shows minimal engagement
with the topic, failing to
recognize multiple
dimensions/ perspectives;
lacking even basic
observations
Paper lacks focus and/or a
discernible thesis.
Some intelligible ideas, but
thesis is weak, unclear, or too
broad.
Evidence
Little to no evidence
Organization
Organization is missing both
overall and within paragraphs.
Introduction and conclusion
may be lacking or illogical.
Some evidence but not
enough to develop argument
in unified way. Evidence may
be inaccurate, irrelevant, or
inappropriate for the purpose
of the essay
Organization, overall and/or
within paragraphs, is
formulaic or occasionally
lacking in coherence; few
evident transitions.
Introduction and conclusion
may lack logic.
Identifiable thesis
representing adequate
understanding of the assigned
topic; minimal irrelevant
material
Evidence accurate, well
documented, and relevant, but
not complete, well integrated,
and/or appropriate for the
purpose of the essay
Style and
Mechanics
Multiple and serious errors of
sentence structure; frequent
errors in spelling and
capitalization; intrusive and/or
inaccurate punctuation such
that communication is
hindered. Proofreading not
evident.
Ideas
Sentences show errors of
structure and little or no
variety; many errors of
punctuation, spelling and/or
capitalization. Errors interfere
with meaning in places.
Careful proofreading not
evident.
Few organizational problems
on any of the 3 levels (overall,
paragraph, transitions).
Introduction and conclusion
are effectively related to the
whole.
Effective and varied
sentences; some errors in
sentence construction; only
occasional punctuation,
spelling and/or capitalization
errors.
Advanced
Demonstrates engagement
with the topic, recognizing
multiple dimensions and/or
perspectives with elaboration
and depth; offers considerable
insight
Clear, narrow thesis
representing full
understanding of the
assignment;
every word counts
Evidence is relevant, accurate,
complete, well integrated,
well documented, and
appropriate for the purpose of
the essay.
Organization is logical and
appropriate to assignment;
paragraphs are well-developed
and appropriately divided;
ideas linked with smooth and
effective transitions.
Introduction and conclusion
are effectively related to the
whole.
Each sentence structured
effectively, powerfully; rich,
well-chosen variety of
sentence styles and length;
virtually free of punctuation,
spelling, capitalization errors.
127
Rubrics have many strengths:
• Complex products or behaviors can be examined efficiently.
• Developing a rubric helps to precisely define faculty expectations.
• Well-trained reviewers apply the same criteria and standards.
• Rubrics are criterion-referenced, rather than norm-referenced. Raters ask, “Did the student
meet the criteria for level 5 of the rubric?” rather than “How well did this student do
compared to other students?” This is more compatible with cooperative and collaborative
learning environments than competitive grading schemes and is essential when using rubrics
for program assessment because you want to learn how well students have met your
standards.
• Ratings can be done by students to assess their own work, or they can be done by others, e.g.,
peers, fieldwork supervisions, or faculty.
Rubrics can be useful for grading, as well as assessment.
Below is a rubric for assessing oral presentation skills, followed by four examples of grading
rubrics based on adapting the assessment rubric. With calibration, these grading rubrics can be
used to assess the program learning outcome by aggregating the results for Organization,
Content, and Delivery across courses.
Organization
Content
Delivery
Below Expectation
Satisfactory
No apparent
organization.
Evidence is not used
to support assertions.
The presentation has a
focus and provides
some evidence which
supports conclusions.
Exemplary
The presentation is
carefully organized
and provides
convincing evidence
to support
conclusions.
The content is
The content is
The content is
inaccurate or overly generally accurate, but accurate and
general. Listeners are incomplete. Listeners complete. Listeners
unlikely to learn
may learn some
are likely to gain new
anything or may be
isolated facts, but they insights about the
misled.
are unlikely to gain
topic.
new insights about the
topic.
The speaker appears The speaker is
The speaker is relaxed
anxious and
generally relaxed and and comfortable,
uncomfortable, and
comfortable, but too
speaks without undue
reads notes, rather
often relies on notes.
reliance on notes, and
than speaks.
Listeners are
interacts effectively
Listeners are largely sometimes ignored or with listeners.
ignored.
misunderstood.
128
Example 1.
Numbers are used for grading; categories (Below Expectation, Satisfactory, Exemplary) are used
for assessment. Individual faculty determine how to assign numbers for their course grading.
Faculty may circle or underline material in the cells to emphasize criteria that were particularly
important during the assessment/grading.
Analytic Rubric for Grading Oral Presentations
Below Expectation
Satisfactory
Exemplary
Organization
Content
Delivery
No apparent
organization.
Evidence is not used
to support assertions.
Score
The presentation has a
focus and provides
some evidence which
supports conclusions.
The presentation is
carefully organized
and provides
convincing evidence
to support
conclusions.
(0-4)
(5-6)
(7-8)
The content is
The content is
The content is
inaccurate or overly generally accurate, but accurate and
general. Listeners are incomplete. Listeners complete. Listeners
unlikely to learn
may learn some
are likely to gain new
anything or may be
isolated facts, but they insights about the
misled.
are unlikely to gain
topic.
new insights about the
topic.
(0-8)
(9-11)
(12-13)
The speaker appears The speaker is
The speaker is relaxed
anxious and
generally relaxed and and comfortable,
uncomfortable, and
comfortable, but too
speaks without undue
reads notes, rather
often relies on notes.
reliance on notes, and
than speaks.
Listeners are
interacts effectively
Listeners are largely sometimes ignored or with listeners.
ignored.
misunderstood.
(0-5)
(6-7)
(8-9)
Total Score
129
Example 2.
Weights are used for grading; categories (Below Expectation, Satisfactory, Exemplary) are used
for assessment. Individual faculty determine how to assign weights for their course grading.
Faculty may circle or underline material in the cells to emphasize criteria that were particularly
important during the assessment/grading.
Analytic Rubric for Grading Oral Presentations
Below Expectation
Satisfactory
Exemplary
Weight
Organization
No apparent
organization.
Evidence is not used
to support assertions.
The presentation has a
focus and provides
some evidence which
supports conclusions.
The presentation is
carefully organized
30%
and provides
convincing evidence
to support conclusions
Content
The content is
inaccurate or overly
general. Listeners are
unlikely to learn
anything or may be
misled.
The content is
generally accurate, but
incomplete. Listeners
may learn some
isolated facts, but they
are unlikely to gain
new insights about the
topic.
The speaker is
generally relaxed and
comfortable, but too
often relies on notes.
Listeners are
sometimes ignored or
misunderstood.
The content is
accurate and
complete. Listeners
are likely to gain new
insights about the
topic.
Delivery
The speaker appears
anxious and
uncomfortable, and
reads notes, rather
than speaks.
Listeners are largely
ignored.
The speaker is relaxed
and comfortable,
speaks without undue
reliance on notes, and
interacts effectively
with listeners.
50%
20%
Comments
130
Example 3.
The faculty member checks off characteristics of the speech and determines the grade based on a
holistic judgment. The categories (Below Expectation, Satisfactory, Exemplary) are used for
assessment. Individual faculty might add scores or score ranges (see Example 1) or a “Weight”
column (see Example 2) for grading purposes.
Analytic Rubric for Grading Oral Presentations
Below Expectation
Satisfactory
Exemplary
Organization
No apparent
organization.
Evidence is not
used to support
assertions.
Content
The content is
inaccurate or
overly general.
Listeners are
unlikely to learn
anything or may
be misled.
Delivery
The speaker
appears anxious
and
uncomfortable.
Speaker reads
notes, rather than
speaks.
Listeners are
largely ignored.
The presentation
has a focus.
Student provides
some evidence
which supports
conclusions.
The content is
generally accurate,
but incomplete.
Listeners may learn
some isolated facts,
but they are
unlikely to gain
new insights about
the topic.
The speaker is
generally relaxed
and comfortable.
Speaker too often
relies on notes.
Listeners are
sometimes ignored
or misunderstood.
The presentation is
carefully organized.
Speaker provides
convincing
evidence to support
conclusions
The content is
accurate and
complete.
Listeners are likely
to gain new insights
about the topic.
The speaker is
relaxed and
comfortable.
Speaker speaks
without undue
reliance on notes.
Speaker interacts
effectively with
listeners.
Comments:
131
Example 4.
Combinations of Various Ideas. As long as the nine assessment cells are used in the same way by
all faculty, grading and assessment can be done simultaneously.
Organization
Content
Delivery
References
Analytic Rubric for Grading Oral Presentations
Below
Satisfactory
Exemplary
Expectation
2
3
1
No apparent
The
The
organization.
presentation has
presentation is
Evidence is not
a focus.
carefully
used to support
Speaker
organized.
Speaker
assertions.
provides some
provides
evidence which
convincing
supports
evidence to
conclusions.
support
conclusions
The content is
The content is
The content is
inaccurate or
generally
accurate and
overly general.
accurate, but
complete.
Listeners are
incomplete.
Listeners are
unlikely to
Listeners may
likely to gain
learn anything
learn some
new insights
or may be
about the
isolated facts,
misled.
but they are
topic.
unlikely to gain
new insights
about the topic.
The speaker
The speaker is
The speaker is
appears
generally
relaxed and
anxious and
relaxed and
comfortable.
uncomfortable.
comfortable.
Speaker
speaks without
Speaker reads
Speaker too
undue reliance
notes, rather
often relies on
on notes.
than speaks.
notes.
Listeners are
Listeners are
Speaker
largely ignored.
sometimes
interacts
ignored or
effectively
misunderstood.
with listeners.
Speaker fails to
Speaker refers
Speaker refers
refer to journal
to 1 or 2 journal
to 3 or more
articles.
articles.
journal
articles.
Weight
20%
40%
20%
20%
132
Assessment vs. Grading Concerns
•
•
•
•
Grading rubrics may include criteria that are not related to the learning outcome being
assessed. These criteria are used for grading, but are ignored for assessment.
Grading requires more precision than assessment.
Assessment rubrics should focus only on the outcome being assessed.
If multiple faculty will use the rubric for grading or assessment, consider calibrating them.
This is especially important when doing assessment.
Rubrics Can:
•
•
•
•
•
•
•
Speed up grading
Provide routine formative feedback to students
Clarify expectations to students
Reduce student grade complaints
Improve the reliability and validity of assessments and grades
Make grading and assessment more efficient and effective by focusing the faculty member on
important dimensions
Help faculty create better assignments that ensure that students display what you want them to
demonstrate
Suggestions for Using Rubrics in Courses
1. Hand out the grading rubric with the assignment so students will know your expectations and
how they'll be graded.
2. Use a rubric for grading student work and return the rubric with the grading on it.
3. Develop a rubric with your students for an assignment or group project. Students can then
monitor themselves and their peers using agreed-upon criteria that they helped develop.
Many faculty find that students will create higher standards for themselves than faculty
would impose on them.
4. Have students apply your rubric to some sample products before they create their own.
Faculty report that students are quite accurate when doing this, and this process should help
them evaluate their own products as they are being developed. The ability to evaluate, edit,
and improve draft documents is an important skill.
5. Have students exchange paper drafts and give peer feedback using the rubric, then give
students a few days before the final drafts are turned in to you. You might also require that
they turn in the draft and scored rubric with their final paper.
6. Have students self-assess their products using the grading rubric and hand in the selfassessment with the product; then faculty and students can compare self- and facultygenerated evaluations.
133
Rubric Category Labels
•
•
•
•
•
Unacceptable, Marginal, Acceptable, Exemplary
Below Expectations, Developing, Meets Expectations, Exceeds Expectations
Novice, Apprentice, Proficient, Expert
Emerging, Developing, Proficient, Insightful
Below Basic, Basic, Proficient, Advanced (AAC&U Board of Directors, Our Students Best
Work, 2004)
Creating a Rubric
1. Adapt an already-existing rubric.
2. Analytic Method
3. Expert-Systems Method
Managing Group Readings
1. One reader/document.
2. Two independent readers/document, perhaps with a third reader to resolve discrepancies.
3. Paired readers.
Before inviting colleagues to a group reading,
1. Develop and pilot test the rubric.
2. Select exemplars of weak, medium, and strong student work.
3. Develop a system for recording scores.
4. Consider pre-programming a spreadsheet so data can be entered and analyzed during the
reading and participants can discuss results immediately.
Inter-Rater Reliability
• Correlation Between Paired Readers
• Discrepancy Index
Scoring Rubric Group Orientation and Calibration
1. Describe the purpose for the review, stressing how it fits into program assessment plans.
Explain that the purpose is to assess the program, not individual students or faculty, and
describe ethical guidelines, including respect for confidentiality and privacy.
134
2. Describe the nature of the products that will be reviewed, briefly summarizing how they were
obtained.
3. Describe the scoring rubric and its categories. Explain how it was developed.
4. Explain that readers should rate each dimension of an analytic rubric separately, and they
should apply the criteria without concern for how often each category is used.
5. Give each reviewer a copy of several student products that are exemplars of different levels
of performance. Ask each volunteer to independently apply the rubric to each of these
products, and show them how to record their ratings.
6. Once everyone is done, collect everyone’s ratings and display them so everyone can see the
degree of agreement. This is often done on a blackboard, with each person in turn
announcing his/her ratings as they are entered on the board. Alternatively, the facilitator
could ask raters to raise their hands when their rating category is announced, making the
extent of agreement very clear to everyone and making it very easy to identify raters who
routinely give unusually high or low ratings.
7. Guide the group in a discussion of their ratings. There will be differences, and this discussion
is important to establish standards. Attempt to reach consensus on the most appropriate rating
for each of the products being examined by inviting people who gave different ratings to
explain their judgments. Usually consensus is possible, but sometimes a split decision is
developed, e.g., the group may agree that a product is a “3-4” split because it has elements of
both categories. You might allow the group to revise the rubric to clarify its use, but avoid
allowing the group to drift away from the learning outcome being assessed.
8. Once the group is comfortable with the recording form and the rubric, distribute the products
and begin the data collection.
9. If you accumulate data as they come in and can easily present a summary to the group at the
end of the reading, you might end the meeting with a discussion of five questions:
a. Are results sufficiently reliable?
b. What do the results mean? Are we satisfied with the extent of student learning?
c. Who needs to know the results?
d. What are the implications of the results for curriculum, pedagogy, or student or faculty
support services?
e. How might the assessment process, itself, be improved?
Assessment Standards: How Good Is Good Enough?
Examples:
1. We would be satisfied if at least 80% of the students are at level 3 or higher.
2. We would be satisfied if no more than 5% of students are at level 1 and at least 80% are at
level 3.
3. We would be satisfied if at least 80% of the students are at level 3 and at least 10% are at
level 4.
135
Lecture / Discussion 6:
Analyzing Evidence of
Student Learning to
Improve Our Practice
Amy Driscoll
136
Analyzing Evidence
of Student Learning
to Improve Our Practice
Amy Driscoll
WASC Educational Seminar
Level I
Sept. 24, 2009
1
Deficiencies in current
academic currency:
• Inability to communicate outcomes of
multiple learning experiences
• Lack of agreed upon achievement
criteria
• Inconsistent faculty judgments
2
Definitions
Reliability: Agreement among faculty that
outcomes have been achieved
Validity: Criteria describes what is intended
as a common referent
Improvements: Usefulness in revising
pedagogy and curriculum for increased
student learning
3
137
Preparation for Analysis
of Student Work
• Outcomes, criteria, standards, evidence
• Student permission
• Systematic collection of representative
student work samples (evidence)
• Faculty motivation, interest and
commitment
• Time and resources ($, expertise)
• Previous assessment experiences- value
for teaching and learning
4
Assessment Protocols
Goal
Outcomes
Evidence
Criteria
Standards:
a)
Exemplary Achievement
b)
Satisfactory Achievement
c)
Unsatisfactory Achievement
5
Assessing Student Learning:
Course, Program and
Institutional Levels 1. Preparation:
7. Revise
outcomes and
criteria, improve
pedagogy and
curriculum for
learner’s success
6. Review and
analyze student
evidence
5. Collect evidence
of student
achievement
4. Make outcomes, evidence,
criteria and standards “public
and visible” (syllabi, programs,
brochures, etc.)
Determine purpose(s)
and definition of
assessment; examine
mission and values
2. Design
assessment:
articulate goals;
develop clear
outcomes, evidence,
criteria and
standards
3. Alignment of curriculum
and pedagogy within
learning outcomes
6
138
Release Form For Use of Student Work Samples
CSUMB is currently collecting samples of student work—work that demonstrates the outcomes and criteria
of the University Learning Requirements. Faculty groups will analyze the work as part of a process of
studying the ULR’s and related assessment processes. Some of the work in our class will be collected for use
in the analysis project. Student names will not appear on the work samples at any time, and analysis will
not occur until the course is complete, student work has been evaluated and grades have been assigned.
You are asked to sign the release form below to indicate your permission for use of your work in this class.
If you choose not to permit use of your work, you are also asked to sign the form below.
Course Instructor: _______________________________________________________________________
Course Name and Number: ________________________________________________________________
RELEASE FORM
DATE: _______________________
I understand that CSUMB is collecting student work samples for analysis in the process of examining the
ULR’s and related assessment processes. My work may be copied and saved for the analysis project.
I understand that my name will not appear on the work samples at any time, and that the analysis of my
work will not occur until after the course is complete, my work has been evaluated and my grade has been
assigned.
___ I give permission to use my work in the ULR analysis project.
___ I do not give permission to use my work in the ULR analysis project.
Print your name: __________________________ Signature: ________________________________
7
Analysis Process
• Holistic reading: Check on achievement of
outcomes (Reliability)
• Verification of criteria (Validity
• Implications for improvement (Usefulness)
8
The Review/Analysis Process
Produced the Following:
• Verification and documentation of student
achievement of most outcomes
• Revision of outcomes
• Revision of criteria
• Changes in courses and pedagogy
• Improvement of assessment
• Decision to have ongoing review/analysis
9
139
Lecture / Discussion 7:
The Administrators’ Role
in Assessment of Student
Learning
Barbara Wright
140
Assessment Tips for the Top
Here are some points for top administrators to think about as they work to implement assessment of
student learning and defuse resistance on their campus.
1. Be informed
a) Assessment of student learning has evolved greatly in the last 15-20 years. It is not the phenomenon
it was when the movement to assess gathered momentum in the mid-80s. As a result, assumptions, methods,
purposes, and even the definitions of basic vocabulary have changed. On a campus, this can lead to enormous
confusion about what people are being asked to do and why. So first and foremost, know what the options are,
and what it is you are trying to accomplish on your campus.
b) Plan to promote what experience has shown is best practice.
Assess for quality improvement, not just quality assurance
Focus on student learning, not surrogates (GPA, satisfaction surveys, etc.)
Pursue outcomes that are educationally important, not just easily measurable
Use authentic methods (i.e., those that reflect what students will need to do once they’re out in
the real world), not just traditional academic ones (e.g., multiple-choice tests)
Gather qualitative as well as quantitative evidence of learning
Promote good, inclusive process as well as products (i.e. findings, changes)
Close the loop not just with feedback but with actions leading to improvement
Support assessment with formal structures, planning, budgeting
2. Communicate
a) Many bodies – state departments of higher education, system offices, regional and professional
accrediting associations – require assessment of student learning. However, the real reason to do it is because
it’s the right thing to do if we care about our students and what they learn. This is a reason that makes sense to
faculty. It can’t be repeated too often.
b) It’s OK to leverage the pressure from external bodies, but don’t overdo it; otherwise the message in
2.a. is undermined.
c) When there are good things to celebrate, we should do it. When there are less than wonderful
findings, we need to acknowledge them candidly, then emphasize that this is a great opportunity for
improvement with maximum value added.
d) We need to report on the findings and results of assessment efforts regularly in publications like
web pages and the student newspaper or alumni magazine,
e) Assessment expectations should be included in the catalogue and view book as well as mission
statements at all levels. Job descriptions, faculty and staff handbooks, employment contracts, and the like
should name assessment as a routine responsibility whenever appropriate.
f) Make sure communication is a two-way street.
3. Provide reassurance
a) The campus needs to know you do not plan to use assessment as a witch hunt or a thinly disguised
ploy to cut lines and terminate programs. This may be the last thing on your mind, but it’s the first thing on a
lot of faculty minds. Experience shows that if faculty do harbor these fears, they will not face problems
candidly but rather seek to conceal them. That undermines the integrity and usefulness of the whole process.
b) Faculty and programs need to know that if those external entities demanding assessment have any
vile plans for the findings, you’re on the side of your faculty and programs. You’ll protect them if push comes
to shove. Again, repetition is key.
c) Give reluctant programs a face-saving way to comply.
d) Assure everyone, especially faculty, that assessment is not an attack on academic freedom, not a
required curriculum. There are many legitimate paths to the same outcome.
141
4. Provide support
a) Don’t ask your campus to plan for assessment or carry it out without some training
b) Provide parameters that reflect the conclusions you’ve come to about the assessment effort on your
campus (see 1.a). Allow flexibility – it’s OK for programs’ plans to reflect their modes of inquiry and the
intellectual traditions of their disciplines – but within those parameters. Don’t make faculty waste time second
guessing you or figuring assessment out entirely for themselves. That just breeds resentment.
c) Use training as an opportunity to get everyone on campus on the same page regarding assumptions,
methods, purposes, and even the definitions of basic vocabulary
d) Use training to clear away misconceptions, reduce fears, attack obstacles
5. Be efficient, inclusive -- and respectful of what's already going on
a) Piggy-back onto existing processes whenever possible, e.g., connect departmental reporting on
assessment to the annual reports programs are used to submitting. Link outcomes and assessment to routine
course approval. But be careful when folding assessment into program review. Traditional program review
focuses on inputs and processes in order to increase inputs -- usually faculty lines and budget -- or to protect
against cuts in inputs. Redefine the review so that the emphasis shifts to learning outcomes and programs are
rewarded for revealing a problem and then fixing it, not just for making themselves look good. (See 6. below.)
b) Draw on existing expertise and models. On every campus there are pockets of assessment activity
and faculty expertise to draw on, even if the work has not been labeled "assessment."
c) Involve existing structures or offices such as the university senate, institutional research, or the
center for teaching and learning.
6. Provide rewards
a) The idea is not to buy compliance by paying for every little bit of faculty work on assessment. In
fact, that’s a vicious circle you don’t want to get caught in. However judicious rewards for special
contributions can help a lot.
b) The idea here is not to reward the programs that keep coming up with proof that “We’re excellent”;
the idea is to reward programs that say “Here’s the problem, and here’s how we solved it.” This message must
be very clear when the choice of program and the reward are announced publicly. In other words, you’re
rewarding “quality improvement,“ not “quality assurance.” To put it another way, the reward is for maximum
value added, not status quo, no matter how good that is.
c) Whenever possible and appropriate, reward programs rather than individuals. One of the biggest
challenges of assessment, but also one of its biggest benefits, is that it requires faculty to act collectively, not
as atomistic individuals responsible solely for their own courses or areas of specialization.
d) It should become institutional policy to expect contributions to assessment as part of reappointment
and tenure dossiers. Across campus, assessment efforts must be acknowledged as a form of scholarship and be
clearly seen to help faculty earn promotion, tenure, and merit increases.
e) The flip side of reward is punishment: doing assessment should not be a “punishment” in the form
of additional workload without some sort of compensation, either for individuals or the program. In other
words, a faculty member who contributes to assessment needs to be relieved of some other ongoing
responsibility. (This also sends a message about the value of work in assessment. Add-ons are seldom taken
seriously and they never last.)
f) The fact that a junior faculty member has worked on assessment should never be allowed to count
against him/her in promotion or tenure proceedings.
7. Provide funding
a) People follow the money and faculty are especially good at this! To be taken seriously, assessment
has to have money behind it. Money is both a practical aid to getting things done and a powerful symbolic
142
message. The pot doesn’t have to be big – some highly effective assessment strategies are actually very cheap
– but it needs to be very visible.
b) Plan on continued budgeting for assessment, not a one-shot infusion. Money is essential for start-up
costs – training, consultants, instrument development – as well as for ongoing activities: retreats, conference
presentations, reports and publications, etc.
c) Think about program-level performance funding. This phrase has horrible connotations at the state
level, but it can work on campus. The key thing here is that “performance” refers to the carrying out of highquality, high-impact assessment, not achievement of high scores or other traditional indicators of quality.
In one model, when departments submit annual budget requests, they have to back up academic
requests with assessment findings. The budgets get reviewed by a faculty/staff budget review committee,
which makes recommendations before passing the budget up to higher levels. The committee looks for
assessment findings to back up requests and bases its recommendations on the quality of the program’s
evidence, analysis, and plans for improvement. Administration generally follows the recommendations, thus
enhancing the status of both the committee and of assessment.
This process is useful for several reasons: 1) it’s highly motivating; 2) it underscores the seriousness of
the assessment effort – and consequences of failure to engage in it; 3) it exerts powerful negative pressure: the
requests of departments not doing assessment really do go unfunded and their programs gradually fall behind;
4) it provides transparency; and 5) it educates the campus, from committee members out, in widening circles,
about how to do assessment well.
8. Aim for broad involvement
To change campus culture, you need broad involvement or the change will remain superficial and fail
to take hold. That means
a) the whole range of campus experience eventually needs to be assessed: not just the major but
general education, first-year experience, student government, internships, community service, dorm life,
extracurricular opportunities, etc.
b) not just faculty but professional academic staff, students, external advisors, alums, employers, etc.
need to participate as appropriate.
c) The whole chain of command needs to be on board and on the same page about the institution’s
philosophy and purpose in doing assessment, from president and AAVP through deans and
department/program chairs
9. Institutionalize
Formal structures will legitimize assessment. Eventually the assessment effort needs to move out of
the realm of a special grant-funded project or one program’s experiment and become standard practice
supported by an office, advisory committee, staff, budget, reporting relationships, and oversight. This can
happen gradually and it needn’t be elaborate, but it needs to happen.
10. Codify
a) Eventually, appropriate documents need to refer explicitly to assessment. (See the list in 2.d, e.)
b) In negotiations with AAUP or other unions, it may be best to emphasize that assessment is not the
end; it is merely the means to a worthy end: better education. Just as computers are a powerful tool, one that
no campus wants to be without, so too assessment is our key tool for improving student learning and
enhancing institutional quality.
Barbara D. Wright
Revised September 2009
[email protected]
143
144
The Administrators’ Role in Assessment of Student Learning:
Setting the Context for Success
Cyd Jenefsky
Setting the Context
¾ Systems view – engaging whole institution in enhancing student learning
¾ Strategic – intentional, aspirational, aligning resources/ processes/goals
¾ Tending to culture as fertile soil for change – setting expectations, priorities
¾ Intentional change management
Integrated System Promoting Student Success
¾ “Consisting of more than isolated cells of activity”
¾ “Institutional way of behaving” (Maki, 2004)
¾ “integrating processes of inquiry into the culture, governance, learners, and
organizational relationships that exist in an institution” (Keeling, et. al., 2008)
¾ “We are all connected by the mutual intellectual endeavor of organizing ourselves to
see that students succeed.”
Bresciani, et. al. (2009)
Stages of Institutional Development
¾ Teaching-Centered – taking teaching seriously
¾ Student/Learner-Centered – taking students seriously
¾ Learning-Centered – taking learning seriously throughout an organization
Promoting continuous organizational learning (including via assessment) in order to
enhance student learning.
Shared Responsibility for Assessment
¾ Who ‘owns’ assessment on campus?
¾ Faculty: “can’t do it without administrative leadership”
¾ Administrators: “it’s faculty who have to do it”
¾ What does it look like as a “shared responsibility”?
¾ What part do administrators own?
Administrators’ Areas of Responsibility
¾ Structures, processes, communication channels at program, institutional levels
¾ Support and resources to initiate, build, sustain core commitment
¾ Valuing assessment of student learning as “core institutional process” (Maki, 2004)
145
Structures, Processes, Communication Channels
Examples:
¾ develop practices, policies, processes for using SLO results (& institutional data) for
evidence-based decision-making at program, college, institutional levels
¾ integrate assessment of SL with strategic planning & budgeting (inc. calendar cycles)
¾ create events, processes for formal & informal discussion of results across disciplines
and divisions
¾ build linkages between existing assessment activities
¾ integrate assessment of student learning into program and performance review
processes
¾ funnel results to existing committees for decision-making and planning
¾ establish expectations for standards of performance
Support & Resources to Initiate, Build Sustain Commitment
Examples:
¾ cultivate ‘champions’ (fac/staff/admin)
¾ build scaffolded professional development for fac/staff/admin (tailored to adult
learners)
¾ allot resources for doing assessment
¾ assist with streamlining assessment processes
¾ provide resources for ‘closing the loop’
¾ share assessment results
¾ guide fac/staff/leadership to develop standards/benchmarks
¾ offer grant competition for innovation in assessment, improvement in student
learning, scholarship on student learning, etc.
¾ support scholarship of teaching, learning and assessment
Valuing Assessment of Student Learning as Core Institutional Process
Examples:
¾ collaboratively develop institutional “principles of commitment” (Maki)
¾ communicate importance of assessment of student learning, organizational learning
and evidence-based decision-making at all levels (& within university materials,
syllabi, catalogue, web, etc.)
¾ articulate coherent vision, clear expectations
¾ frame inquiry into student and organizational learning as collective enterprise
(Shulman’s “community property”)
¾ practice asking questions about student learning
¾ reward initiative, innovation, improvement in assessment of student learning and use
of results, including in RPT (review, promotion, tenure) processes
¾ celebrate achievements!
Start with what matters:
What do you most want to learn about student learning?
About your organization?
146
References:
Allen, M. (2007). Campus support for assessment and educational effectiveness.
Unpublished manuscript.
Bresciani, M.J., Gardner, M.M., & Hickmott, J. (forthcoming, 2009). Demonstrating
student success: A practical guide to outcomes-based assessment of student learning and
development in student affairs. Sterling, VA: Stylus.
D. Cooperrider & D. Whitney (2005). Appreciative Inquiry: A Positive Revolution in
Change. San Francisco, CA: Berrett-Koehler.
Huba, M.E. & Freed, J.E. (2000). Learner-Centered Assessment College Campuses:
Shifting the Focus from Teaching to Learning. Boston: Allyn & Bacon.
Keeling, R.P., Wall, A.F., Underhile, R., & Dungy, G.J. (2008). Assessment
reconsidered: Institutional Effectiveness for Student Success. ICSSIA.
Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across
the institution. Sterling, VA: Stylus.
Shulman, L. (2004). Teaching as Community Property. San Francisco, CA: JosseyBass.
147
Lecture / Discussion 8:
Assessment for
Community College
Career and Technical
Educational Programs
Fred Trapp
148
Assessment for Career &
Technical Education Programs
Presented by
Dr. Fred Trapp
Cambridge West Partnership, LLC
Administrative Dean, Institutional Research/Academic Services
Long Beach Community College (retired)
[email protected]
Overview
„
„
„
„
„
„
„
„
Purposes of career and technical assessment
Types of programs
Assessment plans
The assessment plan
Categories of assessments
Using the grading process
Dale McIver’s problem
ACCJC assessment rubric
2
Purposes of Assessment
„
„
„
Improve learning & instruction
Certify individual mastery
Evaluate program success
3
149
Improve Learning & Instruction
„
„
„
„
Provide information on knowledge &
skills learned
Administered often
Graded quickly
Technical quality of assessments less
critical
4
Certify Individual Mastery
(not our focus in this retreat)
„
„
„
„
Focus on specific skills & knowledge
Multiple & single measures
Used for decisions about selection,
promotion, certification
Quality of measurers (reliability,
validity, fairness) more important
5
Continuum of Knowledge & Skills-
What is your program seeking to accomplish?
General
Workforce
Preparation
Industry Core
Skills &
Knowledge
Occupational
Cluster Skills
Specific
Occupational
Skills
All Workers
Health Services
Health Information
Services
Health Information
Technology
Read, write,
perform
mathematical
operations, listen &
speak
Be aware of the
history of health
care.
Locate information
in medical records.
Evaluate medical
records for
completeness &
accuracy.
Use health care
terminology.
Use computer
programs to
process client
information.
Use a computer
program to assign
patients to a
diagnosis-related
grouping.
6
150
Your Program and Assessment Plan
ƒ
Statement of purpose
ƒ
List of required courses
ƒ
Restricted electives list
ƒ
Curriculum matrix and analysis
ƒ
Plan elements (four parts)
7
Your Program and Assessment Plan
ƒ
Plan elements
1. Intended learning outcomes
ƒ
Long list vs. short list
2. Means of assessment and criteria for success
3. Report of results
4. Use of results
8
Your Program and Assessment Plan
Example of an assessment plan from LBCC
Electricity Program
9
151
Common Outcomes for CTE
„
„
„
„
„
Technical skills (discipline-specific/technology/technical
competency/currency)
Application of discipline skills (theoretical
knowledge/subject matter mastery/observations skills)
Critical thinking & problem solving (assessment
skills)
Communication
Professional behavior (professional
practices/ethics/teamwork)
See Handout- Determining Purposes
10
Strategies for Assessment Work
„
„
„
„
„
„
Selected vs. constructed response
Standard exams, licensing, etc.
Wage data follow up
Opinion data
Use student work in courses to address course and program learning
outcomes
Consider student’s development point in the program
„
Just starting vs. about to complete
„
Capstone project or course
Program portfolio
„
Multiple measures are always recommended
„
„
Specific assignment from key courses
11
Broad Categories of Assessments
Response Type
Category
Selected
Constructed
Written assessments
Multiple choice, true/false, matching
X
Show work, concept map, journal response
X
Essay, problem based, scenario
X
Performance tasks
X
Projects (research paper, project, oral
X
Portfolios
X
presentation)
See Handout- Selecting Assessment Tasks
12
152
Features of Selected & Constructed
Response Assessments
Selected Response
Feature
Rarely
Sometimes
Constructed Response
Usually
Rarely
X
Easy to develop
X
Easy to administer
X
Easy to score
X
X
X
X
Authentic
Sometimes
Usually
X
X
X
Efficient (requires limited time)
Credible to stakeholders
Embodies desired learning activities
X
X
X
X
X
X
Sound basis for determining quality of scores
Effective for factual knowledge
X
Effective for complex cognitive skills
X
X
X
13
Standardized Exams
Job Ready Tests
ƒ
National standardized test for two-year
technical programs
ƒ
Written objective test
ƒ
Performance test - institution administered
14
Fields in Which Job Ready Tests
Are Available
Examples:
Commercial Foods
Diesel Engine Mechanics
Child Care Services
Accounting/Bookkeeping
Refrigeration
Horticulture
Forestry Products
Electronics
This is not an endorsement of NOCTI
National Occupational Competency Testing Institute:
1-800-334-6283
www.nocti.org
15
153
Engineering Technician and
Technologist Certification Programs
National Institute for Certification in Engineering Technologies (NICET)
•
Construction Materials Testing
• Fire Protection
• Electrical Communications System
•
• Geo-synthetic Materials
•
• Land Management and Water Control
•
• Underground Utilities Construction
•
• Building Construction/Water and Waste Water
Low Voltage
Installation Inspection
Geo-technical
Transportation
Plants
Sponsored by
National Society of Professional Engineers Telephone: 888-476-4238
http://www.nicet.org
This is not an endorsement of NICET
16
Standardized Exams
Work Keys
ƒ
Foundational skills
ƒ
ƒ
Applied math, technology, listening, reading for
information, writing, locating information, observation
Interpersonal skills
ƒ
Teamwork
www.ACT.org
1-800/WORKKEY (967-5539)
This is not an endorsement of ACT Work Keys
17
Licensure Examinations Commonly
Found at 2-Year Colleges
ƒ
Nursing (RN, LVN, CNA)
ƒ
Aviation Maintenance
ƒ
Diagnostic Medical Imaging
ƒ
Dietetics
ƒ
Computer Applications
ƒ
Computer Systems Engineering
and Network Management
18
154
Employment/Wage Follow Up as
Assessment
ƒ
Employment Development Department
(EDD) file match
ƒ
By degree/certificate/area of concentration
(12 units or more)
ƒ
ƒ
ƒ
ƒ
ƒ
Wages in final year at community college
Wages one year out of college
Wages three years out of college
Excludes those who transfer
Depends upon unemployment insurance so
does not include self-employed or out of state
19
Opinion as Assessment
Student/Graduate/Alumni
• Affirmation of accomplishments
• Indirect measure
• Primarily support evidence
20
Opinion as Assessment
Student/Graduate/Alumni
„
Completers
„
„
Associate Degree or Certificate of Achievement (18
units plus)
Early Leavers With Marketable Skills (ELMS)
„
12 units in the discipline but not program award
21
155
Opinion as Assessment
Student/Graduate/Alumni
„
Are you employed in the field for which you were
educated at the college?
„
„
„
„
Income range?
To what extent are you able to perform _____
as a result of your education?
„
„
„
How long?
Full-time or part time?
Provide option for N/A
What should we emphasize more/less?
Interest in continuing education opportunities
22
Opinion as Assessment
Employer/Supervisor Survey (primary, direct
evidence)
ƒ Legal restrictions
ƒ Internship/Co-Op Supervisor
ƒ More specific the better
ƒ Use the Chamber of Commerce
ƒ Limit the number of questions asked
ƒ How well do graduates of our program perform?
ƒ Compared to your expectations
ƒ Compared to other educational programs
23
Opinion as Assessment
Mechanics of Survey
ƒ Identify students from college administrative data
ƒ Past three to five years
ƒ Test the name & address (post card or software)
ƒ Mail the survey with cover letter
ƒ How well do graduates of our program perform?
ƒ Include self-addressed, stamped envelope
ƒ Incentive drawing
ƒ Follow up reminder card or phone calls
24
156
Intern Evaluations
Rad. Tech- Positioning Skills
„
„
„
Ratings
Correct orientation of patient
Align part to center of film
Center CR to film and/or angle
1234
1234
1234
CR to part and film
„
„
Remove unnecessary parts from field
Correct placement of film markers, away
from body parts
1234
1234
25
Process Checklist
Procedure
Observed
Comments
Selected approach
Correct equipment used
Measurement accurate
Sought peer help if needed
Recorded observations
Cleaned up after work
26
Problem-Based Exam, Scenario,
Case Study
ƒ Short written answers to longer narratives
ƒ Score in terms of content and conventions
ƒ creativity
ƒ Apply knowledge and skills to new settings
ƒ Level of detail and complexity of problem
varies
27
157
Automotive Technology
Simulation
Program Intended
Educational
Outcomes:
Means of Program Assessment:
2. Graduates of the
Automotive Technology
Program will be
technically proficient.
2a. At the close of their final term, graduates will be able to identify and
correct within a given period of time all of the mechanical problems in five test
cars that have been "prepared" for the students by Automotive Technology
Program faculty.
2b. The Automotive Technology Program graduates will pass the National
Automotive Test.
28
Automotive Technology
Simulation
See handout for elaboration of project
29
Means of Assessment- Grades
„
Evaluation of individual students =
assessment
„
„
„
„
„
Focus is individual not groups of students
A summative, not formative act
Objectivity of single evaluator vs. group
Generally not accepted as direct evidence
Uses of the grading process
„
„
Agreed upon course exam or part of exam
Row and column model for assignments
30
158
Embedded Assessment Strategy
Row and Column Concept
Criteria
Tim Jane Mary Joe Dave Average
Spelling
Grammar
3
2
4
5
1
3
2
2
3
5
2.6
3.4
Punctuation
Structure
4
3
5
2
2
3
3
5
4
3
3.6
3.8
Total
13
17
10
12
15
Student Grade
C
A
D
C
B
Total down the column for individual grading. Analyze across the row for
assessment of intended outcomes from the group.
Jim Nichols
31
The Grading Processes
Implications for Assessment
Using the Grading Process & Existing
„
Assignments (stealth assessment)
„
Build a grading rubric for an assignment
„
„
„
Determine the criteria on which you evaluate student
work (primary trait analysis)
Describe shades of performance quality
As a faculty group sample student work and apply
the rubric
32
Analytic Rubrics-Numerical Scales
„
Number of points on scale
„
„
„
Larger scale- harder to differentiate among points
Smaller scale- less diagnostic information
5 to 6 points*
„
Use prior experience
„
„
„
off task = lowest, excellent = highest
Allocate other three in between
Multiple dimensions
„
„
Use same scale if possible (easier grading)
Use different scales to reflect relative value
*When first doing a rubric it is better to start with a 3-point scale
33
159
Qualitative Scale- Descriptions
„
Descriptions
„
„
„
No, minimal, partial, complete evidence
Task not attempted, partial completion,
completed, goes beyond
Off task, attempts to address task, minimal
attention to task, addresses task but no
elaboration, fully elaborated & attentive to
task and audience
34
Qualitative Scale- Evaluation
„
„
„
„
Evaluate or judge by standards
Criteria embed ideas of excellence,
competence or acceptable outcomes
Discipline-based standards (criteria
referenced)
Comparing students’ relative status
(norm referenced) especially for
developmental considerations
35
Portfolio Assessment Criteria and
Scoring
„
„
„
„
What are the criteria for selecting
samples going into the portfolio?
How or will progress be evaluated?
How will different tasks or productsvideos, art work, essays, journal
entries, etc. be compared or weighted?
What is the role of student reflection?
36
160
Program Portfolio
„
„
For programs without a capstone
Select assignments at key points in the
required courses of the program
„
„
„
„
Beginning, middle, end
Require all students to provide the completed
assignment
Develop an assessment rubric
Sample from the student work at various
stages in the program
37
Reliable Scoring- Train the Raters
„
Scoring guide
„
„
„
„
„
„
Fully explicit scoring criteria
Examples illustrating each scoring point
Abbreviated version of criteria for
reference during scoring
Sample form for recording scores
Orientation and practice scoring
Record scores and discuss
38
Quality of Information of Selected &
Constructed Response Measures
Dimension of Quality
Reliability
Validity
Selected Response
Constructed Response
Automatic scoring is error free
Rating process can increase
errors
Many responses per topic
increases consistency of score
Fewer responses per topic
reduces consistency of score
Strong theoretical basis for
measuring reliability
Greater between-task variability
in student performance
Large inferences from item to
occupational behavior
Greater match between
assessment tasks & real world
demands
Variation in administration
conditions can complicate
interpretation of results
Fairness
Quantitative techniques help
identify potential unfairness
May have greater fairness
because tasks are more
authentic
39
161
Dale Mc’Iver’s Problem
Case study
40
Issues in Assessment Planning
„
„
„
„
„
„
Single vs. multiple measures
High vs. low stakes
Stand-alone vs. embedded tasks
Standardization vs. adaptability
Single vs. multiple purposes
Voluntary vs. mandatory participation
41
ACCJC Assessment Rubric
„
Levels of Implementation
„
„
„
„
Awareness
Development (now)
Proficiency (by 2012)
Sustainable continuous quality
improvement
42
162
Handout Packet
Assessment for Career &
Technical Education
Programs
Fred Trapp, Ph.D.
Cambridge West Partnership, LLC
Administrative Dean,
Institutional Research/Academic Services (retired)
Long Beach City College
September 2009
163
Plan for Electricity Program-Level Assessment
Plan for Electricity Program-Level Assessment
The Program Curriculum
The Electricity program awards both an Associate Degree and a Certificate of Achievement.
Students must complete a total of 45 units, of which 40 are core subject offerings. Certificates of
Completion are awarded at the conclusion of one or several courses in the program to denote
milestones of accomplishment. The requirements for all of these awards are listed in the college
catalog.
st
1 Course
2nd Course
3rd Course
4th Course
Core Required Day Courses
ELECT 200A
ELECT 200B
ELECT 200C
ELECT 200D
ELECT 253
ELECT 225
ELECT 435A
Core Required Evening Courses
ELECT 204, 210A, 240, 202
ELECT 209, 210B, 242
ELECT 212, 210C, 242
ELECT 214, 210D, 245, 250
ELECT 253
ELECT 225
ELECT 435A
Day Program Courses
ELECT 200A
First Semester Industrial Electricity
ELECT 200B
Second Semester Industrial Electricity
ELECT 200C
Third Semester Industrial Electricity
ELECT 200D
Fourth Semester Industrial Electricity
ELECT 253
OSHA Standards for Construction Safety
ELECT 225
Algebra & Trigonometry for Technicians
ELECT 435A
Electrical Motor Control
Evening Program Courses
ELECT 204
First Semester Fundamentals of DC Electricity
ELECT 210A
Laboratory Practices
ELECT 209
Second Sem Fund of Motors/Generators
ELECT 210B
Laboratory Practices
ELECT 212
Third Semester Fund of AC Electricity
ELECT 210C
Laboratory Practices
ELECT 214
Fourth Semester AC Principles & Pract
ELECT 210D
Laboratory Practices
ELECT 240
Electrical Code-Residential
ELECT 202
Electrical Mathematics
ELECT 242
Electrical Code-Grounding
ELECT 245
Electrical Code-Commercial
ELECT 250
Electrical Code-Advanced
Scott Fraser and John Hauck, LBCC
164
Plan for Electricity Program-Level Assessment
Recommended Courses
A listing of 19 additional courses is provided for students to select offerings that are
recommended as instruction that would complement and extend the required curriculum. In all
cases these courses are required elements of other related programs. These courses include:
Networking Cabling Installation
Cisco Networking I, Introduction
Technical Applications of Minicomputers
Electrical Motors and Transformers
Solid State Fundamentals for Electricians
2 Variable Speed Drives
Industrial Drive Systems
Robotics Technology
Electrical Cost Estimating
2 Electrical Pipe Bending
Blueprint Reading for Electricians
Traffic Signals Systems 1
Traffic Systems Communication
Traffic Signal Controllers & Digital Systems
Electrical Motor Control
AutoCAD I, Fundamentals
Basic AutoCAD for Architecture
Scott Fraser and John Hauck, LBCC
165
Plan for Electricity Program-Level Assessment
Assessment Plan Notes
#
Learning Outcomes
1st Crs
2nd Crs
Electricity Core Courses Required
ELECT
253
3rd Crs
4th Crs
ELECT
225
ELECT
435A
Be technically competent:
Perform basic wiring tasks.
1
2
Be technically competent:
Complete circuit box panel
schedules
Demand factors
Load cycles
Be technically competent:
Trouble shoot successfully
Identify symptoms
Diagnose problem
Fix problem
Test fix to verify problem
solution
3
3
3
3
3
2
2
3
3
3
3
3
3
3
3
Be technically competent:
Install electrical wiring or
equipment to national electrical
code standards
3
4
5
Recognize safe work practices
1
1
1
1
3
1
6
Demonstrate safety practice
during lab work
1
1
1
1
1
1
7
8
A possible rating scheme is to code each cell using this approach.
0- Course does not include instruction and assessment of this outcome.
1- Course includes instruction or practice of the outcome, and performance/knowledge of this
outcome is assessed.
2- Course includes instruction or practice in the outcomes of this outcome, performance/
knowledge is assessed, and 20% or more of the course focuses on it.
3- Course includes instruction or practice in the outcome, performance/knowledge is assessed,
and 1/3 or more of the course focuses on it.
The purpose of completing a matrix like the one above is to get an overall feel for the extent to
which each program learning outcome is being addressed throughout the required courses and to
what extent it is addressed where.
Scott Fraser and John Hauck, LBCC
166
Plan for Electricity Program-Level Assessment
Expanded Statement of Purpose: Prepare students for entry level positions in the field of electrical technology.
Intended Educational Outcomes:
Means and Criteria for Assessment:
1. Be technically competent
• Perform basic wiring tasks.
• Accomplish bends in conduit
• Complete circuit box panel
schedules
o Demand factors
o Load cycles
o Wire sizes
o Circuit breakers
• Trouble shoot successfully
o Identify symptoms
o Diagnose problem
o Fix problem
o Test fix to verify
problem solution
• Install electrical wiring or
equipment to national electrical
code standards
1A. Worksheets on parts of the residential site wiring
diagram and other lab projects are scored with an
instructor rubric for accuracy. 85% of the students will
achieve an overall accuracy score of 75% or better (see
the accuracy column of the related example ELECT
200A grading rubric)
Results of Assessment:
Use of Results:
1B. Within either ELECT 200D or 214 students will
successfully create an industrial building wiring plan
which is evaluated by an instructor grading rubric for
accuracy, completeness, and neatness.
• Wiring layout document
• Lighting document
• Title 5 documents
• Panel schedules
• One-line diagram
95% of the students will achieve an overall score of 75%
or better
1C. Within ELECT 435A students will successfully
• Recognize electrical symbols
• Prepare a wiring design
• Demonstrate component knowledge
• Troubleshoot circuit errors
The project is evaluated using an instructor grading
rubric for accuracy, completeness and neatness. (see
the related ELECT 200A grading rubric and the lab task
process notes below for examples of these criteria)
Scott Fraser and John Hauck, LBCC
167
Plan for Electricity Program-Level Assessment
Intended Educational Outcomes:
Means and Criteria for Assessment:
2. Recognizes safe work practices
2. Within ELECT 253 students will complete an exam in
each of eight modules with the OSHA minimum test
score of 80% correct overall.
3. Demonstrates work safety in the
laboratory
3. Instructor observations during labs will result in no
cases of safe work practices.
4. Employers will be satisfied with the
competence of program graduates.
4. Of those employers who respond to a mailed
survey, ___ % will report satisfaction with the
training provided by the program.
Scott Fraser and John Hauck, LBCC
Results of Assessment:
Use of Results:
168
Plan for Electricity Program-Level Assessment
ELECT 435A Laboratory Task/Process and Assessment Procedures Description
1. Students simulate the task on the computer
2. Students wire the lab task which is evaluated by the instructor on
a. neatness
b. proper connections
c. proper routing
3. Students test the operation by
a. completing a check sheet of ohm meter readings
b. complete the expected readings for the balance of the circuit diagram
c. record the actual results
d. synthesize by comparing the actual recorded to the expected results
4. Energize the circuit and verify proper operation after the instructor has signed off on the work in item 3 above
5. Instructor injects faults to the circuit which are typical in the industry without the student present
6. The students trouble shoot and correct faults by doing the following to record their thought processes. All of the faults must be
located or the circuit will not operate.
a. List each fault located
b. Describe the cause of he fault
c. What was malfunctioning that caused the fault?
d. What did the student do to correct the fault
7. Student prepares their box for final inspection.
Scott Fraser and John Hauck, LBCC
169
Questions to Help Determine Learning Outcomes
Write out answers to these questions in sufficient detail that others can agree on what the
outcomes mean and whether or not students have obtained them.
1.
2.
3.
4.
5.
What important cognitive skills do I want my students to develop?
What social and affective skills do I want my students to develop?
What metacognitive skills do I want my students to develop?
What types of problems do I want them to be able to solve?
What concepts and principles do I want my students to be able to apply?
Draw upon the work of national or state professional groups and consult with colleagues.
Generic Skills List
(cross disciplines)
1.
2.
3.
4.
5.
6.
7.
Communicating clearly
Questioning
Formulating problems
Thinking and reasoning
Solving complex, multi-step problems
Synthesizing knowledge from a variety of sources
Using cooperation and collaboration
Big Ideas, Skills, Concepts, Processes & Techniques
(characterize a specific discipline)
1.
2.
3.
4.
5.
6.
7.
Developing a hypothesis (hunch)
Designing experiments (tests)
Drawing inferences from data
Using observation and analyzing similarities and differences in phenomena
Working with laboratory (test) equipment or tools
Re-testing to ensure repair was correct
Complete write up of process, findings, repair if required
Adapted from Herman, Joan, et. Al. A Practical Guide to Alternative Assessment
170
Selecting Assessment Tasks
Selecting Assessment Tasks
Goal- Matching the assessment task to your intended learning outcome (skills knowledge,
attitudes).
1.
2.
3.
4.
Does the task match the specific instructional intention?
Does the task adequately represent the content and skills you expect students to attain?
Does the task enable students to demonstrate their progress and capabilities?
Can the task be structured to provide measures of several outcomes?
Generating good ideas for assessment tasks.
1.
2.
3.
4.
Brainstorm with colleagues.
Draw upon the work of national or state professional groups.
Ask experts in the field who are locally located.
Consider ideas from professional journals, conferences, training sessions, etc.
Describing the assessment task by specifying the following.
1.
2.
3.
4.
5.
6.
7.
8.
What student learning outcomes are intended for the assessment?
What are the content or topic areas?
What is the nature and format of questions to be posed to students?
Is it group or individual work? If group work, what roles are to be filled?
What options/choices are allowed for the finished product? Who makes the choices?
What materials/equipment/resources will be available to the students?
What directions will be given to the students?
What constraints (time allowed, order of tasks, answering student questions, how much
help will be provided) are going to be imposed?
9. What scoring scheme and procedures will be used?
Criteria to critique tasks.
1. Do the tasks match your important instructional goals and student learning outcomes?
2. Do they impose enduring problem types, typical ones students are likely to face
repeatedly?
3. Are the tasks fair and free of bias?
4. Are the tasks creditable to important stakeholders?
5. Will the tasks meaningfully and engaging to students so they will be motivated?
6. Are the tasks instructionally feasible? Do you have the resources and expertise to teach
them?
7. Are the tasks feasible for implementation in the classroom or lab (space, equipment, time,
costs)?
Adapted from Herman, Joan, et. Al. A Practical Guide to Alternative Assessment
Fred Trapp, January 2008
[email protected]
171
SCHOOL NORMS VS. NATIONAL PASSING NORMS
SCHOOL:
FOR
SCHOOL NAME:
CR9T070R
4 QTR
LONG BEACH CITY COLLEGE
AMP
NO APPLS
2
OCT NOV DEC
WP05
WESTERN PACIFIC
CURRENT QUARTER
TYPE TEST
FS 8080-08-147
2007
TWO YEAR ACCUMULATIVE **
NO APPLS PASS
PCT APPLS PASS
2
AVG GRADE
100
APPLS
80
SCHL NORM
NATL APPLS
100
12
NATL NORM
4452
94
COMPUTER TEST SCHOOL NORM VS NATIONAL NORM
A
Recip
Eng
SCHOOL NORM 1 YR
NATIONAL NORM 1 YR
B
C
H
I
Turb Eng Eng Insp Eng Inst Fire Prot
1.17
1.35
0.67
1.61
0.33
0.41
0.50
0.81
0.17
0.44
J
K
L
Eng
Elect
Lub Sys
0.50
1.20
0.67
1.55
M
N
O
P
Q
R
T
Ign Sys
Fuel
Meter
Fuel
Sys
Induct
Sys
Cool
Sys
Exhst
Sys
Prop
APU
1.00
2.07
1.67
1.81
0.50
0.57
0.17
0.76
0.00
0.37
0.33
0.64
1.67
2.59
0.17
0.17
POWERPLANT TEST
2.80
2.40
2.00
1.60
1.20
0.80
0.40
0.00
A
B
C
H
I
J
K
L
M
N
O
P
Q
R
T
School Norm 1 Yr
National Norm 1 Yr
SCHOOL NORM VS NATIONAL NORM - 1 YR
** CURRENT DB: EFF DATE 10/1998.
172
CAPSTONES
Rubric for Assessing the Use of Capstone Experiences for Assessing Program Learning Outcomes
Criterion
Relevant
Outcomes
and Lines of
Evidence
Identified
Initial
It is not clear which program
outcomes will be assessed
in the capstone course.
Valid Results
It is not clear that potentially
valid evidence for each
relevant outcome is
collected and/or individual
faculty use idiosyncratic
criteria to assess student
work or performances.
Reliable
Results
Results Are
Used
The Student
Experience
Emerging
The relevant outcomes are
identified, e.g., ability to integrate
knowledge to solve complex
problems; however, concrete
plans for collecting evidence for
each outcome have not been
developed.
Faculty have reached general
agreement on the types of
evidence to be collected for each
outcome; they have discussed
relevant criteria for assessing
each outcome but these are not
yet fully defined.
Developed
Relevant outcomes are
identified. Concrete plans for
collecting evidence for each
outcome are agreed upon and
used routinely by faculty who
staff the capstone course.
Highly Developed
Relevant evidence is collected; faculty
have agreed on explicit criteria
statements, e.g., rubrics, and have
identified examples of student
performance at varying levels of
mastery for each relevant outcome.
Faculty have agreed on concrete
plans for collecting relevant
evidence for each outcome.
Explicit criteria, e.g., rubrics,
have been developed to assess
the level of student attainment of
each outcome.
Those who review student
work are not calibrated to
apply assessment criteria in
the same way; there are no
checks for inter-rater
reliability.
Results for each outcome
may or may not be are
collected. They are not
discussed among faculty.
Reviewers are calibrated to apply
assessment criteria in the same
way or faculty routinely check for
inter-rater reliability.
Reviewers are calibrated to apply
assessment criteria in the same
way, and faculty routinely check
for inter-rater reliability.
Assessment criteria, such as rubrics,
have been pilot-tested and refined
over time; they usually are shared with
students. Feedback from external
reviewers has led to refinements in the
assessment process, and the
department uses external
benchmarking data.
Reviewers are calibrated, and faculty
routinely find assessment data have
high inter-rater reliability.
Results for each outcome are
collected and may be discussed
by the faculty, but results have
not been used to improve the
program.
Results for each outcome are
collected, discussed by faculty,
analyzed, and used to improve
the program.
Students know little or
nothing about the purpose of
the capstone or outcomes to
be assessed. It is just
another course or
requirement.
Students have some knowledge
of the purpose and outcomes of
the capstone. Communication is
occasional, informal, left to
individual faculty or advisors.
Students have a good grasp of
purpose and outcomes of the
capstone and embrace it as a
learning opportunity. Information
is readily avail-able in advising
guides, etc.
Faculty routinely discuss results, plan
needed changes, secure necessary
resources, and implement changes.
They may collaborate with others,
such as librarians or Student Affairs
professionals, to improve results.
Follow-up studies confirm that
changes have improved learning.
Students are well-acquainted with
purpose and outcomes of the
capstone and embrace it. They may
participate in refining the experience,
outcomes, and rubrics. Information is
readily available.
173
How Visiting Team Members Can Use the Capstone Rubric
Conclusions should be based on discussion with relevant department members (e.g., chair, assessment coordinator, faculty). A variety of capstone
experiences can be used to collect assessment data, such as:
! courses, such as senior seminars, in which advanced students are required to consider the discipline broadly and integrate what they have learned
in the curriculum
! specialized, advanced courses
! advanced-level projects conducted under the guidance of a faculty member or committee, such as research projects, theses, or dissertations
! advanced-level internships or practica, e.g., at the end of an MBA program
Assessment data for a variety of outcomes can be collected in such courses, particularly outcomes related to integrating and applying the discipline,
information literacy, critical thinking, and research and communication skills.
The rubric has five major dimensions:
1. Relevant Outcomes and Evidence Identified. It is likely that not all program learning outcomes can be assessed within a single capstone course or
experience. Questions: Have faculty explicitly determined which program outcomes will be assessed in the capstone? Have they agreed on concrete
plans for collecting evidence relevant to each targeted outcome? Have they agreed on explicit criteria, such as rubrics, for assessing the evidence?
Have they identified examples of student performance for each outcome at varying performance levels (e.g., below expectations, meeting, exceeding
expectations for graduation)?
2. Valid Results. A valid assessment of a particular outcome leads to accurate conclusions concerning students’ achievement of that outcome.
Sometimes faculty collect evidence that does not have the potential to provide valid conclusions. For example, a multiple-choice test will not provide
evidence of students’ ability to deliver effective oral presentations. Assessment requires the collection of valid evidence and judgments about that
evidence that are based on well-established, agreed-upon criteria that specify how to identify low, medium, or high-quality work. Questions: Are
faculty collecting valid evidence for each targeted outcome? Are they using well-established, agreed-upon criteria, such as rubrics, for assessing the
evidence for each outcome? Have faculty pilot tested and refined their process based on experience and feedback from external reviewers? Are they
sharing the criteria with their students? Are they using benchmarking (comparison) data?
3. Reliable Results. Well-qualified judges should reach the same conclusions about individual student’s achievement of a learning outcome,
demonstrating inter-rater reliability. If two judges independently assess a set of materials, their ratings can be correlated. Sometimes a discrepancy
index is used. How often do the two raters give identical ratings, ratings one point apart, ratings two points apart, etc.? Data are reliable if the
correlation is high and/or if the discrepancies are small. Raters generally are calibrated (“normed”) to increase reliability. Calibration usually involves
a training session in which raters apply rubrics to pre-selected examples of student work that vary in quality, then reach consensus about the rating
each example should receive. The purpose is to ensure that all raters apply the criteria in the same way so that each student’s product receives the
same score, regardless of rater. Questions: Are reviewers calibrated? Are checks for inter-rater reliability made? Is there evidence of high inter-rater
reliability?
4. Results Are Used. Assessment is a process designed to monitor and improve learning, so assessment findings should have an impact. Faculty
should reflect on results for each outcome and decide if they are acceptable or disappointing. If results do not meet faculty standards, faculty should
determine which changes should be made, e.g., in pedagogy, curriculum, student support, or faculty support. Questions: Do faculty collect
assessment results, discuss them, and reach conclusions about student achievement? Do they develop explicit plans to improve student learning?
Do they implement those plans? Do they have a history of securing necessary resources to support this implementation? Do they collaborate with
other campus professionals to improve student learning? Do follow-up studies confirm that changes have improved learning?
The Student Experience. Students should understand the purposes different educational experiences serve in promoting their learning and
development and know how to take advantage of them; ideally they should also participate in shaping those experiences. Thus it is essential to
communicate to students consistently and include them meaningfully. Questions: Are purposes and outcomes communicated to students? Do they
understand how capstones support learning? Do they participate in reviews of the capstone experience, its outcomes, criteria, or related activities?
174
Automotive Technology Program
Expanded
Statement of
Institutional
Purpose
Mission Statement:
Your Community
College is an openadmission,
community-based,
comprehensive
college.
Goal Statement:
Serve persons of all
ages in preparing for
job entry and
careers in
automotive
technology.
Program Intended
Educational
Outcomes:
1. Graduates of
the Automotive
Technology
Program will be
successfully
employed in the
field.
(achievement, not
Student Learning
Outcome)
2. Graduates of the
Automotive
Technology
Program will be
technically
proficient.
3. Employers of
the Automotive
Technology
Program graduates
will rate them
competent based
on the education
received in the
program.
Means of Program Assessment and
Program Intended
Educational
Outcomes:
1. Graduates of
the Automotive
Technology
Program will be
successfully
employed in the
field.
(achievement, not
Student Learning
Outcome)
2. Graduates of the
Automotive
Technology
Program will be
technically
proficient.
3. Employers of
the Automotive
Technology
Program graduates
will rate them
competent based
on the education
received in the
program.
Summary of Data
Collected:
1a. 73% reported
employment.
1b. 81% reported
employment one
year after
graduation.
2a. 79% overall
success rate.
Electrical system
malfunction
undetected by 34%
of students.
2b. Pass rate on
National
Automotive Test
was 83%; however,
on “hydraulic
theory” subscale
students missed an
average of 34% of
questions.
3. 90% reported
willingness to
employ graduates,
but only 50% of
body shops.
Use of Results:
1a. Revised criteria
for success to 70%.
1b. No action
necessary at this time,
however, will continue
to monitor.
2a. Expanded
electrical troubleshooting component of
AT 202 to include
automotive electrical
systems.
2b. Modified means
of teaching hydraulic
theory during AT 102
(Basic Auto Systems).
3. Added body shop
representative to
Advisory Committee
and are reviewing
curriculum to
determine if separate
program is needed.
175
Common elements to a rubric
1. One or more traits or dimensions (criteria) that serve as the bases for judging student
responses.
2. Definitions or examples to clarify the meaning of each trait or dimension
3. A scale of values (a counting system) on which to rate each dimension
4. Standards of excellence for specified performance levels accompanied by examples of
each level.
Questions to discovering the dimensions or criteria.
1. By what are the qualities or features will I know whether students have produced an
excellent response to my assessment task?
2. What will they do that shows me the extent to which they have mastered the learning
outcomes?
3. What do I expect to see if the task is done excellently, acceptably, poorly?
4. Do I have examples of student work, from my class or another source, that exemplify the
criteria and I could use to judge the task?
5. What criteria for this or similar tasks exist in the assessment program for the program,
college, etc.?
6. What criteria or dimensions might I adapt from work done at the state or national level or
in professional associations?
Evaluating scoring criteria.
1. All important outcomes are addressed by the criteria.
2. Rating strategy matches decision purposea. Holistic for global, evaluative view
b. Analytic for diagnostic view
3. Rating scale has usable, easily interpreted scores
4. Criteria employ concrete references, clear language understandable to students
5. Criteria reflect current concepts of good work in the field
6. Criteria is reviewed for developmental, ethnic, gender bias
7. Criteria reflect teachable outcomes
8. Criteria are limited to a feasible number of dimensions
9. Criteria are generalizable to other similar tasks
Adapted from Herman, Joan, et. Al. A Practical Guide to Alternative Assessment
176
A rubric is a scoring tool that divides assignments into component parts or criteria used for evaluation and provides a
detailed description of what is acceptable vs. unacceptable qualities of performance. An analytic rubric makes clear
distinctions among the evaluation criteria while a holistic rubric merges the criteria together to stimulate a general
judgment about the quality of student work.
Questions To Ask When Constructing Rubrics
1. What criteria or essential elements must be present in the student’s work to ensure that it is high in quality?
2. How many levels of achievement (mastery) do I wish to illustrate for students?
3. For each criteria or essential element of quality, what is a clear description of performance at each
achievement level?
4. What are the consequences of performing at each level of quality?
5. What rating scheme will I use in the rubric?
6. When I use the rubric, what aspects work well and what aspects need improvement?
Additional Questions To Consider
1. What content must students master in order to complete the task well?
2. Are there any important aspects of the task that are specific to the context in which the assessment is set?
3. In the task, is the process of achieving the outcome as important as the outcome itself?
Source: Huba, Mary E. and Freed, Jann E. Learner-Centered Assessment on College Campuses. Allyn & Bacon,
Boston, MA, 2000. ISBN 0-205-28738-7.
Additional good references:
Moskal, Barbara M. (2000). Scoring Rubrics: What, When and How? Practical Assessment, Research & Evaluation,
7(3). Available online: http://ericae.net/pare/getvn.asp?v=7&n=3.
Definitions
and
construction
ideas
for
http://ericae.net/faqs/rubrics/scoring_rubrics.htm
scoring
rubrics
are
found
at
this
URL
Stevens, Dannelle and Levi, Antonio. Introduction to Rubrics. Stylus Publishing, Herdon, VA 2004. ISBN 1-57922114-9. (forthcoming in September, 2004)
The assessment leader at Winona State University (MN) has an excellent set of rubrics at this URL
http://www.winona.edu/AIR/ Once there click on the sample rubrics link in the left frame.
The Center for Learning and Teaching Excellence at Arizona State University has a bank of rubrics at this URL
http://clte.asu.edu/resources/instructors/ Select the Assessment Web link in the center of the page.
CSU System Office has an excellent rubrics at this URL http://www.calstate.edu/itl/sloa/index.shtml
Example in action:
Johnson County Community College has been making extensive use of rubrics for general education assessment. An
overview is provided at http://www.jccc.net/home/depts.php/6111/site/assmnt/cogout
Raymond Walters College has been making extensive use of rubrics and primary trait assessment, for individual
course assignments. See examples link at http://www.rwc.uc.edu/phillips/index_assess.html
Prepared by Fred Trapp for the Research and Planning (RP) Group Student Learning Outcomes and Assessment Workshops
177
ECCS Department – Web Site Rubric
Course: _______________________________________
Date: _____________________
Student(s): _________________________________________________________________________
URL: ______________________________________________________________________________
Informational
Content
Score: _____
Layout and
Design
Score: _____
Technical
Elements
3 - Expert
2 - Practitioner
1 - Apprentice
0 - Novice
! Presented information is
well-written, concise,
accurate and complete.
! Answers to all expected
questions can be found on
the site.
! Contact information of all
students is present and
will be still valid in the
future.
! Presented information is
accurate and complete.
! Answers to most expected
questions can be found on
the site.
! Contact information of all
students is present.
! Presented information is
incomplete or inaccurate
in places.
! Answers to some
expected questions can be
found on the site.
! Contact information of
some students is not
present.
! Presented information is
grossly incomplete and/or
inaccurate.
! Answers to few expected
questions can be found on
the site.
! Contact information is not
available.
! Pages are attractive and
consistent in style
throughout the site.
! Site is well organized and
is easily navigated from
any page.
! Graphic elements are
appropriate, of high
quality, and are creatively
used to enhance content.
! Pages are attractive, but
not consistent in style
throughout the site.
! Site is well organized.
! Graphic elements are
appropriate and are of
acceptable quality to
enhance content.
! Pages are not attractive,
but do not distract.
! Site is not well organized.
! Graphic elements are not
always appropriate or are
of inferior quality.
! Pages are unattractive.
! Site is not organized or
consists of a single page.
! Graphic elements are not
appropriate or not used,
or are of such poor
quality that they detract
from content.
! All links work.
! Graphics used on page
download quickly;
thumbnails are used as
appropriate.
! All pages have an
appropriate title.
! One broken link is
present.
! Graphics used on page
download in a short
amount of time.
! Most pages have an
appropriate title.
! Two broken links are
present.
! Graphics used on page
causes user to wait for
page to download images
that contain acceptable
content.
! Some pages have an
appropriate title.
! Three or more broken
links are present.
! Graphics used on page
causes user to wait for
page to download images
that do not contain
acceptable content.
! Few if any pages have an
appropriate title.
! Font type used is visually
appealing and is of
appropriate size and style
to make text easy to read.
! Color selection for
foreground and
background are
consistent, visually
pleasing and usable.
! Font type used is of
appropriate size and style
for readable text.
! Color selection for
foreground and
background are mostly
consistent and usable.
! Font type used is not
always of appropriate size
or style, causing some
difficulties in reading.
! Color selection for
foreground and
background cause
difficulties for those with
color-impaired vision
! Font type used is
distracting and/or
unattractive; size and/or
style used makes text
hard to read.
! Color selection for
foreground and
background lack
sufficient contrast, are
clashing, of contain too
many colors to the point
that the text is hard for
anyone to read.
Score: _____
Color and
Typography
Score: _____
Comments:
Evaluator Signature: __________________________________________________________
178
Accreditation Board for Engineering & Technology, Inc. (ABET)
Outcome 1: An ability to apply math & science in engineering
Level 5 performance characterized by:
* Combines mathematical and/or scientific principles to formulate models of chemical,
physical and/or biological processes and systems relevant to civil engineering
* Applies concepts of integral and differential calculus and/or linear algebra to solve civil
engineering problems
* Shows appropriate engineering interpretation of mathematical and scientific terms
* Translates academic theory into engineering applications and accepts limitations of
mathematical models of physical reality
* Executes calculations correctly
o By hand
o Using mathematical software
* Correctly analyzes data sets using statistical concepts
Level 3 performance characterized by:
* Chooses a mathematical model or scientific principle that applies to an engineering problem,
but has trouble in model development
* Shows nearly complete understanding of applications of calculus and/or linear algebra in
problem-solving
* Most mathematical terms are interpreted correctly
* Some gaps in understanding the application of theory to the problem and expects theory to
predict reality
* Minor errors in calculations
o By hand
o Applying math software
* Minor errors in statistical analysis of data
Level 1 performance characterized by:
* Does not understand the connection between mathematical models and chemical, physical,
and/or biological processes and systems in civil engineering
* Does not understand the application of calculus and linear algebra in solving civil
engineering problems
* Mathematical terms are interpreted incorrectly or not at all
* Does not appear to grasp the connection between theory and the problem
* Calculations not performed or performed incorrectly
o By hand
o Does not know how to use math software
* No application of statistics to analysis of data
Department of Civil and Environmental Engineering
University of Delaware | Newark, DE 19716-3120
phone: 302-831-2442 | e-mail CEE | fax: 302-831-3640
179
PORTFOLIOS
Rubric for Assessing the Use of Portfolios for Assessing Program Learning Outcomes
Criterion
Clarification of
Students’
Task
Valid Results
Reliable
Results
Results Are
Used
If e-Portfolios
Are Used
Initial
Instructions to students for
portfolio development provide
insufficient detail for them to
know what faculty expect.
Instructions may not identify
outcomes to be addressed in
the portfolio.
It is not clear that valid
evidence for each relevant
outcome is collected and/or
individual reviewers use
idiosyncratic criteria to assess
student work.
Emerging
Students receive some written
instructions for their portfolios,
but they still have problems
determining what is required of
them and/or why they are
compiling a portfolio.
Appropriate evidence is
collected for each outcome,
and faculty have discussed
relevant criteria for assessing
each outcome.
Developed
Students receive written
instructions that describe faculty
expectations in detail and include
the purpose of the portfolio, types
of evidence to include, role of the
reflective essay (if required), and
format of the finished product.
Appropriate evidence is collected
for each outcome; faculty use
explicit criteria, such as agreedupon rubrics, to assess student
attainment of each outcome.
Rubrics are usually shared with
students.
Those who review student
work are not calibrated to
apply assessment criteria in
the same way, and there are
no checks for inter-rater
reliability.
Results for each outcome are
collected, but they are not
discussed among the faculty.
Reviewers are calibrated to
apply assessment criteria in
the same way or faculty
routinely check for inter-rater
reliability.
Reviewers are calibrated to apply
assessment criteria in the same
way, and faculty routinely check for
inter-rater reliability.
Results for each outcome are
collected and discussed by the
faculty, but results have not
been used to improve the
program.
Results for each outcome are
collected, discussed by faculty, and
used to improve the program.
There is no technical support
for students or faculty to learn
the software or to deal with
problems.
There is informal or minimal
formal support for students
and faculty.
Formal technical support is readily
available and proactively assists in
learning the software and solving
problems.
Highly Developed
Students in the program understand the
portfolio requirement and the rationale for it,
and they view the portfolio as helping them
develop self-assessment skills. Faculty may
monitor the developing portfolio to provide
formative feedback and/or advise individual
students.
Assessment criteria, e.g., in the form of
rubrics, have been pilot-tested and refined
over time; they are shared with students,
and student may have helped develop them.
Feedback from external reviewers has led to
refinements in the assessment process. The
department also uses external
benchmarking data.
Reviewers are calibrated; faculty routinely
find that assessment data have high interrater reliability.
Faculty routinely discuss results, plan
needed changes, secure necessary
resources, and implement changes. They
may collaborate with others, such as
librarians or Student Affairs professionals, to
improve student learning. Students may also
participate in discussions and/or receive
feedback, either individual or in the
aggregate. Follow-up studies confirm that
changes have improved learning.
Support is readily available, proactive, and
effective. Tech support personnel may also
participate in refining the overall portfolio
process.
180
How Visiting Team Members Can Use the Portfolio Rubric
Portfolios can serve many purposes besides assessment; in fact, these other purposes are actually much more common. Portfolios may be compiled so
students can share their work with family and friends. They may be designed to build students’ confidence by showing development over time or by
displaying best work. They may be used for advising and career counseling, or so students can show their work during a job interview. The first thing a
team needs to do is determine that the portfolios are used for assessment, and not for another purpose.
Conclusions about the quality of the assessment process should be based on discussion with relevant department members (e.g., chair, assessment
coordinator, faculty, students) and a review of the program’s written portfolio assignment. Two common types of portfolios are:
! Showcase portfolios—collections of each student’s best work
! Developmental portfolios—collections of work from early, middle, and late stages in the student’s academic career that demonstrate growth
Faculty generally require students to include a reflective essay that describes how the evidence in the portfolio demonstrates their achievement of
program learning outcomes. Sometimes faculty monitor developing portfolios to provide formative feedback and/or advising to students, and sometimes
they collect portfolios only as students near graduation. Portfolio assignments should clarify the purpose of the portfolio, what kinds of evidence should
be included, and the format (e.g., paper vs. e-portfolios); and students should view the portfolio as contributing to their personal development.
The rubric has five major dimensions and a fifth dimension limited to e-portfolios:
1. Clarification of Students’ Task. Most students have never created a portfolio, and they need explicit guidance. Questions. Does the portfolio
assignment provide sufficient detail so students understand the purpose, the types of evidence to include, the learning outcomes to address, the role
of the reflective essay (if any), and the required format? Do students view the portfolio as contributing to their ability to self-assess? Do faculty use
the developing portfolios to assist individual students?
2. Valid Results. Sometimes portfolios lack valid evidence for assessing particular outcomes. For example, portfolios may not allow faculty to assess
how well students can deliver oral presentations. Judgments about that evidence need to be based on well-established, agreed-upon criteria that
specify (usually in rubrics) how to identify work that meets or exceeds expectations. Questions: Do the portfolios systematically include valid
evidence for each targeted outcome? Are faculty using well-established, agreed-upon criteria, such as rubrics, to assess the evidence for each
outcome? Have faculty pilot tested and refined their process? Are criteria shared with students? Are they collaborating with colleagues at other
institutions to secure benchmarking (comparison) data?
3. Reliable Results. Well-qualified judges should reach the same conclusions about a student’s achievement of a learning outcome, demonstrating
inter-rater reliability. If two judges independently assess a set of materials, their ratings can be correlated. Sometimes a discrepancy index is used.
How often do the two raters give identical ratings, ratings one point apart, ratings two points apart, etc.? Data are reliable if the correlation is high
and/or if discrepancies are small. Raters generally are calibrated (“normed”) to increase reliability. Calibration usually involves a training session in
which raters apply rubrics to pre-selected examples of student work that vary in quality, then reach consensus about the rating each example should
receive. The purpose is to ensure that all raters apply the criteria in the same way so that each student’s product would receive the same score,
regardless of rater. Questions: Are reviewers calibrated? Are checks for inter-rater reliability made? Is there evidence of high inter-rater reliability?
4. Results Are Used. Assessment is a process designed to monitor and improve learning, so assessment findings should have an impact. Faculty
should reflect on results for each outcome and decide if they are acceptable or disappointing. If results do not meet their standards, faculty should
determine what changes should be made, e.g., in pedagogy, curriculum, student support, or faculty support. Questions: Do faculty collect
assessment results, discuss them, and reach conclusions about student achievement? Do they develop explicit plans to improve student learning?
Do they implement those plans? Do they have a history of securing necessary resources to support this implementation? Do they collaborate with
other campus professionals to improve student learning? Do follow-up studies confirm that changes have improved learning?
5. If e-Portfolios Are Used. Faculty and students alike require support, especially when a new software program is introduced. Lack of support can
lead to frustration and failure of the process. Support personnel may also have useful insights into how the portfolio assessment process can be
refined. Questions: What is the quality and extent of technical support? Of inclusion in review and refinement of the portfolio process? What is the
overall level of faculty and student satisfaction with the technology and support services?
181
Dale McIver’s Problem
Dale McIver teaches in the office automation program at Watson Technical, an area vocational
technical school in Dade County, Florida. She teaches a five-course sequence leading to a
certificate in office computer applications software. However, few of her students complete the
full sequence. Instead, her classes are primarily composed of students wanting to gain some
initial familiarity with some computer applications and very basic information system concepts
or wanting to upgrade their skills in particular ways.
Dale is frustrated with her current grading system, which is based on unit tests from the class
workbooks and textbook. The test scores do not give her or the students enough information
about the students’ abilities to respond to realistic office demands that involve several software
programs and problem solving.
She is looking for an assessment system that will engage her students more, help them
understand their own strengths and weaknesses, and provide her with information to improve her
instruction. She believes there is too much emphasis on rote learning of commands and
functions. She wants her students to be better problem solvers when it comes to using computers
in the office environment.
What assessment plan would you recommend to Dale?
182
Accrediting Commission for Community and Junior Colleges
Western Association of Schools and Colleges
Rubric for Evaluating Institutional Effectiveness – Part III: Student Learning Outcomes
(See attached instructions on how to use this rubric.)
Levels of
Implementation
Characteristics of Institutional Effectiveness in
Student Learning Outcomes
(Sample institutional behaviors)
Awareness
Development
• There is preliminary, investigative dialogue about student learning outcomes.
• There is recognition of existing practices such as course objectives and how they relate to
student learning outcomes.
• There is exploration of models, definitions, and issues taking place by a few people.
• Pilot projects and efforts may be in progress.
• The college has discussed whether to define student learning outcomes at the level of
some courses or programs or degrees; where to begin.
• College has established an institutional framework for definition of student learning
outcomes (where to start), how to extend, and timeline.
• College has established authentic assessment strategies for assessing student
learning outcomes as appropriate to intended course, program, and degree learning
outcomes.
• Existing organizational structures (e.g. Senate, Curriculum Committee) are
supporting strategies for student learning outcomes definition and assessment.
• Leadership groups (e.g. Academic Senate and administration), have accepted
responsibility for student learning outcomes implementation.
• Appropriate resources are being allocated to support student learning outcomes and
assessment.
• Faculty and staff are fully engaged in student learning outcomes development.
Proficiency
• Student learning outcomes and authentic assessment are in place for courses, programs
and degrees.
• Results of assessment are being used for improvement and further alignment of
institution-wide practices.
• There is widespread institutional dialogue about the results.
• Decision-making includes dialogue on the results of assessment and is purposefully
directed toward improving student learning.
• Appropriate resources continue to be allocated and fine-tuned.
• Comprehensive assessment reports exist and are completed on a regular basis.
• Course student learning outcomes are aligned with degree student learning outcomes.
• Students demonstrate awareness of goals and purposes of courses and programs in
which they are enrolled.
Sustainable
Continuous
Quality
Improvement
• Student learning outcomes and assessment are ongoing, systematic and used for
continuous quality improvement.
• Dialogue about student learning is ongoing, pervasive and robust.
• Evaluation and fine-tuning of organizational structures to support student learning is
ongoing.
• Student learning improvement is a visible priority in all practices and structures across the
college.
• Learning outcomes are specifically linked to program reviews.
JP;DB: cg 8/2007
183
Cambridge West Partnership, LLC
Vocational Information Center
http://www.khake.com/page50.html
Competency Model Clearinghouse
Authentic Assessment Toolbox
Career and Technical Education National Research Center
http://www.nccte.org/repository/
Repository of skills standards
U.S. Department of Labor
http://www.careeronestop.org/ and
http://www.careeronestop.org/CompetencyModel/default.aspx and
http://www.careeronestop.org/CompetencyModel/learnCM.aspx
General
• Learn about competency models
• Find competency model resources
• Build a competency model
Exploring industries
• Advanced manufacturing
• Financial services
• Retail industry
• Hospitality/hotel & lodging
• Energy/generation, transmission & distribution
184
Plenary
The Learning-centered
Institution: Curriculum,
Pedagogy and Assessment
for Student Success
Amy Driscoll
185
The Learning-centered Institution:
Curriculum, Pedagogy and
Assessment for Student Success
Amy Driscoll
WASC EDUCATIONAL SEMINAR
September 25 2009
Outcomes for Plenary
„
„
Articulate and discuss the rationale for and
impact of learning-centered curriculum,
pedagogy, and assessment on student
success with colleagues, administration,
students, and others
Design assessment that promotes learningcentered curriculum and pedagogy and
ultimately student success
2
Questions for your campus
team:
„
„
What are four indicators (or more) of
student success at your institution?
Is there an indicator that is distinctive
to your institution? That we would not
use to describe student success at other
institutions?
3
186
National Indicators of Student Success
„
„
„
„
„
Academic Achievement
Citizenship
Retention and Graduation
Collaboration
Well-being and Health (American Council on
Education, AACU’s LEAP, NSSE)
4
Success Indicators for
Graduate/Doctoral Programs
„
„
„
„
„
National rankings (U S News, NRC periodic
review)
Career trajectories of alumni
Student report/self evaluations (Survey of
Doctoral Education, Nat’l Assoc of Graduate
and Professional Student survey)
Graduation/attrition rates - “time to degree”
External Review Committees
5
Key Ideas
„
„
„
„
„
Defining Student Success
Hallmarks of a Learning-centered Education
A Learning-centered Assessment Process
Designing Learning-centered Courses,
Programs, and Pedagogy
Cycle of Assessment for Improving Learning
6
187
Learning)-Centered = Student
Success
WHY?
Reasons for Student Success and
Learning-centered Focus:
„
„
Institutions that focus on student success
(learning-centered) are better positioned to
help their students attain their educational
objectives (or goals and outcomes)
Assessment and accountability efforts need to
be focused on what matters to student
success (learning-centered)(Kuh, 2006)
8
Hallmarks of Learningcentered Education
„
Curriculum
„
Pedagogy
„
Assessment
9
188
Learning-centered Curriculum
„
„
„
„
„
Synthesizes content
Builds on previous learning
Integrates education and experience
Communicates values for and connection to
student lives
Attends to learning needs
= RELEVANCE, LONG-TERM MEMORY,
MOTIVATION, RESPONSIVE
10
How do we get to know our
students?
Their previous learning, their
experiences, their needs, their
lives, their assets, their
challenges, and so on?
Learning-centered Pedagogy
„
„
„
„
„
„
Students have clear expectations
Students are actively involved
Students apply knowledge to important issues and
problems
Student find relevance and value
Students experience support and feedback for
learning
Students are able to practice and take risks
= CONFIDENCE, ENGAGEMENT, PERSONAL LEARNING,
SECURE ENVIRONMENT
12
189
Learning-centered Assessment
„
„
„
„
„
Assessment is ongoing not episodic.
Students understand and value the criteria,
standards, and methods by which they are
assessed.
The purpose of assessment is to improve
student learning.
Faculty consider student perspectives.
The assessment activity makes sense to
students.
= OWNERSHIP, SECURITY, CLARITY
13
Processes for Developing
Learning-centered Education
„
„
„
„
„
„
Reflect on purpose of assessment and define
Align pedagogy/assessment with mission and
values of institution, department, faculty
Articulate goals, outcomes, evidence, criteria,
and standards
Design curriculum and pedagogy to achieve
learning outcomes
Conduct collaborative review of student
evidence
Use review to improve learning
14
Developing Learning-centered
Assessment
„
„
„
„
„
Describe educational culture - fit?
Engage in inquiry process - focus on
student success
Study list of possible purpose(s)
Consider how assessment can support
your intentions for student success
Define assessment with the inclusion of
student success
15
190
An Inquiry Process for
Assessment:
„
What questions are you trying to
answer with your assessment?
„
What questions do you care about?
„
What answers do you already have?
16
Possible Purposes for Assessment:
which purposes support student success?
„
„
„
„
„
„
„
„
„
Provide feedback to students
Classify or grade achievement
Enable students to correct errors and improve
learning
Motivate students focusing on their sense of
achievement
Consolidate and summarize student learning
Estimate students’ potential to progress to other
courses
Help students apply learning to practical contexts
Give us feedback on how effective we are at
promoting learning
Provide data for internal and external accountability17
Possibilities:Purpose/Definition
„
„
“The purpose of assessment is to
improve learning” (Angelo, 2000)
“Assessment is a dynamic pedagogy
that extends, expands, enhances, and
strengthens learning” (Driscoll, 2001)
18
191
Thinking about Assessment
„
„
„
„
Does assessment flow from the institution’s
mission and reflect the educational values?
from the department’s mission?
Does assessment address questions that
people really care about?
Does assessment help faculty fulfill their
responsibilities to students, to the public?
Does assessment promote student success?
19
Fundamental questions for
graduate/doctoral education:
„
„
„
What is the purpose of the doctoral program?
What is the rationale or educational purpose
of each element of the doctoral program?
How do you know what is working? What
should be changed or eliminated? What
should be affirmed or retained?
(Carnegie Institute on the Doctorate, 2007)
20
Starting with Mission…
Missions contribute meaning to
our definition of student success
with the uniqueness of the
institution.
192
Aligning Mission with
Educational Goals for
Assessment
„
Our central mission is to develop lifelong learning skills, impart society’s
cultural heritage, and educate and
prepare for both the professions and
advanced study.
22
Aligning Values With
Educational Goals
„
ESU has a commitment to academic
and personal integrity.
GOALS: Academic Integrity
Personal Integrity
23
School/Departmental Missions
„
„
“respond effectively to issues of diversity,
ambiguity, and conflict as natural parts of
American politics”
(Division of Political Science)
“work toward influencing health behaviors
through modification of lifestyles and changes
to the environment” (School of Community
Health)
24
193
Important Foundational
Questions:
„
„
What goals or indicators of student
success emerge or pop out of you
mission, values, vision, etc.
What promises do your brochures or
websites make about student success?
25
Assessment Protocols
for Learning-centered Assessment
OUTCOMES
GOAL
Evidence
Criteria
Standards:
a) Exemplary Achievement
b) Satisfactory Achievement
c) Unsatisfactory Achievement
26
Goals
„
Broad descriptions
„
Categories of learning outcomes
„
End toward which efforts are
directed
27
194
Examples of Goals
„
„
„
„
Critical Thinking
Citizenship in a Democracy (Grad.
School of Education)
Team work and Collaboration (School of
Community Health
Ethics
28
Goals for Graduate Programs
„
„
„
„
“State of the Art” Disciplinary
Knowledge
Leadership
Scholarly Communication
Assessment and Evaluation
29
Impact of Goals on Student
Learning & Success
„
„
„
Focuses student learning efforts for increased
success
Translates mission and values to help make
sense of learning
Provides rationale for and makes meaning of
curriculum and pedagogy to motivate for
success
30
195
Student Learning Outcomes
„
Refer to Results in Terms of Specific Student
Learning, Development, and Performance (Braskamp
and Braskamp, 1997)
„
„
Answer the Question – “What Do We Expect of
Our Students?” (CSU Report 1989)
Describe Actual Skills, Understandings, Behaviors,
Attitudes, Values Expected of Students
31
Examples of Outcomes
Math: Use arithmetical, algebraic, geometric and statistical
methods to solve problems.
Ethics: Identify and analyze real world ethical problems or
dilemmas and identify those affected by the dilemma.
Culture and Equity: Analyze and describe the concepts of
power relations, equity, and social justice and find examples
of each concept in the U.S. society and other societies.
Team work: Listens to, acknowledges, and builds on the ideas of
others.
32
Examples of Outcomes for
Graduate Programs
„
„
„
„
Generate new knowledge through
research/scholarship and transmit that
knowledge to others
Describes and applies techniques,
technologies, and strategies that promote
required or desired change
Works effectively with individuals from
diverse cultural backgrounds
Articulates and follows ethical standards
consistent with professional commitment
33
196
Impact of Outcomes on
Student Learning & Success
„
„
„
Directs student learning efforts for greater
success
Motivates student learning efforts for greater
success
Promotes deep learning due to understanding
of expectations which leads to greater
success
34
Evidence
„
„
„
Student Work that Demonstrates
Achievement of Outcomes
(Assignments, Projects, Presentations,
Papers, Responses to Questions, Etc.)
Designed for appropriate level of
learning expectations (outcomes)
Opportunity for Different Ways of
Demonstrating Learning
35
Examples of Evidence
Teamwork
Role play or case study
Project or problem solving assignment
Math
Mathematical and statistical projects and papers
Ethics
A written account
A multi-media presentation or display board
An audio tape
36
197
Impact of Evidence on
Student Learning & Success
„
„
„
Limit or expand the ways they demonstrate
learning - success for all students?
Enrich and enhance learning and success
Provide opportunity to integrate experience
with learning for enhanced success
37
Criteria
„
„
Qualities Desired in Student Work (Evidence)
Represent Powerful Professional
Judgment of Faculty
„
Guide Student Learning Efforts
„
Promote Lifelong Learning
„
Support Faculty in Making Objective
Evaluations
38
Examples of Criteria
Math
„
„
„
Accuracy
Complexity
Clarity and Coherence
Ethics
„
„
Complexity (broad, multifaceted, interconnected)
Conscious Awareness
Culture and Equity
„
„
Range of Cultures
Reflectivity and Integration
Teamwork
ƒ Respect
ƒ Flexibility
39
198
Criteria for Leadership
(Graduate program goal)
„
„
„
„
Balance
Change
Self-appraisal
Strategic engagement (planning,
evaluation, implementation, and
assessment)
(Maki & Borkowski, 2006)
40
Impact of Criteria on Student
Learning & Success
„
„
„
„
Promotes confidence in their learning
efforts = success
Promotes qualities of life-long learning
= life success
Promotes habits of self assessment =
success
Promotes student’s sense of fairness of
evaluation = increased effort = success
41
Important Question
„
„
What criteria would be distinctive of
your institution? To cut across most of
student work? Or be distinctive for
your graduates?
IDEAS: scholarship, multiple
perspectives, reflection, commitment
42
199
Standards/Rubrics
„
Describe Different Levels of Criteria
„
Describe Specific Indications of Criteria
„
Promote Understanding of Criteria
„
Support Faculty in Making Objective
Evaluations
43
Examples of Standards/Rubrics
Math (Accuracy)
„
„
Satisfactory: Contains few errors and those errors do not significantly
undermine the quality of the work.
Considers and uses data, models, tools or processes that
reasonably and effectively address issues or problems.
Unsatisfactory: One or more errors that significantly undermine the quality
of the work.
Uses data, models, tools or processes in inappropriate or
ineffective ways.
Ethics (Complexity)
„
Standard for Excellent: Consistently views sophisticated and significant
„
Standard for Satisfactory: Usually views sophisticated and significant
„
dilemmas and issues with a broad focus and from multiple perspectives.
dilemmas and issues with a broad focus, but may sometimes use a more narrow focus
and may use fewer perspectives.
Standard for Unsatisfactory: Mainly views issues and dilemmas in simple
terms and usually does so with a limited focus and minimal perspectives.
44
Rubric for Leadership: Balance
in Facilitating Group Processes
„
Exemplary: Leads and empowers group members
towards consensual solutions that maximize
members’ commitment to and satisfaction with
agreed-upon responses.
ƒ Proficient:
Is hesitant but able to lead and
empower group members in…
ƒ Marginal:
Requires significant assistance in leading
and empowering group members…after an extended
time period.
ƒ Unacceptable: Is not able to lead or empower a
group…
45
200
Impact of Standards on
Student Learning & Success
„
„
„
Guides level of student investment =
focused success
Provides insight into the assessment
process – understanding leads to
success
Promotes confidence in how their work
will be evaluated – security in learning
46
Assessing Student Learning:
Course, Program and
Institutional Levels
7.
Revise outcomes
and criteria,
Improve pedagogy
and curriculum for
learner success
6.
Review and
analyze student
evidence
5. Collect evidence of
student achievement
4.
Make outcomes, evidence, criteria, and
standards “public and visible” (syllabi,
programs, brochures)
1.
Preparation: Determine
purpose(s) and
definition of
assessment; Examine
mission and values
2.
Design
assessment:
Articulate goals,
Develop clear
outcomes,
evidence, criteria,
and standards
3. Alignment of
curriculum and
pedagogy with learning
outcomes
47
Step 3: Aligning Curriculum
and Pedagogy with Learning
Outcomes
„
„
„
Outcomes and Criteria as Planning Focus
Faculty Alignment Grids
Learner Grids
48
201
Student Learning Outcome
Example: Implications?
Students describe and assume personal
responsibility in collaborative
endeavors, and respect and support the
contributions of others.
Students analyze ethical issues from a
variety of cultural perspectives.
49
Step 4 Communicating Learning
Outcomes ---Leads to Success
„
Public and Visible
ƒ Relevant and Meaningful
ƒ Motivating and Supportive of Learning
50
Step 5: Collect Evidence of
Student Achievement
Collect representative samples from each
course
Organize collaborative faculty teams for
review
Use outcomes, criteria, and directions for
assignments or assessments
51
202
Step 5: Review and Analyze
Evidence
„
„
„
Read holistically to determine whether
outcomes are achieved (reliability).
Several readings to identify examples of
criteria (validity).
Final reading for insights about
pedagogy, class structure and
environment, and learning supports.
52
Changes in Teaching, Assessment and
Reflection on Pedagogy to Promote
Student Success
„
Scaffolding
„
Iterative assessment
„
Assessment used as a teaching tool
53
Processes for Developing Learningcentered Education and Student Success
„
„
„
„
„
„
„
„
Develop purpose and definition
Review/analyze mission and values
Articulate goals, outcomes, evidence, criteria,
and standards
Designing curriculum and pedagogy
Make assessment public and visible
Systematically collect student evidence
Conduct collaborative review of student
evidence
Use review to improve learning
54
203
SUMMARY
Outcomes-based assessment will
intentionally focus your teaching,
curriculum, and assessment on student
learning in ways that are authentic,
helpful to you and your students,
provide accountability for you and
others, and actually increase student
success.
55
INSPIRATIONS:
Let’s begin to think of students as
scholars and teach accordingly.
Let’s model how to learn from mistakes.
Let’s work to elevate learning to the level
of identity rather than level of
accomplishment.
56
204
RESOURCES
205
BIBLIOGRAPHY
206
Literature on Learning and Assessment
Allen, M. J. (2004). Assessing Academic Programs in Higher Education. Bolton, MA:
Anker.
Allen, M. J. (2006). Assessing General Education Programs. Bolton, MA: Anker.
American Association for Higher Education. (1998). Powerful Partnerships: A Shared
Responsibility for Learning (www.aahe.org/assessment/joint.htm). Washington
DC: AAHE, ACPA, & NASPA.
Anderson, L. W. & Associates. (2000) A Taxonomy for Learning, Teaching, and
Assessing. A Revision of Bloom’s Taxonomy of Educational Objectives. Boston:
Allyn & Bacon
Angelo, T. A., & Cross, K. P. (1993). Classroom Assessment Techniques: A Handbook
for College Teachers. San Francisco: Jossey-Bass Publishers.
Astin, A. W. (1991). Assessment for Excellence: The Philosophy and Practice of
Assessment and Evaluation in Higher Education. New York: Macmillan.
Astin, A. W. (1993). What Matters in College? Four Critical Years Revisited. San
Francisco: Jossey-Bass.
Banta, T. W. & Associates. (2002). Building a Scholarship of Assessment. San
Francisco: Jossey-Bass.
Bourne, J. & Moore, J., eds. (2004). Elements of Quality Online Education: Into the
Mainstream. Needham: Sloan Consortium.
Brown, J. S. & Duguid, P. (2000). The Social Life of Information. Boston: Harvard
Business School Press.
Burke, J. C. & Associates. (2005). Achieving Accountability in Higher Education:
Balancing Public, Academic, and Market Demands. San Francisco: Jossey-Bass.
Cohen, A. M. (1998). The Shaping of American Higher Education: Emergence and
Growth of the Contemporary System. San Francisco: Jossey-Bass Publishers.
Cross, K. P., & Steadman, M. H. (1996). Classroom Research: Implementing the
Scholarship of Teaching. San Francisco: Jossey-Bass.
DeZure, D. (Ed.). (2000). Learning from Change: Landmarks in Teaching and Learning
in Higher Education from Change Magazine 1969-1999. Sterling, VA: Stylus.
DiStefano, A., Rudestam, K. E., & Silverman, R. J., eds. (2004). Encyclopedia of
Distributed Learning. Thousand Oaks, CA: Sage.
Doherty, A., Riordan, T., & Roth, J. (Eds.). (2002). Student Learning: A Central Focus
for Institutions of Higher Education. Milwaukee, WI: Alverno College Institute.
Updated: 1/5/2009
207
Driscoll, A. & Cordero de Noriega, D. (2006). Taking Ownership of Accreditation:
Assessment Processes that Promote Institutional Improvement and Faculty
Engagement. Sterling, VA: Stylus.
Driscoll, A. & Wood, S. (2007). Developing Outcomes-Based Assessment for LearnerCentered Education: A Faculty Introduction. Sterling, VA: Stylus.
Erwin, T.D. (1991). Assessing Student Learning and Development. San Francisco:
Jossey-Bass.
Fried, J. & Evans, N. (2004). Learning Reconsidered: A Campus-Wide Focus on the
Student Experience. Washington, DC: National Association of Student Personnel
Administrators and The American College Personnel Association.
Gaff, J. G., & Ratcliff, J. L. (1997). Handbook of the Undergraduate Curriculum: A
Comprehensive Guide to Purposes, Structures, Practices, and Change. San
Francisco: Jossey-Bass.
Gardiner, L. (1994) Redesigning Higher Education: Producing Dramatic Gains in Student
Learning. East Lansing, MI: ASHE Higher Education Report Series
Hakel, M. & Halpern, D. F., eds. (2002). Applying the Science of Learning to University
Teaching and Beyond: New Directions for Teaching and Learning. San
Francisco: Jossey-Bass.
Halpern, D. F. (2002). Thought and Knowledge: An Introduction to Critical Thinking.
Mahwah, NJ: Lawrence Erlbaum.
Huber, M. T., & Morreale, S. P., eds. (2002). Disciplinary Styles in the Scholarship of
Teaching and Learning: Exploring Common Ground. Washington, DC: American
Association for Higher Education.
Huba, M. E. & Freed, J. E. (2000). Learner-Centered Assessment on College
Campuses: Shifting the Focus from Teaching to Learning. Boston: Allyn &
Bacon.
Jarvis, P., ed. (2001). The Age of Learning: Education and the Knowledge Society.
London: Kogan Page.
Leskes, A. & Wright, B. (2005). The Art & Science of Assessing General Education
Outcomes. Washington, DC: Association of American College and Universities.
Lewis, R. G. and Smith, D. H. (1994). Total Quality in Higher Education. Delray Beach:
St. Lucie Press.
Maki, P. (2006). The Assessment of Doctoral Education. Sterling, VA: Stylus.
Maki, P. (2004) Assessing for Learning: Building a Sustainable Commitment Across the
Institution. Sterling, VA: Stylus
Massy, W. F. (2003). Honoring the Trust: Quality and Cost Containment in Higher
Education. Bolton, MA: Anker.
Updated: 1/5/2009
208
Mentkowski, M. & Associates. (2000). Learning That Lasts: Integrating Learning,
Development, and Performance in College and Beyond. San Francisco: JosseyBass.
Mestre, J. (2005). Transfer of Learning: Research and Perspectives.
Miller, R. (2007). Assessment in Cycles of Improvement. Faculty Designs for Essential
Learning Outcomes. Washington, DC: Association of American Colleges and
Universities.
Miller, R. & Leskes, A. (2005). Levels of Assessment. From the Student to the Institution.
Washington, DC: Association of American Colleges and Universities.
Musil, C. M. (2006) Assessing Global Learning. Matching Good Intentions with Good
Practice. Washington, DC: Association of American Colleges and Universities.
Nichols, James O. A Road Map For Improvement of Student Learning and Support
Services Through Assessment, Agathon Press, New York, 2005.
National Research Council. (2003). Evaluating And Improving Undergraduate Teaching:
In Science, Technology, Engineering, and Mathematics. Washington, DC:
National Academy Press.
National Research Council. (2000). How People Learn: Brain, Mind, Experience, and
School. Washington, DC: National Academy Press.
National Research Council. (2001). Knowing What Students Know: The Science and
Design of Educational Assessment. Washington, DC: National Academy Press.
O’Banion, T. (1997). A Learning College for the 21st Century. Phoenix, AZ: Oryx &
American Council on Education.
Palmer, P. J. (1993). To Know As We Are Known: Education As A Spiritual Journey. San
Francisco: Harper.
Palomba, C. A. & Banta, T. W. (1999). Assessment Essentials: Planning, Implementing
and Improving Assessment in Higher Education. San Francisco: Jossey-Bass.
Riordan, T. & Roth, J. (2005). Disciplines as Frameworks for Student Learning. Sterling,
VA: Stylus.
Schuh, John H. and Associates, M. Lee Upcraft (Foreword by) (2009) Assessment
Methods for Student Affairs San Francisco: Jossey-Bass.
Shavelson, R. J. (2007) A Brief History of Student Learning Assessment. How We Got
Where We Are and a Proposal for Where to Go Next. Washington, DC:
Association of American Colleges and Universities.
Silverman, S. L. & Casazza, M. E. (2000). Learning and Development: Making
Connections to Enhance Teaching. San Francisco: Jossey-Bass.
Stevens, D. & Levi, A. J. (2005). Introduction to Rubrics. An Assessment Tool to Save
Grading Time, Convey Effective Feedback and Promote Student Learning.
Sterling, VA: Stylus.
Updated: 1/5/2009
209
Stiehl, Ruth and Lewchuk, Les. The Outcomes Primer: Reconsidering the
College Curriculum 3rd edition. The Learning Organization Press,
Corvalis, Oregon, 2008. ISBN 978-0-9637457 Available for purchase only via
http://www.outcomesnet.com
Stiehl, Ruth and Lewchuk, Les. The Mapping Primer: Tools for
Reconstructing the College Curriculum. The Learning Organization Press,
Corvalis, Oregon, 2005. ISBN 978-0-9637457-3-6 Available for purchase only via
http://www.outcomesnet.com
Stiehl, Ruth and Lewchuk, Les. The Assessment Primer: Creating A Flow
of Learning Evidence. The Learning Organization Press, Corvalis,
Oregon, 2008. ISBN 978-0-9637457-5-0 Available for purchase only via
http://www.outcomesnet.com
Suskie, L. (2004). Assessing Student Learning: A Common Sense Guide. Bolton, MA:
Anker.
Urban Universities Portfolio Project. (2002). Metropolitan Universities: An International
Forum. Special Issue, September, 13:3.
Vaill, P. B. (1996). Learning As A Way Of Being: Strategies for Survival in a World of
Permanent White Water. San Francisco: Jossey-Bass Publishers.
Walvoord, B. E. (2004). Assessment Clear and Simple : A Practical Guide for
Institutions, Departments and General Education. San Francisco: Jossey-Bass
Publishers.
Walvoord, B. A. & Anderson, J.A. (1998). Effective Grading: A Tool for Learning and
Assessment. San Francisco: Jossey-Bass.
Wellman, Jane V., Ehrlich, Thomas, ed. (2003). How the Student Credit Hour Shapes
Higher Education : The Tie That Binds. San Francisco: Jossey-Bass Publishers.
Wiggins, G. (1998). Educative Assessment: Designing Assessment to Inform and
Improve Student Performance. San Francisco: Jossey-Bass.
Updated: 1/5/2009
210
RUBRICS
ACSCU
211
PROGRAM LEARNING OUTCOMES
Rubric for Assessing the Quality of Academic Program Learning Outcomes
Criterion
Comprehensive
List
Initial
The list of outcomes is
problematic: e.g., very incomplete,
overly detailed, inappropriate,
disorganized. It may include only
discipline-specific learning,
ignoring relevant institution-wide
learning. The list may confuse
learning processes (e.g., doing an
internship) with learning outcomes
(e.g., application of theory to realworld problems).
Outcome statements do not
identify what students can do to
demonstrate learning. Statements
such as “Students understand
scientific method” do not specify
how understanding can be
demonstrated and assessed.
There is no clear relationship
between the outcomes and the
curriculum that students
experience.
Emerging
The list includes reasonable
outcomes but does not specify
expectations for the program
as a whole. Relevant
institution-wide learning
outcomes and/or national
disciplinary standards may be
ignored. Distinctions between
expectations for
undergraduate and graduate
programs may be unclear.
Most of the outcomes indicate
how students can demonstrate
their learning.
Developed
The list is a well-organized set of
reasonable outcomes that focus on
the key knowledge, skills, and
values students learn in the
program. It includes relevant
institution-wide outcomes (e.g.,
communication or critical thinking
skills). Outcomes are appropriate
for the level (undergraduate vs.
graduate); national disciplinary
standards have been considered.
Each outcome describes how
students can demonstrate learning,
e.g., “Graduates can write reports
in APA style” or “Graduates can
make original contributions to
biological knowledge.”
Highly Developed
The list is reasonable, appropriate, and
comprehensive, with clear distinctions
between undergraduate and graduate
expectations, if applicable. National
disciplinary standards have been
considered. Faculty have agreed on
explicit criteria for assessing students’
level of mastery of each outcome.
Students appear to be given
reasonable opportunities to
develop the outcomes in the
required curriculum.
Assessment
Planning
There is no formal plan for
assessing each outcome.
The program relies on shortterm planning, such as
selecting which outcome(s) to
assess in the current year.
The curriculum is designed to
provide opportunities for students
to learn and to develop increasing
sophistication with respect to each
outcome. This design may be
summarized in a curriculum map.
The program has a reasonable,
multi-year assessment plan that
identifies when each outcome will
be assessed. The plan may
explicitly include analysis and
implementation of improvements.
The Student
Experience
Students know little or nothing
about the overall outcomes of the
program. Communication of
outcomes to students, e.g. in
syllabi or catalog, is spotty or
nonexistent.
Students have some
knowledge of program
outcomes. Communication is
occasional and informal, left to
individual faculty or advisors.
Pedagogy, grading, the curriculum,
relevant student support services, and cocurriculum are explicitly and intentionally
aligned with each outcome. Curriculum
map indicates increasing levels of
proficiency.
The program has a fully-articulated,
sustainable, multi-year assessment plan
that describes when and how each
outcome will be assessed and how
improvements based on findings will be
implemented. The plan is routinely
examined and revised, as needed.
Students are well-acquainted with
program outcomes and may participate in
creation and use of rubrics. They are
skilled at self-assessing in relation to the
outcomes and levels of performance.
Program policy calls for inclusion of
outcomes in all course syllabi, and they
are readily available in other program
documents.
Assessable
Outcomes
Alignment
Students have a good grasp of
program outcomes. They may use
them to guide their own learning.
Outcomes are included in most
syllabi and are readily available in
the catalog, on the web page, and
elsewhere.
Outcomes describe how students can
demonstrate their learning. Faculty have
agreed on explicit criteria statements,
such as rubrics, and have identified
examples of student performance at
varying levels for each outcome.
212
How Visiting Team Members Can Use the Learning Outcomes Rubric
Conclusions should be based on a review of learning outcomes and assessment plans. Although you can make some preliminary judgments
about alignment based on examining the curriculum or a curriculum map, you will have to interview key departmental representatives, such as
department chairs, faculty, and students, to fully evaluate the alignment of the learning environment with the outcomes.
The rubric has five major dimensions:
1. Comprehensive List. The set of program learning outcomes should be a short but comprehensive list of the most important knowledge, skills,
and values students learn in the program, including relevant institution-wide outcomes such as those dealing with communication skills, critical
thinking, or information literacy. Faculty generally should expect higher levels of sophistication for graduate programs than for undergraduate
programs, and they should consider national disciplinary standards when developing and refining their outcomes, if available. There is no strict
rule concerning the optimum number of outcomes, but quality is more important than quantity. Faculty should not confuse learning processes
(e.g., completing an internship) with learning outcomes (what is learned in the internship, such as application of theory to real-world practice).
Questions. Is the list reasonable, appropriate and well-organized? Are relevant institution-wide outcomes, such as information literacy,
included? Are distinctions between undergraduate and graduate outcomes clear? Have national disciplinary standards been considered when
developing and refining the outcomes? Are explicit criteria – as defined in a rubric, for example – available for each outcome?
2. Assessable Outcomes. Outcome statements should specify what students can do to demonstrate their learning. For example, an outcome
might state that “Graduates of our program can collaborate effectively to reach a common goal” or that “Graduates of our program can design
research studies to test theories and examine issues relevant to our discipline.” These outcomes are assessable because faculty can observe
the quality of collaboration in teams, and they can review the quality of student-created research designs. Criteria for assessing student
products or behaviors usually are specified in rubrics, and the department should develop examples of varying levels of student performance
(i.e., work that does not meet expectations, meets expectations, and exceeds expectations) to illustrate levels. Questions. Do the outcomes
clarify how students can demonstrate learning? Have the faculty agreed on explicit criteria, such as rubrics, for assessing each outcome? Do
they have examples of work representing different levels of mastery for each outcome?
3. Alignment. Students cannot be held responsible for mastering learning outcomes unless they have participated in a program that
systematically supports their development. The curriculum should be explicitly designed to provide opportunities for students to develop
increasing sophistication with respect to each outcome. This design often is summarized in a curriculum map—a matrix that shows the
relationship between courses in the required curriculum and the program’s learning outcomes. Pedagogy and grading should be aligned with
outcomes to foster and encourage student growth and to provide students helpful feedback on their development. Since learning occurs within
and outside the classroom, relevant student services (e.g., advising and tutoring centers) and co-curriculum (e.g., student clubs and campus
events) should be designed to support the outcomes. Questions. Is the curriculum explicitly aligned with the program outcomes? Do faculty
select effective pedagogy and use grading to promote learning? Are student support services and the co-curriculum explicitly aligned to
promote student development of the learning outcomes?
4. Assessment Planning. Faculty should develop explicit plans for assessing each outcome. Programs need not assess every outcome every
year, but faculty should have a plan to cycle through the outcomes over a reasonable period of time, such as the period for program review
cycles. Questions. Does the plan clarify when, how, and how often each outcome will be assessed? Will all outcomes be assessed over a
reasonable period of time? Is the plan sustainable, in terms of human, fiscal, and other resources? Are assessment plans revised, as needed?
5. The Student Experience. At a minimum, students should be aware of the learning outcomes of the program(s) in which they are enrolled;
ideally, they should be included as partners in defining and applying the outcomes and the criteria for levels of sophistication. Thus it is
essential to communicate learning outcomes to students consistently and meaningfully. Questions: Are the outcomes communicated to
students? Do students understand what the outcomes mean and how they can further their own learning? Do students use the outcomes and
criteria to self-assess? Do they participate in reviews of outcomes, criteria, curriculum design, or related activities?
213
Rubric for Evaluating General Education Assessment Process (Draft for Use as Pilot, June 7, 2008)
Criterion
GE Outcomes
Initial
GE learning outcomes
have not yet been
developed for the entire
GE program; there may
be one or two common
ones, e.g., writing,
critical thinking.
Emerging
Learning outcomes have been
developed for the entire GE
program, but the list is too
long, too short, or inappropriate. Outcomes do not lend
themselves to demonstrations
of student learning.
Developed
The list is a well-organized set of
reasonable outcomes that focus on
the most important knowledge, skills,
and values students learn in the GE
program. Outcomes express how
students can demonstrate their
learning. Work to define levels of
performance is beginning.
Highly Developed
The list of outcomes is reasonable and
appropriate. Outcomes describe how
students can demonstrate their learning.
Faculty have agreed on explicit criteria,
such as rubrics, for assessing students’ level
of mastery and have identified exemplars of
student performance at varying levels for
each outcome.
Curriculum
Alignment with
Outcomes
There is no clear
relationship between the
outcomes and the GE
curriculum. Students may
not have the opportunity
to develop each outcome.
Students appear to be given
reasonable opportunities to
develop each of the GE
outcomes. Curriculum map
may indicate opportunities to
acquire outcomes.
The curriculum is explicitly designed
to provide opportunities for students
to learn and to develop increasing
sophistication with respect to each
outcome. Design may be summarized
in a curriculum map that shows
“beginning,” “intermediate” and
“advanced” treatment of outcomes.
Pedagogy, grading, the curriculum, and
relevant student support services and the cocurriculum are explicitly aligned with GE
outcomes.
Assessment
Planning
There is no formal plan
for assessing each GE
outcome.
GE assessment relies on shortterm planning, such as
selecting which outcome(s) to
assess in the current year.
Interpretation and use of
findings for improvement are
implicit rather than planned or
funded.
The campus has a reasonable, multiyear assessment plan that identifies
when each GE outcome will be
assessed. The plan includes specific
mechanisms for interpretation and
use of findings for improvement.
Assessment
Implementation
It is not clear that
potentially valid
evidence for each GE
outcome is collected
and/or individual
reviewers use
idiosyncratic criteria to
assess student work.
Appropriate evidence is
collected and faculty have
discussed relevant criteria for
assessing each outcome.
Those who assess student
work are calibrated to apply
assessment criteria in the same
way or faculty routinely check
for inter-rater reliability.
Appropriate evidence is collected and
faculty use explicit criteria, such as
rubrics, to assess student attainment
of each outcome. Those who assess
student work are calibrated to apply
assessment criteria in the same way,
and faculty routinely check for interrater reliability.
The campus has a fully-articulated,
sustainable, multi-year assessment plan that
describes when and how each outcome will
be assessed. The plan is routinely examined
and revised, as needed, based on experience
and feedback from external reviewers. The
campus uses some form of comparative data
(e.g., own past record, aspirational goals,
external benchmarking).
Assessment criteria, such as rubrics, have
been pilot-tested and refined over time; and
they usually are shared with students. Those
who assess student work are calibrated, and
faculty routinely find high inter-rater
reliability. Faculty take comparative data
into account when interpreting results and
deciding on changes to improve learning.
214
Use of Results
Results for GE outcomes
are collected, but they are
not discussed among
relevant faculty. There is
little or no collective use
of findings.
Results for each GE outcome
are collected and discussed by
relevant faculty; results have
been used occasionally to
improve the GE program.
Results for each outcome are
collected, discussed by relevant
faculty and others, and regularly used
to improve the GE program.
Relevant faculty routinely discuss results,
plan needed changes, secure necessary
resources, and implement changes. They
may collaborate with others, such as
librarians or Student Affairs professionals,
to improve the program. Follow-up studies
confirm that changes have improved
learning.
215
How Visiting Team Members Can Use the GE Assessment Rubric
Conclusions should be based on review of the GE program’s written assessment record and discussion with relevant campus representatives (e.g.,
GE chair, GE Assessment Coordinator, faculty who teach GE courses). Discussion should validate that the reality matches the written record.
The rubric has five major dimensions:
1.
GE Outcomes. The set of GE learning outcomes should be a comprehensive list of the most important knowledge, skills, and values
students learn in the GE program. There is no strict rule concerning the optimum number of outcomes, but quality is more important than
quantity. Faculty should not confuse learning processes (e.g., completing a science lab) with learning outcomes (what is learned in the
science lab, such as ability to apply the scientific method). Outcome statements should specify what students do to demonstrate their
learning. For example, an outcome might state that “Students who complete the GE program can explain major concepts and theories in at
least two social science disciplines.” This outcome is assessable because faculty can rate the quality of students’ explanations. Criteria for
assessing student work usually are specified in rubrics, and faculty should identify examples of varying levels of student performance, such
as work that does not meet expectations, that meets expectations, and exceeds expectations. Questions. Is the list of outcomes reasonable
and appropriate? Do the outcomes express how students can demonstrate learning? Have faculty agreed on explicit criteria, such as rubrics,
for assessing each outcome? Do they have examplars of work representing different levels of mastery for each outcome?
2.
Curriculum Alignment. Students cannot be held responsible for mastering learning outcomes unless the GE program systematically
supports their development. The GE curriculum should be explicitly designed to provide opportunities for students to develop increasing
sophistication with respect to each outcome. This design often is summarized in a curriculum map—a matrix that shows the relationship
between GE courses and GE learning outcomes. Pedagogy and grading should align with outcomes to foster growth and provide students
helpful feedback on their development. Relevant student services (e.g., advising and tutoring centers) and the co-curriculum (e.g., student
clubs and campus events) should also be designed to support development of the learning outcomes, since learning occurs outside the
classroom as well as within it. Questions. Is the GE curriculum explicitly aligned with program outcomes? Do faculty select effective
pedagogies and use grading to promote learning? Are student support services and the co-curriculum explicitly aligned to promote student
development of GE learning outcomes?
3.
Assessment Planning. Faculty should develop explicit, sustainable plans for assessing each GE outcome. They need not assess every
outcome every year, but they should have a plan to cycle through the outcomes over a reasonable period of time, such as the period for
program review cycles. Experience and feedback from external reviewers should guide plan revision. Questions. Does the campus have a
GE assessment plan? Does the plan clarify when, how, and how often each outcome will be assessed? Will all outcomes be assessed over a
reasonable period of time? Is the plan sustainable? Supported by appropriate resources? Are plans revised, as needed, based on experience
and feedback from external reviewers? Does the plan include collection of comparative data?
4.
Assessment Implementation. GE assessment data should be valid and reliable. A valid assessment of a particular outcome leads to
accurate conclusions concerning students’ achievement of that outcome. Sometimes campuses collect assessment data that do not have the
216
Western Association of Schools and Colleges
Accrediting Commission for Senior Colleges and Universities
potential to be valid. For example, a multiple-choice test may not collect information that allows faculty to make judgments about students’
ability to explain phenomena. Assessment requires the collection of valid evidence and judgments about that evidence that are based on
agreed-upon criteria that specify how to identify work that meets or exceeds expectations. These criteria usually are specified in rubrics.
Well-qualified judges should reach the same conclusions about individual student’s achievement of a learning outcome, demonstrating
inter-rater reliability. If two judges independently assess a set of materials, their ratings can be correlated. Sometimes a discrepancy index
is used. How often do the two raters give identical ratings, ratings one point apart, ratings two points apart, etc.? Data are reliable if the
correlation is high and/or if the discrepancies are small. Raters generally are calibrated (“normed”) to increase reliability. Calibration
usually involves a training session in which raters apply rubrics to pre-selected examples of student work that vary in quality; then they
reach consensus about the rating each example should receive. The purpose is to ensure that all raters apply the criteria in the same way so
that each student’s product would receive the same score, regardless of rater. Faculty may take external benchmarking data or other
comparative data into account when interpreting results. Questions: Do GE assessment studies systematically collect valid evidence for
each targeted outcome? Do faculty use agreed-upon criteria such as rubrics for assessing the evidence for each outcome? Do they share the
criteria with their students? Are those who assess student work calibrated in the use of assessment criteria? Does the campus routinely
document high inter-rater reliability? Do faculty pilot test and refine their assessment processes? Do they take external benchmarking
(comparison) data into account when interpreting results?
5.
Use of Results. Assessment is a process designed to monitor and improve learning, so assessment findings should have an impact. Faculty
should reflect on results for each outcome and decide if they are acceptable or disappointing. If results do not meet faculty standards,
faculty (and others, such as student affairs personnel, librarians, tutors) should determine which changes should be made, e.g., in pedagogy,
curriculum, student support, or faculty support. Questions: Do faculty collect assessment results, discuss them, and reach conclusions about
student achievement? Do they develop explicit plans to improve student learning? Do they implement those plans? Do they have a history
of securing necessary resources to support this implementation? Do they collaborate with other campus professionals to improve student
learning? Do follow-up studies confirm that changes have improved learning?
217
PORTFOLIOS
Rubric for Assessing the Use of Portfolios for Assessing Program Learning Outcomes
Criterion
Clarification of
Students’
Task
Valid Results
Reliable
Results
Results Are
Used
If e-Portfolios
Are Used
Initial
Instructions to students for
portfolio development provide
insufficient detail for them to
know what faculty expect.
Instructions may not identify
outcomes to be addressed in
the portfolio.
It is not clear that valid
evidence for each relevant
outcome is collected and/or
individual reviewers use
idiosyncratic criteria to assess
student work.
Emerging
Students receive some written
instructions for their portfolios,
but they still have problems
determining what is required of
them and/or why they are
compiling a portfolio.
Appropriate evidence is
collected for each outcome,
and faculty have discussed
relevant criteria for assessing
each outcome.
Developed
Students receive written
instructions that describe faculty
expectations in detail and include
the purpose of the portfolio, types
of evidence to include, role of the
reflective essay (if required), and
format of the finished product.
Appropriate evidence is collected
for each outcome; faculty use
explicit criteria, such as agreedupon rubrics, to assess student
attainment of each outcome.
Rubrics are usually shared with
students.
Those who review student
work are not calibrated to
apply assessment criteria in
the same way, and there are
no checks for inter-rater
reliability.
Results for each outcome are
collected, but they are not
discussed among the faculty.
Reviewers are calibrated to
apply assessment criteria in
the same way or faculty
routinely check for inter-rater
reliability.
Reviewers are calibrated to apply
assessment criteria in the same
way, and faculty routinely check for
inter-rater reliability.
Results for each outcome are
collected and discussed by the
faculty, but results have not
been used to improve the
program.
Results for each outcome are
collected, discussed by faculty, and
used to improve the program.
There is no technical support
for students or faculty to learn
the software or to deal with
problems.
There is informal or minimal
formal support for students
and faculty.
Formal technical support is readily
available and proactively assists in
learning the software and solving
problems.
Highly Developed
Students in the program understand the
portfolio requirement and the rationale for it,
and they view the portfolio as helping them
develop self-assessment skills. Faculty may
monitor the developing portfolio to provide
formative feedback and/or advise individual
students.
Assessment criteria, e.g., in the form of
rubrics, have been pilot-tested and refined
over time; they are shared with students,
and student may have helped develop them.
Feedback from external reviewers has led to
refinements in the assessment process. The
department also uses external
benchmarking data.
Reviewers are calibrated; faculty routinely
find that assessment data have high interrater reliability.
Faculty routinely discuss results, plan
needed changes, secure necessary
resources, and implement changes. They
may collaborate with others, such as
librarians or Student Affairs professionals, to
improve student learning. Students may also
participate in discussions and/or receive
feedback, either individual or in the
aggregate. Follow-up studies confirm that
changes have improved learning.
Support is readily available, proactive, and
effective. Tech support personnel may also
participate in refining the overall portfolio
process.
218
How Visiting Team Members Can Use the Portfolio Rubric
Portfolios can serve many purposes besides assessment; in fact, these other purposes are actually much more common. Portfolios may be compiled so
students can share their work with family and friends. They may be designed to build students’ confidence by showing development over time or by
displaying best work. They may be used for advising and career counseling, or so students can show their work during a job interview. The first thing a
team needs to do is determine that the portfolios are used for assessment, and not for another purpose.
Conclusions about the quality of the assessment process should be based on discussion with relevant department members (e.g., chair, assessment
coordinator, faculty, students) and a review of the program’s written portfolio assignment. Two common types of portfolios are:
• Showcase portfolios—collections of each student’s best work
• Developmental portfolios—collections of work from early, middle, and late stages in the student’s academic career that demonstrate growth
Faculty generally require students to include a reflective essay that describes how the evidence in the portfolio demonstrates their achievement of
program learning outcomes. Sometimes faculty monitor developing portfolios to provide formative feedback and/or advising to students, and sometimes
they collect portfolios only as students near graduation. Portfolio assignments should clarify the purpose of the portfolio, what kinds of evidence should
be included, and the format (e.g., paper vs. e-portfolios); and students should view the portfolio as contributing to their personal development.
The rubric has five major dimensions and a fifth dimension limited to e-portfolios:
1. Clarification of Students’ Task. Most students have never created a portfolio, and they need explicit guidance. Questions. Does the portfolio
assignment provide sufficient detail so students understand the purpose, the types of evidence to include, the learning outcomes to address, the role
of the reflective essay (if any), and the required format? Do students view the portfolio as contributing to their ability to self-assess? Do faculty use
the developing portfolios to assist individual students?
2. Valid Results. Sometimes portfolios lack valid evidence for assessing particular outcomes. For example, portfolios may not allow faculty to assess
how well students can deliver oral presentations. Judgments about that evidence need to be based on well-established, agreed-upon criteria that
specify (usually in rubrics) how to identify work that meets or exceeds expectations. Questions: Do the portfolios systematically include valid
evidence for each targeted outcome? Are faculty using well-established, agreed-upon criteria, such as rubrics, to assess the evidence for each
outcome? Have faculty pilot tested and refined their process? Are criteria shared with students? Are they collaborating with colleagues at other
institutions to secure benchmarking (comparison) data?
3. Reliable Results. Well-qualified judges should reach the same conclusions about a student’s achievement of a learning outcome, demonstrating
inter-rater reliability. If two judges independently assess a set of materials, their ratings can be correlated. Sometimes a discrepancy index is used.
How often do the two raters give identical ratings, ratings one point apart, ratings two points apart, etc.? Data are reliable if the correlation is high
and/or if discrepancies are small. Raters generally are calibrated (“normed”) to increase reliability. Calibration usually involves a training session in
which raters apply rubrics to pre-selected examples of student work that vary in quality, then reach consensus about the rating each example should
receive. The purpose is to ensure that all raters apply the criteria in the same way so that each student’s product would receive the same score,
regardless of rater. Questions: Are reviewers calibrated? Are checks for inter-rater reliability made? Is there evidence of high inter-rater reliability?
4. Results Are Used. Assessment is a process designed to monitor and improve learning, so assessment findings should have an impact. Faculty
should reflect on results for each outcome and decide if they are acceptable or disappointing. If results do not meet their standards, faculty should
determine what changes should be made, e.g., in pedagogy, curriculum, student support, or faculty support. Questions: Do faculty collect
assessment results, discuss them, and reach conclusions about student achievement? Do they develop explicit plans to improve student learning?
Do they implement those plans? Do they have a history of securing necessary resources to support this implementation? Do they collaborate with
other campus professionals to improve student learning? Do follow-up studies confirm that changes have improved learning?
5. If e-Portfolios Are Used. Faculty and students alike require support, especially when a new software program is introduced. Lack of support can
lead to frustration and failure of the process. Support personnel may also have useful insights into how the portfolio assessment process can be
refined. Questions: What is the quality and extent of technical support? Of inclusion in review and refinement of the portfolio process? What is the
overall level of faculty and student satisfaction with the technology and support services?
219
CAPSTONES
Rubric for Assessing the Use of Capstone Experiences for Assessing Program Learning Outcomes
Criterion
Relevant
Outcomes
and Lines of
Evidence
Identified
Initial
It is not clear which program
outcomes will be assessed
in the capstone course.
Valid Results
It is not clear that potentially
valid evidence for each
relevant outcome is
collected and/or individual
faculty use idiosyncratic
criteria to assess student
work or performances.
Reliable
Results
Results Are
Used
The Student
Experience
Emerging
The relevant outcomes are
identified, e.g., ability to integrate
knowledge to solve complex
problems; however, concrete
plans for collecting evidence for
each outcome have not been
developed.
Faculty have reached general
agreement on the types of
evidence to be collected for each
outcome; they have discussed
relevant criteria for assessing
each outcome but these are not
yet fully defined.
Developed
Relevant outcomes are
identified. Concrete plans for
collecting evidence for each
outcome are agreed upon and
used routinely by faculty who
staff the capstone course.
Highly Developed
Relevant evidence is collected; faculty
have agreed on explicit criteria
statements, e.g., rubrics, and have
identified examples of student
performance at varying levels of
mastery for each relevant outcome.
Faculty have agreed on concrete
plans for collecting relevant
evidence for each outcome.
Explicit criteria, e.g., rubrics,
have been developed to assess
the level of student attainment of
each outcome.
Those who review student
work are not calibrated to
apply assessment criteria in
the same way; there are no
checks for inter-rater
reliability.
Results for each outcome
may or may not be are
collected. They are not
discussed among faculty.
Reviewers are calibrated to apply
assessment criteria in the same
way or faculty routinely check for
inter-rater reliability.
Reviewers are calibrated to apply
assessment criteria in the same
way, and faculty routinely check
for inter-rater reliability.
Assessment criteria, such as rubrics,
have been pilot-tested and refined
over time; they usually are shared with
students. Feedback from external
reviewers has led to refinements in the
assessment process, and the
department uses external
benchmarking data.
Reviewers are calibrated, and faculty
routinely find assessment data have
high inter-rater reliability.
Results for each outcome are
collected and may be discussed
by the faculty, but results have
not been used to improve the
program.
Results for each outcome are
collected, discussed by faculty,
analyzed, and used to improve
the program.
Students know little or
nothing about the purpose of
the capstone or outcomes to
be assessed. It is just
another course or
requirement.
Students have some knowledge
of the purpose and outcomes of
the capstone. Communication is
occasional, informal, left to
individual faculty or advisors.
Students have a good grasp of
purpose and outcomes of the
capstone and embrace it as a
learning opportunity. Information
is readily avail-able in advising
guides, etc.
Faculty routinely discuss results, plan
needed changes, secure necessary
resources, and implement changes.
They may collaborate with others,
such as librarians or Student Affairs
professionals, to improve results.
Follow-up studies confirm that
changes have improved learning.
Students are well-acquainted with
purpose and outcomes of the
capstone and embrace it. They may
participate in refining the experience,
outcomes, and rubrics. Information is
readily available.
220
How Visiting Team Members Can Use the Capstone Rubric
Conclusions should be based on discussion with relevant department members (e.g., chair, assessment coordinator, faculty). A variety of capstone
experiences can be used to collect assessment data, such as:
• courses, such as senior seminars, in which advanced students are required to consider the discipline broadly and integrate what they have learned
in the curriculum
• specialized, advanced courses
• advanced-level projects conducted under the guidance of a faculty member or committee, such as research projects, theses, or dissertations
• advanced-level internships or practica, e.g., at the end of an MBA program
Assessment data for a variety of outcomes can be collected in such courses, particularly outcomes related to integrating and applying the discipline,
information literacy, critical thinking, and research and communication skills.
The rubric has five major dimensions:
1. Relevant Outcomes and Evidence Identified. It is likely that not all program learning outcomes can be assessed within a single capstone course or
experience. Questions: Have faculty explicitly determined which program outcomes will be assessed in the capstone? Have they agreed on concrete
plans for collecting evidence relevant to each targeted outcome? Have they agreed on explicit criteria, such as rubrics, for assessing the evidence?
Have they identified examples of student performance for each outcome at varying performance levels (e.g., below expectations, meeting, exceeding
expectations for graduation)?
2. Valid Results. A valid assessment of a particular outcome leads to accurate conclusions concerning students’ achievement of that outcome.
Sometimes faculty collect evidence that does not have the potential to provide valid conclusions. For example, a multiple-choice test will not provide
evidence of students’ ability to deliver effective oral presentations. Assessment requires the collection of valid evidence and judgments about that
evidence that are based on well-established, agreed-upon criteria that specify how to identify low, medium, or high-quality work. Questions: Are
faculty collecting valid evidence for each targeted outcome? Are they using well-established, agreed-upon criteria, such as rubrics, for assessing the
evidence for each outcome? Have faculty pilot tested and refined their process based on experience and feedback from external reviewers? Are they
sharing the criteria with their students? Are they using benchmarking (comparison) data?
3. Reliable Results. Well-qualified judges should reach the same conclusions about individual student’s achievement of a learning outcome,
demonstrating inter-rater reliability. If two judges independently assess a set of materials, their ratings can be correlated. Sometimes a discrepancy
index is used. How often do the two raters give identical ratings, ratings one point apart, ratings two points apart, etc.? Data are reliable if the
correlation is high and/or if the discrepancies are small. Raters generally are calibrated (“normed”) to increase reliability. Calibration usually involves
a training session in which raters apply rubrics to pre-selected examples of student work that vary in quality, then reach consensus about the rating
each example should receive. The purpose is to ensure that all raters apply the criteria in the same way so that each student’s product receives the
same score, regardless of rater. Questions: Are reviewers calibrated? Are checks for inter-rater reliability made? Is there evidence of high inter-rater
reliability?
4. Results Are Used. Assessment is a process designed to monitor and improve learning, so assessment findings should have an impact. Faculty
should reflect on results for each outcome and decide if they are acceptable or disappointing. If results do not meet faculty standards, faculty should
determine which changes should be made, e.g., in pedagogy, curriculum, student support, or faculty support. Questions: Do faculty collect
assessment results, discuss them, and reach conclusions about student achievement? Do they develop explicit plans to improve student learning?
Do they implement those plans? Do they have a history of securing necessary resources to support this implementation? Do they collaborate with
other campus professionals to improve student learning? Do follow-up studies confirm that changes have improved learning?
The Student Experience. Students should understand the purposes different educational experiences serve in promoting their learning and
development and know how to take advantage of them; ideally they should also participate in shaping those experiences. Thus it is essential to
communicate to students consistently and include them meaningfully. Questions: Are purposes and outcomes communicated to students? Do they
understand how capstones support learning? Do they participate in reviews of the capstone experience, its outcomes, criteria, or related activities?
221
PROGRAM REVIEW
Rubric for Assessing the Integration of Student Learning Assessment into Program Reviews
Criterion
Required
Elements of
the Self-Study
Initial
Program faculty may be
required to provide a list of
program-level student
learning outcomes.
Emerging
Faculty are required to provide
the program’s student learning
outcomes and summarize annual
assessment findings.
Developed
Faculty are required to provide the
program’s student learning outcomes,
annual assessment studies, findings,
and resulting changes. They may be
required to submit a plan for the next
cycle of assessment studies.
Process of
Review
Internal and external
reviewers do not address
evidence concerning the
quality of student learning
in the program other than
grades.
Internal and external reviewers
address indirect and possibly
direct evidence of student
learning in the program; they do
so at the descriptive level, rather
than providing an evaluation.
Planning and
Budgeting
The campus has not
integrated program
reviews into planning and
budgeting processes.
Annual
Feedback on
Assessment
Efforts
No individual or committee
on campus provides
feedback to departments
on the quality of their
outcomes, assessment
plans, assessment
studies, impact, etc.
The campus has attempted to
integrate program reviews into
planning and budgeting
processes, but with limited
success.
An individual or committee
occasionally provides feedback
on the quality of outcomes,
assessment plans, assessment
studies, etc.
Internal and external reviewers analyze
direct and indirect evidence of student
learning in the program and offer
evaluative feedback and suggestions
for improvement. They have sufficient
expertise to evaluate program efforts;
departments use the feedback to
improve their work.
The campus generally integrates
program reviews into planning and
budgeting processes, but not through a
formal process.
The Student
Experience
Students are unaware of
and uninvolved in program
review.
Program review may include
focus groups or conversations
with students to follow up on
results of surveys
A well-qualified individual or committee
provides annual feedback on the quality
of outcomes, assessment plans,
assessment studies, etc. Departments
use the feedback to improve their work.
The internal and external reviewers
examine samples of student work, e.g.,
sample papers, portfolios and capstone
projects. Students may be invited to
discuss what they learned and how they
learned it.
Highly Developed
Faculty are required to evaluate the
program’s student learning outcomes, annual
assessment findings, bench-marking results,
subsequent changes, and evidence
concerning the impact of these changes.
They present a plan for the next cycle of
assessment studies.
Well-qualified internal and external reviewers
evaluate the program’s learning outcomes,
assessment plan, evidence, benchmarking
results, and assessment impact. They give
evaluative feedback and suggestions for
improve-ment. The department uses the
feedback to improve student learning.
The campus systematically integrates
program reviews into planning and budgeting
processes, e.g., through negotiating formal
action plans with mutually agreed-upon
commitments.
A well-qualified individual or committee
provides annual feedback on the quality of
outcomes, assessment plans, assessment
studies, benchmarking results, and
assessment impact. Departments effectively
use the feedback to improve student
learning. Follow-up activities enjoy
institutional support
Students are respected partners in the
program review process. They may offer
poster sessions on their work, demon-strate
how they apply rubrics to self-assess, and/or
provide their own evaluative feedback.
222
How Visiting Team Members Can Use the Program Review Rubric
Conclusions should be based on a review of program-review documents and discussion with relevant campus representatives, such as department
chairs, deans, and program review committees.
The rubric has five major dimensions:
1. Self-Study Requirements. The campus should have explicit requirements for the program’s self-study, including an analysis of the program’s
learning outcomes and a review of the annual assessment studies conducted since the last program review. Faculty preparing the self-study should
reflect on the accumulating results and their impact; and they should plan for the next cycle of assessment studies. As much as possible, programs
should benchmark findings against similar programs on other campuses. Questions: Does the campus require self-studies that include an analysis of
the program’s learning outcomes, assessment studies, assessment results, benchmarking results, and assessment impact, including the impact of
changes made in response to earlier studies? Does the campus require an updated assessment plan for the subsequent years before the next
program review?
2. Self-Study Review. Internal reviewers (on-campus individuals, such as deans and program review committee members) and external reviewers (offcampus individuals, usually disciplinary experts) should evaluate the program’s learning outcomes, assessment plan, assessment evidence,
benchmarking results, and assessment impact; and they should provide evaluative feedback and suggestions for improvement. Questions: Who
reviews the self-studies? Do they have the training or expertise to provide effective feedback? Do they routinely evaluate the program’s learning
outcomes, assessment plan, assessment evidence, benchmarking results, and assessment impact? Do they provide suggestions for improvement?
Do departments effectively use this feedback to improve student learning?
3. Planning and Budgeting. Program reviews should not be pro forma exercises; they should be tied to planning and budgeting processes, with
expectations that increased support will lead to increased effectiveness, such as improving student learning and retention rates. Questions. Does the
campus systematically integrate program reviews into planning and budgeting processes? Are expectations established for the impact of planned
changes?
4. Annual Feedback on Assessment Efforts. Campuses moving into the culture of evidence often find considerable variation in the quality of
assessment efforts across programs, and waiting for years to provide feedback to improve the assessment process is unlikely to lead to effective
campus practices. While program reviews encourage departments to reflect on multi-year assessment results, some programs are likely to require
more immediate feedback, usually based on a required, annual assessment report. This feedback might be provided by an Assessment Director or
Committee, relevant Dean or Associate Dean, or others; and whoever has this responsibility should have the expertise to provide quality feedback.
Questions: Does someone have the responsibility for providing annual feedback on the assessment process? Does this person or team have the
expertise to provide effective feedback? Does this person or team routinely provide feedback on the quality of outcomes, assessment plans,
assessment studies, benchmarking results, and assessment impact? Do departments effectively use this feedback to improve student learning?
5. The Student Experience. Students have a unique perspective on a given program of study: they know better than anyone what it means to go
through it as a student. Program review should take advantage of that perspective and build it into the review. Questions: Are students aware of the
purpose and value of program review? Are they involved in preparations and the self-study? Do they have an opportunity to interact with internal or
external reviewers, demonstrate and interpret their learning, and provide evaluative feedback?
223
The Educational Effectiveness Framework:
Capacity and Effectiveness as They Relate to Student and Institutional Learning
Key Descriptive Terms Î
Ð ELEMENT & DEFINITION
INITIAL
EMERGING
DEVELOPED
HIGHLY DEVELOPED
Learning
A. Student learning outcomes
established; communicated in
syllabi and publications; cited
and used by faculty, student
affairs, advisors, others
(CFRs 2.2, 2.4):
B. Expectations are established
for how well (i.e., proficiency
or level) students achieve
outcomes (CFRs 2.1, 2.4,
2.5):
C. Assessment plans are in
place; curricular and cocurricular outcomes are
systematically assessed,
improvements documented
(CFRs 2.4, 2.7):
For only a few programs and units; only
vaguely (if at all) for GE; not
communicated in syllabi, or publications
such as catalogues, view books, guides
to the major; only a few faculty know
and use for designing curriculum,
assignments, or assessment
For many programs and units, most
aspects of GE; beginning to be
communi-cated in basic documents;
beginning to be used by some faculty for
design of curriculum, assignments,
assessments
For all units (academic & co-curricular),
and for all aspects of GE; cited often but
not in all appropriate places; most
faculty cite; used in most programs for
design of curriculum, assignments, and
assessment
For all units (academic and cocurricular), and for all aspects of GE;
cited widely by faculty and advisors;
used routinely by faculty, student
affairs, other staff in design of
curricula, assignments, co-curriculum,
and assessment
Expectations for student learning have
not been set beyond course completion
and GPA; level of learning expected
relative to outcomes unclear
Expectations for level of learning explicit
in a few programs; heavy reliance on
course completion and GPA
Expectations for student learning explicit
in most programs
Expectations for student learning are
explicit in all programs, widely known
and embraced by faculty, staff, and
students
No comprehensive assessment plans.
Outcomes assessed occasionally using
surveys and self reports, seldom using
direct assessment; rarely lead to
revision of curriculum, pedagogy, cocurriculum, or other aspects of
educational experience
Some planning in place. Outcomes
assessed occasionally, principally using
surveys; beginning to move toward
some direct assessment; occasionally
leads to improvements in educational
experience; improvements sporadically
documented, e.g., in units’ annual
reports.
Plans mostly in place. Assessment
occurs periodically, using direct methods
supplemented by indirect methods and
descriptive data; educational experience
is frequently improved based on
evidence and findings; improvements
are routinely documented, e.g. in units’
annual reports
D. Desired kind and level of
learning is achieved (CFR
2.6):
Possible that learning is not up to
expectations, and/or expectations set by
institution are too low for degree(s)
offered by the institution
Most students appear to achieve at
levels set by the institution; faculty and
other educators beginning to discuss
expectations and assessment findings
Nearly all students achieve at or above
levels set by institution; assessment
findings discussed periodically by most
faculty and other campus educators
Teaching/Learning
Environment
A. Curricula, pedagogy, cocurriculum, other aspects of
educational experience are
aligned with outcomes (2.1,
2.2, 2.3, 2.4, 2.5, 4.6):
B. Curricular and co-curricular
processes (CFRs 2.1, 2.2,
2.3, 2.11, 2.13) are:
Conceived exclusively or largely in
terms of inputs (e.g. library holdings, lab
space), curricular requirements (e.g., for
majors, GE) and availability of cocurricular programs; not visibly aligned
with outcomes or expectations for level
of student achievement; evidence of
alignment processes lacking
Educational experience beginning to be
aligned with learning outcomes and
expectations for student achievement;
evidence of alignment efforts available
in some academic and co-curricular
programs
Educational experience generally
aligned with learning outcomes,
expectations for student achievement;
alignment becoming intentional,
systematic, supported by tools (e.g.
curriculum maps) and processes.
Evidence of alignment efforts generally
available
Assessment plans throughout
institution. Assessment occurs on
regular schedule using multiple
methods; strong reliance on direct
methods, performance-based;
educational experience systematically
reviewed and improved based on
evidence and findings; documentation
widespread and easy to locate.
All students achieve at or above levels
set by institution; findings are
discussed regularly and acted upon by
all or nearly all faculty and other
campus educators
Educational experience fully aligned
with learning outcomes, expectations;
alignment is systematic, supported by
tools and processes as well as broader
institutional infrastructure. Evidence of
alignment efforts readily available
Rarely informed by good learning
practices as defined by the wider higher
education community; few curricular or
co-curricular activities reviewed, mostly
without reference to outcomes or
evidence of student learning
Informed in some instances by good
learning practices; curricula and cocurricular activities occasionally
reviewed and improved but with little
reference to outcomes or assessment
findings
Informed in many cases by good
learning practices; reviewed and
improved by relevant faculty and other
campus educators; often based on
outcomes and assessment findings
Regularly informed by good learning
practices; improvements consistently
result from scholarly reflection on
outcomes and assessment findings by
relevant faculty and other campus
educators
224
The Educational Effectiveness Framework:
Capacity and Effectiveness as They Relate to Student and Institutional Learning
C. Professional development,
rewards (CFRs 2.8, 2.9):
Little or no support for faculty, other
campus educators to develop expertise
in assessment of student learning,
related practices; work to assess,
improve student learning plays no
positive role in reward system, may be
viewed as a negative
Some support for faculty, other
educators on campus to develop
expertise in assessment of student
learning, related practices; modest,
implicit positive role in reward system
Some support for faculty, other campus
educators to develop expertise in
assessment of student learning, related
practices; explicit, positive role in reward
structure
Significant support for faculty, other
campus educators to develop expertise
in assessment of student learning,
related practices; explicit, prominent
role in reward structure
Organizational Learning
A. Indicators of educational
effectiveness are (CFRs 1.2,
4.3, 4.4):
Notable by their absence or considered
only sporadically in decision-making
Found in some areas; dissemination of
performance results just beginning; no
reference to comparative data
Multiple, with data collected regularly,
disseminated, collectively analyzed;
some comparative data used. Some
indicators used to inform planning,
budgeting, other decision making on
occasional basis
Occasional, in some departments or
units; heavy reliance on traditional
inputs as indicators of quality; findings
occasion-ally used to suggest
improvements in educational
effectiveness; weak linkage to
institution-level planning, budgeting
Limited collection, dissemination,
disaggregation, or access. Campus at
beginning stages of use for decisions to
improve educational effectiveness at
program, unit, and/or institutional level
Frequent, affecting most academic and
co-curricular units, with growing
inclusion of findings about student
learning; unit uses findings to
collectively reflect on, improve
effectiveness; some linkage to
institution-level planning, budgeting
Systematic collection and dissemination,
wide access; sometimes disaggregated;
usually considered by decision-making
bodies at all levels, but documentation
and/or linkage to educational
effectiveness may be weak
Campus knowledge and support for a
culture of inquiry and evidence fairly
consistent across administration, faculty,
professional staff but may not be
uniformly deep
Multiple, with data collected regularly,
disseminated widely, collectively
analyzed; comparative data used, as
appropriate, in all programs. Indicators
consistently used to inform planning,
budgeting, other decision making at all
levels of the institution
Systematic and institution-wide, with
learning assessment findings a major
component; units use findings to
improve student learning, program
effectiveness, and supporting
processes; close linkage to institutionlevel planning, budgeting
Systematic collection and
dissemination, and access, purposeful
disaggregation; consistently used by
decision-making bodies for program
improvement at all levels, with
processes fully documented
Consistent, knowledgeable, deep
commitment to creating and sustaining
a culture of inquiry and evidence in all
appropriate functions at all levels
B. Formal program review (CFRs Rare, if it occurs at all, with little or no
useful data generated. Assessment
2.7, 4.4) is:
findings on student learning not
available and/or not used
C. Performance data, evidence,
and analyses (CFRs 4.3, 4.5,
4.6) are:
Not collected, disseminated,
disaggregated, or accessible for wide
use. Not evident in decision-making
processes; do not appear to be used for
improvement in any programs
D. Culture of inquiry and
evidence (CFRs 4.5, 4.6, 4.7):
Faculty, other educators, staff,
institutional leaders, governing board
not visibly committed to a culture of
inquiry and evidence except in isolated
cases; not knowledgeable about learnercenteredness, assessment, etc.
Little or no data, findings, analyses from
assessment of student learning
available within the institution or to
external audiences
Campus knowledge is minimal; support
– at top levels and/or grass roots – for
development of a culture of inquiry and
evidence is sporadic and uneven
Some data, findings, analyses from
assessment of student learning
available but may be incomplete, difficult
to access or understand for internal or
external audiences
Data, findings, analyses from
assessment of student learning
generally available, easily accessible;
chosen for relevance to multiple
audiences
Data, findings, analyses from learning
assessment are widely available and
skillfully framed to be understandable,
useful to multiple audiences
Committed to isolated aspects of
educational effectiveness; if other areas
are not addressed, continuing
reaffirmation of accreditation is
threatened
Committed to educational effectiveness
in some areas; significant number of
areas require attention, improvement
Mostly well-established commitment to
educational effectiveness; a few areas
require attention, improvement
Fully committed to and going beyond
WASC recommendations; operates at
an exemplary level in addressing its
Core Commitments to capacity as it
relates to learning and to educational
effectiveness
E. Communication and
transparency (CFR 1.2, 1.7):
Overall: The institution can best
be described as:
225
RUBRICS
ACCJC
226
Accrediting Commission for Community and Junior Colleges
Western Association of Schools and Colleges
Rubric for Evaluating Institutional Effectiveness – Part I: Program Review
(See attached instructions on how to use this rubric.)
Levels of
Implementation
Awareness
Development
Proficiency
Sustainable
Continuous
Quality
Improvement
Characteristics of Institutional Effectiveness in Program Review
(Sample institutional behaviors)
• There is preliminary investigative dialogue at the institution or within some departments
about what data or process should be used for program review.
• There is recognition of existing practices and models in program review that make use of
institutional research.
• There is exploration of program review models by various departments or individuals.
• The college is implementing pilot program review models in a few programs/operational
units.
• Program review is embedded in practice across the institution using qualitative and
quantitative data to improve program effectiveness.
• Dialogue about the results of program review is evident within the program as part of
discussion of program effectiveness.
• Leadership groups throughout the institution accept responsibility for program review
framework development (Senate, Admin. Etc.)
• Appropriate resources are allocated to conducting program review of meaningful quality.
• Development of a framework for linking results of program review to planning for
improvement.
• Development of a framework to align results of program review to resource allocation.
• Program review processes are in place and implemented regularly.
• Results of all program review are integrated into institution- wide planning for
improvement and informed decision-making.
• The program review framework is established and implemented.
• Dialogue about the results of all program reviews is evident throughout the institution as
part of discussion of institutional effectiveness.
• Results of program review are clearly and consistently linked to institutional planning
processes and resource allocation processes; college can demonstrate or provide specific
examples.
• The institution evaluates the effectiveness of its program review processes in supporting
and improving student achievement and student learning outcomes.
• Program review processes are ongoing, systematic and used to assess and improve
student learning and achievement.
• The institution reviews and refines its program review processes to improve institutional
effectiveness.
• The results of program review are used to continually refine and improve program practices
resulting in appropriate improvements in student achievement and learning.
1
227
Accrediting Commission for Community and Junior Colleges
Western Association of Schools and Colleges
Rubric for Evaluating Institutional Effectiveness – Part II: Planning
(See attached instructions on how to use this rubric.)
Levels of
Implementation
Awareness
Development
Characteristics of Institutional Effectiveness in Planning
(Sample institutional behaviors)
• The college has preliminary investigative dialogue about planning processes.
• There is recognition of case need for quantitative and qualitative data and analysis in
planning.
• The college has initiated pilot projects and efforts in developing systematic cycle of
evaluation, integrated planning and implementation (e.g. in human or physical resources).
• Planning found in only some areas of college operations.
• There is exploration of models and definitions and issues related to planning.
• There is minimal linkage between plans and a resource allocation process, perhaps
planning for use of "new money"
• The college may have a consultant-supported plan for facilities, or a strategic plan.
• The Institution has defined a planning process and assigned responsibility for
implementing it.
• The Institution has identified quantitative and qualitative data and is using it.
• Planning efforts are specifically linked to institutional mission and goals.
• The Institution uses applicable quantitative data to improve institutional effectiveness in
some areas of operation.
• Governance and decision-making processes incorporate review of institutional
effectiveness in mission and plans for improvement.
• Planning processes reflect the participation of a broad constituent base.
Proficiency
• The college has a well documented, ongoing process for evaluating itself in all areas of
operation, analyzing and publishing the results and planning and implementing
improvements.
• The institution's component plans are integrated into a comprehensive plan to achieve
broad educational purposes, and improve institutional effectiveness.
• The institution effectively uses its human, physical, technology and financial resources to
achieve its broad educational purposes, including stated student learning outcomes.
• The college has documented assessment results and communicated matters
of quality assurance to appropriate constituencies (documents data and analysis of
achievement of its educational mission).
• The institution assesses progress toward achieving its education goals over time
(uses longitudinal data and analyses).
• The institution plans and effectively incorporates results of program review in all areas of
educational services: instruction, support services, library and learning resources.
• Program review processes are ongoing, systematic and used to assess and improve
student learning and achievement.
Sustainable
Continuous
Quality
Improvement
• The institution uses ongoing and systematic evaluation and planning to refine its key
processes and improve student learning.
• There is dialogue about institutional effectiveness that is ongoing, robust and pervasive;
data and analyses are widely distributed and used throughout the institution.
• There is ongoing review and adaptation of evaluation and planning processes.
• There is consistent and continuous commitment to improving student learning;
and educational effectiveness is a demonstrable priority in all planning structures and
processes.
2
228
Accrediting Commission for Community and Junior Colleges
Western Association of Schools and Colleges
Rubric for Evaluating Institutional Effectiveness – Part III: Student Learning Outcomes
(See attached instructions on how to use this rubric.)
Levels of
Implementation
Characteristics of Institutional Effectiveness in
Student Learning Outcomes
(Sample institutional behaviors)
Awareness
Development
• There is preliminary, investigative dialogue about student learning outcomes.
• There is recognition of existing practices such as course objectives and how they relate to
student learning outcomes.
• There is exploration of models, definitions, and issues taking place by a few people.
• Pilot projects and efforts may be in progress.
• The college has discussed whether to define student learning outcomes at the level of
some courses or programs or degrees; where to begin.
• College has established an institutional framework for definition of student learning
outcomes (where to start), how to extend, and timeline.
• College has established authentic assessment strategies for assessing student learning
outcomes as appropriate to intended course, program, and degree learning outcomes.
• Existing organizational structures (e.g. Senate, Curriculum Committee) are supporting
strategies for student learning outcomes definition and assessment.
• Leadership groups (e.g. Academic Senate and administration), have accepted responsibility
for student learning outcomes implementation.
• Appropriate resources are being allocated to support student learning outcomes and
assessment.
• Faculty and staff are fully engaged in student learning outcomes development.
Proficiency
• Student learning outcomes and authentic assessment are in place for courses, programs
and degrees.
• Results of assessment are being used for improvement and further alignment of
institution-wide practices.
• There is widespread institutional dialogue about the results.
• Decision-making includes dialogue on the results of assessment and is purposefully
directed toward improving student learning.
• Appropriate resources continue to be allocated and fine-tuned.
• Comprehensive assessment reports exist and are completed on a regular basis.
• Course student learning outcomes are aligned with degree student learning outcomes.
• Students demonstrate awareness of goals and purposes of courses and programs in
which they are enrolled.
Sustainable
Continuous
Quality
Improvement
• Student learning outcomes and assessment are ongoing, systematic and used for
continuous quality improvement.
• Dialogue about student learning is ongoing, pervasive and robust.
• Evaluation and fine-tuning of organizational structures to support student learning is
ongoing.
• Student learning improvement is a visible priority in all practices and structures across the
college.
• Learning outcomes are specifically linked to program reviews.
JP;DB: cg 8/2007
3
229
SLO, Rubrics, & Assessment
http://www.apus.edu/Learning-Outcomes-Assessment/Initiatives/Rubrics-Program/RubricsProgram-Overview.htm
http://www4.nau.edu/assessment/assessment/liberal/documents/Oral_Comm_discussion_rubric.p
df
http://www.greenriver.edu/learningoutcomes/StudentLearningOutcomes.htm
http://pandora.cii.wwu.edu/cii/resources/writing/writing_rubric.asp
http://www.uncwil.edu/cas/documents/Elaboratedcompetencies3.pdf
http://www.mscd.edu/~ssac/SLO-IdentificationMatrix.pdf
http://www.rcampus.com/indexrubric.cfm
http://www.uwlax.edu/gened/GEAC%20Overview%20of%20Course%20Embedded%20Assess
ment%20Tasks%20&%20Rubrics.pdf
http://www4.nau.edu/assessment/assessment/liberal/documents/Effective_writing_rubrics.pdf
http://online.fresnocitycollege.edu/senate/curriculum/slo.html
http://www.cabrillo.edu/~tsmalley/learneroutcomes.html
http://pro.cabrillo.edu/slos/
http://www.montgomerycollege.edu/outcomes/documents/sloa_handbook.pdf
http://www.elcamino.edu/academics/slo/
http://columbia.yosemite.cc.ca.us/slo/slohome.htm
http://www.oxnardcollege.edu/faculty_staff/student_learning_outcomes.shtml
http://www.league.org/league/projects/lcp/lcp3/Learning_Outcomes.htm
http://www.contracosta.edu/Library/sites/slos.htm
http://www.aacu.org/documents/New_Leadership_Statement.pdf
http://www.laccd.edu/inst_effectiveness/Student_Learning/
http://www.cos.edu/view_page.asp?nodeid=3138&parentid=933&moduleid=5
230
http://www.laney.peralta.edu/apps/comm.asp?Q=31028
http://www.pima.edu/acad_services/slo/
http://www.uri.edu/assessment/media/public/page_files/uri/outcomes/student/outcomes/outcome
s_tools/SLO%20201%202.pdf
http://aslo.lbcc.edu/
http://pro.cabrillo.edu/slos/docs/SLO_InstructionalPlanningAssessmentProcess.pdf
http://css.rpgroup.org/ka.php?ka_id=7
http://research.crc.losrios.edu/Instructions%20for%20Writing%20Student%20Learning%20Outc
omes.htm
http://www.grossmont.edu/student_learning_outcomes/writing_slos.asp
http://www.lcc.hawaii.edu/userfiles/file/accreditation/presentations/retreat_06_3_dashboards.pdf
http://www.dvc.edu/org/departments/research/slo/
http://www.pasadena.edu/slo/
http://www.foothill.fhda.edu/schedule/learning_outcomes.php
http://www.merritt.edu/apps/comm.asp?$1=40770
http://www.mtsac.edu/instruction/outcomes/
http://www.goldenwestcollege.edu/slo/
http://www.compton.edu/studentservices/slo.aspx
http://www.deltacollege.edu/dept/dsps/StudentLearningOutcomes.html
http://4faculty.org/docs/slo_resources.htm
https://plan.elac.edu/public/Shared%20Documents/Student%20Learning%20Outcomes.aspx
http://www.uwlax.edu/gened/Outcomes.htm
http://cai.cc.ca.us/workshops/SLOFocusOnResults.doc
http://www.sloassessment.com/
231
Rubric Resources
http://www.rubrician.com/writing.htm
http://www.teach-nology.com/web_tools/rubrics/
http://www.shambles.net/pages/staff/rubrics/
http://www.scribd.com/doc/11541381/Persuasive-Writing-Rubric
http://www.teach-nology.com/web_tools/rubrics/persuade/
http://www.pz.harvard.edu/research/RubricsSelfPE.htm
http://www.google.com/#hl=en&q=problem+solving+rubric&revid=1640595612&ei=Yf8_SsLY
N4HKsQOwjM2VDw&sa=X&oi=revisions_inline&resnum=0&ct=broadrevision&cd=5&fp=cbonjxemLj4
http://www.schreyerinstitute.psu.edu/pdf/ProblemSolvingRubric1.pdf
http://www.nden.k12.wi.us/tlcf/prob3.htm
http://www.uen.org/Rubric/rubric.cgi?rubric_id=13
http://www.google.com/#hl=en&q=sentence+writing+rubric&revid=1640595612&ei=Yf8_SsL
YN4HKsQOwjM2VDw&sa=X&oi=revisions_inline&resnum=0&ct=broadrevision&cd=4&fp=cbonjxemLj4
http://www.education.vermont.gov/new/html/pgm_curriculum/literacy/writing/benchmarks.html
http://www.google.com/#hl=en&q=writing+conventions+rubric&revid=1640595612&ei=Yf8_S
sLYN4HKsQOwjM2VDw&sa=X&oi=revisions_inline&resnum=0&ct=broadrevision&cd=3&fp=cbonjxemLj4
http://www.neiu.edu/~neassess/gened.htm
http://www.englishcompanion.com/pdfDocs/foundationskills.pdf
232
Assessment
Quickies:
Student Learning
Outcome
Assessment
in Ten Easy Steps
233
ASSESSMENT QUICKIES:
STUDENT LEARNING OUTCOME ASSESSMENT IN TEN EASY STEPS
By
Michelle Saint-Germain, Director, Program Review and Assessment
California State University, Long Beach*
What is it?
Where is it?
What do I need?
How do I get it?
What do I get?
Can I share it?
Feedback?
File#
A set of ten podcasts on how to do assessment
Available (free!) on iTunesU (http://itunesu.csulb.edu/)
Download iTunes player on your computer or iPod
Find the podcasts and subscribe to all
Power-point slides and voice narration
Yes, it is open to anyone who wants to use it
Send questions/comments to [email protected]
Title
File Type
001
What Are Student Learning Outcomes?
MPEG-4 video file
001T
What Are Student Learning Outcomes?
Transcript (Word)
002
Writing Student Learning Outcome Statements
MPEG-4 video file
002T
Writing Student Learning Outcome
Transcript (Word)
003
Levels of Student Learning
MPEG-4 video file
003T
Levels of Student Learning
Transcript (Word)
004
Mapping Student Learning Outcomes to the Curriculum
MPEG-4 video file
004T
Mapping Student Learning Outcomes to the Curriculum
Transcript (Word)
005
Choosing Assessment Measures
MPEG-4 video file
005T
Choosing Assessment Measures
Transcript (Word)
006
Matching Assessment to Teaching and Learning
MPEG-4 video file
006T
Matching Assessment to Teaching and Learning
Transcript (Word)
007
Collecting Assessment Evidence of Student Learning
MPEG-4 video file
007T
Collecting Assessment Evidence of Student Learning
Transcript (Word)
008
Analyzing Evidence of Student Learning
MPEG-4 video file
008T
Analyzing Evidence of Student Learning
Transcript (Word)
009
Using Evidence of Student Learning for Program
Improvement
MPEG-4 video file
009T
Using Evidence of Student Learning for Program
Improvement
Transcript (Word)
010
Why Assess Student Learning?
MPEG-4 video file
010T
Why Assess Student Learning?
Transcript (Word)
*These podcasts reflect the opinions of the author only and do not constitute any official
policy or process of the California State University, Long Beach.
234
Notes
235
Notes
236
ARC
s p o n s o r e d by ac s c u i n c o l l a b o r at i o n w i t h acc j c
academic resource conference
2010
S U S TA I N A B I L I T Y: A V I S I O N
F O R H I G H E R E D U C AT I O N
April 21-23, 2010, The Westin Long Beach, CA