“Closing the Loop:”
Using Assessment Data for
Improvement
Margaret Kasimatis, PhD
Associate VP for Institutional Effectiveness
OVERVIEW
• Brief review of assessment process
• Barriers to closing the loop
• Steps for closing the loop
– Organizing your data
– Analyzing and interpreting data
– Linking results to courses, experiences,
practices
– Feeding results back into curricula, programs,
or pedagogies
ASSESSMENT PROCESS
IDENTIFY
SPECIFIC
OUTCOMES
DETERMINE
PRACTICES USED
TO ACHIEVE
OUTCOMES
GATHER
EVIDENCE
ARTICULATE
MISSION/
GOALS
RECOMMEND
ACTIONS
REVIEW &
INTERPRET
RESULTS
The goal is to understand and improve student learning.
Barriers to Closing the Loop
• Measures not originally designed with
specific outcomes in mind
• Practices, curricula, pedagogies not mapped
to outcomes
• Low self-efficacy regarding data analysis or
quantitative research
ORGANIZING DATA
• Assemble “ingredients:”
– All data or possible sources of data
– List of learning outcomes/research questions
– Curriculum, experiences, practices
• Determine level of analysis
– E.g., course-level or program-level
• Map data sources to outcomes
• Map results to outcomes
– Present in “user-friendly” way
Mapping Data Sources to Outcomes:
Engineering Example
ASSESSMENT MEASURES
LEARNING
OUTCOMES
Alumni
Survey
Ability to apply knowledge in math,
science, and engineering
National Survey of
Student Engagement
(NSSE)
Evaluation of
Clinic
Presentations
Evaluation of
First-Year
Presentations
X
Rubric Evaluation
of Upper-level
Coursework
X
Ability to design a system, component,
or process to meet desired needs
X
Ability to identify, formulate, and
solve engineering problems
X
X
Ability to communicate effectively
X
X
Ability to use techniques, skills, and
modern engineering tools necessary
for engineering practice
X
X
X
X
X
X
X
Ability to function on
multidisciplinary teams
X
X
Recognition of need for and ability
to engage in lifelong learning
X
X
Understanding of professional and
ethical responsibility
X
X
Broad education necessary to
understand the impact of engineering
solutions in a global & societal context
X
X
Map Results to Outcomes
• Options for how to do this:
– Map all results to outcomes in tabular form
– Present results one outcome at a time
• It’s not recommended to present results primarily by
measure
Presenting Results by Learning Outcome
Engineering Example
NSSE: Data on Communication Skills
HMC Senior Engineers
AITU Seniors
Liberal Arts Seniors
Mean Response (1-4 scale)
4
3
2
1
Experience contributed to ability
to write clearly and effectively
Experience contributed to ability
to speak clearly and effectively
Presenting Results by Learning Outcome
Engineering Example
Alumni Survey: Data on Communication Skills
Percent Indicating "Very Much"
60
Biology
Chemistry
Comp Sci
Engineering
Math
Physics
50
40
30
20
10
0
To what extent was an emphasis on
writing skills present while you were at
HMC?
To what extent was an emphasis on oral
communiction skills present while you
were at HMC?
Presenting Results by Learning Outcome
Engineering Example
Rubric Evaluation of Student Presentations
4
Delivery
Language
Organization
Content
3
2
1
First-Year Presentation
Clinic Presentation
Presenting Results by Learning Outcome
Engineering Example
Mean Rubric Score on Writing Style (1-5 scale)
Rubric Evaluation of Student Work: Writing Style
5
4
3
2
1
Course #1
Course #2
INTERPRETING DATA
• What type of criterion?
• What is “significant?”
• Are the findings reliable/valid?
INTERPRETING DATA
• What type of criterion?
– Norm-referenced
• Need appropriate comparison group
– Avoid percentile rankings
• Good for more nebulous findings
– Absolute standard
• Usually more appropriate for performancebase measures
INTERPRETING DATA
• What is “significant?”
– Important to test when making comparisons
• Numbers that look different may not really be
different
– Just because a difference is significant
doesn’t mean it’s important
INTERPRETING DATA
• Are the findings reliable/valid?
– Can we believe student self-reports?
– Are standardized measures more valid?
– Single measure vs. multiple measures
LINKING RESULTS TO
PRACTICES
• Where in the curriculum are
outcomes addressed?
– Complete curriculum map, or
– Inventory for particular outcome
• At what level?
– E.g., introduce, reinforce, emphasize
• How? What practices/techniques
are used?
Linking Results to Practices:
Engineering Example
CURRICULUM
Core
COURSE
SIGNIFICANT WRITING
EXPERIENCES
Humanities 1 & 2
Short Research Papers – feedback on drafts
Integrative Experience
Research Paper
Humanities or Social Science Seminar
Research Paper
E4: Introduction to Engineering Design
Short Report
E8: Engineering Representation and Realization
E59: Introductions to Engineering Systems
E80: Experimental Engineering
E82: Chemical, Thermal Processes
Engineering
Major
E83: Continuum Mechanics
E84: Electronic & Magnetic Circuits & Devices
E85: Digital Electronics
E101/102: Advanced Systems Engineering
E106: Materials Engineering
E111/112/113: Engineering Clinic
Engineering Seminar
Proposal, Mid-year Report, Final Clinic
Report (team written)
CLOSING THE LOOP
• Where are the “gaps?”
– Either in curriculum or in pedagogy
• What can be changed?
– In the short-term vs. the long-term
• Think small at first (“low-hanging fruit”)
– Adding vs. modifying
• Be sure to document changes
– And plan on measuring effectiveness of change
QUESTIONS?