Evaluation for Learning
Paul Ramsden
University of Sydney
The problem
To evaluate university teaching
effectively, we should apply
the principles of good
practice in assessing student
learning.
Sounds elementary...
“Chapter 2 has demonstrated that even the best of
current practices are by and large not good
practice...”
Lewis Elton and Brenda Johnston
‘Assessment in universities: a critical review’
(http://www.ltsn.ac.uk/genericcentre/docs/Critical%20review%20of
%20assessment %20research.rtf)
67
Teaching is good
66
38
Staff available to
discuss work
45
1999
1994
25
Helpful feedback
28
Source:
McInnis et al, 2000
“There are no practice exercises – every piece of work is assessed,
which tends to focus me on attaining marks rather than exploring
ideas”
“Assessment and marking have not been good. It’s quite subjective,
and group work assignments are sometimes unfair in their assessment
criteria (for example, everyone gets the same mark even though one
person may do less work than another)”
“The amount of multiple choice questions in some subjects does not
provide opportunities to show how much you have learnt and
understood.”
“We need more feedback during semester. It is not reasonable to get a
mark at the end of the year and have no idea what it is based on. It is
also not ideal to front up to an exam not knowing how well we have
been going in earlier assessments.”
Bad practice remains common

Multiple purposes, same techniques

No formative assessment at all

No match to learning objectives

Feedback too little, too late

Narrow range of methods unreflectively chosen

No student choice

Incorrect use of group assessment

Ignorance of Heisenberg!

Standards assumed to depend on norm referencing
Bad practice remains common

Interaction reliability x validity not understood

Unreliable assessments weighted less

Limited application of grade descriptors

Use of MCQs without professional training

Plagiarism opportunities not designed out – focus on
technical fixes
Underlying all this is...




a focus on the producer’s concerns
a continuing emphasis on teaching (rather than learning)
a remarkable neglect of existing evidence
(people seem to prefer ‘dissemination and projects’
despite overwhelming proof that it doesn’t work)
and a tendency to blame the student
We should try adapting these conclusions to the evaluation
(and management) of university teaching.
THEORY 1
THEORY 2
THEORY 3
Teaching as telling
Teaching as
orga nising
Teaching as making
learnin g possibl e
FOCUS
Teache r and
content
Teaching techn iques
that will result in
learning
Relation between
students and subj ect
matter
STRATEGY
Transmit
information
Manage t eaching
process; transmit
conc epts
Engage ; challenge ;
im agine onese lf as the
student
ACTIONS
Chiefl y
presentation
‘Active le arning’;
organ ising activit y
Systematically
adapted to suit stud ent
unde rstand ing
REFLECTION
Unreflective;
taken for gran ted
App ly skill s to
im prove teaching
Teaching as a
research-li ke,
scholarly process
Sydney 1999-2003
Assignment:
Design a system to evaluate teaching which
leads to a better student experience and
improved learning outcomes.
… and make it work.
The response
1.
Break the problem into manageable parts
2.
Design systems that resonate with values
and leverage strengths
3.
Align evaluation with outcomes
4.
Benchmark good practice
5.
Test the impact against the evidence
The response
1.
Break the problem into manageable parts
2.
Design systems that resonate with values
and leverage strengths
3.
Align evaluation with outcomes
4.
Benchmark good practice
5.
Test the impact against the evidence
Break the problem into manageable parts

Make goals, methods, measurement and outcomes
cohere ( aka ‘alignment’)

Recognise importance of perceptions/ theories in use

Design the system around the culture (avoid onesize-fits-all solutions)

Learn from others’ mistakes
The response
1.
Break the problem into manageable parts
2.
Use strategies that resonate with values
and leverage strengths
3.
Align evaluation with outcomes
4.
Benchmark good practice
5.
Test the impact against the evidence
Principles
• Adopt a single, but flexible, SAL perspective, derived
from research evidence.
Use it to inform every policy and process
• Plan for coherence between collegial and managerial
strategies
• Use an evidence-based approach to change and
leadership, aligned with academic values
Mechanisms to leverage strengths
• Academic Board reviews
• Rigorous, peer review-driven QA process
•
Manage teaching proactively
• Funding drivers aligned with research measures and
national indicators
• Plans that work
• New role for academic development unit
• Strategic projects
Management of teaching
• Teaching Dividend
currently $4.5M
• Scholarship Index
c. $650,000
• Teaching Improvement Fund
$1.3M
• Required T&L plans
– Annually updated operational plans
– Assess progress against targets
– Condition of access to performance-based funds
– Interrogated in Academic Board reviews
Criterion
Description of Indicators
Weight
Student progress
Unit of study pass rate
2
First to second
year retention
Percentage of students still enrolled after
one year
2
Graduate
experiences
Scores on three measures: good teaching,
generic skills, overall satisfaction
3x1
Student
experiences
Scores on three measures: good teaching,
generic skills, overall satisfaction (annual
student course experience questionnaire)
3x2
Employability
First destinations survey employment rate
2
Further study
First destinations survey rate of further study
2
Strategic projects
• Re-engineered academic development unit
• First year experience: learning community
• Expansion of training opportunities
• including mandatory 21 hour training
• Research-led teaching, including PIs
• Evaluation and QA working group
• Graduate attributes for a research university
• Research supervision initiatives
The response
1.
Break the problem into manageable parts
2.
Use strategies that resonate with values
and leverage strengths
3.
Align evaluation with outcomes
4.
Benchmark good practice
5.
Test the impact against the evidence
Align evaluation with desired outcomes
• Rewards and recognition at multiple levels
• SI rewards trained staff and scholarly outputs
• Array of student-focused evaluation instruments,
consistent with SAL theory
• New teaching awards
• Performance linked to funding at Faculty level
• Material support for changes
• New promotions policy
Evidence-based academic promotions criteria
1. Fundamentals
2. Criteria
Performance
Research-led
Student-focused
Scholarship
Leadership
3. Evidence
Fundamentals
Fundamental things have got to be
simple… we must look for simplicity
in the system first.
Ernest Rutherford
Fundamentals

Interest and explanation

Respect for students

Appropriate assessment

Clear goals and challenge

Independence: student control

Learning from students
(Learning to Teach in Higher Education, Chapter 6)
Even more fundamental...

Positive attitude towards students

Ability to communicate well

Lively interest in improving teaching
And plainer still...
The aim of teaching is simple: it is to make
student learning possible.
Performance
A lecturer should appear easy and
collected, undaunted and unconcerned,
his thoughts about him and his mind
clear for the contemplation and
description of his subject … His whole
behaviour should evince a respect for
his audience
Michael Faraday
Performance

Planning (e.g. effective subject design, clear objectives)

Process (e.g. presentation technique, WebCT design)

Assessment (e.g. use of variety of appropriate methods)

Outcomes (some evidence of link to learning)

Evaluation (some evidence of use of evaluation to improve)
Research-led teaching
This atmosphere of excitement,
arising from imaginative
consideration of knowledge,
transforms knowledge.
A. N. Whitehead
Research-led teaching

Imagination and enthusiasm: a shared journey to
understanding rather than delivery of content

Effective design of curricula to engage students in
inquiry

Materials make use of primary sources, recent
discoveries, progress in field
(“If you can’t explain it to the charlady, you don’t know
anything about it”)
Student-focused teaching
The two secrets of lecturing from
which everything else follows: first,
to believe that you have something
worth telling your audience; second,
to imagine yourself as one of that
audience.
R.V. Jones
Student-focused teaching

Use of evaluation evidence to redesign curriculum

Use of assessment data to modify teaching strategy

Focus on relation between students and subject matter

Choice of technique reflects level of student knowledge
(From “Did I make the goals clear?” to “Are the goals clear to
the students?”)
Scholarship in teaching
What is needed is for teachers in
higher education to bring to their
teaching activities the same critical,
doubting and creative attitude which
they bring habitually to their
research activities.
Lewis Elton
Scholarship in teaching

Systematic use of best available evidence to
select and deploy teaching and assessment strategies

Publication of refereed journal articles on
university teaching in discipline

Invitations to address international conferences
on university teaching
Leadership in teaching
She successfully inspired us to
transform the course and to re-focus
on our students. She melded a
diverse group of academics into a
team of great teachers.
A lecturer
Leadership in teaching

Policy development and implementation

Successful re-design and coordination of
courses; team leadership in teaching

Mentoring of junior academics as teachers
 Application
of teaching strategies and curriculum
designs in other institutions

Coordination of benchmarking activity with other
universities
Criteria are hierarchically ordered...
Non-negotiable basis:
Performance
Second level:
Research-led
Third level:
Student-focused
Fourth level:
Scholarship
Fifth level:
Leadership
… leading to a structure that can be mapped on to
promotion at different levels.
And the evidence?

Are the basics in place?

Use multiple sources (never rely on student
evaluations alone)

Evaluate teaching like research
– Use peer review if possible
– Use hard data when available (e.g. S of T publications)

Do the different sources tell a similar story?

Do the claims made by the teacher match the
evidence?
Required
Administrative
Academic
Recognised and encouraged
Required
Fundamental values:
Administrative
• research intensive
• academic-led
• self-regulation
• evidence base
• international
referencing
• focus on student
experience
Academic
Recognised and encouraged
Required
Surveys
TPIs and performance
funding
External QA benchmarks
Required training in teaching
Teaching & Learning Plans
Administrative
Academic Board reviews
(self-evaluation, visit,
report)
Policies on teaching:
evaluation,
assessment,
ICT QA, promotions
Academic
University teaching
awards
Supervision awards
Teaching Improvement
Fund
Scholarship Index
Research-led teaching
(policy and indicators)
‘Guidelines for Good
Practice’ (teaching,
learning with ICT);
ITL courses and support
groups: quality,
graduate attributes,
first year experience,
research-led teaching
Recognised and encouraged
The response
1.
Identify the problem
2.
Use strategies that resonate with values
and leverage strengths
3.
Align evaluation with outcomes
4.
Benchmark good practice
5.
Test the impact against the evidence
Benchmarks
Oxford
Student surveys (SCEQ), QA policies
Lund
QA policies
UCL
QA policies
OU
ICT evaluation and QA
ANU
ICT in T&L for research universities
Monash
Research-led teaching PIs
Queensland
Student surveys (SCEQ)
Hong Kong
Academic development standards
The response
1.
Identify the problem
2.
Use strategies that resonate with values
and leverage strengths
3.
Align evaluation with outcomes
4.
Benchmark good practice
5.
Test the impact against the evidence
Conventional vs. Innovative
60
Old
New (SMP)
50
40
30
20
10
0
-10
-20
-30
Good
Teaching
Generic
Skills
Assessment
Satisfaction
with Support
Overall
Satisfaction
Demand indicator for high quality students
(percentage of offers to students with UAIs 95 or greater
1999-2003)
40
35
30
1999
25
2000
20
2001
2002
15
2003
10
5
0
Sydney
Uni 1
Uni 2
Uni 3
Others
Changes in the Sydney first year experience, 1999-2002
Teaching staff give helpful feedback
Teamwork skills developed
Motivated to do best work
More confident to tackle new problems
IT supports my learning
Problem solving skills developed
Feel part of a learning community
Satisfied with dept/faculty admin.
Overall course satisfaction
Overall satisfaction (services & admin)
Percentage agreements, annual survey
1999
2002
% change
39
48
38
47
56
52
39
51
66
57
48
58
48
54
64
58
53
68
71
65
+9
+10
+10
+7
+8
+6
+14
+17
+5
+8
70
60
50
40
30
20
10
0
1999
Helpful feedback
2000
2001
Feedback only marks
2002
All you need is memory
70
60
50
40
30
20
10
0
1999
Satisfied with faculty admin.
Teamwork skills
2000
2001
Overall satisfaction
Part of a learning community
2002
IT supports learning
Oxford vs. Sydney, 3rd year Undergraduates
(broad agreement)
10 0
90
80
70
60
Sydney
Oxford
50
40
30
20
10
0
Good
Te ach ing
Clea r Goals
Ap propri ate
As sess ment
Ap propri ate
Wo rklo ad
Gene ric Skil ls
Overall
Satisfa cti on
Obstacles to an evidence-based approach

Rationality in an audit society (Smith, J.R.Statist.Soc. 1996)

EBM is an example of a very uncommon
phenomenon

Motivation – what’s in it for me?

Focus on teaching and methods

Antediluvian attitude to ‘staff development’
– yet more projects, ‘action research’, ‘dissemination’,
‘skills’, centralist policies …. itself not evidence-based

Management inadequacies
Advice to the National Institute for L & T
 ‘Staff development & dissemination’ is the front
end only: management & structures are required
 Design for your users: focus on learning; have a
bold vision; capture academic imaginations
through trust and credible leadership
 Celebrate diversity in the sector: be inclusive
 Operate at multiple levels: use a systems
perspective
 Forget about “accreditation”
 Make it easy to share good practice
 Don’t become an arm of the audit society
Further reading
(1)
Management of T&L, teaching quality
Paul Ramsden
Learning to Teach in Higher Education
Second Edition 2003 Foreword by Sir David Watson
London: RoutledgeFalmer
– the classic text fully revised and updated
(2)
Practical advice for heads
Paul Ramsden
Learning to Lead in Higher Education 1998
London: RoutledgeFalmer
www.routledgefalmer.com