Higher learning skills,
good feedback
EASiHE Workshop
10th February 2010
Introduction
• Welcome
• Introductions
• Overview of the day
2
Competency Modelling
Supporting Higher-Level
Outcomes and Better Feedback
Lester Gilbert
Learning Societies Lab
University of Southampton
Competence
Discriminate learning & teaching
Teaching
Learning
Technology support
The big picture
Competence hierarchy
Higher to
lower
capabilities
Merrill
Bloom
Gagne
Find
Synthesis
Problem
solving
Analysis,
Evaluation
Use
Recall
Prerequisite
competencies
Common
enabling
competencies
Indefinite
depth
Rules
Application
Comprehension
Concepts
Knowledge
Discriminations
Competency tree
C
K
J
A
I
Prerequisites
Linked, but no
implied sequence
B
G
H
D
E
Comprises
F
A competency may
itself be a tree
Common
Example nursing competencies
Competency No.
C00
C10
C11
C12
C20
C21
C22
C23
C24
Capability
Use
Apply
Demonstrate
Identify
Use
Evaluate
Identify
Recognize
Recall
Subject Matter Content
Ethical principles
Ethical issues
Client confidentiality
Ethical Issues
Professional regulation
Professional regulation
Limitation in own practice
Need for referral
Professional regulations
Sitthisak, Gilbert, & Davis (2007)
Attitude
Values
Actively
Respectfully
Sensitively
Actively
Considerately
Values
Willingly
Willingly
Example competency 3D tree
Sitthisak, Gilbert, & Davis (2007)
Questions, comments,
discussion…
Gilbert, L., & Gale, V. (2008).
Principles of E-learning Systems Engineering.
Chandos Publishing. ISBN 1-84334-290-1
HLOs (Higher-order
Learning Outcomes)
Bill Warburton
EASiHE/iSolutions
A formative taxonomy
• For FORMATIVE evaluation, Bloom et al. (1971) distinguished a
hierarchy of levels of behaviour parallel to the cognitive taxonomy
• Added value to the cognitive taxonomy by making it easier for teachers
to identify ‘mal-rules’ – flaws in reasoning
ABILITY TO MAKE APPLICATIONS
ABILITY TO MAKE TRANSLATIONS
SKILL IN USING PROCESSES & PROCEDURES
KNOWLEDGE OF RULES & PRINCIPLES
Increasing level
of abstraction
KNOWLEDGE OF FACTS
KNOWLEDGE OF TERMS
16
Testing KNOWLEDGE OF FACTS
(Bloom 1956: KNOWLEDGE RECALL)
– Memorise something
– Recognise something
– Recall of information
or identity:
• Who, what, when,
where, how ...?
• Describe…
(School of Oceanography
and Earth Science,
University of Southampton)
17
KNOWLEDGE OF RULES & PRINCIPLES
– Memorise and recall a general rule
• What, when, where, how ...?
• Describe…
– Describe interrelationships among many items
– Memorise and recall applications of a rule
– Memorise and recall exceptions to a rule
– Does *not* deal with Application of a rule
18
Testing KNOWLEDGE OF RULES & PRINCIPLES
(Bloom 1956: COMPREHENSION)
Example:
In order to write a chemical formula, you have to know
a. only the symbols of the elements that are in the compound
b. only the proportions in which the atoms of elements
combine
c. both the symbols of the elements that are in the compound
and the proportions in which the atoms of elements
combine
d. the atomic weight of the elements that form the compound
(Bloom et al. 1971)
19
Skill in using Processes & Procedures
• Steps along the route to Mastery
• The difference between ‘Knowing That’ and ‘Knowhow’?
• Accuracy in use
• Practice makes perfect
• Expression of (justified) confidence
• Not formally represented in Bloom’s Taxonomy of the
Cognitive domain but important for assessing Mastery
• Formative drills
20
Skill in using Processes & Procedures
Example of process-based item (JISC/REAQ Report,2009)
21
ABILITY TO MAKE TRANSLATIONS
• Put idea in own words or use new examples of what is learned
• Transform a term, fact, rule, principle, process or procedure
from one form to another
• Take a phenomenon presented in one mode/form and
represent it by an equivalent form/mode
• Move from a verbal to a symbolic form
• Determine when a new illustration is appropriate or not
• Move from a concrete to a more abstract form, or from a
general to a more specific illustration, and vice versa.
22
Testing ABILITY TO MAKE TRANSLATIONS
(Bloom 1956: COMPREHENSION)
Example: testing a participant’s
ability to
(i) Interpret (native
Spanish speaker) or
(ii) Translate (second
language speaker)
parts of speech from
English to Spanish
(School of Humanities,
University of
Southampton)
23
ABILITY TO MAKE APPLICATIONS
• Recognize the essentials of the problem
• Use of a rule/principle learned in one context to solve a
problem presented in a new or unfamiliar context
• Identify rules/principles/generalisations relevant to a problem
• Use ideas to solve a problem which is different from those
previously encountered in the instruction or instructional
materials.
• Most complex of the Formative categories - depends on other
classifications but requires application of ideas in new
situations or problems
24
Testing ABILITY TO MAKE APPLICATIONS
(Bloom 1956: APPLICATION)
Example: testing a participant’s ability to
(i) solve problems (in this case, using Wien’s Law) …
(ii) use facts, rules and principles
(School of Physics and Astronomy, University of Southampton)
25
Testing ANALYTICAL skills
• Subdivide something to show how it is put together
• Find the underlying structure of a communication
• Identify motives
• Separate something into component parts:
• What are the parts or features of...?
• Classify...according to...
• Outline/diagram...
• How does...compare/contrast with...?
• What evidence can you list for...?
26
26
Testing ANALYTICAL skills
This example tests a participant’s ability classify data
according to specific criteria:
Q. Which countries' statistics are being reported in A, B and C?
GNP growth
p.c. p.a. ‘80-’91
Pop. growth
80-91
Total employment 1980-85 (%)
Agriculture
Industry
Services
A
GNP per
capita 1991 ($
USA)
500
2,5%
1,5%
51
20
29
B
1570
5,8%
1,6%
74
8
8
S.A.
2560
0,7%
2,5%
17
36
36
C
25110
1,7%
0,3%
6
32
32
1. A=South Korea; B=Kenya; C=Canada
3. A=Sri Lanka; B=Thailand; C=Sweden*
2. A=Sri Lanka; B=Germany; C=Thailand
4. A=Namibia; B=Portugal; C=Botswana
(CASTLE project, University of Leicester)
27
Testing SYNTHETIC skills
– Create original products in verbal or physical form
– Combine ideas to form a new whole:
•
•
•
•
•
What would you predict/infer from...?
What ideas can you add to...?
How would you create/design a new...?
What might happen if you combined...?
What solutions would you suggest for...?
28
28
Testing SYNTHETIC skills
Example: testing a participant’s ability to predict/infer the 3D
appearance of a 2D net. Requires the abstract abilities to
accurately reconstruct solids, rotate them about three axes and
combine the results with a predicted model.
Q. The picture shows a cube that I have made. Which
one of the shapes below, if cut out and folded, could
make a cube the same as mine?
A*
B
C
D
E
(Thinking Skills Admission Tests, University of Cambridge)
29
Testing EVALUATION skills
– Make value decisions about issues
– Resolve controversies or differences of opinion
– Develop opinions, judgements or decisions:
•
•
•
•
•
Do you agree that...?
What do you think about...?
Place the following in order of priority...
What criteria would you use to assess...?
What are the most important aspects of...?
30
Testing Evaluation skills
Example: testing a participant’s ability to evaluate the link between cause
and effect in terms of predefined criteria
"The United States took part in the Gulf War against Iraq BECAUSE of the
lack of civil liberties imposed on the Kurds by Saddam Hussein's regime."
a.
The assertion and the reason are both correct, and the reason is valid
b.
The assertion and the reason are both correct, but the reason is invalid*
c.
The assertion is correct but the reason is incorrect
d.
The assertion is incorrect but the reason is correct
e.
Both the assertion and the reason are incorrect.
(CASTLE toolkit, University of Leicester)
31
Mastering the art of item authoring
Three principle challenges for new authors:
• How to reducing the scope for guessing
• How to testing ILOs at higher levels of abstraction
• How to varying the difficulty of items
32
Effective questions
and feedback
Veronica Gale
EASiHE/Independent Consultant
HLO questions
• Write questions and feedback after finalising the ILO
• Do students have enough information to be able to answer
the question correctly?
• Is the question focused on one main learning point?
• If there is a score, is it clear to students how the scoring
works?
• If there is a ‘correct answer’ is this the only one that can be
considered correct?
• Does the question use terminology that is suitable for this
stage in the course?
• Have you piloted the assessment?
34
Example question from Bloom
FACT: The number of shareholders in most large corporations
has increased considerably during the last 30 years.
CONCLUSION: Control of corporations has become more
democratic in the last 30 years.
A: The fact is good evidence to support the conclusion
B: The fact is good evidence to disprove the conclusion
C: Neither A nor B applies clearly
To help them learn, what do the students who select A
need in their feedback?
35
Effective feedback
• Whether the response is correct or not
• If the student is incorrect: an explanation of why, what the
correct answer is and why this is correct
• If the student is correct: confirmation of why this is correct
• Recommended action e.g., reference back to the learning
materials, remedial tuition, further questions for practice
All to be:
• Presented immediately
• With the question, their response and the feedback visible
at the same time
36
When there are no ‘right’ and ‘wrong’ answers
• Scoring may be inappropriate
• Discuss the merits and drawbacks of the different options
• Give an expert’s opinion on the best choice, with full
explanation
• Consider a model answer, range of sample answers or
demonstration
• Ask a follow-up question on the rationale for the response
37
Visual feedback
38
Activities
Group discussions
Activity One
Review the questions provided and identify:
• what works well and
• what needs to be improved
40
Activity Two
Drawing on your own experiences and thinking
about the discussions in the workshop so far,
create a poster listing:
• what to do, and
• what to avoid
when writing e-assessments and feedback for
higher learning outcomes.
Your readership are academics creating new
formative questions.
41
Lessons learned from
project managing high
volume e-assessment
David Bacigalupo
EASiHE/ECS
Conclusion and next
steps
Lester Gilbert
EASiHE/ECS