TDU Talk
ISSUE 3 ▪ APRIL 2010
REFLECTIVE PRACTICE
Reflecting on Your Place in Teaching and Learning
Dr Trudy Harris, Appraisals Administrator, Teaching Development Unit, University of Waikato
The WMS Pilot Project
Dr Trudy Harris, Appraisals Administrator, Teaching Development Unit, University of Waikato
Closing the Feedback Loop
Robina Ann Cummins, Centre for Sport and Exercise Science, The Waikato Polytechnic
Different Ways of Recording Reflection
Dr Pip Bruce Ferguson, Teaching Developer, Teaching Development Unit, University of Waikato
A Personal Journey: Introducing Reflective Practice into Pre-service
Teacher Education to Improve Outcomes for Students
Anne Hume, Faculty of Education, University of Waikato
Developing Students’ Skills in Reflective Practice: Design and
Assessment
Kathryn Pavlovich, Eva Collins & Glyndwr Jones, Waikato Management School, University of Waikato
“
Staff will be aware that the Minister for Tertiary Education,
Steven Joyce, recently stated that government was going to
shift at least some of its funding to tertiary institutions to a
‗payment on retention and completion of qualification‘ basis
rather than payment by enrolments. The impact of this on staff
could be considerable, and is likely to result in a greater emphasis
on the quality of teaching as a way of attracting students to our
university in the first place, and supporting them through to
graduation.
It‘s a significant shift in government practice and its impacts may be widespread.
In this issue of TDU Talk, we look at how staff might reflect on various aspects of
their practice, using the reflective practitioner model advocated by Laurence Kolb,
and advanced by Donald Schön in his book, ―The Reflective Teacher‖. However,
building on the work of both those authors, we advocate reflection at the
institutional as well as the classroom level. Accordingly, the issue commences
with an article by Dr Trudy Harris on the importance of reflecting on how staff ‗fit
in‘ to the processes and policies of the university. In this article, Trudy presents
both verbally and diagrammatically how institutional policies and processes
impact upon staff, students and the reputation of the university. With our Cycle 4
audit coming up, we need to recognize how and why certain policies and
processes are required by the university, and how these contribute to a better
teaching, learning and assessment experience for our students. Trudy‘s article
encourages staff to see how important their actions are in contributing to this
experience, supported by and supporting the institution‘s quality assurance
processes.
ā
APRIL 2010 • TDU TALK
In a second article, Trudy examines a recent trial of appraisal processes within a
School, and reflects on the effectiveness of this trial for staff within the School
and the changes that have been made as a result of their feedback. Following
Trudy‘s work, we have included a paper written by a Wintec Sports Science
lecturer, Robina Cummins, who was moved, through involvement in professional
development programmes, to investigate how her appraisal results might be
improved. She took seriously the comments made by students in her appraisal that
suggested a couple of areas in which she might better meet their needs, and
systematically worked to improve in those areas. Along with being a very
competent and published writer in her discipline area, she had a paper accepted for
an educational conference at which she discussed the intervention she had used
with her students. In this way, she was able both to improve her practice and to
meet PBRF publication requirements.
•2•
Moving on to other uses of reflective practice, we refer you to ways in which
authors reflect on how educators can use (a) memos and (b) a journal, to capture
their thoughts about how to improve their practice. Staff who attended the
WCELFest in 2009 may have heard keynote presenter Lorraine Stefani talk of
how she reframed the notion of ‗written reflection‘ to encourage engineers to
capture their reflections by calling the process ‗project notes‘, a term with which
they were familiar. In the two papers cited in this section, Dr Shankar Sankaran
uses ‗Memos to Myself‘ to show how he worked to extend and consolidate his
PhD study and educational practice. Shankar‘s work is compatible with his
experience as a business lecturer, using the methods with which he is familiar.
Emeritus Professor David Boud includes the assessment of reflective journals in
his paper on why, how and for whom journals may be written, published in New
Directions for Adult and Continuing Education journal.
Finally, the magazine concludes by presenting the work of Waikato University
lecturers on ways that they have used reflective journaling with their students. We
cite the abstract for a paper by Anne Hume from the Faculty of Education, which
discusses the reflection processes that she and her students used, as she worked to
support education students to become better reflectors. Finally, the paper by
Katherine Pavlovich, Eva Collins and Glyndwr Jones from Waikato Management
School presents evidence on how they used learning journals as a tool to develop
self- awareness within a business context, in their students. Both papers have been
published and will have boosted the authors‘ PBRF ratings alongside improving
their teaching practice.
We hope that you enjoy this edition of TDU Talk, and that you will think of ways
of incorporating reflection in and on practice in your own work, and to consider
publishing this in either disciplinary education-based, or more general education,
journals.
All the best
”
Pip Bruce Ferguson, Trudy Harris, Dorothy Spiller, Preetha Pratapsingh and Shant Lochan
Teaching Development Unit
•3•
APRIL 2010 • TDU TALK
Dr Trudy Harris, Appraisals Administrator, Teaching Development Unit
The last two years have seen the development of a number of strategic documents
at the University of Waikato related to teaching and learning. These documents
have been developed for the University through the Teaching Quality Committee
(in consultation with academic staff) and then communicated to teachers via
Boards of Study (BOS). However, communication of these documents through
this process can lead to confusion and the context in which they were developed
can be lost. Staff may be left wondering how these documents relate to them.
This article looks at a number of documents relating to teaching and learning that
have recently been developed and communicated in this way, and re-focuses the
attention on the individual teacher. This will be achieved by showing the
interconnectedness of some of the strategic documents, especially in the area of
appraisals and the evaluation of teaching and papers.
Before we look at the strategic documents this is perhaps a good place to provide
definitions of appraisal and evaluation. Many people use these terms
synonymously but they have very specific meanings:
Appraisal – the act of measuring teaching and paper quality
Evaluation – the use of information gained through appraisal and formative
feedback to provide continuous improvement in the quality of teaching and
papers.
While these terms are closely connected, they have very different outcomes and
uses. Appraisal data is used for audit and promotion purposes, while evaluation
provides evidence and follow up processes that allow for continuous improvement
and consequently the development of teaching and papers. Many people believe
that the areas of audit and development are mutually exclusive. However a
balance can be achieved (see the work of Carmel McNaught, 2003) that will allow
for the collection of data for both quality and developmental needs. In fact, a
number of studies have shown that an improvement in teaching quality is not
likely to happen when using appraisal data alone (Edstrom 2008; Kember, Leung
and Kwan 2002). For improvement in teaching quality, a form of intervention has
to occur (Piccinin, Cristi and McCoy 1999). Intervention in this case can be talks
with colleagues around teaching; implementing changes to your own practice (see
Cummins, 2000, in this edition); visits to the TDU or the reading of literature
around teaching. But do not believe that there are no development benefits from
the appraisal system because it too can be part of the evaluation process. However,
its summative nature means that it is better used as an indicator of levels of
teaching and paper quality. Therefore, its use at the end of the semester should be
the final act of evaluation rather than the only method as is currently the case.
APRIL 2010 • TDU TALK
•4•
The Teaching and Learning Plan 2010-2013
So how do we use formative evaluation? The development of a framework for
formative evaluation has been supported through a number of key University
documents, so this is a good point to return to the Teaching and Learning Plan.
Within the Teaching and Learning Plan there are a number of goals which relate
directly to the evaluation of teaching and papers. For example:
Goal 1: The University demonstrates that it values and supports effective
teaching and supervision
Action 1.2: ensure that the importance, effectiveness and ongoing development
of teaching and supervision is recognised in academic position descriptions, staff
recruitment, induction, professional goal setting, professional development
provision and academic promotions processes and reinforce the expectation that
academic staff maintain evidence of effective teaching and postgraduate
supervision.
Goal 2: Effective teaching and supervision are consistently demonstrated
Action 2.1: ensure that all departments comply with the University policy in the
Evaluation of Teaching and Papers and use student evaluations and other
summative and formative approaches in developing papers, teaching and
postgraduate supervision.
It will be helpful if you can note and remember the highlighted text in the goals
above. We will be revisiting these phrases in later strategic documents.
Through the Teaching and Learning plan, we see that there is now support and a
requirement for formative evaluation of teaching. But what does this mean for the
individual teacher, i.e. you?
The Policy for the Evaluation of Teaching and Papers
Within the Teaching and Plan are embedded a number of policies which provide
the supports for the framework around teaching and learning. In 2008 the
Academic Board approved the Policy for the Evaluation of Teaching and Papers.
Full implementation of the policy occurs in 2010. The Policy emphasises the goals
of the University with regard to excellence in teaching, and provides the support
for the framework around formative evaluation. The policy document outlines the
responsibilities for both the CODs and teaching staff. Listed below are the
responsibilities for teaching staff:
•5•
APRIL 2010 • TDU TALK
Teaching and papers are to be evaluated every two calendar years.
The evaluation must be supported by evidence from one or more
methods, one of which must be Teaching and Paper appraisal.
Maintain a portfolio of evidence of teaching and paper evaluations
for Professional Goal Setting (PGS) and Academic Promotion
processes.
Again we see similar wording to the Teaching and Learning Plan, but now we see
the emergence of the portfolio and the mechanism for ensuring improvement and
development of teaching and papers. So how can we do this?
A possible model
As already mentioned, evaluation leads to the continuous improvement and
development of teaching through a process of collecting evidence around an
activity, assessment or lecture. The methods used for the collection of this
evidence are varied. The choice of method will depend on a number of factors
such as the intended outcomes; the timing/resources available for collecting the
information; and the fit with your activity/discipline. However, there are a number
of excellent resources available, which outline some of the better-known methods,
see for example Angelo and Cross (1993) and George and Cowan (1999).
From conversations with staff around formative evaluation, it is reassuring that
staff use a number of different evaluative techniques within their teaching i.e.
reflective journals, and regular anonymous feedback. If you already do this, you
need to keep a record of the student feedback, but most importantly record the
process you use to act on the information that you receive.
Therefore, we can better define evaluation as:
… the continuous improvement and development of teaching through a
process of collecting evidence around an activity, assessment or lecture
and then acting on the evidence to improve and develop.
Another outcome for this approach to evaluation is that immediate action allows
for greater engagement with your students and the learning process.
APRIL 2010 • TDU TALK
•6•
So to a model…
What we have outlined is a very simple continuous improvement model as
described by the PIRI model shown in Figure 1.
Figure 1 The PIRI Continuous
Improvement Cycle
The PIRI (Plan, Implement, Review, Improve) continuous
improvement cycle can be used across all aspects of an individual‘s
academic role, within a team environment and at a Department,
Faculty or University level to ensure effectiveness and ongoing
development. Note that this is consistent with Kolb‘s learning cycle
of active experimentation, concrete experience, reflective
observation and abstract conceptualisation as used in action
research and is therefore an effective formative evaluation and
development mechanism.
(9-11 June 2010) Singapore.
Register now for Temasek Polytechnic's
International Conference on Learning and
Teaching (9-11 June 2010) to be held in Singapore.
The theme is Creative Pedagogies. Changing
Perspectives. Crossing Boundaries. The conference
is held in conjunction with Temasek Polytechnic's
20th anniversary.
Please visit the conference website for more
details.
http://www.tpconference2010.com
Keynote speakers: David Boud, Diana Laurillard,
Stephen Brookfield, Gary Poole
•7•
So let us collect all the pieces together and look at how this could
work over a semester of teaching. Using the appraisal comments
from the last occurrence of a hypothetical paper, the teacher decides
to re-design the paper and its assessments (see Cummins, 2000, in
this edition). To ensure that the assessments are delivering on the
outcomes required, a system of formative evaluation was embedded
into the course design as part of the assessment exercises. The
evaluation method chosen identifies issues with student
understanding of the requirements around the assessment. The
teacher responds to the feedback provided and allows extra time to
explain in more depth the requirements of the assessment. Using this
methodology throughout the assessment process provides a greater
level of engagement with the students and their learning process.
This engagement with the students is reflected in the comments
from a customised appraisal at the end of the semester. A pictorial
form of the model is shown in Figure 2.
APRIL 2010 • TDU TALK
Activities such as assessment or lectures that you want feedback on.
Can occur as many times as you require throughout semester.
Appraisal from
current occurrence paper
Appraisal
from former
occurrence
of paper
Semester
Figure 2 Model for the
evaluation of teaching
Conclusion:
I have presented in this brief paper some issues around teaching from the
development of a number of University strategic documents. I have also outlined
one possible model that will allow for the collection of evidence as required in
both the Teaching and Learning Plan and the Policy on the Evaluation of Teaching
and Papers. A number of references on formative evaluation have been provided
as a good place to start your research in this area. Please feel free to contact the
TDU if you would like further advice, assistance or exemplars of how other staff
have worked in their own practice to carry out the University‘s policies and plans.
References
Angelo, T. A. & Cross. P. (1993). Classroom Assessment Techniques. San
Francisco: Jossey Bass.
Edstrom, K. (2008). Doing course evaluations as if learning matters most. Higher
Education Research & Development. 27 (2), 95-106.
George, J. W & Cowan, J. (1999). A Handbook of Techniques for Formative
Evaluation. London: Kogan Page.
Kember, D. Leung, D. Y. P & Kwan, K. P. (2002). Does the use of student
feedback questionnaires improve the overall quality of teaching? Journal of
Assessment & Evaluation in Higher Education, 27 (5) 411-425.
APRIL 2010 • TDU TALK
•8•
McNaught, C. (2003). Innovation and Change in higher education: managing
multiple polarities. Perspectives: Policy and Practice in Higher Education. 7 (3),
76 -82.
Piccinin, S. Cristi, C. & McCoy, M. (1999). The impact of individual
consultations on student ratings of teaching. International Journal for Academic
Development. 4 (2), 75-88.
ā ā
ā
ā ā
•9•
ā
APRIL 2010 • TDU TALK
Dr Trudy Harris, Appraisals Administrator, Teaching Development Unit
“
Hello Everybody, I thought I would take the opportunity to update you on a
project that the Teaching Development Unit is collaborating on with
colleagues from the Faculty of Management, so that we can eventually
enhance practice around appraisals across the University.
What is the project?
Late in 2009, as part of ongoing work by the Teaching Quality Committee (TQC),
it was suggested that the Faculty of Management and the Teaching Development
Unit (TDU) undertake a pilot project to enhance the use of appraisal data with the
aim of improving quality processes around teaching and the student learning
experience. Key areas to consider were:
 Implementing the optimum reporting and use of appraisal data
 Determining the effectiveness of the current policy for the evaluation of
Teaching and Papers (up for review 2010/11)
 Determining if further policy or guidelines are necessary to address issues that
occur as the results of reporting appraisal data
 Determining the effectiveness of the current appraisal instrument
Why are we running the project?
As some of you may remember, 2006 saw the publication of the Cycle 3 Audit
Report. A number of recommendations were made regarding the reporting of
appraisal data, namely:
Quality assurance R 2 [p.6]
The panel recommends that the University implements an institutional quality
assurance system that includes planning, monitoring, review and
improvement, and that feeds back into professional development and
enhancement.
Evaluation of teaching and feedback R 6 [p.14]
The panel recommends that the University gives priority to developing and
promulgating the guidelines related to peer appraisal of teaching as well as
formal and informal appraisal of teaching and papers as was stated in the
report of the Working Party on the appraisal of papers and teaching: final
report (May 2002).
To facilitate some of these recommendations the policy on the Evaluation of
Teaching and Papers and associated guidelines were developed. These
documents were approved by Academic Board in 2008, and are due for full
implementation this year.
• 10 •
APRIL 2010 • TDU TALK
Table 1 The reporting structure
The guidelines to the policy document require that data from the appraisal of
papers should be provided to a range of key people as well as the individual
teachers. Table 1 shows the reporting structure for this data.
Who
Teacher
What
Paper and individual teacher report
Paper Convenor
Paper report for that particular paper
COD
Aggregated data for all papers relevant to department. Teaching reports through
PGS.
Dean/Faculty/School/APC
Aggregated paper results for each department in Faculty/School
University
Aggregated results for each Faculty/School
Appraisals for Demonstrators, Sessional Assistants and Tutors will go to the paper convenor for distribution.
Table 2 Appraisal benchmarks in
the Teaching and Learning Plan
2010-2012
Recently the Teaching and Learning Plan 2010-2012 has added further
requirements for appraisal data with the development of a number of internal
benchmarks. These are outlined in Table 2:
Goal 2:
Effective teaching and supervision are consistently demonstrated
Action 2.1:
ensure that all departments comply with the University policy on the evaluation of Teaching and Papers
and use students evaluations and other summative and formative approaches in developing papers,
teaching, and postgraduate supervision
The mean for robust teaching and paper evaluations each year is less than 1.75, with fewer than 10%
greater than 2.5 and more than 33% below 1.5
KPI 2.1:
Benchmarks are an important part of improving the quality around teaching. The
quote below from Jackson and Lund (2000) defines benchmarking succinctly:
―Benchmarking is first and foremost, a learning process structured so as to
enable those engaging in the process to compare their services/activities/
products in order to identify their comparative strengths and weaknesses as
a basis for self-improvement and/or self-regulation.‖ (p.6)
What have we found so far?
In February 2010 the Dean and CODs of the Faculty of Management were
presented with appraisal data covering the period 2007-2009. A hypothetical set of
data is shown in Figure 3. The data has been broken down by level to align with
external benchmarks provided by the Australasian Universities Survey of Student
Engagement (AUSSE) that will run in 2010.
• 11 •
APRIL 2010 • TDU TALK
Teaching Development Unit
Departmental Appraisal Report
School: Teaching Development
Feb-10
2009Data
Department: Appraisals
Paper Occurrence
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
1 APP101-09A (HAM)
1.6
1.6
1.6
1.6
1.6
2.1
2.1
1.8
2 APP101-09B (HAM)
1.6
1.7
2.0
1.8
1.8
2.3
2.1
2.0
3 APP102-09A (HAM)
1.3
1.6
1.6
1.6
1.7
2.3
1.9
1.9
4 APP102-09B (HAM)
1.3
1.3
1.6
1.3
1.6
2.2
1.9
1.7
4
4
4
4
4
4
4
4
Average:
1.5
1.6
1.7
1.6
1.7
2.2
2.0
1.9
Standard Deviation
0.2
0.2
0.2
0.2
0.1
0.1
0.1
0.1
Paper Occurrence
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
2.5
2.2
2.3
Level 100 papers
5 APP102-09S (HAM)
Level 100 Totals:
Level 200 papers
6
APP202-09A (HAM)
2.0
1.9
2.3
2.0
2.0
7
APP202-09B (HAM)
1.3
1.3
1.8
1.6
1.6
8
APP202-09S (HAM)
9
APP209-09A (HAM)
1.0
1.0
1.0
1.0
1.0
1.0
1.6
1.0
10
APP209-09B (HAM)
1.3
1.3
1.3
1.6
1.3
1.7
1.3
1.6
11
APP209-09S (HAM)
12
APP231-09A (HAM)
1.7
1.6
1.9
1.7
1.8
2.2
1.8
1.9
13
APP231-09B (HAM)
2.0
2.3
3.1
2.4
2.6
2.5
2.6
2.6
6
6
6
6
6
5
5
6
Average:
1.6
1.6
1.9
1.7
1.7
2.0
1.9
1.9
Standard Deviation
0.4
0.5
0.7
0.5
0.6
0.6
0.5
0.6
Level 200 Totals:
Table 3 Dummy appraisal data in
a format similar to that used in
the Pilot Project
• 12 •
1.7
Alongside this data an overview of how each department was benchmarked within
the Faculty was also provided. This is shown in Table 4. In alignment with the
KPI of the Teaching and Learning Plan (2010-2012), a distribution of papers was
also presented, please see figure 5.
APRIL 2010 • TDU TALK
Department of
Department of Department of
Appraisals Tertiary Teaching Higher Education
Research
n
Number of possible Paper Appraisals
64
Number of Paper Appraisals conducted
35
(%)
n
%
67
54.7
41
n
(%)
51
61.2
34
Totals
n
%
182
66.7
110
60.4
Distribution of paper quality
n
(%)
n
%
n
(%)
n
%
Excellent1 -1.5
Very Good 1.6 - 2.5
Satisfactory2.6 - 3.5
Unsatisfactory 3.6 - 4.5
Poor 4.6 - 5
11
20
3
1
0
31.4
57.1
8.6
2.9
0.0
19
22
0
0
0
46.3
53.7
0.0
0.0
0.0
17
16
1
0
0
50.0
47.1
2.9
0.0
0.0
47
58
4
1
0
42.7
52.7
3.6
0.9
0.0
Average Paper Quality
Standard deviation
1.9
0.5
1.9
0.5
1.7
0.4
1.8
0.1
Table 4 The dummy data
presented as a Faculty overview
Table 5 Distribution of paper
quality taken from the dummy
data shown in Table 4
• 13 •
We actively encouraged feedback on a number of subjects connected with the
project, namely: the data provided; the appraisal instrument used; and the policy
on the Evaluation of Teaching and Papers that is due for review at the end of this
year. We have received responses from the majority of CODs, as well as verbal
feedback and observations from teaching staff within the Faculty. In most cases
staff found the data useful, and appreciated the fact that they were presented with
data covering a number of years. Some typical feedback is outlined below, and the
recommendations made to address the issues that have been raised.
APRIL 2010 • TDU TALK

Exclude papers with codes 399/499/599/590/591/592 and 594 from the
‗number of papers that could be appraised‘ statistic. This then focuses the
data on taught papers only.
Action: This is a recurrent theme and these papers have been removed from the
data.

It has been suggested that similar data be presented for teaching appraisals,
and should be ready to report with the 2011 appraisal report..
Discussion: It is necessary to discuss this point at TQC to ascertain whether it is
possible.

There was discussion about the inclusion of school and university norms for
courses at each level (i.e., 100, 200, etc) so that there was a better basis for
assessing whether a department was doing well or in need of improvement.
It was also suggested that such norms be included in individual lecturers'
reports, so that staff could see how a particular paper compared with school
and university averages.
Recommendation: The norms be developed and incorporated into the reporting
template when University wide appraisal data is rolled out.
In terms of quality assurance and continuous improvement a number of
suggestions were made concerning the use of the appraisal data. For example:

Plotting the appraisal data against the median grade of assessment for each
paper, and also retention data.

For this information to be fully utilised for improvement purposes it was
suggested that it would be necessary for the Faculties and Departments to
receive this information by the 31st of January each year.
Recommendation: The necessary resources be provided to TDU to enable
information to be provided to Faculties/Schools and Departments within two
months of completing processing for the purpose of quickly implementing
improvements in paper preparation.

There were a number of different viewpoints concerning policy
development. For example, there has been discussion about follow up
processes for papers that score 2.5 (as outlined in the Teaching and
Learning Plan 2010-2012) and it has been suggested that policy should be
developed institutionally as this provides an organisational focus on quality
teaching and learning, and there is an expectation that something should be
done.
Recommendation: that the University reinforce the expectation that all papers
with an overall paper quality of 2.5 and above will be examined for contributing
• 14 •
APRIL 2010 • TDU TALK
factors and the development of actions to address any causes identified
including the development of policy and/or guidelines to support those papers.
There were very few comments concerning the appraisal
questionnaire. One COD thought that the current set of questions are
adequate and the guidelines were appropriate, while another said that
the questions did not apply to all situations, especially in NET papers.

Recommendation: Consideration needs to be given to the
appropriateness of the questionnaire for NET papers and discussion
needs to be held about the further developments that are required.
19-22 October 2010, Liverpool, UK
You are invited to join the 7th annual conference of
the International Society for the Scholarship of
Teaching and Learning in Liverpool, UK,19—22
October 2010. This multidisciplinary, international
community of scholars
will convene to share evidence-based insights and
theoretical frameworks that enhance our
understanding of student learning and guide our
teaching practices. Incorporated into this year's
conference, is the 18th Improving Student Learning
Symposium, an established annual event on the
International calendar organized by the Oxford
Centre for Staff and Learning Development, Oxford
Brookes University. The major aim of the Improving
Student Learning Symposia is to provide a forum,
which brings together those who are primarily
researchers into learning in higher education and
those that are primarily practitioners concerned
more pragmatically with improving their practice.
The conference will be held in the heart of Liverpool
on the historic, world heritage waterfront, at the BT
Convention Centre, Monarchs Quay, Liverpool UK
which sits alongside the Grade 1 listed Albert Dock
complex on the eastern bank of the river Mersey.
The conference will feature workshops facilitated by
leading scholars in the field, distinguished
international plenary speakers, panel presentations,
individual paper and poster presentations, and
roundtable discussions. Please join us!
To book your place and for more information,
please go to [http://issotl10.indiana.edu/ ]
• 15 •
Verbal comments have indicated that both Faculty and the individual
departments are now having conversations around teaching. There
have also been requests for questionnaire development for some of the
papers. This has come about through the identification of papers that
had not been appraised in the last two years (in line with the Policy on
Evaluation of Teaching and Papers).
Where to next?
The next stage of the project is to collate the feedback so that it is
ready to incorporate into a number of projects that will run during
2010. These are:
1. The review of the Policy on the Evaluation and Review of
Teaching and Papers and associated guidelines documents
2. The Review of the Appraisal Instrument, templates and item bank
3. The development of a postgraduate survey for level 399 – 599
papers.
Acknowledgement
The Teaching Development Unit acknowledges and appreciates the
support, interest and engagement provided by the Faculty of
Management for this pilot study to improve teaching and
learning quality at Waikato, the Dean (Frank Scrimgeour),
John Tressler, Chairs of Department and staff.
References
.”
Jackson, N & Lund, H (2000) "Introduction to Benchmarking", in N,
Jackson & H, Lund. (Eds). Benchmarking for Higher Education,
Buckingham: Open University Press.
APRIL 2010 • TDU TALK
Robina Ann Cummins Centre for Sport and Exercise Science, The Waikato
Polytechnic, Hamilton, New Zealand (now Waikato Institute of Technology)
Paper Presented at Effective Teaching and Learning at University Duchesne College, The University
of Queensland. 9 and 10 November 2000. A conference for university teachers with particular focus
on effective teaching and learning.
Abstract
Feedback on teaching practice may be formal or informal (Zepke, Nugent and
Roberts, 1996). In many tertiary institutions formal evaluations of teaching
practice are performed on completion of teaching the course. The lecturer is left
with feedback which may be, but is not necessarily, relevant to the next cohort of
students.
In this paper I reflect upon two changes that I made with respect to the gathering
of formal feedback on teaching practice from a class of 48 students. Feedback was
obtained by administration of a SETPAC (Student Evaluation of Teaching,
Programmes and Courses) questionnaire (The Waikato Polytechnic, 1999). The
questionnaire included ten questions that required a quantitative rating on a scale
of 1 (strongly disagree) to 5 (strongly agree) and three open ended questions. The
SETPAC questionnaires were administered according to guidelines
specified by the Academic Unit of The Waikato Polytechnic.
The first change I made to the collection of the formal feedback was to
administer the SETPAC questionnaires twice (once mid-way through
the semester and once at the end of the semester) rather than just once at
the end of the semester when the course had finished. The second
change was to share the collective collated feedback with the students. I
subsequently made minor changes to my teaching style in consultation
with the students.
Comparison of the responses to the two sets of SETPAC questionnaires
indicated a significant improvement in the students' perception of my
teaching from the first to the second administration of the questionnaire
(Wilcoxon Matched Pairs Signed Ranks Test, p=0.01). The mean score across ten
questions increased from 4.2 to 4.4. The score for the question on overall teaching
of the course increased from 4.3 to 4.5.
Although there was no control group against which to make a comparison, it is my
perception that 'closing the feedback loop' in the manner described allowed for an
efficient use of feedback and that this contributed to the students' experience of
increased satisfaction with the course as it progressed.
APRIL 2010 • TDU TALK
• 16 •
Introduction
One definition of feedback is 'the modification or control of a process or system
by its results or effects, especially by the difference between the desired and the
actual result' (1). Feedback on teaching practice from students to educator may be
formal or informal (2). In many tertiary institutions, formal evaluations of
teaching practice are performed on completion of teaching the course. The lecturer
is left with feedback that may be, but is not necessarily, relevant to the next cohort
of students.
The situation that led me to reflect upon the degree of effectiveness of the formal
feedback process was a perusal of data from evaluations of my teaching of a broad
range of courses over a period of five years at The Waikato Polytechnic. During
that period two systems had been available to academic staff who wished to gather
data on their teaching from evaluations by students. The systems were a SETPAC
(Student Evaluation of Teaching, Programmes and Courses) questionnaire and a
non-SETPAC questionnaire. The SETPAC system has a wide range of items
available for staff members to choose from. However if the staff member wishes
to go for promotion the items to be included in the SETPAC evaluation are
specified. These items are considered to supply the same information as the nonSETPAC questionnaire which is the other option available for evaluation of staff
going for promotion. The promotion specified SETPAC questions specified for
promotion and the non-SETPAC questions are shown in Table 1.
Good Practice Publication Grants
At this early stage of the year, the northern hub staff of Ako Aotearoa
would like to bring your attention to another opportunity to obtain
funding for the writing up and publishing of your good practice in the
area of teaching and learning. This might be practice which you, your
organisation or department has instigated and can be shown to have
a demonstrable positive impact on the educational outcomes of your
learners.
Both questionnaires have ten items or statements to be
ranked on a scale of 1- 5 (strongly disagree to strongly agree)
and room for extra comments. Although item for item the
two systems are supposed to provide similar information to
each other I believe there is some loss of equivalence
between the two systems in items 3 and 5 and that within the
SETPAC system the distinction between items 3 and 5 is
somewhat blurred.
The new Good Practice Publication Grants flyer outlines the scope of
this grant. For the guidelines and application form, follow this link to
the Ako Aotearoa website: http://akoaotearoa.ac.nz/good-practicepublication-grants. The closing date for applications is 30 April
2010. Please feel free to contact Ruth Peterson on (09) 921 9752 or
Nigel Field on (09) 921 9750, in the northern hub, if you would like to
talk through any aspect of this grant.
During the five years in which I had collected data via one or
other method, I had noticed trends emerging. The item for
which I invariably received the highest rating was
'knowledge' (item 1). Other items which consistently scored
highly were item 2 (organisation), item 6 (concern for
students), item 7 (respect for students) and item 10 (overall
Resource Centre
performance). The lowest ratings tended to come from item 3
Another Ako Aotearoa service we would like to highlight is the
and item 5. Item 3 in the non-SETPAC system reads: 'This
Resource Centre contained on our website. Much practical, recent
tutor (lecturer) really gets the message across'. Its SETPAC
research in the various disciplines and topics is published here. You can
counterpart is 'The tutor (lecturer) presents the subject matter
be kept up to date by receiving monthly email bulletins, highlighting
what is new, through simply going to the website and registering
clearly.‘ Item 5 in the non-SETPAC system is 'This tutor's
yourself on the site www.akoaotearoa.ac.nz .
(lecturer's) teaching style suits me'. Its SETPAC counterpart
• 17 •
APRIL 2010 • TDU TALK
is 'The tutor presented the content satisfactorily.' For items 3 and 5 the range of
responses was greater than was the case for the other items. The bulk of students
gave me high scores in items 3 and 5 but there were subsets of students who were
not satisfied with my performance in these areas. Although I have an ongoing
invitation to students to let me know whenever my teaching is unclear, or noneffective, on no occasion had students approached me with such concerns.
My challenge was to set up a process that would enable me to ferret out areas of
discontent that existed for those students who would not approach me personally.
The system would need to provide sufficient feedback to enable me to make
constructive modifications that would benefit discontented students while the
course was in progress.
APRIL 2010 • TDU TALK
• 18 •
In this paper I reflect upon two changes that I made to the gathering of formal
feedback on my teaching practice form a class of 49 first year students in the
Bachelor of Sport and Exercise Science. The first change I made was with regard
to collection of formal feedback. Rather than administer the formal feedback once,
at the end of the course, I carried out two formal SETPAC evaluations (one midway through the semester and one at the end of the semester). The second change
was to share the collective collated feedback with the students and to subsequently
make minor changes in my teaching style in consultation with them. Throughout
the process I undertook to treat any piece of feedback with respect and with a
commitment to working out a solution.
Method
a. Ethical Considerations This paper is a retrospective reflection rather than a pre
-planned experiment and as such was not put before the Ethics Committee prior to
the gathering of data. However, prior to submitting my abstract and paper to the
Conference, I placed my intention before The Waikato Polytechnic Ethics
Committee and was given their informal approval.
b. Participants The students were 49 first year students within the Bachelor of
Sport and Exercise Science at The Waikato Polytechnic. There were 33 males and
16 females. They ranged in age from 18 years to 62 years with the mean age being
22 years. In terms of ethnicity, 35 students classified themselves as being
European/Pakeha, six students classified themselves as New Zealand Māori, and
five students classified themselves as being a combination of European Pakeha
and New Zealand Māori. One student classified his/herself as being a combination
of New Zealand Māori and Samoan. One student classified his/herself as South
African.
• 19 •
APRIL 2010 • TDU TALK
c. The Course The course in which I taught the students was SCBS105 Sport
Anatomy and Physiology. SCBS105 (Sport Anatomy and Physiology) is a
compulsory core paper that is undertaken by students in the first semester of their
first year. The semester is 15 weeks in duration. Each week students attend one
lecture (of two hours) and one practical class (of two hours). Students are all
together for the lecture and in groups of less than 20 for the practical classes. I
taught all lectures and all practical classes. In lectures, students are given a
standard lecture accompanied by typed lecture notes and overhead transparencies.
Interspersed within the lecture time are videos (accompanied by a bank of video
questions), quizzes and short activities. In practical classes, student dissect organs
(eg hearts, lungs, kidneys, brains) and learn skills (eg measuring blood pressure ,
testing reflexes, assembling bones of a skeleton), manipulate models and refer to
their text books in order to research answers to questions. Assessment is made up
of practical work (30%), seminar presentation (10%), and a theory exam (60%).
d. The Evaluation Tool - SETPAC Questionnaire
The evaluation tool used was a SETPAC (Student
Evaluation of Teaching, Programmes and Courses)
questionnaire (3). The items used were as follows:
1. The tutor seems to know the subject matter well.
2. The tutor is well-organised and prepared for classes.
3. The tutor presents the subject matter clearly.
4. The tutor clearly defines my responsibility in the
subject.
5. The tutor presented the content satisfactorily
6. The tutor shows genuine concern for student progress
and needs.
7. The tutor treats students with respect.
8. The tutor provides feedback that is constructive and
helpful.
9. The tutor welcomes student feedback on the classes.
10. I would rate the overall teaching of this tutor in this
course as good.
The instructions on the SETPAC evaluation form request
the student to allocate a rank from 1 (strongly disagree) to
5 (strongly agree) next to each item. In addition, space is
left for the student to write three open-ended comments in
response to the following three statements:
i
ii
iii
APRIL 2010 • TDU TALK
aspects which are done well;
aspects in need of improvement;
other comments.
• 20 •
e. Administration and Processing of SETPAC Evaluations
The administration and processing of the SETPAC evaluation forms were carried
out according to strict guidelines laid down by the Academic Unit of The Waikato
Polytechnic. On both occasions when the evaluations were carried out, I left the
room and another member of staff distributed the forms, read the instruction form
provided by the Academic Unit, allowed the students adequate time to fill in their
responses, and collected the forms (either folded or face down) in a brown
envelope. They signed a document and sealed this inside the envelope with the
student responses. They sent the envelope to the Academic Unit via Internal mail.
Staff of the Academic Unit processed the forms. They produced a statistical
analysis of the quantitative data gained from the rankings assigned to items 1-10.
This included:
i the mean of all responses, calculated by weighting each response by the
numerical value and dividing by the total number of responses;
ii the maximum number of valid responses for that item;
iii the number of valid responses for the question expressed as a percentage of
the students responding to the evaluation;
iv a pictorial representation of the spread of responses across the five point
scale showing the positions of the median and the upper and lower
quartiles..
The "Feedback Environment" within the Classroom
It is my belief that the teaching/learning process flourishes when there is rapport,
confidence, trust, mutual respect and openness between the lecturer and students.
With all classes that I teach, I undertake to respect and to consider carefully each
piece of feedback that I receive be it anonymous or identifiable, formal or
informal.
The Innovations
The first innovation I used was to administer the evaluation forms twice (once mid
-way through the semester and once at the end of the semester) rather than just
once at the end of the semester.
The second evaluation was performed one week prior to the examination period.
The second innovation I used was that once I had received the typed summary of
the first set of feedback, I brought it to each practical class and allowed students to
see it if they so desired.
Other aspects of gathering feedback remained the same as they had been
previously. As always, I encouraged students to communicate openly with me
about the course and my teaching whenever they wished to. I listened carefully to
their comments and genuinely tried to "take their comments on board".
• 21 •
APRIL 2010 • TDU TALK
Results
a. Response Rate Forty-eight (48) students responded to the evaluation at Time 1
(mid-semester). There was 100% response to items 1 to 3 and 5 - 9 and 98%
response to Item 5 and Item 10 and a total of 54 open ended responses. Fortyseven (47) students responded to the evaluation at Time 2 (the end of the
semester). There was 100% response to items 1 to 10. There were 79 open ended
responses.
b. Qualitative Data from Items 1 to 10 Table 2 shows the mean score at Time 1
(mid-semester) and at Time 2 (end of semester) for each item evaluated by the
students. The items are listed from top to bottom in descending order of the rank
of their scores at Time 1. In addition, Table 2 shows the magnitude and the
direction of any change in mean score of a particular item at Time 2 with respect
to Time 1. Two items remained unchanged in their mean score across the two
times. Item 1 (knowledge) had a score of 4.7 and a rank of 1 on both occasions.
Item 4 (definition of student responsibility) retained a score of 4.0 at both times
but changed rank from 8th to 10th by virtue of increases in the scores of other
items. In all, eight items showed an increase in mean score from Time 1 to Time
2. This increase was significant (Wilcoxon Matched Pairs Signed Ranks Test,
APRIL 2010 • TDU TALK
• 22 •
p=0.01 (4)). The two lowest ranked items at Time 1 showed two of the largest
increases in score at Time 2. These were items 3 which increased from 3.9 to 4.2
and item 5 which increased from 3.8 to 4.3. Item 6 (concern for students)
increased from 4.1 to 4.5.
c. Qualitative Data from Open ended responses
Table 3 shows the main themes that emerged from the open-ended responses
and includes representative examples of these comments. In 'aspects which are
well done' the theme 'practical classes' was very strongly apparent across both
sets of evaluations. The other theme that was very strong across, both times, was
'interactions with students'. Within the theme 'interactions with student' a subtheme 'responding to student feedback' emerged at the second set of evaluations.
Presentation/interest level was present as a theme within 'aspects well done' at
Evaluation 1 and strengthened further in Evaluation 2.
• 23 •
APRIL 2010 • TDU TALK
Discussion
As stated in the introduction, my intention was to establish a process that would
allow me to uncover and address any areas of dissatisfaction. Analysis of formal
feedback from previous years, over a range of courses, had indicated that although
the majority of students were satisfied with my teaching, there were 'small
pockets' of students who felt my clarity and presentation could be improved.
Early notice of Action
Learning and Action Research Association conference to be held in Melbourne in November.
Abstracts for Paper or Catalyst Presentations, Workshop proposals, proposals
for Poster Sessions and discussion rounds are invited;
the Organising Committee
will receive abstracts for
proposed refereed contributions (up to 300-400 words)
from interested individuals
and groups till May 31st
2010. The proposals will be
reviewed by the ALARA
World Congress Editorial
review Panel, who will also
offer some feedback as to
the linkages of the proposed
presentations or discussions
with the themes of the Congress. See www.alara.net.au
for further details.
APRIL 2010 • TDU TALK
My first innovation was to invite students of SCBS105 to perform a formal
evaluation of my teaching via SETPAC mid-way through the semester. I let them
know that this was the first of two such evaluations and that the second one would
be at the end of the semester. From this first evaluation I found that the responses
followed a similar pattern to those set by previous students in other courses.
Although there was overall satisfaction the discontent or uncertainty occurred with
respect to clarity (mean score 3.9) and presentation (mean score 3.8).
The open ended responses consistent with this theme were comments such as 'it‘s
too boring' and 'make classes more interesting'. In an attempt to make class more
interesting, and in the absence of written comments from the class on how to do
this, I sought advice from a member of The Waikato Polytechnic Professional
Development Unit. As a consequence I was introduced to 'mind mapping' and I
took the concept into a one-hour slot in my lectures on the endocrine system. I
arranged the students in small groups and made each group responsible for a
different portion of the lecture notes handout. I gave each group large sheets of
paper and coloured marker pens. The one-hour segment was apportioned into
putting together their mind map and then presenting it as a group to the rest of the
class.
My second innovation was to show the students the final analysed collated copy of
collective evaluations. This enabled them to find their individual comments in
print, to see what other class members thought and to generally see further along
the 'feedback chain'. I thanked them for the many positive comments and invited
them to approach me with ideas they had that might improve the course for them
in any way. I believe this strategy helped the students to perceive openness on my
part to gain and then act upon feedback.
Possibly as a consequence of the combination of the two innovations, two students
were forthcoming with suggestions for improving my presentation of lecture
material. Each student approached me individually with his request several weeks
before the end of the semester. The first request was for me to hand out a list of
important terms to accompany each set of lecture notes. The students would then
be able to fill in the meaning of each term. The second request was to recap major
concepts at the end of each lecture. I fulfilled both requests in the lecture that
remained for the semester and publicly acknowledged the students who had made
the suggestions.
• 24 •
In summary, the changes in presentation that I made comprised the inclusion of
one session of 'mind mapping', three word lists and two concluding summaries of
major points. In addition, whenever I changed activities within the lecture I
mentioned that we were swapping in order to 'keep things interesting'!
The changes I made were not major ones, and yet there was a significant shift in
the degree of the satisfaction the students reported with respect to clarity and
presentation. It is my belief that there was a subtle 'extra something' that
contributed to the shift. Indeed, it is my perception that part of the increased
satisfaction came from the students gaining a sense of empowerment in terms of
their contribution to the course that, in turn, empowered them in the whole process
of their learning. Certainly in the second evaluation the fact that student feedback
had been acted on emerged as a theme. I believe 'effective teaching' and 'effective
learning' are two sides of one process in which commitment from and
communication between 'teachers' and 'learners' is essential. My most treasured
comment from the open-ended responses of the second set of evaluations of the
course is "Well done. A game of two halves".
References
1. Hughes, J.M; Mitchell, P.A., & Ramson, W.S. (Eds) (1992)The Australian
Concise Oxford Dictionary. 2nd edition. Melbourne: Oxford University Press.
2. Zepke, N; Nugent, D; & Roberts, C. (1996) The New Self Help Book for
Teachers. Wellington, New Zealand: WP Press.
3. The Waikato Polytechnic (1999). SETPAC. Obtaining Student feedback on
Teaching, Programme and Courses. A Guide to The Waikato Polytechnic Student
Evaluation Services and Procedures. Hamilton, New Zealand: Academic Unit,
The Waikato Polytechnic.
4. Siegel, S. Non-parametric Statistics for the Behavioural Sciences (1956).
International Student Edition. New York: McGraw-Hill Book Co. Inc.
Acknowledgements
I acknowledge Don Hewison for development of SETPAC, Andrew Wright for
introducing me to the concept of 'mind mapping' and Dr Pip Bruce Ferguson for
drawing my attention to the 'Effecting Teaching and Learning at University' 2000
Conference. I thank Dr. Gordon Paterson for commenting on a rough draft of this
paper and for displaying an interest in the subject matter. I also thank Landmark
Corporation for introducing to me the concept of 'listening from nothing', an
invaluable skill in allowing free flow of feedback. Most of all I thank the students
of SCBS105 for their generosity in giving feedback and their enthusiastic
involvement in the challenges of Sport Anatomy and Physiology.
Downloaded 12 May 2009 from http://www.tedi.uq.edu.au/conferences/teach_conference00/
abstractsA-H.html; used with permission of author.
• 25 •
APRIL 2010 • TDU TALK
In the introduction, we indicated that it can sometimes be helpful for staff wanting
to reflect on practice to use ways of recording reflection that work best with their
own discipline. In the two articles cited below, the authors indicate how they have
used such methods to capture their own (Sankaran) or their students‘ (Boud)
reflections.
Dr Shankar Sankaran was teaching in a business context and studying for his PhD
when he wrote Memos to myself. It‘s an occasionally humorous description of his
adaptation of ‗business memos‘ to reflect on different aspects of his personality as
he worked to complete his PhD. You can access it from the internet, and Shankar
has given us permission to use this paper although space constraints preclude its
publishing in full here.
Sankaran, S. (1997) Memos to myself: a tool for improving reflection during an
action research project. Downloaded from http://www2.fhs.usyd.edu.au/arow/o/
m01/rshankar.htm on March 31, 2010.
The second article is quite different. In this chapter, Emeritus Professor David
Boud, who has been involved in education for many years, considers at a general
level why, how and for whom journals may be written. Using Journal Writing to
Enhance Reflective Practice, published in New Directions for Adult and
Continuing Education, no. 90, Summer 2001. Downloaded from:
http://www3.interscience.wiley.com/cgi-bin/fulltext/90513645/PDFSTART on
March 31, 2010. This edition also contains a wide variety of relevant chapters,
including the pluses and minuses of assessing reflective journals (see the chapters
by English and by Fenwick). While David was happy for the chapter to be
reproduced here, we were unable to get clearance from the publishers in time for
this to happen.
We think that both articles are valuable contributions to the literature, presenting
quite different ways of looking at and using reflection.
• 26 •
APRIL 2010 • TDU TALK
Anne Hume - Faculty of Education
Hume, A. (2010). A personal journey: introducing reflective practice into pre-service teacher
education to improve outcomes for students. Teachers and Curriculum, 11, 21-28
A personal journey: introducing reflective practice into pre-service teacher education to improve
outcomes for students
Abstract
This paper traces the development over several years of an initiative, involving
student journals, that was introduced into a tertiary science education course for
pre-service teachers to improve communication between the lecturer and students.
The narrative recounts how the nature and uses of the journals subsequently
evolve as a result of reflective practice by the course lecturer and students. This
introduction of intentional reflection by the course lecturer, informed by ongoing
action research, is providing valuable insights into the nature and extent of student
learning and the actions required to improve outcomes for students.
We had hoped to reproduce Anne‟s article in full here, and had permission
to do so. It is an excellent paper that describes critical reflection by both
Anne and her students on various aspects of their practice. Unfortunately,
space constraints in this edition preclude our publishing this paper in its
entirety so, as we have recently published another full paper by Anne, we
have opted to reproduce just the abstract here in the hopes that interested
readers will access this paper from its original source, Teachers and
Curriculum (2010), 11, 21—28.
Alongside the Pavlovich, Collins and Jones paper which is paraphrased
next, it may give readers some ideas about how they might reflect on their
own practice and its improvement to gain research publications as well as
improving practice.
APRIL 2010 • TDU TALK
• 27 •
Kathryn Pavlovich, Eva Collins, Glyndwr Jones, Waikato Management School,
University of Waikato
Note: This article was published in the Journal of Management Education, 33 (1), 37—58. Small parts
of it are reprinted here with permission of the authors, but you are encouraged to access the original
to see the full scope of the work, as copyright issues have not enabled us to get clearance to
reproduce the full paper. The paraphrased material appears in italics, to separate it from the
authors’ work, cited from the online version http://jme.sagepub.com/cgi/
rapidpdf/1052562907307640v1
Abstract
This article examines learning journals as a method for developing selfawareness
within a business education context, exploring ―how can effective design and
assessment of reflective journals assist the development of students‘ selfknowledge?‖ The authors describe three different approaches to learning journals,
with each case study outlining the purpose of the course and the learning journal
within it, the design and assessment of the journal, and an evaluation of this
experience. The authors‘ aim is to illustrate how journals can be implemented in
management education. Although each case study is distinct, three interconnecting
themes also emerge that underlie why this approach to learning is important:
finding the subjective voice that enables students to access their inner learning;
accepting that learning is mutually constructed within a cocreative space rather
than something ―done to the student‖; and that a more reflective self-awareness
engages a higher sense of personal purpose. These significant outcomes illustrate
the success of this learning approach.
Education is longing for a deeper more connected, more inclusive, and more
aware way of knowing. One that connects heart and hand and head and does
not split knowledge into dualities of thought and being, mind, and body,
emotion and intellect, but resonates with a wholeness and fullness that engages
every part of one’s being.
—Kind, Irwin, Grauer, and de Cosson (2005, p. 33)
At the start of the article, the authors cite literature pertaining to the nature of
teaching, and questions such as „who‟ is the teacher. They indicate that the use of
reflective questions can help people to develop „a deeper sense of learning and
experience‟ (page 2). Building on the work of such authors as Brearley
(2000),Cunliffe (2004), and Schön (1983) they encourage staff to see their role as
educators being to help students to develop their own self-awareness and
appreciation. They describe, via three case studies of the use of learning journals
in different courses, how they have encouraged students „to think more holistically
than conventional neo-liberal institutions‟ (page 2).
• 28 •
APRIL 2010 • TDU TALK
The paper then proceeds to describe how learning journals were used to promote
reflection and learning in management education. As these case studies wrestled
with the assessment of reflective journals and it is an important issue in the
literature, this entire section is reproduced below.
Assessment
The biggest issue encountered with reflective journals is their assessment. Varner
and Peck (2003) note that students invest much of their personality and self-image
into the journal, and this makes them sensitive pieces of assessment to mark.
Crème (2005) too claims that in writing these journals, we ask our students to
open themselves up to us by using their individual voice, expressing a sense of
honesty, and taking a risk in the content they write. This makes grading and
evaluation difficult, as their subjective nature defies the standardized criteria of
more objective forms of assessment. Furthermore, when journals are graded, the
grade becomes the emphasis that constrains free expression and creativity. On the
other hand, grading encourages classroom preparation and participation, with
other studies concluding that when journals are not counted toward a grade,
students do not put in the work (Kennison & Misselwitz, 2002). As Crème (2005)
notes, what we assess is pedagogically important as it powerfully influences
student learning and also sends signals as to what we, as teachers, believe is
important.
Moon (1999) concludes that detailed assessment criteria can play a central part in
the success of the journal writing, as they provide the structure and foundation for
what is expected. Clarity of objectives assists students to move beyond descriptive
accounts of their experiences. In a study on learning journals, Crème (2005)
identifies the following guidelines for assessing journals. A good record of study
she claims is
a) comprehensive as it meets requirements of an introduction, conclusion and
demonstrates syllabus coverage; b) shows understanding of the material, with the
ability to select, summarize, analyze and show relationships between concepts,
both within the course and outside of it; c) shows self-awareness of the writer as
learner, both in relation to the ideas on the course, and to course activities,
processes and colleagues; and d) demonstrates that the writer is prepared to take
risks with the material in relation to their own political and intellectual position (p.
290).
Embedded in these guidelines is a mix of cognitive skills in knowing what content
should be selected as important, while also writing in a manner that emotionally
and holistically connects the student with the context.
Dilemmas regarding issues surrounding the subjectivity of the assessment process
notwithstanding, we share Dewey‘s (1933) view that reflective habits must be
taught if we wish to foster critical thinking. We believe that learning journals
sharpen our ability to reflect. Most important, we agree that reflection as a skill
APRIL 2010 • TDU TALK
• 29 •
can be developed and follow the Habermas tradition in seeing it as a tool for
personal empowerment and emancipation (Moon, 1999). Our aim in this article is
to examine how the design and assessment of learning journals may be better
employed to encourage student self-awareness (pages 4—5).
The case study method used was then described, followed by a presentation of the
management of Case Study 1, “Managing with spirit.” This described how the
journal was developed, and its design, including specific questions that the
students were encouraged to cover. Picking up the assessment issue cited at length
above, the authors then explained how they designed marking criteria to assist
with the assessment of journals. This material is also cited verbatim below.
Assessing the journal. As noted above, a very specific structure has been
developed to guide and assess students in their maturing of reflective practice. The
marking criteria closely followed the structure of the journal (see Table 1) and was
placed in the course outline at the beginning of the course. Describing the event
gave the students a C grade. Including some analysis and making meaning of it
moved the grade to a B—this included a good analysis of the course readings.
Demonstrating how this understanding would create new behavior in the future
was necessary for an A grade. Being able to demonstrate that learning and new
action took place was the essence of this assessment. [Table 1 appears on p.31]
Yet new learning and action also emerged for us as teachers, conforming to what
Ramsey (2002) and Williams and Wessel (2004) recalled as benefits for teachers.
Through the development of this article, we developed an ―engaged pedagogy‖ to
enhance and refine this marking criteria. Subsequently, this framework is now
being utilized by each of the three case study authors and authenticates its
applicability in other contexts (pages 7 and 8).
The authors next presented students‟ evaluations of
their reactions to the reflective journals. These are
very interesting and informative, and show responses
from a range of students, including a long journal
entry from a Chinese student who concluded ―Every
night when I lay on bed, I can hear my inner voice and I can
sum up important things to my brain. As long as I study this
course, I can feel my heart and my spirit is light‘ (page 8).
Case Study 2 covered Human Resource Management
Theory and Practice. As with the first case study, the
authors described the purpose and design of the
journal, its assessment and student responses to the
journaling task. Again, the entries of an international
student were cited. In this case study, students had
been asked to share ideas from their journals with
each other through an e-channel. Although initially
reserved and uncomfortable about this, „Polly‟ later
wrote:
• 30 •
APRIL 2010 • TDU TALK
―The difference between Kiwis and me is that I am afraid that my opinion might be wrong
and kiwis like showing their opinions. This is what I want to learn from them, so that I can
participate in more discussion and have really fun‖ (page 12).
Case study 3 was a graduate-level course aiming to enhance understanding of
sustainability issues from a manager‟s perspective. Again, the purpose, design
and assessment of the journal were covered, and student responses to its use were
included. In this case study, students had a choice of assessment options, and over
subsequent years, between a third and a ninth of students opted to do the journal
(6/18 in the first year; 2/18 the second/;4/12 the third and 3/13 the fourth).
One student wrote:
―I picked the learning journal option because I thought it would be an easy
assessment. I learned so much more than I expected! You should change
this from an optional assessment to a required assessment‖ (page 13).
In their discussion and conclusion, the authors identified common themes
that had come through all three case studies, showed how these aligned with
the literature and described the difficulties of assessment. The table that they
included and which summarises how the case studies contributed to reflection and
education appears on page 26. Their conclusions were that the journaling process
helped students to find their voice; made „a new space for learning‘ (page 17), and
enabled students to “connect with a more purposeful life” (page 18). This
summary of Pavlovich, Collins and Jones concludes with the voice of a student, as
did their paper.
―When my grandmother passed away I was asked to do a reading at her funeral by my
Dad. I found myself at a loss about what to read to truly represent how I felt about her. A
friend suggested that I read the paragraph from my second reflective journal as part of the
reading to show the importance of my Grandmother in our lives and the difficulties that all
my family encountered when she became ill. I was skeptical at first, thinking about myself
and how the people there would perceive me as being selfish and self-centred. I then
realized that I was falling into the same trap. It wasn‘t about me, it was about my
Grandmother, my family and all of her friends celebrating her life. They would be touched
and proud to hear of the respect and significance I reserved for her (page 19)‖.
APRIL 2010 • TDU TALK
• 31 •
April
Professional Development at a Glance
Make a space at your place for teaching
Some of the best learning happens through conversation and most of the working life of academics is focused around the
department. So why not make the occasional space for conversation about teaching in your department?
ĀHANGA