EDUCATION FOR
PRACTICE TOPICS:
7
8
New SES WPL Items: Using
Them to Enhance Learning
By Edwina Adams
Reference for this occasional paper:
Adams, E. (2013). New SES WPL items: Using them to enhance
learning (Occasional Paper 7). Sydney: The Education For Practice
Institute, Charles Sturt University.
ISSN 2201-8395 (Online)
© EFPI
Contact details:
The Education For Practice Institute
Charles Sturt University – Sydney
Locked Bag 450
Silverwater NSW 2128, Australia
http://www.csu.edu.au/division/landt/efp/
[email protected]
Contact details:
The Education For Practice Institute
Charles Sturt University – Sydney
Locked Bag 450, SILVERWATER NSW 2128, Australia
http://www.csu.edu.au/division/landt/efp/
New SES WPL items: Using them to enhance learning
Aim
This paper is designed to provide users of the new subject experience survey (SES) workplace learning (WPL) items with some practical
information on how to make the most of the instrument that replaces the online evaluation of subjects (OES). The information provided is in
addition to the power point presentation given at the EFPI/DLTS VC seminars on the 4 and 5 December 2012and the FAQs developed from those
sessions, see http://csusap.csu.edu.au/~fcolquho/wp_events.htm. Ideally, review the power point and FAQs before reading this paper.
How is the SES different to the OES?
The SES employs two sets of core items designed for different subject types whereas the OES was used for all subjects. One SES survey is
designed for subjects not classified as WPL and consists of 14 items plus two open-ended items. The other survey designed for WPL subjects
comprises 16 items and two open-ended items. Each of these surveys uses a 5-point Likert response format with scores ranging from a score of 1
for the least positive to a score of 5 for the most positive response. Note, the SES uses a continuous scale from least to most where the OES had
a midpoint response of 'uncertain'. In addition to the change in scoring, the new surveys have two response formats – one format addresses
frequency e.g. never to always while the other evaluates the extent to which something occurred e.g. to a very little extent to a very large
extent. When designing the SES surveys the working party premised the items on the following two key principles:
1. Learning focus: the items are designed to evaluate learning in the subject not satisfaction
2. Actionable change focus: the SES report will provide data that can inform the subject coordinator of where to make change. In other words
the items were designed to provide data that could be acted upon.
What learning constructs are evaluated by the WPL items?
The working party carefully considered the constructs fundamental for all good learning before designing the SES core items and these provided
the basis for the work. Table 1 presents the initial constructs and their dimensions. Reviewing these provides a greater understanding of what is
required to enhance learning and therefore achieve more highly rated responses in your subject.
Table 1 Good Learning Constructs and Dimensions
Construct
Design
Content
Implementation
Students
Context
Dimensions
Clear learning outcomes
Learning in situ
Meaningful learning
Collaborative and cooperative learning
Recognition of prior learning
Reflection in and on learning
Clearly articulated learning activities
Learning as a social process
Constructive alignment of learning outcomes with assessment
Currency
Appropriate
Scope
Depth
Blended learning
Flexible learning
Responsiveness to students
Timeliness of response
Assessment practices
Feedback and remediation
Learning how to learn
Reflection in and on learning
Recognition of prior learning
Interest
Learning in context
Meaningful learning
Coherent learning experience
Social presence
Problem solving
Authentic learning
Amount
Subject–Course Linkages
Approach to teaching
Engagement
The fundamental constructs for learning in Table 1 premised the development of the WPL items. In order to make the items specific to the WPL
learning environment however, more discrete constructs were considered to add to those in Table 1. Table 2 shows the WPL specific constructs.
Table 2 WPL Specific Constructs Building on the General Learning Constructs
Construct
Preparation
Learning
Specific skills deemed essential to
practice
Apply theory to practice
Assessment
Support for learning
Dimension
Learning provided before placement that develops basic skills and an understanding of the placement
requirements including OH&S
An environment that provides adequate breadth and experience of learning at the required subject level
All WPL should include the attainment of generic core skills necessary for any professional practice e.g.
"professional decision making".
The learning environment enables the application of theory to practice. In other words, there is sufficient time
allocated to do work (practise work).
Assessment should take on the dual role of providing a means for evaluation and the promotion of learning
Learning is enabled by feedback and resources
How can report data inform learning strategies?
Each time you receive a subject evaluation you should employ a critically reflective approach to analysing the data. This approach will allow you
to make a more considered and informed approach to making change in your subject. A review of the mean scores for each item and the
corresponding school mean should not be your only form of analysis.
Interrogating the data by looking at the results in detail i.e. percentages of students responding to an item, the subject mean and standard
deviation, and comparing these against the school mean and standard deviation all help to more accurately interpret the data. In addition, you
should reflect on the student cohort or any other variables that are not seen by the quantitative results. In other words, a full review including
contextual aspects (or other feedback) will assist you to make an informed decision about strengths and weaknesses in the subject.
In general, the following considerations should be made when interpreting the quantitative results:
 Core items are mandatory and a student will be unable to submit the survey until all core items have been complete. Therefore, the response
rate reported for the subject report is the same for all core items.
 Remember, the responses relate to a student's perception of their learning not an actual attainment of it. Assessment tasks and the student's
achievement in these indicate the actual learning that has taken place.
 The subject mean provides an average score of all responses to that item.
1
 The percentage distribution of item responses is useful in gauging whether the greater proportion of students report a positive or negative.
 The standard deviation provides an indication of how tightly the responses fall around the mean. A larger standard deviation indicates a
greater spread (distribution) of responses from the mean, this indicates that there were a number of students who responded quite
differently to the average response. If you would like to read more on standard deviation the University of New England School of Psychology
is a useful source http://www.une.edu.au/WebStat/unit_materials/c4_descriptive_statistics/standard_deviation.htm A greater depth of reflection
regarding these outlying students is required to determine why they encountered a different learning experience to the average student. This
is particularly necessary if these students respond in a negative direction.
Mock report interpretation and reflection
The following mock report is also presented in the power point presentation. This section will expand on this scenario for the purpose of
presenting some ideas on how to interpret the data and then reflect upon and design strategies to enhance the learning related to the item
presented. Increasing awareness of this process of interpretation and reflection in relation to SES WPL items is a valuable process with benefits
for both staff and students. This process enables staff to become more practically engaged and active with ways of enhancing learning.
Scenario
Subject identifier: MG10 Number enrolled: 80
Item 2
This subject prepared me to
participate in the work placement.
Number responded: 40 Response percentage: 50%
To a very
great
extent
To a
great
extent
To a
moderate
extent
To a
little
extent
To a very
little
extent
Subject
core
mean
Subject
standard
deviation
School core
mean
2
5
10
25
20
50
8
20
0
0
3.15
0.8
3.5
Count
%
School
standard
deviation
Interpretation of Item 2
Preparation is generally positive but 20% (or 1/5) of the students responding did not consider they were prepared. The standard deviation is
quite high indicating a spread of results.
Reflection of Item 2
The subject coordinator could reflect on the following questions and possible strategies:
1. Were briefing sessions compulsory, did all students attend? If not, could this be changed to ensure a greater proportion of students are in
attendance thereby raising the perception of preparedness.
2. Were the briefing sessions conducted in a manner that was not readily understood by all students e.g. unnecessarily complex or an
assumption of prior knowledge making it more difficult for less diligent students to pick up on the key messages? Often multiple formats (e.g.
online, face to face) repeating key messages in a different way are required to ensure comprehension by the vast majority of students.
Item 4
The learning activities in this subject (e.g.
briefings, simulations, labs, lectures, forum
discussions, group work) prepared me to
complete my assessment tasks.
Count
%
Always
Often
Sometimes
Rarely
Never
0
0
5
12.5
25
62.5
8
20
2
5
Subject
core
mean
2.82
Subject
standard
deviation
0.71
School
core
mean
School
standard
deviation
3.3
Interpretation of Item 4
The majority (62.5%) of students only reported they were sometimes prepared. A greater proportion report on the negative side of 'sometimes'
i.e. 25% report rarely or never as compared to the positive side i.e. 12.5% report often. Given a quarter of the students perceive they were
infrequently prepared it is important to give this learning construct thought to determine why this is the case and design an intervention that
could enhance the preparedness of students. In addition, the subject mean is much lower than the School mean which indicates other WPL
subjects in the School have a more positive perception of preparedness than this subject.
Reflection of Item 4
The subject coordinator could reflect upon the following:
1. Does this subject include structured activities that help develop the learning/skills required by the assessment task? Have the related
principles or skills of the assessment task been taught? An analysis of student results in the assessment tasks may indicate where they did not
grasp the key learning required to be successful. Once the area/s have been identified, additional learning strategies can be put in place.
2. If the assessment tasks are very focussed on the students' activities in the workplace the question needs to be asked as to whether some
workplaces are not providing (exposing the student to or allowing them to participate in) the learning required for the assessment tasks? If
this is the case the assessment task/s may need to be redesigned to be more equitable i.e. allowing all students an equal opportunity for
success.
Item 6
The goals of the assessment tasks in this
subject were made clear to me.
Count
%
Always
Often
Sometimes
Rarely
Never
Subject core
mean
Subject
standard
deviation
School core
mean
1
8
25
6
0
3.1
0.61
3.7
2.5
20
62.5
15
0
School
standard
deviation
Interpretation of Item 6
Responses fall in a positive direction with only a small proportion of students reporting negatively. The mean for the subject is lower than the
School mean even with this more positive direction so perhaps there are some very successful WPL subjects within the School or alternatively
the other WPL subjects have small numbers of students which has helped to set a high School mean.
2
Reflection of Item 6
1. Is the wording in the subject outline describing the assessment tasks and the learning outcomes easy for students to interpret? If not,
rewording or a rethink of what is actually being assessed. Alignment between content and learning outcome/s should also be reviewed.
2. The inclusion of a marking rubric describing the grading of the assessment tasks may help students to understand how to achieve well in the
assessment task and understand what is required without leading them to the answer.
Item 9
Feedback I received during the
work placement helped me to
improve my professional
practice.
Count
%
To a very
great
extent
To a
great
extent
To a
moderate
extent
To a
little
extent
To a very
little
extent
Subject
core
mean
Subject
standard
deviation
School
core
mean
0
8
21
9
2
2.87
0.79
3.4
0
20
52.5
22.5
5
School
standard
deviation
Interpretation of Item 9
In this item there is a slightly more negative response (22.5%) outside the response 'to a moderate extent' than positive (20%). The standard
deviation indicates there is a fairly wide distribution of results. Feedback is typically provided by workplace educators and although strategies
are employed to encourage useful feedback it is not always provided. In addition, the number of workplace educators providing feedback is an
issue because the greater the number of workplace educators, the greater the time and resources required in supporting these individuals
especially if they do not frequently take students.
Reflection of Item 9
Conducting continuing professional development opportunities or providing resources for educators in this area is helpful in raising the ability of
workplace educators. It may also be useful to try and identify specific centres where feedback is not so well delivered and conduct more
specialised strategies to overcome the shortcoming.
How can I make the most of the SES WPL data in my performance review?
SES WPL data should be reviewed in consideration of the response rate and any other feedback or information available regarding this subject. A
comprehensive approach to feedback from multiple sources provides a more effective indication of performance related to the design, content,
assessment and implementation of learning in a subject. In WPL subjects, a comprehensive approach is even more important because not all
learning is conducted by university staff, in fact the greater proportion is by staff in the workplace (i.e. subjects with >75% work placement).
Response rate is important for drawing conclusions about the population being surveyed. A high response rate gives greater confidence that the
feedback is representative of the whole population i.e. all students in the subject. A low response rate however, may only represent a subset of
the population e.g. students who had a poor learning experience and therefore do not represent the majority viewpoint. It is important then to
try and obtain a high response rate. The University of Wisconsin-Madison report the typical response rate for online surveys at 30 - 40%
http://oqi.wisc.edu/resourcelibrary/viewResource1.aspx?id=754 p.14.
A greater response rate can often be achieved when students understand why they are being surveyed and that their feedback is taken
seriously. Time taken in class or online to explain the value of providing feedback is often rewarded with a higher response rate. A CSU paper,
Strategies for raising Online Evaluation of Subjects response rates across the University (2005) has some useful strategies, see the section 'Role
of Academic Staff' http://www.csu.edu.au/__data/assets/pdf_file/0015/201156/resprate.pdf. Note, you can only access this paper if you are logged onto the CSU network.
Seeking other sources of feedback regarding student learning is recommended because it aids in understanding the factors influencing the
students' perceptions and therefore the ratings in the SES. A mini report outlining issues that could have affected learning should be
incorporated with your SES results to provide context and an explanation as to why specific items may be rated less favourably. Feedback
obtained at de-briefing sessions or from any other source should be included in your mini report.
In WPL subjects, the workplace and the staff employed there have a substantial impact on student learning. Factors such as the number of new
workplaces and workplace educators used in the subject or using sites that are not ideal due to shortages, all contribute to the effectiveness of
learning. Workplace educators who are new to their role may not fully comprehend the requirements for good learning nor have had the
opportunity to attend professional development sessions on this topic. Sometimes, due to an increase in student numbers or a reduction in
available sites, subject coordinators are forced to use sites that do not provide the full extent of work experiences required for the subject or
have some other deficiency. All factors influencing learning should be recorded and included in a performance review.
Obtaining feedback is the first step to enhancing learning in your subject but careful consideration of this information and then a well planned
means for implementing change is most important. Taking a comprehensive approach, investigating the causes, outlining changes to be made
and why, demonstrates a critical and reflective approach to subject coordination.
Conclusion
The new SES WPL items have been designed to provide useful data regarding the perception of student learning in a subject. Understanding how
to use the data effectively i.e. interpretation and reflection is important to making change that will enhance learning. This paper has provided an
additional resource to aid staff in this area.
For further information on aspects of WPL in curricula see:
Adams, E., (2012). Workplace Learning (WPL) Placements in Curricula: Strengths and Challenges, Occasional Paper 2. Sydney: The Education For Practice Institute,
Charles Sturt University.
Adams, E., (2012). Workplace Learning in Curricula: Assessment Design, Occasional Paper 3. Sydney: The Education For Practice Institute, Charles Sturt University.
Adams, E., (2013). Workplace Learning (WPL) in Curricula: Supervision that enables good learning, Occasional Paper 4. Sydney: The Education for Practice Institute,
Charles Sturt University.
3