ATTACHMENT
Committee: Academic Senate
Date: 16 May 2012
Item: 4.2
CHARLES STURT UNIVERSITY ACADEMIC SENATE REPORT OF THE ‘MODERATION WORKING PARTY’ ON A PROPOSED UNIVERSITY-­‐WIDE “MODERATION POLICY” 1. INTRODUCTION At its December 2010 meeting, Academic Senate “in principle” approved the CSU Draft Moderation Policy (now referred to as the ‘Policy’), but further consultation, feedback and review was requested. During 2011 four schools, one from each faculty, evaluated the Policy through consultation, feedback and review within their respective faculties. Two schools trialed the Policy and two schools reviewed the Policy. The purpose of this paper is to provide Academic Senate with feedback from the school reports and provide a recommendation for the implementation of a CSU Moderation Policy. 2. BACKGROUND AND MEMBERSHIP At its meeting on 26 May 2010 (AS 10.48), Academic Senate established a Working Party to consider ways of addressing the AUQA Panel’s Affirmation (number 2): AUQA affirms the findings of CSU’s Academic Senate Self-­‐Review, including the development of a University-­‐wide moderation standard. The AUQA Panel encouraged the development of a university-­‐wide moderation standard because “moderation is employed quite widely as the University uses a significant number of external sessional markers in some disciplines”. Further, although not mentioned specifically by the AUQA Panel, the development of a university-­‐wide moderation policy may be important as the University confers awards on CSU graduates taught by teaching partners in other countries and at Australian study centres. The recommended membership of the Academic Senate’s Moderation Working Party comprised: Associate Professor J. Atkinson (Chair) – Faculty of Business Dr. H. Cavanagh, Faculty of Science Dr. A. Crampton, Faculty of Science Dr. N. Drengenberg, Academic Secretary Ms S. Thacker – Faculty of Business Emeritus Professor R. Coombes Other people to join the working party have been: Associate Professor A. Bain, Dr J. Munday (School of Education), Dr. F. Trede, (Deputy Director, EFPI), and Mr. C. Sharp (Planning and Audit). Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 1 of 15 The final membership of the Academic Senate’s Moderation Working Party was: Associate Professor J. Atkinson (Chair) – Faculty of Business Dr. H. Cavanagh, Faculty of Science Professor R. Coombes – Faculty of Business Associate Professor Barney Dalgarno – Faculty of Education Dr. A. Crampton, Faculty of Science Dr. N. Drengenberg, Academic Secretary Associate Professor Joy Wallace – Faculty of Arts The members of this Working Party commend the Recommendation made in this report to Academic Senate. 3. RECOMMENDATION The following recommendation is made to Academic Senate: RECOMMENDATION: To recommend to Academic Senate that the Draft Moderation Policy (Appendix 1), which will apply to all undergraduate (including honours’ courses) and coursework postgraduate students at Charles Sturt University, be approved effective from the commencement of domestic Session 1 in 2013. 4. BACKGROUND The methodology used by the Moderation Working Party (now referred to as the “Working Party”) to identify options for the Policy included: an examination of some existing moderation processes within CSU, consultations with relevant CSU staff, a review of the AUQA good practice database as well as similar policies at other institutions. The Working Party met on several occasions between May 2010 and May 2011 to discuss issues and develop drafts of the document. Although the University does not have a “moderation” policy, individual faculties do have moderation processes and practices that have emerged over time, especially in circumstances where external professional accreditation is involved. Moderation procedures are also included in contracts (operations’ manuals) with teaching partners. The Working Party developed a Draft Moderation Policy covering undergraduate (including honours’ courses) and postgraduate coursework offerings although the Policy does not cover postgraduate research higher degrees. In addition to the moderation of “assessment” practices, the Policy expanded the definition of moderation was to include pre-­‐assessment and post-­‐assessment processes. This is consistent with practices at some other universities. The Policy requires the faculties and schools to develop specific guidelines/requirements, consistent with the general principles/requirements of the Policy, and suggested guidelines are Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 2 of 15 attached to the policy. The guidelines indicate suggested processes and provide insights into the practical implications of the proposed policy. The Policy complements very well recent CSU initiatives to enhance the quality of teaching and learning and the student experience: such as the Academic Senate approved recommendations on the “CSU Subject”, the five Common Teaching Standards (which contain requirements on the currency of materials and assessment practices), as well as the University’s emerging strategic direction which focuses on courses, the student experience and research. The Working Party believes that effective moderation processes should help to reassure students that their performances are being evaluated with independent checks during the teaching period, and this may increase the integrity of the assessment and grading processes, as well as reduce the number of grade appeals by students. Further, the Policy, once implemented, can be monitored and adjusted over time for “policy gaps” and for implementation issues such as unintended workload implications and other resource matters. From June to December 2010, the Working Party conducted several consultations and policy revision processes, including the following: (a) The draft policy, without the School “guidelines”, was provided to many staff and groups for feedback: the groups consulted included the Heads of School, the Deans, and the Sub-­‐Deans (Learning and Teaching). There was general support for the need for a policy on moderation, but a great deal of the feedback was concerned about the practical aspects of the proposal (especially due to expected “‘turnaround times” for assessments), the clarification of the responsibilities of various staff (such as Heads of School and course coordinators/ directors), and the impact of the policy’s requirements on the workloads of staff. The Deans also requested that the working party provide “some practice guidelines” for the Schools. (b) At its September 2010 meeting, the Academic Programs Committee (APC) considered an earlier version of the draft policy that did not contain the suggested School guidelines. The Committee agreed that, prior to further consideration, the working party should provide clarification on the resourcing implications of the proposed policy (10/76). However, the incremental resource implications of the policy (i.e. additional to those resources already employed) are difficult to measure with certainty. The Working Party then developed some suggested guidelines in order to provide some of the details of the possible resource implications. (c) It was noted that some of the moderation phases of the policy could also be carried out by experienced casual staff, retired academics and others experienced in moderation. However, as one staff member commented: “to ensure integration with other CSU activities, …….. moderators should have an understanding of relevant CSU standards (e.g. EFPI and flexible learning), the CSU Degree Initiative, the CSU subject, first year curriculum principles and Assessment 2020”. The draft policy and guidelines were considered by the Academic Programs Committee (APC) and Academic Senate in November and December 2010 respectively. At its November meeting, the APC provided some amendments to the policy. Members affirmed the importance of induction and training for moderators, and the pressure of time in completing moderation on time for grade release. However, the Academic Programs Committee agreed (APC 10/82) to recommend to Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 3 of 15 Academic Senate that the CSU Moderation Policy and general guidelines for moderation for faculties, schools and course teams ..... be approved.” The APC expressed concern in relation to the impact on workloads, and also suggested that partner agreements may need to be checked to ensure that they align with the draft policy. At its December 2010 meeting, Academic Senate approved “in principle” the Moderation Policy (see AS 10/156), but also requested further consultation, feedback and review. In particular, Academic Senate requested that the draft Policy and the guidelines should be referred to the Heads of School Forum and the members of the Working Party for further consideration and revision and then resubmission to Academic Senate for final approval. After the December 2010 meeting of Senate, the Deans and others were requested to seek and provide feedback on the Policy and guidelines. Feedback was received from: a. Faculty of Education b. Staff member in the Faculty of Science c. Heads of School Forum A clear message from this consultation and feedback was that a trial should be undertaken to evaluate the proposed Policy and associated guidelines. The Working Party supported this recommendation to conduct the trial in four different types of Schools, including one that offers workplace learning opportunities for students. The four Deans were contacted to nominate a school to trial the Policy and the following were recommended: 1.
2.
3.
4.
School of Accounting School of Education and School of Teacher Education School of Theology School of Biomedical Sciences The trials / reviews of the Policy were conducted in the 201260 session. The reports from each review / trial was submitted and then considered by be Working Party. Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 4 of 15 5. TRIAL / REVIEW OF THE DRAFT MODERATION POLICY A summary of each review / trial conducted by the four schools is presented in the following section. 5.1 TRIAL BY SCHOOL OF EDUCATION AND SCHOOL OF TEACHER EDUCATION RECOMMENDATIONS: •
•
USE OF EXTERNAL MODERATORS o Trials of the use of external moderators are seen as important particularly because they may become part of TEQSA’s quality assurance requirements o The cost to the Faculty of external moderation was significant but it is envisaged that the time required and therefore cost will go down once the process is bedded down. The inclusion of student focus groups or interviews as part of the external moderation process, is seen as worthy of further exploration/evaluation. TIME FACTORS Workload / time implications of implementing the Policy need to further investigated. For example: o Standard moderation practice The time to undertake the requirements of the ‘Policy’ is expected to reduce in the future through the use of standard moderation documents e.g. use of common templates across a Faculty. o Pre-­‐moderation: Longer lead time in subject outline preparation The pre-­‐moderation process may be very time intensive because of the discussions required for the development of effective rubics or standards. Where sessional staff fill the subject coordinator role they may be excluded from the discussions because they are not employed till the beginning of the session. A suggestion is that pre-­‐
moderation is carried out as part of post-­‐moderation with any changes implemented in the following teaching session. o During-­‐moderation A quick turn around is required in the during subject moderation to ensure timely return of assignments to students. o Post Delivery The time required for post delivery moderation mean that it will not normally be concluded until after grades are finalized and consequently the findings will inform the design of the subjects for their next offering not the composition of final grades. Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 5 of 15 5.2. TRIAL BY SCHOOL OF ACCOUNTING RECOMMENDATIONS: •
•
•
•
The moderation of a selection of DE Study Guides be specifically included in the Policy. That the Policy be amended to explicitly include the moderation of the assessment rationale. That positive feedback is provided on exemplars of best practice identified during the moderation process. That the use of the term ‘culturally appropriate by the University in connection with subject materials, assessment tasks, and on other contexts, be more precisely defined in the Policy and other relevant University documents. 5.3 REVIEW BY SCHOOL OF BIOMEDICAL SCIENCES RECOMMENDATIONS: PRE-­‐DELIVERY MODERATION •
•
•
•
3.1 and 3.2 – these activities are currently being undertaken as part of MSI QA External moderators The use of external moderators has not been costed into the Policy. It is recommended that these duties be undertaken by School staff. 3.1.2 – it is impractical to exclude current teaching teams. 3.4 – this step is picked up in ACPRs and can be held at a course level. DURING MODERATION • This section should be reworked to focus on ensuring design of task and marching marking schemes well before the task is used. The setting up of marking schemes and training of markers will minimize the need for ‘during moderation’. • There is no time during the session to do this. • 4.4 -­‐ why does the moderator need to do this – this should be done by the subject convenor. • 4.6 -­‐ should be done on every offering of a subject, not every 3 years. POST DELIVERY MODERATION • Documents are unclear in this section particularly how it links with the normal processes around subject and course reviews. OVERALL COMMENTS: • Need clear guidelines on the structure of the reports. • Time will be an issue with the number of the reports that need to be produced. Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 6 of 15 5.4 REVIEW BY SCHOOL OF THEOLOGY RECOMMENDATIONS: •
•
•
•
•
•
Moderation processes: The School already has in place moderation processes but this policy has formalized their thinking and requirements for moderation. Staff require professional development in terms of developing assessment requirements within the School. Highlights the need for staff PD to assist in implementing the Policy. Clear direction needs to be provided to sessional staff Pre-­‐moderation o Pre-­‐moderation is essential to ensure, besides other things, currency of subjects, agreed learning outcomes, syllabus and assessment tasks. o The reference to culturally appropriate assessment is a most apposite inclusion in the guidelines. During moderation o Use of guidelines / templates would be very valuable to the School. Post-­‐moderation o Will require resources to implement. 6.0. CONCLUSION The members of the Moderation Working Party commend the Recommendation made in this report to Academic Senate. Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 7 of 15 APPENDIX 1 DRAFT MODERATION POLICY APPROVED BY: DATE EFFECTIVE: As approved by the Academic Senate on 16 May 2012 (AS 12/…). Commencement of domestic Session 1 in 2013 1. THE SCOPE OF THIS POLICY 1.1
1.2
This policy applies to all undergraduate (including undergraduate honours’ courses) and coursework postgraduate courses and subjects The policy establishes a framework of minimum requirements for the moderation of learning materials and assessment within which Faculties will devise specific guidelines for moderation procedures, allowing for the contractual arrangements involving teaching partners of Charles Sturt University. 2. PURPOSE OF THIS POLICY The purpose of this Policy is to ensure that: 2.1 Course and subject materials provided to students contain clearly-­‐stated objectives, learning outcomes and current learning resources, and that students understand what is expected to complete satisfactorily the requirements of courses/subjects 2.2 Subject assessment tasks are stated unambiguously, consistent with the internal and external accepted curriculum principles for the course and the objectives and learning outcomes for the subject, and that they are appropriately and fairly weighted and are applied consistently irrespective of the place and modes of delivery 2.3 Everyone involved in delivery and assessment has a shared understanding of the CSU Assessment Policy, and will make informed assessment judgments that are transparent and applied consistently to all students. 3. THE OBJECTIVES OF MODERATION Moderation seeks to ensure that: 3.1
The subject convenors/coordinators and course teams comply with the assessment regulations of Academic Senate Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 8 of 15 3.2
Standards to be achieved by students are transparent, widely-­‐understood and observed 3.3
Learning tasks, activities and assessments are consistent with stated learning objectives and outcomes and are set at the appropriate AQF level (for the award) 3.4
Assessment procedures and practices are fair, culturally-­‐appropriate and incorporate clearly-­‐defined assessment (and marking) criteria that are fairly and consistently applied for all students in the same course and subject. 4. THE NATURE AND DEFINITION OF MODERATION The minimum requirement at Charles Sturt University is that moderation of all learning materials and assessments for each subject will be undertaken at least once per year, taking into account course-­‐wide objectives and using course level documents and reports. At Charles Sturt University, moderation has three phases (pre-­‐delivery moderation; moderation during delivery; post-­‐delivery moderation): 4.1 PRE-­‐DELIVERY MODERATION: 4.1.1 Moderation of Subject Learning Materials: involves a comprehensive analysis of the content of the subject outline across all subject offerings, including: The appropriateness of the learning materials, including web-­‐based resources, to the assumed knowledge for the subject and the level of the award The currency and suitability of the learning materials An appraisal of the relation between the subject and the intended course objectives An appraisal of the extent to which previous moderation reviews, feedback and comments have been addressed. •
•
•
•
4.1.2. Review of Assessment Tasks: reaffirms the fairness, clarity and standards of the assessment tasks before they are used. Assessment tasks will be subject to pre-­‐assessment moderation to ensure that: •
•
•
•
They are appropriately aligned to the learning outcomes of the course Assessments are fair and feasible, culturally-­‐appropriate and that reasonable weightings are applied for each task They are appropriately spaced throughout the study period and achievable by students in the allocated timeframe Their content and instructions are presented clearly so that students understand what is required of them in order to achieve a given grade, in accordance with criterion-­‐referencing and Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 9 of 15 •
The academic challenge they demand of students is consistent with the level of the award for the course. 4.2 MODERATION DURING DELIVERY: Moderation During Delivery includes checking the consistency of marking during the assignment, examination and grading process against the assessment/marking criteria and related standards in order to ensure consistency across groups of students in the same subject, as well as the review of grades before approval and communication to students. 4.3 POST-­‐DELIVERY MODERATION: Post Delivery Moderation is an examination of the effectiveness of the moderation process and includes a formal reflection on moderation methods used. 5. THE FREQUENCY OF MODERATION OF THE THREE PHASES Pre-­‐Delivery, During-­‐Delivery and Post-­‐Delivery Moderation should be carried out at least once per year for each subject. 6. RESPONSIBILITIES AND REPORTING Faculties, Schools and Course Teams and Course Directors are responsible for developing their own moderation guidelines and internal reporting practices 6.1 The Dean or nominee (such as a Course Director) will be responsible for moderation of courses and subjects taught by CSU teaching partners, in consultation with Heads of Schools 6.2 The Dean or nominee will provide a summary report to Academic Senate each year, in a format determined by Academic Senate 6.3 The Head of School or nominee will be responsible for the moderation of subjects, and will maintain a record of the moderation processes. 7. MODERATORS 7.1 Moderators will be appointed by the Head of School or nominee 7.2 Moderators will be experienced staff, with appropriate skills, and may be people who are not currently employed as staff of the University or are otherwise independent of the teaching team responsible for the subject 7.3 It is expected that moderators will be provided with appropriate induction training. Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 10 of 15 8. THE ROLE OF ACADEMIC SENATE Academic Senate is responsible for: 8.1 Approving and amending this policy 8.2 With the assistance of the Deputy Vice-­‐Chancellor (Academic), monitoring the implementation and the regular reporting on the outcomes of this Policy. END OF POLICY
Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 11 of 15 GENERAL GUIDELINES FOR FACULTIES, SCHOOLS AND COURSE TEAMS THESE GUIDELINES COULD VARY OVER TIME (NOTE: THESE GUIDELINES ARE NOT PART OF THE UNIVERSITY-­‐WIDE POLICY) 1. INTRODUCTION The policy was developed in response to the 2009 AUQA report which suggested the need for a University-­‐wide moderation policy and the policy was approved “in principle” by Academic Senate in December 2010. It is acknowledged that most Schools have moderation and quality assurance processes already in place and this policy will complement those processes. These guidelines have been developed after the careful consideration of the considerable and comprehensive feedback of staff as well as an analysis of the results of the trialling of the moderation policy in several Schools in 201160. They are intended to be informative rather than prescriptive, and Faculties and Schools will develop their own approaches to satisfying the intent of the University-­‐wide policy. 2. PRE-­‐DELIVERY MODERATION 2.1 MODERATION OF SUBJECT LEARNING MATERIALS 2.1.1 The moderator (or subject team) will examine the details included in the subject outline, including the currency of learning materials and the accuracy of information provided to students as well as other matters included in 4.1.1. This could also be facilitated via the normal QA process for the Subject Outline. 2.1.2 The moderator will provide feedback to the subject convenor/coordinator on the quality of the materials inspected. This may be facilitated via the current QA process for the Subject Outline. 2.1.3 The subject convenor/coordinator will consider the feedback provided. 2.2 REVIEW OF ASSESSMENT TASKS 2.2.1 The moderator (such as MSI teams and QA officers) will form judgments on the items outlined in 4.1.2 2.2.2 The moderator will examine the assessment tasks according to the information communicated to students and will provide feedback to the subject convenor and the Head of School or nominee, on the quality of the materials inspected 2.2.3 The subject convenor/coordinator will consider the feedback provided Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 12 of 15 3. MODERATION DURING DELIVERY 3.1 The moderator can be members of staff involved in teaching the subject in the current teaching period, and could be appointed by the subject convenor/coordinator. The moderator could be the QA officer for the subject (this is a current process, and so should not involve extra work or expense) 3.2 During the assessment moderation process, the moderator could check that, and confirm the extent to which, the specified assessment criteria have been applied to all students in the subject (and the pre-­‐grading “checking” process may include School rules such as double-­‐marking (of say final exam papers) for all potential “fail” grades). 4. POST-­‐DELIVERY MODERATION This stage of the moderation process involves undertaking post-­‐assessment moderation activities in accordance with University policy and School guidelines. The specific processes to support post-­‐assessment moderation will vary according to the nature of the assessment tasks (e.g. exams), the sampling techniques and sample size, and the locations and modes of subject offerings. 4.1 The moderator should not be a member of staff involved in teaching the subject in the current teaching period, and can be a person from outside the University 4.2 The moderator may prepare a brief report on the overall moderation process that can be in the form of a standard template, but there will be provision for written narrative. END OF GUIDELINES Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 13 of 15 CONTACT OFFICER: Associate Professor John Atkinson Convenor, Moderation Working Party 1 May 2012
Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 14 of 15 Report to Academic Senate from its Moderation Working Party – 1 May 2012 Page 15 of 15