Reversing the process: Inventing as a means of effective learning Shoshanah R. Jacobs, Department of Integrative Biology, University of Guelph Abstract “Inventing” rather than the traditional “tell and practice” approach to teaching has been shown to promote deeper learning and increase retention time in students in addition to other benefits. Presented here are the results of a study on the effectiveness of the Reverse Journal Assignment (RJA), a learning tool designed to guide fourth year university students to ‘invent’ the rules of scientific writing and data analysis in two targeted article sections (Materials & Methods and Statistics), sections where student performance was identified as weak. The results show an improvement in the performance of students in these areas after having completed the RJA, indications of higher level learning, and positive attitudes towards the value of the RJA as a learning tool. Keywords: Inventing, Reverse Learning, Instructional Assessment Introduction With so much free and readily accessible information available to students, the university curriculum is shifting its focus from teaching content to teaching critical thinking through student-­‐centered active learning pedagogy. Evidence of this shift includes the popularisation of the ‘flipped classroom’, where content is made available prior to class time and students are expected to arrive to scheduled classes prepared; class time is then used for practice and engaging with other students and the instructor prior to assessment. In this case, however, the ‘order’ of learning in the flipped classroom, namely ‘tell and practice’ (the most prevalent pedagogy in the United States; Hiebert, J. Stigler, 2004) is unaltered (Lage, Platt, & Treglia, 2000). In the traditional model, students learn content during class time, practice the concepts (during labs, on their own or in study groups, with or without instructors and/or teaching assistants) and are then assessed. The primary improvement with flipped classrooms is that the practice component of learning is guided by the instructor, an activity that could most certainly promote deeper learning. A different approach to teaching is not to change the address at which the ‘tell and practice’ pedagogy occurs (flipped classroom), but to change the actor doing the telling. It has been argued that, within a traditional classroom format (tell and practice), it is the instructor who learns the most (Huba and Freed 2000) because in order to prepare a lecture the instructor must gather the content, organise it in a meaningful way, develop visual materials to support the content, and then teach it to others. With careful design, the student can, in large part, become the instructor by focusing the assessment on skills-­‐
based learning outcomes, encouraging the selection of content of personal interest, and allowing creativity, the student can then ‘devise the rules’ in a personally meaningful way. Allowing students to invent concepts, rather than telling them, can promote deeper learning (i.e. Hiebert & Wearne 1996) with longer retention (i.e. Schwartz, Chase, Oppezzo, & Chin, 2011), engage students more quickly in problem solving activities (Taylor, Smith, Stolk, & Spiegelman, 2010) that leads to better performance on (appropriate) assessments (Eddy, Crowe, Wenderoth, & Freeman, 2013). What follows are the results of a small scale study on the effects of ‘reversing’ the order of learning with respect to a class assignment. Research Rationale In the Fall of 2012, the instructor taught a core fourth year undergraduate biology course in which an independent research project was a part of the curriculum with an assessed Final Report. Students had to devise their own hypotheses, methods, collect and analyse data and write a report in the style of a manuscript for submission to a peer reviewed journal.. That year, the instructor observed that though the grades were generally quite high (average of 74.5%), there was a distinct unevenness in the distribution of those grades among the different categories in the rubric. Generally, those categories assessing literacy (i.e. Introduction, Discussion) were well done, while those related to scientific inquiry and numeracy (i.e. Materials & Methods 73.5%, Results 64.4%, Statistics 62.4%) were rather poorly done and were responsible for lowering the average overall grade. Based upon the Reverse Journal Club assignment presented at the Western Conference for Science Education (Ju, 2013), the instructor designed the RJA (Appendix 1) to help students in future cohorts improve their reporting of scientific methods and data and devise appropriate statistical analyses. This assignment guides students in learning the rules of scientific writing by challenging them to write the Materials & Methods and Results sections in the format of a peer reviewed scientific journal article articles given a series of data graphs. The graphs were taken from a published paper and the axes labels and other identifiers were removed so that students had to ‘invent’ the axes and recreate what happened in order to yield such graphs. It was not at all necessary for the interpretation to be based upon a real scenario; it had only to be plausible. It was up to the student to invent the rules regarding what information is required by reading published scientific literature and it allowed them to creatively construct this information in a way that complements the provided data. It was intended that performing the RJA would increase their abilities to describe, evaluate and interpret scientific information. Therefore, it was predicted that performing the RJA would increase students’ assessment performance in the previously weak sections of the Final Report (namely Materials & Methods, Results and Statistics) while having little impact on their grades in the other sections. Materials & Methods In the Fall 2012 and 2013, 41 and 45 students completed the course for credit, respectively. The researcher has been the instructor of the course since Fall 2012. In addition to the instructor, there was a lab instructor and teaching assistant. Both were aware that the instructor was conducting this study in 2013 but neither knew the specific details of the study including the hypotheses and methods. Both the 2012 and 2013 course offerings were taught in the same way with the same materials and activities with the exception of including the RJA prior to conducting the independent research and some minor modifications to the other assessments; a small assignment, apart from the RJA was required of students in Fall 2013 to replace an ineffective question on the final exam from previous years (the Dichotomous Key Assignment). This assignment had the same learning outcomes as the question on the final exam. The independent project and the Final Report required students to devise their own research questions, sampling protocols and data analyses. They had to conduct the research and write up the results following the submission guidelines of a known peer-­‐reviewed journal. The selection of the topic for the project and the hypotheses were devised early in the semester, prior to completing the RJA. Of the 41 students registered in the 2012 cohort, 32 did not reclaim their graded Final Reports to review the feedback. These were therefore available as a ‘control’ for this study. These reports were stripped of all personal and year identifiers, assigned a number code, markings were erased and the reports were photocopied to ensure that the Teaching Assistant could not determine to which cohort the Final Report belonged. During the 2013 course offering, the RJA was presented to the class as though it was a regular assessed component of the course, with no special explanation for why it was being required or extra instructions apart from those provided in the assignment. The RJA was worth 10% of the final grade and the weight of the final research report was worth 30% (in 2012 it was worth 40%, Table 1). Table 1: Assessed activities in the two offerings (2012, 2013) of the fourth year undergraduate biology course. Fall 2012 Activity Field observation notebook Independent Project Abstract % final grade 20% 40% Fall 2013 Activity Field observation notebook Reverse Journal Assignment Independent Project Abstract % final grade 15% 10% 30% Introduction Methods Results Discussion References Statistics Format Creativity Laboratory Exam 40% Introduction Methods Results Discussion References Statistics Format Creativity Laboratory Exam Dichotomous Key Assignment 40% 5% At the end of the semester and on the same day, students were required to personally submit their final reports for their independent project by making an appointment with the instructor. Once the student handed in the Final Report, they were informed of the study and asked to consider providing their consent to participate in the study. They were told of the rationale, the methodology, and given 1) a copy of the consent form, 2) the student survey (see Appendix 2), 3) the alternative assignment, and 4) an envelope. The consent form contained the option to opt out or participate. Students were offered 1% on their final grade for participating. An alternative assignment was provided should the student choose not to participate but wanted access to the 1% incentive. Each student was asked to consider the information in the envelope, make their decision, and return the sealed envelope with their name on it to the administration office within 1 week. An administrator then provided the instructor with a list of those names on envelopes so that the 1% credit could be added to the final grades without the instructor knowing the decisions of individual students prior to the submission of the final grades. 41 students from the 2013 class provided the necessary consent to participate in the study. The Final Reports for their independent projects from 2013 were stripped of all personal and year identifiers, assigned a number code, and photocopied for the Teaching Assistant to mark. The Teaching Assistant kindly accepted to do the extra marking from the 2012 cohort in addition to those from 2013. Remarking the Final Reports eliminated the inter-­‐marker variability and allowed the use of the 2012, as a quasi-­‐control to test for any effects of the RJA on learning. In addition, it provided the opportunity to gauge the level of inter-­‐marker reliability between the instructor, who marked all the Final Reports in 2012, and the Teaching Assistant, who re-­‐marked 32 of them. Once the final course grades were submitted to the office administrator, the sealed envelopes were given to the instructor for the purposes of the study and their names were replaced with the corresponding number code assigned to their Final Reports. The comments and the results from the questionnaire were coded and entered in association with the grades. The open-­‐ended questions from the student survey were coded using NVivo version 10. The questions asked whether anything was learned from the RJA regarding the Materials & Methods and Results sections, or overall, for the Final Report (appendix 2). It was observed that a gradient of responses were given from rather low-­‐orders to rather high-­‐orders of learning (six categories), or three categories of no learning (Table 2). These coded categories were assigned to the comments, or parsed comments. Table 2: Coded types of learning or no learning identified in the open ended responses in the questionnaire. Learning Code Learning 1 – Professor expectations Learning 2 – Used Feedback Learning 3 – Formatting Definition Learned about professor’s expectations for future graded assignments Used feedback for subsequent assignments Learned how to format the respective sections Learning 4 – Analysis Techniques Learning 5 – Comprehensiveness Learned about different methods of analysis Learned how to write the respective sections, what information to include, level of details, and the importance of each section and how they are linked to other sections Learned how to evaluate the information presented in each section reflected on the importance of the information, identified challenges or weaknesses, and made connections among the sections to increase understanding of scientific inquiry. Did not learn anything due to other “I don't think it helped mainly commitments, limitations, challenges because I am terrible at stats and outside of the Assignment although we all had to take statistics they never really taught us how to implement them.” Did not learning anything due to having “I feel I had a solid understanding learned the same concept previously of how to write a results section before this assignment.” Did not learn anything due to a challenge “I think with a bit more within the Assignment information, so less time/effort was spent scrounging for information or guessing at ways to cover every angle that we weren't even sure existed this could have been helpful.” Learning 6 – Interpretational No-­‐Learning 1 – Outside Distractor No-­‐Learning 2 – Learned Previously No-­‐ Learning 3 – Internal Distractor Sample comment “... only in terms of knowing what is expected of us...” “… gave some notes on missing details to include in report” “It helped me gain better knowledge of the standard formatting of this section…” “I had to research many types of data collection tools…” “It helped put extra emphasis on the process and continuity of the methods and results. You had to be sure everything was in order and flowed.” “I critically analysed my data and I paid a lot more attention to relationships that may help explain these findings” Statistical and qualitative analyses Inter-­‐marker reliability The distributions of the grades given on the 2012 Final Reports among markers were tested using a G-­‐
Test of goodness of fit. We tested the null hypothesis that there was no difference between the two distributions. Inter-­‐year differences in grades To allow for statistical analysis, the grades given in the Final Report categories (Abstract, Introduction, Materials, Results, Discussion, References, Statistics, Format, Creativity) were binned (Table 3) to eliminate zero-­‐value cells. Table 3: Grade bins used for the analysis of differences in distributions of grades among years. Section Abstract Introduction Materials Results Discussion References Statistics Format Creativity Bins 0-­‐1, 1.5-­‐2 0-­‐1, 1.5-­‐2, 2.5-­‐3 0-­‐1, 1.5-­‐2, 2.5-­‐3 0-­‐1, 1.5-­‐2, 2.5-­‐3, 3.5-­‐4, 4.5-­‐5 0-­‐1, 1.5-­‐2, 2.5-­‐3, 3.5-­‐4, 4.5-­‐5 0-­‐1, 1.5-­‐2, 2.5-­‐3 0-­‐1, 1.5-­‐2, 2.5-­‐3, 3.5-­‐4, 4.5-­‐5 0-­‐1, 1.5-­‐2, 2.5-­‐3, 3.5-­‐4 0-­‐1, 1.5-­‐2, 2.5-­‐3, 3.5-­‐4, 4.5-­‐5 The distributions of the grades within each section were tested for differences using G-­‐Tests of goodness of fit. We tested the null hypothesis that there was no difference among these distributions between years. Questionnaire analyses The questions were grouped into four themes: 1) stress, 2) excitement, 3) connecting the RJA to the Final Report, and 4) performance and experience. The frequencies of the responses to the coded possible answers were graphically represented. Each student’s answers to questions 4-­‐6 from the questionnaires were linked in an Excel Spreadsheet to the distribution of grades that they earned so that an analysis of identified level of learning relative to grade earned could be conducted. Using NVivo’s queries option, matrices were generated to depict distributions of “learning level” associated with each grade on the Materials & Methods, and Statistics sections and their respective answer. Also, an overall analysis was done, with all comments and their associated “learning level” and the final grade received on the Final Report. Differences for distributions in “learning level” with grade were tested for using Chi-­‐square test. In all cases, p-­‐values of 0.05 or less were accepted as demonstrating significant statistical differences. Results and Discussion Inter-­‐marker variation Overall, there was no significant difference in the distribution of grades assigned by either the Course Instructor or the Teaching Assistant on the ‘control’ group of final reports from the 2012 cohort (G8=0.9, p=0.99, n=32). Both markers found that the results and statistics sections were rather weak and though there was no overall difference in the final grades or the distribution of grades, further examination of the data revealed that disagreement was noted in the Materials & Methods and Creativity sections Materials & Methods (Figure 1). 100 95 Grades (%) 90 85 80 75 70 65 60 55 50 Abstract Intro Materials Results Discussion Refs Stats Format Creanvity Figure 1: Inter-­‐marker variation (+/-­‐ SE) of the final reports from the 2012 class between Course Instructor (dark) and Teaching Assistant (light). Assessing inter-­‐marker variation can be difficult. In this case, the likely explanation for this discrepancy is that of a difference in the level of familiarity of the markers with the current discipline-­‐specific techniques and published literature; in both cases, the less experienced marker gave higher grades in those sections than did the more experienced marker. In addition, the more experienced marker had greater knowledge of the available resources regarding the sampling techniques and availability of samples which may have caused the marker to provide a less sympathetic assessment of their choice of methods. Inter-­‐year differences in grades Of the nine assessed components of the Final Report, there was a significant shift in the distribution of grades in five sections including Materials & Methods, Statistics, Format, Discussion, and Creativity (Table 4). There were more students that received higher grades in the Materials & Methods, Statistics, Formatting, and Creativity sections in 2013 than in 2012 Table 4: Average grades of students in the ‘control’ group (2012) and students who completed the RJA prior to the Final Report (2013). A shift in distribution of the grades to the right indicates that more students in 2013 received higher grades in that section; a shift to the left indicates that more students received lower grades. P-­‐values of less than 0.05 are considered significant. Average grade (%) Section
2012
2013
Abstract Introduction Materials Results Discussion References Statistics Format Creativity 89.1 89.1 89.1 69.7 90.0 77.1 56.9 81.6 50.6 93.0 88 91.0 71.0 83.0 81.0 61.0 83.0 53.0 TOTAL 74.5 75 NA *denotes statistically significant differences Distribution Shift Direction NA NA Right NA Left NA Right Right NA G test
p
0.4 0.5 10.2 0.9 8.7 3.2 7.5 7.9 2.2 0.33 0.4 0.003* 0.4 0.02* 0.14 0.05* 0.02* 0.3 2.0 0.3 In all assessed sections of the Final Report the distribution of grades either improved or did not change in 2013 except for the Discussion section, where the grade distribution was shifted towards the lower end of the scale (Table 4). An explanation for the shift in Discussion grades would be purely speculative; for example, did the RJA shift student focus to numeracy skills at the expense of literacy? It is unclear but further research may provide insight. Questionnaire analyses Students expressed having felt some degree of stress about completing both the RJA and the independent project with Final Report (Figure 2A). However, the RJA was a significantly less stressful experience than the Final Report (G3=28.2, p<0.001). Most students felt that completing the RJA did not affect their stress levels regarding writing the Final Report (Figure 2B). Though most students were not excited about completing the RJA (n=15), 11 students expressed some degree of excitement (Figure 2C). Most students felt that completing the RJA improved their understanding of how to write the Final Report (n=26), Materials & Methods (n=26) and the Results (n=21) and most of them said that they recognised the instructor’s intention in designing the RJA (n=25; Figure 2D). Most student reported having felt that completing the RJA helped them to achieve a higher grade on their Final Report (n=24) and none felt as though it was detrimental to their learning or performance (Figure 2D). A
How stressed were you about complenng the final report? How stressed were you about complenng the reverse journal exercise? -­‐20 -­‐15 -­‐10 -­‐5 Extremely A lot 0 5 10 Frequency Stressed 15 20 A little 25 30 Not at all B
Did complenng the reverse journal exercise affect your stress level regarding the final report? -­‐5 0 5 10 15 20 25 30 35 Frequency Yes, it increased No, it had no effect
Yes, it decreased
C
How excited were you about complenng the reverse journal exercise? -­‐20 -­‐15 -­‐10 -­‐5 0 5 10 15 A lot Extremely Frequency Not at all A little Excited D
Do you think that the reverse journal exercise improved your understanding of how to write a final report? Do you think that the reverse journal exercise improved your understanding of how to write results? Do you think that the reverse journal exercise improved your understanding of how to write materials and methods? Did you recognise that the reverse journal exercise was linked to teaching you how to write your final reports? -­‐20 -­‐15 -­‐10 -­‐5 0 5 10 15 20 25 30 Frequency No Yes E
Do you think that performing the reverse journal exercise affected the quality of your final report? -­‐15 -­‐10 -­‐5 0 5 10 15 Frequency No 20 25 Yes Figure 2. Frequency distributions of responses to questionnaire questions. Learning and assessment Most students identified higher orders of learning (learning associated with critical thinking and analysis rather than knowledge of content) from the RJA that was subsequently applied to the writing of both the Materials & Methods and Results sections of the Final Report (Tables 5 and 6). Table 5: Frequency distribution of participant self-­‐reported learning level by their corresponding grade on the Materials & Methods assessment. Learning Level Learning 1 Learning 2 Learning 3 Learning 4 Learning 5 Learning 6 No Learning 1 No Learning 2 No Learning 3 Expectations Used Feedback Formatting Analysis Techniques Comprehensiveness Interpretational Outside distractor Learning Previously Internal Distractor 1-­‐1.5 0 0 0 1 2 0 0 0 0 Grade Bin 2-­‐2.5 1 0 1 1 3 2 0 1 1 3 0 0 1 0 8 12 3 3 1 Total 1 0 2 2 13 14 3 4 2 Table 6: Frequency distribution of participant self-­‐reported learning level by their corresponding grade on the Statistics assessment. Learning Level Learning 1 Learning 2 Learning 3 Learning 4 Learning 5 Learning 6 No Learning 1 No Learning 2 No Learning 3 Expectations Used Feedback Formatting Analysis Techniques Comprehensiveness Interpretational Outside distractor Learning Previously Internal Distractor 0-­‐1 0 0 0 1 0 1 0 1 0 Grade Bin 2-­‐3 4-­‐5 1 0 2 0 0 1 5 1 7 4 13 2 2 0 5 2 4 0 Total 1 2 1 7 11 16 2 8 4 There were significant differences in the distributions of self-­‐reported Learning Level (X2=38.2, p=0.008) and of No-­‐Learning reasons (X2=18.6, p=0.02) with overall grade on the Final Report. Self-­‐reported Learning Level increased towards higher order learning as the grade on the Final Report increased (Table 7). Interestingly, this trend stopped at A-­‐grade students; those students who received an A+ identified lower order levels of learning as having come from the RJA and 3/7 A+ students reported having learned the relevant skills previously. Lower grade students also reported having learned the relevant skill previously; 9/14 students reporting previous learning received a grade of B or lower. Table 7: Frequency distribution of participant self-­‐reported learning level by their corresponding overall grade on the Final Report. Bolded values are higher than predicted. Learning Level Learning 1 Learning 2 Learning 3 Learning 4 Learning 5 Learning 6 No Learning 1 No Learning 2 No Learning 3 Expectations Used Feedback Formatting Analysis Techniques Comprehensiveness Interpretational Outside distractor Learning Previously Internal Distractor D 1 0 0 2 2 0 1 2 1 C 1 1 1 3 14 10 0 4 0 B 2 1 1 4 12 14 5 3 0 Letter grade A 3 2 0 1 10 11 2 2 5 A+ 1 0 2 0 1 0 0 3 0 Total 8 4 4 10 39 35 8 14 6 Most of the students who reported to have not learned anything from the RJA because of a distractor inherent within the design of the Assignment itself, received a grade of A on the Final Report. These comments included: “I think with a bit more information, so less time/effort was spent scrounging for information or guessing at ways to cover every angle that we weren't even sure existed this could have been helpful. Also needed to be done earlier so they could be discussed.” “This is easier to do and makes sense, to form a report based on own story” “Truthfully, no, because I think it contradicts what we are taught about writing scientific papers; that is, to start with a hypothesis, rather than starting from results” Though these students performed well on the Final Report, they seem to be expressing confusion about the objectives of the assignment; the first comment describes the objective of the RJA (to invent so as to ‘cover every angle’) nearly perfectly and the last comment confuses scientific method with scientific writing. Examples of Learning 5 and 6 statements included: “Graphs can mean many different things depending on their context, it's important how they are described” “It forces you to think creatively and analyse all possible aspects of the study” “Needed a lot of thought/analysis in order to effectively interpret and describe the visual representation of the data” Conclusion Both quantitatively and qualitatively, there was supporting evidence for the prediction that the RJA improves student learning in targeted areas of scientific writing and suggests that giving students the opportunity to ‘invent’ the concept is an effective means. ‘Tell and practice’ methods of teaching, though effective at ensuring that students learn a concept, promote ‘brittle learning’, where the student is unable to transfer learning to unfamiliar scenarios or apply knowledge in a different context (Moore and Schwartz 1998 in Schwartz & Martin, 2004). This study complements existing literature that shows that ‘inventing’ teaches students how to evaluate unfamiliar data, allowing them to let go of old interpretations and build new, more flexible schemata (Schwartz & Martin, 2004). Of note in these past studies is that in most cases, students were asked to ‘invent’ while working in small groups of their peers. Perhaps the overall positive effect on learning is, in part, attributable to this dynamic and this is not a confounding variable in this study. Further study with a control student group will be required to isolate the impact of inventing vs telling, though the 2012 cohort in this study represent a quasi-­‐control because they were ‘told’ rather than given the opportunity to invent. Acknowledgements The author would like to thank Bill Ju for thoughtful conversations and helpful information. Much appreciation goes to Mike Wells, Lori Ferguson and Ayla Visser. This study was completed as a requirement of the EnLITE II program offered at the University of Guelph. Many thanks to the members of my Action Learning Set group: Joanne Hewson, Peter Hausdorf, and Jeji Varghese for regular brainstorming sessions, valuable advice, and thoughtful critique. And above all, many thanks to my enthusiastic and supportive students. References Eddy, S. L., Crowe, A. J., Wenderoth, M., & Freeman, S. (2013). How should we teach tree-­‐thinking? An experimental test of two hypotheses. Evolution: Education and Outreach, 6(1), 13. doi:10.1186/1936-­‐6434-­‐6-­‐13 Hiebert, J., & Wearne, D. (1996). Instruction, Understanding, and Skill in Multidigit Addition and Subtraction. Cognition and Instruction, 14(3), 251–283. doi:10.1207/s1532690xci1403_1 Hiebert, J. Stigler, J. W. (2004). A world of difference. Journal of Staff Development, 25, 10–15. doi:10.1126/science.344.6186.820 Huba, M.E., and Freed, J.E. 2000. Learner-­‐Centered Assessment on College Campuses: Shifting the Focus from Teaching to Learning. Boston: Allyn and Bacon, 35. Ju, W. 2013. Reverse journal club: Giving students less but asking for more. Western Conference on Science Education, London, July, 2013. Lage, M. J., Platt, G. J., & Treglia, M. (2000). Inverting the Classroom : A Gateway to Creating an Inclusive Learning Environment. The Journal of Economic Education, 31(1), 30–43. Schwartz, D. L., Chase, C. C., Oppezzo, M. a., & Chin, D. B. (2011). Practicing versus inventing with contrasting cases: The effects of telling first on learning and transfer. Journal of Educational Psychology, 103(4), 759–775. doi:10.1037/a0025140 Schwartz, D. L., & Martin, T. (2004). Inventing to Prepare for Future Learning: The Hidden Efficiency of Encouraging Original Student Production in Statistics Instruction. Cognition and Instruction, 22(2), 129–184. doi:10.1207/s1532690xci2202_1 Taylor, J. L., Smith, K. M., Stolk, A. P. Van, & Spiegelman, G. B. (2010). Using Invention to Change How Students Tackle Problems, 9, 504–512. doi:10.1187/cbe.10 Appendix 1
The Reverse Journal Exercise Have you ever read a scientific article from the very first word in the Abstract, to the very last of the Acknowledgments? Or do you skip around from section to section? Whichever way you do it, it isn’t wrong and, in fact, these formats are designed to allow for either approach. Getting it ‘right’ isn’t easy; there is a real art to writing scientific papers and the Reverse Journal Exercise is designed to help teach: 1) how to construct a paper around the data 2) that it really isn’t too difficult once you know what story you want to tell. The ingredients: % body fat (% body H2O)/10 The mission: Above is a graph series from a scientific article. Your mission is to reconstruct the article (the Materials & Methods and the Results) based upon the data presented above. The method: 1) Spend considerable time looking at these graphs. Tape it to your refrigerator. Each time that you get something to eat, look at the graphs and ask yourself questions about what each piece of data means, about the significance of the axes, the changes in values, what the variables are and what they mean. Do this over several days and each time you get a snack, the data should become clearer. Follow up any ‘gaps’ in your knowledge with literature searches. 2) Stop thinking about the exercise. In fact, put it away, have a bbq with friends, go for a walk… 3) Return to the graphs and repeat step 1. What new thing have you discovered? 4) Figure out a way that those data could have been collected. What would you need? How would you do it? How do you collect those types of samples? How do you care for them, treat them? (hint: this is where scholar.google.com becomes super useful!) 5) Write a draft of the Materials & Methods. Be creative but make sure that you can get the data presented in the graph from your experimental design. 6) Write a draft of the results section. Include at least four statistical analyses. Obviously you will have to make up the numbers, but by looking at the graphs you will be able to determine what types of tests are appropriate and what their probable outcomes should be. This is the ‘tricky’ part. To do this you will have to reacquaint yourselves with stats. And that is what we are here for so please take advantage, ask us questions but before you do, come up with an idea and use us as a sounding board rather than just asking us for the answers. You will discover that, in this exercise there isn’t only one right answer. You can present more data, fictitious data … whatever you want. Just make sure that it all makes sense. 7) Put the exercise away and do something fun. 8) Read the entire exercise starting with the Materials & Methods and evaluate whether everything makes sense and flows nicely. 9) Hand it in on October 24th and hope for the best! The rubric: Materials & Methods: -­‐ consistent with collected the samples for the data presented in the graphs: -­‐ specific enough to be repeatable by anyone reading it: 2% 2% Results: -­‐ text complements the graphs and ‘places’ them within a context -­‐ statistical analysis is appropriate and supports the graphs 3% 3% -­‐-­‐-­‐-­‐-­‐-­‐-­‐ 10% Appendix 2 Dear ZOOL4920 student, Thank you for your participation in this study. It was designed to determine whether the Reverse Journal Exercise is helpful in teaching how to write a scientific article. The principle is based on a study by Schwartz et al 2011 who determined that students learned more effectively by ‘inventing’ the concept rather than just ‘practicing’ it. Please take the next few minutes to answer the following questions and return this questionnaire along with the consent form in the sealed envelope provided. These documents will NOT be accessed until after your final grades have been submitted. If you have any questions about this study please do not hesitate to contact me directly or anyone at the Research Services Office. 1. How stressed were you about completing the Reverse Journal Exercise? a. Not stressed at all b. A little stressed c. Stressed d. A lot stressed e. Extremely stressed 2. How excited were you about completing the Reverse Journal Exercise? a. Not excited at all b. A little excited c. Excited d. A lot excited e. Extremely excited 3. Did you recognise that the Reverse Journal Exercise was linked to teaching you how to write your final report? a. Yes b. No 4. Do you think that the Reverse Journal Exercise improved your understanding of how to write Materials & Methods? a. Yes b. No Please comment: _____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________ 5. Do you think that the Reverse Journal Exercise improved your understanding of how to write Results? a. Yes b. No Please comment: _____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________ 6. Do you think that the Reverse Journal Exercise improved your understanding of how to write a Final Report? a. Yes b. No Please comment: _____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________ 7. Do you think that performing the Reverse Journal Exercise affected the quality of your final report? a. yes, it improved the quality of my final report b. yes, it decreased the quality of my final report c. no, it did not affect the quality of my final report 8. How stressed were you about completing the Final Report? a. Not stressed at all b. A little stressed c. Stressed d. A lot stressed e. Extremely stressed 9. Did completing the Reverse Journal Exercise affect your stress level regarding the Final Report? a. yes, it increased my stress level b. yes, it decreased my stress level c. no, there was no effect on my stress level 10. Please describe your opinion regarding involving students in the study of teaching techniques during ongoing classes: _____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________ Thank you for your participation; it will help to improve the way that we teach and we look forward to seeing the results!