2013-­‐2014 Annual Program Assessment Report Please submit report to your department chair or program coordinator, the Associate Dean of your College, and to [email protected], director of assessment and program review, by Tuesday, September 30, 2014. You may submit a separate report for each program that conducted assessment activities. College: Science and Mathematics Department: Chemistry and Biochemistry Program: Chem BA, BS; Biochem BS, MS Assessment liaison: Thomas Minehan 1. Overview of Annual Assessment Project(s). Provide a brief overview of this year’s assessment plan and process. The following assessment activities were planned for this year: a. Assess basic knowledge in general, organic, analytical and biochemistry (SLO1) using standardized exam questions in course finals. b. Review evidence pertaining to the effectiveness of a curricular change to improve student learning: implementation of mandatory discussion sessions for Chemistry 333 (Organic Chemistry 1). c. Review evidence pertaining to SLO4: Work effectively and safely in a laboratory environment, including the ability to follow experimental chemical procedures and maintain a proper lab notebook. d. Review evidence pertaining to SLO6: Perform qualitative and quantitative chemical analysis e. Refine the signature assignment for longitudinal assessment to be given students in Chem 321 (gateway course) and Chem 401 (capstone course) and establish protocols for administration and evaluation of the assessment. 2. Assessment Buy-­‐In. Describe how your chair and faculty were involved in assessment related activities. Did department meetings include discussion of student learning assessment in a manner that included the department faculty as a whole? 1 Assessment activities and plans for assessment were discussed at departmental meetings on a regular basis, with the majority of faculty receptive to active participation in gathering and analyzing assessment data related to our program learning outcomes. In addition, many of our department faculty also contributed to the development of a signature assignment for longitudinal assessment of student learning. 3. Student Learning Outcome Assessment Project. Answer items a-­‐f for each SLO assessed this year. If you assessed an additional SLO, copy and paste items a-­‐f below, BEFORE you answer them here, to provide additional reporting space. 3a. Which Student Learning Outcome was measured this year? SLO 1: Demonstrate basic knowledge in the following areas of chemistry: analytical, biochemistry, organic, and general chemistry 3b. Does this learning outcome align with one or more of the university’s Big 5 Competencies? (Delete any which do not apply) • Critical Thinking • Quantitative Literacy 3c. What direct and/or indirect instrument(s) were used to measure this SLO? The instruments used to measure this SLO were multiple choice questions taken from standardized American Chemical Society exams in organic, analytical, and biochemistry, either administered as a course pre-­‐test or embedded in course final exams. Organic: In Chem 333 (Organic Chemistry I, 44 students) 5 questions from the ACS standardized exam in organic chemistry were embedded in the course final exam. In Chem 334 (Organic Chemistry II, 120 students) the entire ACS standardized exam in organic chemistry (70 questions) was administered on the last day of class; in two other sections of Chem 334 (67 students and 59 students, respectively), 10 questions from the ACS standardized exam in organic chemistry were embedded in the course final exam. Analytical: In Chem 321 (Analytical Chemistry I, 30 students) and Chem 422 (Analytical Chemistry II, 26 students) a pretest consisting of five questions from the ACS standardized exam in analytical chemistry was given on the first day of class; the same questions were used for both courses for a comparison of student performance across the analytical sequence. Biochemistry: in two sections of Chem 462 (Biochemistry II, 25 students and 22 students, respectively) 8 and 10 questions from the ACS 2 standardized exam in biochemistry were embedded in the course final exam. General Chemistry: In Chem 101 (General Chemistry I, 65 students), a pretest consisting of 24 multiple-­‐choice questions selected from the textbook author’s testbank material was administered via Moodle at the beginning of the semester. 47 students completed the assessment. 3d. Describe the assessment design methodology: For example, was this SLO assessed longitudinally (same students at different points) or was a cross-­‐sectional comparison used (Comparing freshmen with seniors)? If so, describe the assessment points used. A cross-­‐sectional design methodology, in which freshman (Chem 101, Chem 102) are compared with sophomores/juniors (Chem 333, 334, 321, 422) and seniors (Chem 411, 433, 461, 462, 464), was used: General Chemistry: Chem 101 and Chem 102 are taken by both majors and non-­‐majors, and are typically populated by students in the first year of required chemistry courses for their major. The foundational concepts introduced in these courses are crucial for student success in all subsequent courses in the major. Thus, assessment at this introductory stage will allow us to establish a baseline level of student performance useful for comparison with assessments done in later courses in our undergraduate program. Organic Chemistry and Analytical Chemistry: the majority of our majors take Chem 321, Chem 333, Chem 334 and Chem 422 after completion of Chem 102. Assessment in these courses may provide information on how well our students retain fundamental concepts and apply foundational principles as they progress through the program. Inorganic Chemistry (Chem 401), Biochemistry (Chem 461, 462/L and Chem 464), and upper division lab courses (Chem 411, 433): for the chemistry and biochemistry seniors who typically make up a significant portion of the students in Chem 401, Chem 411, Chem 433, Chem 461, Chem 462 or Chem 464, these classes typically would be among the last courses they take in the majors, and thus assessment in these classes may provide information on how our students have been able to both understand and apply fundamental concepts in chemistry they have learned throughout the program. 3e. Assessment Results & Analysis of this SLO: Provide a summary of how the results were analyzed and highlight findings from the collected evidence. Organic Chemistry: In Chem 333 (spring 2014), 22 out of 44 students achieved the benchmark level of performance, which is 50% or more of the assessment questions answered correctly. In two sections of chem 334 (spring 2014), two instructors selected questions to embed in their final exam: in one section, 31 out of 59 students (53% of the class) achieved the benchmark level of performance (50% or more of the assessment questions answered correctly); in another section 35 out of 67 students (52% of the class) achieved the established benchmark level of performance (60% or more of the assessment questions answered correctly). 3 Thus, for instructors who selected questions to embed in their final exams, a very consistent level of performance of our students is observed: approximately half of a given class is able to achieve the benchmark for success. Another Chem 334 instructor (fall 2013) gave the entire ACS standardized exam in organic chemistry on the last day of class to a total of 120 students. 24 students out of 120 (20%) achieved the benchmark level of performance, which is 50th percentile (41 out of 70 questions correct or more), the national average (50% of students nationwide taking the exam scored 41 or higher). The average score was 31/70, which is 21st percentile. These results are essentially the same as the 2012 Chem 334 assessment (20% of students at or above benchmark), but significantly better than what was recorded for chem 334 previously (12.5% of students at benchmark in 2011). These results suggest that while the proficiency level of our students in organic chemistry has improved over the years, much work still needs to be done: most of our students are performing well below the national average when looking at data from the entire ACS exam, and even with instructor-­‐selected questions, approximately half of a given class is still not achieving the established benchmark level of success. Although the majority of students in Chem 333 and Chem 334 are not Chem/Biochem majors, these results also suggest our students are having difficulty in extrapolating concepts they learn in the lecture and lab courses to questions they haven’t specifically studied for, which represents a weakness in critical thinking ability. Analytical Chemistry: In Chem 321 (fall 2014), only 20% of the students (6/30 students) achieved the benchmark level of performance (3/5 or higher) on the course pre-­‐test. In contrast, in Chem 422 (fall 2014), 50% of the students (13/26) achieved the benchmark level of performance on the same pre-­‐test. The results indicate that students learn the material from Chem 321 and seem to perform better on the ACS questions after having taken the course. Selection should be considered, as the only students taking Chem 422 successfully passed Chem 321 and the students that did not pass Chem 321 were not assessed. Biochemistry: in Chem 462 (Biochemistry II), a set of 8-­‐10 assessment questions were embedded in the course final exam. In Fall 2013, 21 out of 25 students (84%) achieved the benchmark level of 5 out of 8 questions correct; in Spring 2014, 15 out of 22 students (68%) achieved the benchmark level of 5 out of 10 questions correct. These results are in line with previous years’ assessments in biochemistry and it appears that in general a greater percentage of students in our biochemistry courses are achieving benchmark levels of success relative to those in 300-­‐level organic chemistry. Results from the administration of the entire ACS exam in biochemistry would be revealing in that a direct comparison with performance in Chem 334 could be made. General Chemistry: In Chem 101 (fall 2014), 66% of students (31/47) achieved the benchmark of 50% or more assessment questions correct. With a benchmark of 66% or more assessment questions correct instead, only 15% (7/47) are performing at or above the benchmark level. Since this was a pre-­‐test only, the assessment gives vital information about the retention of concepts students learned in the preceding course, Chem 100 (Principles of Chemistry). 4 3f. Use of Assessment Results of this SLO: Describe how assessment results were used to improve student learning. Were assessment results from previous years or from this year used to make program changes in this reporting year? (Possible changes include: changes to course content/topics covered, changes to course sequence, additions/deletions of courses in program, changes in pedagogy, changes to student advisement, changes to student support services, revisions to program SLOs, new or revised assessment instruments, other academic programmatic changes, and changes to the assessment plan.) Organic Chemistry: The problem of student performance in organic chemistry has been highlighted in previous assessments and has prompted the departmental assessment and curriculum committees to propose the creation of a mandatory problem-­‐solving session, Chem 333D, to accompany the Chem 333 lecture course. This course was implemented for the first time in Spring 2013 and results for the new discussion course in terms of student performance in the corresponding lecture are positive (vide infra). However, it is now clear that students would also benefit from a discussion/problem solving session linked to the Chem 334 lecture course. While Chem 334R is mandatory for our majors, at present we have not offered 334R sections linked to the 334 lectures. This has caused major problems with students in 334R that are enrolled in different 334 lecture sections operating at different paces and different topic sequences. A solution (subject to a department budget increase) would involve offering a 334R section linked to each 334 lecture every semester. One suggestion to even further improve student performance in organic chemistry is to incorporate an online homework component, as several examples of successful implementation of this strategy already exist in the department. Instructors in the organic division are now considering the use of the Sapling online homework system which serves as a tutor for students by guiding them from wrong answers to right answers on each homework problem assigned. Analytical Chemistry: A broader selection of questions will be used in upcoming assessments since new ACS Analytical Exams have been acquired by the department. This past year’s assessment using post-­‐tests was a pseudo-­‐longitudinal study (the same set of students was not used). However, members of the analytical faculty plan on tracking the Chem 321 students through Chem 422 in future semesters to provide a fully longitudinal study as students progress through the major. Furthermore, as has been done in the past, comparison of pre-­‐test results with student performance on assessment questions embedded in course final exams will provide an indication of “value added” learning through each course. Biochemistry: For Chem 461, 462 and 464, increased use of the Sapling on-­‐line homework system is being implemented in the current academic year as well as increased use of in-­‐class problem solving sessions geared toward concepts that appeared 5 difficult for the students based on the assessment questions. There are also plans to expand the numbers of questions being used for assessment. The prior question cohort may not have been capturing a clear picture of student knowledge gaps and gains. General Chemistry: Examination of the most often selected incorrect answer for each assessment question reveals underlying misconceptions that need to be addressed in classroom discussion throughout the semester. In the future the assessment will be extended to include a post-­‐test to measure improvements in student understanding of fundamental general chemistry concepts at the completion of Chem 101. 3a. Which Student Learning Outcome was measured this year? SLO 4: Work effectively and safely in a laboratory environment, including the ability to follow experimental chemical procedures and maintain a proper lab notebook. 3b. Does this learning outcome align with one or more of the university’s Big 5 Competencies? (Delete any which do not apply) • Written Communication • Quantitative Literacy 3c. What direct and/or indirect instrument(s) were used to measure this SLO? In Chem 422L (Analytical Chemistry II Lab) A Laboratory notebook rubric adapted from a general department notebook rubric template was the instrument used to measure this SLO. In Chem 411 (Synthesis Lab) and Chem 433 (Organic Analysis) points are assigned for notebook maintenance. Safety in department teaching labs was assessed by evaluating accident logs maintained by our department safety officer. 3d. Describe the assessment design methodology: For example, was this SLO assessed longitudinally (same students at different points) or was a cross-­‐sectional comparison used (Comparing freshmen with seniors)? If so, describe the assessment points used. A cross-­‐sectional design methodology, in which sophomores/juniors (Chem 422L) and seniors (Chem 411, Chem 433), were compared: 6 Chem 422L is a laboratory course taken as a co-­‐requisite with the corresponding lecture course in analytical chemistry. The majority of the students in this lab are chemistry or biochemistry majors. All students taking this course have completed general chemistry labs (Chem 101L and Chem 102L) as prerequisites, and thus they have already been introduced to safety protocols and procedures in the lab as well as the importance of maintaining a proper lab notebook and following chemical procedures. Assessment in these courses thus reveals how well students have learned to implement these basic procedures and techniques as they encounter more challenging experiments. Chem 411 and Chem 433 are typically among the last courses taken by students in our degree programs. Assessment in these courses should confirm that students are able to maintain a professional lab notebook while performing advanced experiments in chemistry with adequate regard for their safety and the safety of others. 3e. Assessment Results & Analysis of this SLO: Provide a summary of how the results were analyzed and highlight findings from the collected evidence. Chem 422L: Two of the 10 Chem 422L experiments are graded exclusively on notebook performance. The average score on these experiments was 11.81/15 and 13.35/15. Only 17% of the class failed to score at least 75% on these experiments, demonstrating that students can adequately keep lab notebooks. For all ten laboratory experiments in CHEM 422L, the average performance was 79.9%. All students achieved a grade of at least 50%, and only 17% of the class failed to achieve 75% or better. This demonstrates that students can follow directions. No safety incidents were reported during F13 and S14 semesters, demonstrating that students can work safely in lab. Chem 411 and Chem 433: In one instructor’s perfunctory evaluation of lab notebooks in Chem 411 and Chem 433 (2012-­‐2014), only 50% of the enrolled students appear to be able to maintain readable and sufficiently detailed notebooks. For another Chem 411 instructor (Fall 2009-­‐Fall 2011), lab notebooks were randomly spot-­‐checked during lab periods 3-­‐4 times during the semester and points (out of a total of 20) were assigned. One point was deducted for each for the following: missing table of contents entry, missing date entry for each experiment, blank page, missing procedure entry, missing data entry (yields, mass, volume, etc), experiment specific critical observation entry (if any). The average notebook score for the first check was between 15.1-­‐
18.1; the average notebook score for the final check was 19.9-­‐20.0. Thus, even students having very low notebook scores in the beginning of the semester learned how to maintain a proper lab notebook to score perfect (20 out of 20) by the end of the semester. However, rubrics were not used for notebook evaluation in these upper division labs. All students completed their synthesis projects in Chem 411, indicating the ability to follow experimental chemical procedures. Also, no safety incidents were reported for either Chem 411 or Chem 433 during the period F09-­‐Sp14, again demonstrating that students can work safely in lab. 7 3f. Use of Assessment Results of this SLO: Describe how assessment results were used to improve student learning. Were assessment results from previous years or from this year used to make program changes in this reporting year? (Possible changes include: changes to course content/topics covered, changes to course sequence, additions/deletions of courses in program, changes. While it appears that students in analytical chemistry labs are able to maintain adequate lab notebooks, information from the upper division courses suggests that more improvement could be achieved in this area, and that a good strategy may be to incorporate random notebook spot checks several times a semester. For certain it will be important for all lab instructors to use some form of notebook rubric in the future, so that results in different courses may be more easily compared, and also so that students know how they will be evaluated and what the expectations of each instructor are with respect to notebook maintenance. Data from Chem 422 as well as Chem 411 and Chem 433 indicate that students in general are able to follow experimental chemical procedures and can work in the laboratory safely. Emphasis on safety lectures at the beginning of all department lab courses (as well as safety quizzes following those lectures) may be the reason for student awareness of and attention to safety issues. These efforts will continue so that the department maintains its safety record in the upcoming years. 3a. Which Student Learning Outcome was measured this year? SLO 6: Perform qualitative and quantitative chemical analysis 3b. Does this learning outcome align with one or more of the university’s Big 5 Competencies? (Delete any which do not apply) • Critical Thinking • Quantitative Literacy 3c. What direct and/or indirect instrument(s) were used to measure this SLO? ACS exam questions in Analytical Chemistry (Chem 321, Chem 422) Accuracy of results in Chem 422L Correct vs. partial structural analyses in Chem 433 8 3d. Describe the assessment design methodology: For example, was this SLO assessed longitudinally (same students at different points) or was a cross-­‐sectional comparison used (Comparing freshmen with seniors)? If so, describe the assessment points used. A cross-­‐sectional design methodology, in which sophomores/juniors (Chem 321, 422/4L) and seniors (Chem 433), were compared: Chem 321 and Chem 422/L: are courses taken by chem/biochem majors typically after completion of general and organic chemistry. At this point, students have been extensively exposed to problem-­‐solving activities in both general and organic chemistry, and thus assessment in the analytical sequence is designed to evaluate if students can apply critical thinking and analytical skills to practical, real-­‐world chemical problems. Chem 433: is typically taken by senior chem/biochem majors and requires students to use a wide battery of analytical and spectroscopic techniques to identify the structure of unknown organic compounds. Assessment in this course should thus confirm that students are able to successfully interpret a body of experimental data to establish a logical chemical structure. 3e. Assessment Results & Analysis of this SLO: Provide a summary of how the results were analyzed and highlight findings from the collected evidence. Chem 422L: Eight of the 10 CHEM 422L (Fall 2013 and Spring 2014) experiments are at least partly graded on accuracy of results. Of these eight, the average score for experiment reports was 11.18/15 (78.6%). Of all 96 graded reports, 92 reports achieved a minimum of 50% and 67% of all students scored at least 75%, demonstrating that students can perform quantitative chemical analysis at the intermediate stage in their progress through the degree program. Chem 433: For the last two years, the success rate for students correctly identifying their unknown compounds has been 75-­‐80%. For incorrect structures, students still come up with a significant amount of correct partial structural analysis. This again demonstrates that the majority of our students can perform chemical analysis by the time they complete their degree program. Chem 321 and Chem 422: The course topics in analytical chemistry involve a significant amount quantitative and qualitative analysis, and standardized ACS exam questions have been used recently for assessment of student learning in these courses (vide infra): in Chem 321, only 20% of the students (6/30 students) achieved the benchmark level of performance (3/5 or higher) on the course pre-­‐test. In contrast, in Chem 422, 50% of the students (13/26) achieved the benchmark level of performance on the same 9 pre-­‐test. The results indicate that students learn the material from Chem 321 and seem to perform better in the area of quantitative and qualitative analysis after having taken the course. 3f. Use of Assessment Results of this SLO: Describe how assessment results were used to improve student learning. Were assessment results from previous years or from this year used to make program changes in this reporting year? (Possible changes include: changes to course content/topics covered, changes to course sequence, additions/deletions of courses in program, changes As mentioned above, future assessment in Chem 321 and Chem 422 will involve comparison of pre-­‐test results with student performance on assessment questions embedded in course final exams, thus providing an indication of “value added” learning through each course. 4. Assessment of Previous Changes: Present documentation that demonstrates how the previous changes in the program resulted in improved student learning. We have implemented mandatory discussion (recitation) sessions for Chemistry 333 (Organic Chemistry 1) in the Spring of 2013, responding to previous assessment data suggesting that students attending problem solving sessions in addition to lecture performed better in the course than students who did not attend such sessions. With a full year of data so far, the results are encouraging: For one of our instructors, in Fall 2012 Chem 333 with optional Chem 333R, the class D/F percentage was 31%, while for Summer 2013 Chem 333 with mandatory Chem 333D, the class D/F percentage was only 9%! For another instructor, in Spring 2012 Chem 333 with optional Chem 333R, the class D/F percentage was 60%, while for Spring 2013 Chem 333 with mandatory Chem 333D, the D/F percentage was 42%. These preliminary results are quite positive and encourage the department faculty to find ways to implement linked discussion sessions for other courses such as Chem 334, Organic Chemistry II. Recent assessment data for the optional Chem 334R sections again emphasize the need to make the course mandatory for all students in the course: in Spring 2014, 57% of students (20 out of 35) enrolled in Chem 334R achieved the benchmark level of performance on the selected ACS standardized exam questions (5/10 or more correct), compared with only 46% of students (11 out of 24) without Chem 334R who achieved the same benchmark. 10 5. Changes to SLOs? Please attach an updated course alignment matrix if any changes were made. (Refer to the Curriculum Alignment Matrix Template, http://www.csun.edu/assessment/forms_guides.html.) N/A 6. Assessment Plan: Evaluate the effectiveness of your 5 year assessment plan. How well did it inform and guide your assessment work this academic year? What process is used to develop/update the 5 year assessment plan? Please attach an updated 5 year assessment plan for 2013-­‐2018. (Refer to Five Year Planning Template, plan B or C, http://www.csun.edu/assessment/forms_guides.html.) We are currently on schedule with our five-­‐year plan, starting the third year of the five-­‐year cycle. This year we were to assess SLO’s 4 and 6, and thus assessment data from as many courses in our program as possible were obtained. We have been assessing SLO1 on a yearly basis, since volumes of assessment data are generated each semester from the use of ACS exam questions. Revisions to the 5-­‐year plan will be discussed at future departmental meetings, and the suggested plan for 2014-­‐2019 is attached. 7. Has someone in your program completed, submitted or published a manuscript that uses or describes assessment activities in your program? Please provide citation or discuss. N/A 8. Other information, assessment or reflective activities or processes not captured above. We are in the process of developing a signature assignment for longitudinal assessment of student learning. We have collected appropriate questions from each subdivision of chemistry (physical, organic, analytical, inorganic and biochemistry) to include in the assignment that will be administered to students in our gateway course (Chem 321) and then again in our capstone course (Chem 401). Although a question bank was developed by faculty last year, it was felt that multiple-­‐choice questions would be more appropriate, making the evaluation phase more rapid (and involving less faculty time commitment) while still providing the relevant information on student “value-­‐added” learning in our program. As a result, we have been re-­‐crafting our assessment questions in multiple-­‐choice format and a sample of the assignment is attached. Previous plans were to administer the assignment online via Moodle; however, after extensive faculty discussion over fears that students will have access to outside 11 information in an out-­‐of-­‐class online setting while doing the assignment, it was decided that the assessment should be given in class. It is anticipated that the assignment will be administered in both gateway and capstone courses this semester so that results may be evaluated for next year’s assessment cycle. 12