Tier 1 and Tier 2 Programs – Assessment Report
Department or program name:
Degree/program assessed:
Submitted by:
Date submitted:
Molecular Biology
Molecular Biology Undergraduate Program
Rachel Watson/Mark Stayton
October 1, 2011
What is your research question about student learning?
This assessment project, the Knowledge Survey Project, assesses certain sub-sections of
each of the three major learning outcomes for the department (as more fully described
under the next question). It was conducted in MOLB 2210/MICRO 2210, General
Microbiology, a core foundational course required of all majors. This project attempted to
answer the following research questions:
1) What level of knowledge and confidence exists for student in General
Microbiology at the beginning and end of the semester?
2) Is there a relationship between confidence and actual knowledge?
3) Are knowledge surveys a useful tool for assessing student learning in the
sciences?
What program or department-level student learning outcomes were assessed by this
project?
The Molecular Biology Department has adopted three learning outcomes:
1) Graduates will be able to propose hypotheses that explain novel biological
phenomena;
2) Graduates will be able to propose experimental tests of hypotheses; and
3) Graduates will be able to effectively communicate the significance of classic
biological principles and emerging biological developments to both expert and lay
audiences.
Describe your assessment project and provide pertinent background information.
In General Microbiology, during the spring semester of 2011 and continuing into this fall
semester of 2011, knowledge surveys were pilot tested (Nuhfer & Knipp, 2003). These
surveys present a series of questions that can be skill- or content-based, that unlike
traditional pre-/post-tests, do not ask students to answer the questions, but instead ask
them to rate their confidence in answering the questions. These surveys have touted
advantages for encouraging student and instructor preparation as well as tracking missing
concepts in curriculum (Nuhfer & Knipp, 2003).
1
The Knowledge Survey Project began in late 2009 with the development of a list of
concepts of which molecular biology majors are expected to have knowledge upon
graduating. Pursuant to this has been the development of a large database of questions
(many appropriate to both Molecular and Microbiology majors) that have been coded into
Bloom’s six levels of taxonomy (Bloom, 1956) (Figure 1 below). This was exploratory
research with a two-fold goal: 1) to determine if knowledge surveys would have value for
our department/program and 2) to determine whether there would be a positive
correlation between class confidence and class correctness as measured using a combined
knowledge survey and pre- / post- test.
Questions on the survey were coded into Bloom’s six levels of taxonomy and thus
include recall, comprehension, application, analytical, synthesis and evaluation questions
(Bloom, 1956). The second and third sub-sections of learning outcome 1 are,
“Appropriate application of fundamental principles” and “Sub-discipline specific
application of facts and principles”. These are assessed by application questions on the
knowledge survey. Evaluative knowledge survey questions assess learning outcome 1,
sub-point #2, “Consideration of alternative hypotheses.” Synthesis questions on the
knowledge survey assess learning outcome 2 as they ask students to produce an original
idea, propose an experiment or design a construct. Although learning outcome 3 is less
heavily assessed, comprehension questions ask students to interpret scientific literature
and communicate biological principles.
Forty questions from the database were selected and used to generate a pre-test /
knowledge survey and post-test / knowledge survey that were administered on a secure
classroom management system (eCompanion). Students were asked to answer each
question and to rate their corresponding confidence in their answer for each question.
Students answered A if they were confident that their answer was sufficient for grading
purposes; they answered B if they felt they could answer at least 50% of the question or
could research the answer in 20 minutes or less; they answered C if they felt unconfident
that they could answer the question for grading purposes. The pre-test and knowledge
survey was administered on the first week of class (January 10-14, 2011). The post-test
and knowledge survey was administered near the completion of the course (April 20- 27,
2011). Data for those students signing informed consent were analyzed (N = 70).
Provide relevant data to answer your research question. What are the key findings?
Scatter plot trend analysis of correctness versus confidence for both pre- and post-test
data show a strong, positive correlation. When pre-test correctness is plotted as a function
of pre-test confidence, a correlation coefficient (Pearson’s R) of 0.70 is found. The same
analysis for post-test data shows an even stronger positive correlation (Pearson’s R =
0.85). These data are summarized in data graph 1. To further verify the relationship
between confidence and correctness, the mean correctness for all students answering
either A, B or C on the confidence survey was determined. Descriptive statistics were
performed using an ANOVA with alpha set to 0.05. Findings show that for students
2
answering A on the pre-test confidence survey, the mean correctness was 73% (standard
error of ±1.7%). Those students answering B on the pre-test confidence survey had a
mean correctness of 42% (±1.1%). Finally, a mean correctness of 3.3% (±0.6%) was
found for students answering C on the pre-test confidence survey. The same analysis
done for post-test data showed similar trends. Those who answered with an A-level
confidence had a mean correctness of 77 % (±1.1%), B-level confidence had a mean
correctness of 54% (±1.2%), and C-level confidence had a mean correctness of 8%
(±1.3%). These data are summarized in data graphs 2 and 3. Both the pre-test and posttest analyses demonstrated significant differences (p<0.005) in correctness between
confidence levels.
Data were also analyzed for changes in correctness and confidence (pre- versus post-test)
within the six levels of Bloom’s taxonomy (Figure 1). Data show an overall increase in
both confidence and correctness in each level of Bloom’s taxonomy from pre- to post-test
(Data graphs 4 & 5). Confidence and correctness is lowest in the synthesis category on
both pre- and post-tests. The smallest degree of improvement for both measurements is
seen in this level as well. The greatest improvement in correctness and confidence is seen
at the evaluative level. Improvements in both confidence and correctness were greater at
the recall and comprehension levels, with application lagging behind.
Initial research questions have thus been addressed. Findings show a positive correlation
between correctness and confidence and an increase in both measures from pre to post
test. There was also general usefulness of this method to the Molecular and Microbiology
programs. Because the pilot study was performed in General Microbiology, a
traditionally sophomore-level course, the greater improvements in recall and
comprehension are expected. However, the outstanding increase in confidence and
correctness in Bloom’s highest level (evaluation) is very positive and worthy of note. On
a less optimistic note, synthesis and application skills clearly need to be cultivated and
thus changes in instructional emphasis relating to these levels are already being
considered. A second pilot project is underway in General Microbiology this semester.
Changes include the replacement of the A, B, C confidence scale with a 1 through 5
numeric scale as well as clarifications in certain questions and instructions.
The strong, positive relationship between correctness and confidence is a key finding.
This finding provides tentative data supporting the use of confidence surveys in their
traditional form in which students do not answer the questions, but instead simply rate
their confidence. However, further pilot testing is needed to assure that even when
students are not simultaneously asked to answer the question, this positive correlation is
still seen.
Equally important findings include the changes in confidence and correctness from preto post-test in Bloom’s six levels of taxonomy. First and foremost, students showed an
increase in both confidence and correctness on every level. Students seem to display the
lowest confidence and correctness in the synthesis category. At this level and at that of
analytical and application, even on the post-test, the number of students relating lack of
confidence still outnumbers those with the greatest confidence. On an opposite note, the
3
increase in both confidence and correctness at Bloom’s highest level (evaluation) shows a
clear strength. Thus, some weaknesses and strengths in instruction have been identified
and instructional goals can now be evolved to address both.
Describe the meaning of your results as they relate to the program strengths and
challenges. What changes to the program or curriculum have been made, are planned, or
contemplated in the future as a result from this assessment project?
In the General Microbiology class, as mentioned above, the pilot data are already being
considered to make changes in instructional emphasis relating to Bloom’s taxonomy. A
second pilot project is underway in General Microbiology this semester to fine tune the
knowledge survey and the process itself. Overall, the response from other faculty within
Molecular Biology and Microbiology has been positive. It is hoped that with increasing
pilot data, the Molecular Biology department will launch a programmatic knowledge
survey project (across the curriculum) in the fall semester of 2012. The Microbiology
program has adopted knowledge surveys in a slightly different format as a part of
programmatic assessment and professors in some upper-division molecular biology
classes have also begun to integrate knowledge surveys into their courses.
References:
Bloom, B. S. (1956). Taxonomy of Educational Objectives- The Classification of
Educational Goals. Handbook I. - Cognitive Domain, New York, NY: David
McKay.
Nuhfer, E. B and Knipp, D. (2003) The knowledge survey: a tool for all reasons. To
Improve the Academy. 21, 50-78. accessed on August 17, 2009 at
http://www.isu.edu/ctl/facultydev/KnowS_files/KnowS.htm
4
Figure 1: Bloom’s Six Levels of Taxonomy
Figure created by Rachel Watson and adapted from Nuhfer (2003) as well as Overbaugh &
Schultz (n. d.)
5
Data Graph 1. Scatter plot and trend analysis of correctness versus confidence
Data Graph 2. Pre-test comparison of confidence rating to correctness.
6
Data Graph 3. Post-test comparison of confidence rating to correctness.
Legend for data graphs 1 and 2:
X-axis – Confidence survey answer.
Y-axis – Mean correctness of answers. This is set of a scale from 0 to 1.0, with 1.0 being 100%
correct.
7
Data Graph 4. Confidence rating according to Bloom’s Taxonomy Level. Pre-test and
Post-test.
Legend:
X-axis – Levels in Bloom’s Taxonomy, with the first bar in each category being the pre-test
confidence score and the second bar being the post-test confidence score.
Bloom’s Taxonomy Levels:
1) Recall
2) Comprehension
3) Application
4) Analytical
5) Synthesis
6) Evaluation
Y-axis – Mean confidence score with 5 being equivalent to a confidence rating of A.
8
Data Graph 5. Correctness according to Bloom’s Taxonomy Level. Pre-test and Post-test.
Legend:
X-axis – Levels in Bloom’s Taxonomy, with the first bar in each category being the pre-test
correctness score and the second bar being the post-test correctness score.
Bloom’s Taxonomy Levels:
1) Recall
2) Comprehension
3) Application
4) Analytical
5) Synthesis
6) Evaluation
Y-axis – Mean correctness of answers. This is set of a scale from 0 to 1.0, with 1.0 being 100%
correct.
9