Will this be on the test? Characterizing the cognitive levels

4/23/13 Will this be on the test?
Characterizing the cognitive levels
of undergraduate biology
Jennifer Momsen, North Dakota State University
Undergraduate
Biology
Classes
Facts
Facts
More facts
Why?
1 4/23/13 Student expectations of biology
1. 
2. 
What is the nature of the knowledge I am
learning?
What should I do to learn in biology?
Will this
be on the
test?
Will this
be on the
test?
Will this
be on the
test?
Hall et al 2011
Course expectations of students
1. 
Learning goals and outcomes
Students will…
define, describe, explain, identify, label, etc
2. 
Nature of assessments
Students will…
memorize terms, processes, examples
bring a number 2 pencil
3. 
Classroom activity
Students will…
take copious, accurate notes (and maybe complete the crossword)
2 4/23/13 Characterizing assessments
 
Concept and content knowledge
o 
o 
 
Vision and Change Core Concepts
Map to a textbook
Skills
Cognitive skills using Bloom’s taxonomy, Webb’s
depth of knowledge
o  Vision and Change Core disciplinary practices
o 
Bloom’s taxonomy, cognitive
domain
o 
o 
o 
Identifies 6* levels
of cognitive skills
Nested or hierarchical
Higher levels require
more complex cognitive
processes
Synthesis &
Evaluation
Analysis
Application
Comprehension
Knowledge
*Note: Synthesis and Evaluation are collapsed into
one level as they are not truly hierarchical.
3 4/23/13 Applying Bloom’s taxonomy
Independent raters assign a Bloom’s level to Assessment items weight by Point value discuss to Reach consensus on each item Does the MCAT stress
rote memorization?
N
AP Biology
Undergraduate
GRE
MCAT
Medical School
157
111
108
109
101
Weighted
ratings
2.35 ± 0.12
2.43 ± 0.14
2.32 ± 0.10
2.57 ± 0.09
1.93 ± 0.08
Zheng et al 2008
4 4/23/13 Study objectives
 
Characterize and compare the cognitive skills
routinely assessed in
o 
o 
 
Introductory biology courses (Study 1, 2)
Upper division biology courses (Study 2)
Characterize the relationship between
cognitive skill level and difficulty in biology
(Study 2)
Study 1
Introductory biology, nationally
56 faculty (from FIRST II or SI) taught 77 introductory biology courses 2 Independent Raters assigned a Bloom’s level to Exam/quiz quesKons (n=9,317) IRR = 0.65
RaKngs averaged 5 4/23/13 Study 1
Course demographics
Momsen et al 2010
Study 1
Introductory biology, nationally
100%
Syllabus Goals
Assessment Items
80%
60%
40%
20%
0%
1
Knowledge
2
3
4
Bloom's Level
5
6
Synthesis
Momsen et al 2010
6 4/23/13 Study 2
Characterizing the biology major
Large, public university, very high research activity
o 
Single department of biological sciences
• 
• 
• 
17 research-active faculty
~500 undergraduate majors
Courses serve science and engineering majors
Focus on a single academic year (2010-11)
o  Collected all exams, term papers, portfolios, high
stake homework from courses within major
o 
No. Courses No. Items
Intro biology
5
1100
Upper division
21
2051
Study 2
Applying Bloom’s taxonomy
3 independent raters assigned a Bloom’s level to Assessment items (n=3151) Agreement of 83%
discussed to Reach consensus on each item 7 4/23/13 Study 2
Just the facts?!
Introductory
100%
Upper Division
80%
Level
60%
N
Average Level
Introductory
1100
1.55 ± 0.67
Upper Division
2051
1.42 ± 0.87
40%
20%
Take-home exams
Course portfolio
0%
1
2
Knowledge
3
4
5
6
Synthesis
Bloom's Level
Study 2
Bloom’s level versus performance*
1 course,
n = 3 exams
Percent Correct
100
80
60
40
20
0
0
1
Knowledge
*For introductory biology only
2
3
4
Bloom's Level
5
6
Synthesis
Momsen et al 2013
8 4/23/13 Study 3
(bonus!)
Good news?
100%
Items with figures
Items without figures
80%
60%
Still a lot of identify
questions…
40%
But a small yet significant shift
towards higher-level cognitive
tasks
20%
0%
1
Knowledge
2
3
4
Bloom's Level
5
6
Synthesis
Good news?
Study 3
(bonus!)
Questions with figures ask students to label,
interpret, reason with, create, evaluate, etc.
9 4/23/13 Assessment drives learning
Provides practice with
•  Disciplinary skills
•  Concept understanding
•  Metacognitive skills
New
information
Perceived
Will this
be on the
test?
Organize
Working
memory
Elaborate
Visualize
Processed
Retrieval
Long-term
memory
Based on Kolb 1984
Moving beyond the facts
Synthesis &
Evaluation
Create, integrate,
assess, convince
Analysis
Separate, classify,
order, interpret
Application
Show, solve, modify,
relate, calculate
Comprehension
Knowledge
10 4/23/13 Ready to LEVEL UP?
Coding assistance
  Erika Offerdahl
  Lisa Montplaisir
  Elizabeth Anderson
  Amy Williams
Method development
  Tammy Long
  Sara Wyse
  Elena Bray Speth
  Diane Ebert-May
  Scott Freeman
Thanks to over 100 faculty who’ve graciously
shared exams, quizzes, and homework
assignments.
NSF-DUE
#0833268
11