Gerald Kruse and David Drews
Juniata College
Huntingdon, PA
[email protected]
[email protected]
MA 103, Quantitative Methods, aka “QM”
 “Mathematics 103 prepares students to be quantitatively
literate citizens in today's world. By learning to think
critically about quantitative issues, students will be
able to make responsible decisions in their daily lives. …as
well as to present quantitative output and verbal
arguments. ”
MA 103, Quantitative Methods, aka “QM”
 “Mathematics 103 prepares students to be quantitatively
literate citizens in today's world. By learning to think
critically about quantitative issues, students will be
able to make responsible decisions in their daily lives. …as
well as to present quantitative output and verbal
arguments. ”
 Three Projects during the semester
- began using CLA performance tasks Spring 2009
- authentic and open-ended
MA 103, Quantitative Methods, aka “QM”
 “Mathematics 103 prepares students to be quantitatively
literate citizens in today's world. By learning to think
critically about quantitative issues, students will be
able to make responsible decisions in their daily lives. …as
well as to present quantitative output and verbal
arguments. ”
 Three Projects during the semester
- began using CLA performance tasks Spring 2009
- authentic and open-ended
 Pre and Post Assessment (Skills and Attitudes)
- 55 min exam given on the first and last class of
semester
- Fall 2009 transition from math skills to CLA
performance task
“The Collegiate Learning Assessment (CLA)
provides one possible example of an
assessment that fits a situated notion of
QR.”
Richard Shavelson, “Reflections on Quantitative Reasoning, an
Assessment Perspective,” In B.L. Madison & L.A. Steen (Eds.),
Calculation vs. context: Quantitative literacy and its implications for
teacher education. MAA.
“The Collegiate Learning Assessment (CLA)
provides one possible example of an
assessment that fits a situated notion of
QR.”
Richard Shavelson, “Reflections on Quantitative Reasoning, an
Assessment Perspective,” In B.L. Madison & L.A. Steen (Eds.),
Calculation vs. context: Quantitative literacy and its implications for
teacher education. MAA.
Being quantitatively literate is being “able to
think and reason quantitatively when the
situation so demands”
Higher-Order Skills Assessed
 Evaluating Evidence
Higher-Order Skills Assessed
 Evaluating Evidence
 Analysis / Synthesis / Conclusion
Higher-Order Skills Assessed
 Evaluating Evidence
 Analysis / Synthesis / Conclusion
 Presenting / “creating” evidence
Higher-Order Skills Assessed
 Evaluating Evidence
 Analysis / Synthesis / Conclusion
 Presenting / “creating” evidence
 Acknowledging alternatives to THEIR conclusion
Higher-Order Skills Assessed
 Evaluating Evidence
 Analysis / Synthesis / Conclusion
 Presenting / “creating” evidence
 Acknowledging alternatives to THEIR conclusion
 Completeness
Experimental Design
Instrument
Section 01
Section 02
Section 03
Pre-Assessment
QR Assessment
QR Assessment
Traditional Math
Skills
Project 01
Performance Task,
based on CLA
Traditional
Traditional
Project 02
Performance Task,
based on CLA
Traditional
Traditional
Project 03
Performance Task,
based on CLA
Traditional
Traditional
Post-Assessment
QR Assessment
QR Assessment
QR Assessment
Fall 2009, Version 1.o
•
Explicit scoring guidelines (based on rubric) established.
Fall 2009, Version 1.o
•
Explicit scoring guidelines (based on rubric) established.
•
Scoring guidelines gave “good” reliability.
Fall 2009, Version 1.o
•
Explicit scoring guidelines (based on rubric) established.
•
Scoring guidelines gave “good” reliability.
•
Encouraging, but not statistically significant, results
indicated that students in the section with performance task
based projects showed more improvement in critical
thinking skills.
Fall 2009, Version 1.o
•
Explicit scoring guidelines (based on rubric) established.
•
Scoring guidelines gave “good” reliability.
•
Encouraging, but not statistically significant, results
indicated that students in the section with performance task
based projects showed more improvement in critical
thinking skills.
•
Use results to prepare for next round of assessment in
Spring 2011.
Modifications for V2.0
“How was the student experience different in Spring 2011 vs. Fall 2009?”
•
The content of the course Fall 2009 vs. Spring 2011 remained
the same, as did 95% of the “classroom experience,” but the
course was reframed with a focus on quantitative reasoning
(q. r.) and critical thinking (c. t.)
Modifications for V2.0
“How was the student experience different in Spring 2011 vs. Fall 2009?”
•
The content of the course Fall 2009 vs. Spring 2011 remained
the same, as did 95% of the “classroom experience,” but the
course was reframed with a focus on quantitative reasoning
(q. r.) and critical thinking (c. t.)
- syllabus
- assignments
- opportunities during lecture
- “salt and pepper”
Modifications for V2.0
“How was the student experience different in Spring 2011 vs. Fall 2009?”
•
The content of the course Fall 2009 vs. Spring 2011 remained
the same, as did 95% of the “classroom experience,” but the
course was reframed with a focus on quantitative reasoning
(q. r.) and critical thinking (c. t.)
- syllabus
- assignments
- opportunities during lecture
- “salt and pepper”
•
The pre/post assessment was modified:
- better linkage with specific learning outcomes
- more open-ended scenario
- names
- one prompt
Modifications for V2.0 continued
“How was the student experience different in Spring 2011 vs. Fall 2009?”
•
Present the idea of a rubric:
- objective assessment, “trust”
- familiarize with elements
- create one for “chips”
Scoring Guidelines
Higher Order Skill
Evaluating evidence provided
None (0)
Emerging (1, 2)
Developing (3, 4)
Mastering (5, 6)
Evaluation of 1 document with:
acceptable evaluations (1)
Evaluation of 3 documents with:
acceptable evaluations (3)
Evaluation of 4 documents with:
acceptable evaluations (5)
If discussing documents A and D, and
ignoring B and C, there is some implied
reason that these documents are
relevant.(1)
Evaluation of 3 documents with:
good evaluations (4)
Evaluation of 4 documents with:
good evaluations (6)
Document A is relevant and from an
acceptable source
Document B is anecdotal
Document C is biased and not relevant
Document D is relevant and from an
acceptable source
Document A is relevant and from an
acceptable source
Document B is anecdotal
Document C is biased and not relevant
Document D is relevant and from an
acceptable source
Implies or directly disagrees with Sauer, or
concludes “more research needed,”
analyzing Document A (outliers) OR
Document D (headaches and aspartame
independent), but analysis is
inaccurate/unclear/incomplete (3)
or
uses A AND D but misinterprets one of
them (3)
or
analysis of A OR D is good (4).
Implies or directly disagrees with Sauer, or
concludes “more research needed,”
analyzing Document A (outliers) AND
Document D (headaches and aspartame
independent), but analysis is:
Tries to reorganize information to new
format but does so poorly (1).
Makes some reasonable effort to
reorganize information but could do
more/better, hitting % (2)
Reorganizes information to create
compelling graphs/tables or conditional
probability, Pr(A|H) (3)
Acknowledges possibility of alternative(s)
to their conclusion, but does not elaborate
or with poor reason(s) (1).
Describes an alternative
possibility/interpretation to their
conclusion, without acceptable rationale
or support (2).
Describes an alternative
possibility/interpretation to their
conclusion, with acceptable rationale or
support (3).
Completeness
Disagree with Sauer or argue for more research
Only makes conclusion about Sauer’s claim,
but uses no or inappropriate evidence (1).
Deals with Sauer’s claim plus minimal
attention to strengths and weaknesses of
argument (2)
Deals with Sauer’s claim plus good
attention to strengths and weaknesses of
argument (3)
Completeness
Agree with Sauer
Only makes conclusion about Sauer’s claim,
but uses no or inappropriate evidence (1).
Deals with Sauer’s claim and gives a plan
OR strengths and weaknesses of argument
(2)
Deals with Sauer’s claim and gives a plan
AND strengths and weaknesses of
argument or plan (3)
Mechanics/persuasiveness
Distracting errors, poor sentence structure,
poor organization (1).
Few errors, generally well-constructed
sentences, fair-good organization (2).
Hardly any errors, good sentence structure,
good overall organization
AND
Memo format, From:/To: etc. (3).
Identifying relevant evidence, evaluating evidence
credibility, reliability, relevance.
Keep scores low for people that misunderstand what
evidence says/can say.
Document A is relevant and from an acceptable source
Document B is anecdotal
Document C is biased and not relevant
Evaluation of 2 documents with:
acceptable evaluations (2)
Document D is relevant and from an acceptable source
Analysis/synthesis /conclusion
I. Analysis
1. Document A contains outliers, must conclude that
no correlation between headaches and Aspartame,
Sucralose, or Placebo consumption
2. Document D has conditional probability table which
shows that headaches and Aspartame consumption
are independent
II. Conclusion - must make one
Blowing it on one of the documents is worse than ignoring
the document altogether
Presenting/”creating” evidence
Creating a graph, calculating conditional probability
Acknowledging alternatives to their conclusion
If disagree w/Sauer and not ban Aspartame, acknowledge
that Aspartame could be bad, just substituting sucralose is
not an alternative
Exhibiting facility with the English language, especially
sentence structure and overall organization.
Implies or directly incorrectly agrees with
Sauer, or conclusion is unclear:
with no evidence (1)
or
some evidence, not just agreeing/quoting
Sauer (2).
If analyzed Document A (outliers) or
Document D (headaches and aspartame
independent) well, but no conclusion (2)
unclear or incomplete(5)
OR
good (6).
Modifications for V2.0 continued
“How was the student experience different in Spring 2011 vs. Fall 2009?”
•
Present the idea of a rubric:
- objective assessment, “trust”
- familiarize with elements
- create one for “chips”
•
Improved feedback for projects completed during the
semester:
- students used guideline to score their work
- compared this to my scoring
- general trends discussed with entire class
- scheduled time to meet for specific feedback
•
Each of the three projects emphasized different quantitative
content (what we were doing at the time) as well as different
categories in the rubric
Results for V2.0
Total Score
14.00
13.00
Total Score
Pre
Post
my section
9.46
13.25
other section
8.2
9.24
% of possible 12.00
% change improvement
40.06
12.68
32.84
8.13
my section
11.00
other section
10.00
9.00
8.00
Pre
Post
Evaluating Evidence
3.00
2.50
Evaluating
Evidence
Pre
Post
% change
% of possible
improvement
my section
1.68
2.5
48.81
18.98
other section
0.96
1.24
29.17
5.56
2.00
my section
1.50
other section
1.00
0.50
0.00
Pre
Post
Results for V2.0
Analysis / Synthesis / Conclusion
4.10
3.90
Analysis,
Synthesis,
Conclusion
my section
other section
3.70
3.50
Pre
2.86
2.6
Post
4
3.28
% of possible
% change improvement
39.86
36.31
26.15
20.00
3.30
my section
3.10
other section
2.90
2.70
2.50
Pre
Post
Presenting / Creating Evidence
1.20
1.00
Presenting,
Creating
Evidence
my section
other section
0.80
Pre
0.38
0.16
Post
1
0.6
% of possible
% change improvement
163.16
23.66
275.00
15.49
my section
0.60
other section
0.40
0.20
0.00
Pre
Post
Results for V2.0
Acknowledging Alternatives to
Their Conclusion
1.00
Acknowledging
Alternatives to
Their
Conclusion
my section
other section
0.90
0.80
Pre
0.71
0.8
Post
0.96
0.44
% of possible
% change improvement
35.21
10.92
-45.00
-16.36
my section
0.70
other section
0.60
0.50
0.40
Pre
Post
Completeness
2.35
2.25
Completeness
my section
other section
Pre
1.89
1.84
Post
2.28
1.84
% of
possible
improve% change
ment
20.63
35.14
0.00
0.00
2.15
my section
2.05
other section
1.95
1.85
1.75
Pre
Post
Rubric Reliability
Dimension % spot on
% +/- 1
Evaluation
56.7
Anal/Synth/Concl
58.6
Create
73.3
Alternatives
51.7
Completeness 62.1
Total Score
39.2
40.0
34.5
26.7
37.9
34.5
35.7
96.7
93.1
100.0
98.7
96.6
75.0
Rubric Reliability
Correlation of Total Scores
Pearson correlation of D and J = 0.927
P-Value = 0.000
Scatterplot of D vs J
22.5
20.0
17.5
D
15.0
12.5
10.0
7.5
5.0
5.0
7.5
10.0
12.5
J
15.0
17.5
20.0
22.5
Critical thinking involves
evaluating/making good arguments
Making good arguments involves. . .
 Clearly stating a conclusion
 Evaluating and selecting evidence
 Creating links between evidence and conclusion
We consider quantitative reasoning as critical
thinking involving numbers/data…
MA 103, Quantitative Methods
at Juniata College
•
Juniata has a Quantitative Skills Requirement,
Q = QM + QS
•
MA 103, Quantitative Methods, is offered for
students who don’t fulfill the “Q” in their POE
•
Three Projects during the semester
- began using CLA performance tasks Spring 2009
- authentic and open-ended
Critical thinking involves
evaluating/making good arguments
Making good arguments involves. . .
 Clearly stating a conclusion
 Evaluating and selecting evidence
 Creating links between evidence and conclusion
We can then consider quantitative reasoning as
critical thinking involving numbers/data…
Evaluating Evidence Category on Rubric
Identifying relevant evidence, evaluating evidence credibility,
reliability, relevance
 Not Attempted (0)
 Emerging (1,2)
Mentions one or two documents, with:
- No or wrong evaluation on both (1)
- cursory-to-OK eval on document C, flawed on other (2)
 Developing (3,4)
Mentions two documents (one must be C), with:
- cursory-to-OK evaluation on both (3)
- good evaluation on both (4)
 Mastering (5,6)
Evaluation of C is good, and evaluates two other doc with:
- acceptable evaluations (5)
- good evaluations (6)
Analysis/Synthesis/Conclusion Category on Rubric
Identifying relevant evidence, evaluating evidence credibility, reliability,
relevance
 Not Attempted (0)
 Emerging (1,2)
Incorrectly implies , or states directly, agrees that “banning aspartame would
improve the health of the state’s citizens” with:
- no evidence (1)
- evidence (2)
 Developing (3,4)
Implies, or directly disagrees, with Sauer, noting inconsistency of claim
with data in doc C but reason is:
- inaccurate/unclear or incomplete(3)
- good (4)
 Mastering (5,6)
- Says C doesn’t support claim and is clear about reason and uses F
reasonably well (5)
- Satisfactorily uses conditional probability when discussing relationship
between headaches and aspartame usage (6)
Performance Task Scenario for Pre
and Post-Assessment
•Sen. Nathan Dulce is running for re-election vs. Pat Sauer
•Proposed bill to ban aspartame, an artificial sweetener,
from being added to any soft drink or food product, Dulce
opposes, Sauer approves.
•Pat Sauer made two arguments during a recent TV
interview:
(1) Strong correlation between the number of people who
consume aspartame and headaches, so,“banning aspartame
would improve the health of the state’s citizens.”
(2)“Aspartame should be banned and replaced with
sucralose.” Pat Sauer supported this argument by referring to
a news release.
Scatterplot of D vs J
22.5
20.0
17.5
D
15.0
12.5
10.0
7.5
5.0
5.0
7.5
10.0
12.5
J
15.0
17.5
20.0
22.5
Scatterplot of D vs J
22.5
20.0
17.5
D
15.0
12.5
10.0
7.5
5.0
5.0
7.5
10.0
12.5
J
15.0
17.5
20.0
22.5