COGNITIVE TRANSFER
OUTCOMES FOR A
SIMULATION-BASED
INTRODUCTORY STATISTICS
CURRICULUM
Matthew Beckman
Pennsylvania State University
Robert delMas
Joan Garfield
University of Minnesota
OVERVIEW
 Learning Statistics: Intrinsic & Extraneous Cognitive
Load
 Cognitive Transfer: Near and Far
 Simulation-Based Methods: How they address
Cognitive Load
 Comparative Study: Simulation-Based course and NonSimulation-Based courses
 Discussion of Results
LEARNING STATISTICS
 Discipline of statistics has high intrinsic cognitive load (Paas,
1992; Sweller, 1994) – concepts are difficult and highly
interrelated
 Extraneous cognitive load: Cognitive load added by teaching
methods (e.g., covering too much content too quickly for
students to adequately process)
 Need teaching methods (e.g., scaffolding) that help students
learn concepts and connections between concepts
 But, teaching statistical concepts and procedures may not
be enough to develop transfer (Garfield, 2000; Garfield et
al., 2012; Lovett & Greenhouse, 2000).
COGNITIVE TRANSFER
 Ability to apply skills and knowledge students learn in a
course to novel problems (Cox, 1997; Singley & Anderson,
1989).
 Distinction between Near and Far transfer
 Near: tasks & contexts highly similar to curriculum tasks
 Far: little or no explicit training in the context of the new task (in
this study)
 Transfer requires a rich schema for the content area built
slowly by experiencing numerous problems (e.g., Bransford,
Brown, & Cocking, 2000)
 Transfer requires intentional training (Perkins & Salomon,
1988) and understanding constructed by the students
(Broers, Mur, & Bude, 2004)
SIMULATION-BASED METHODS
 Argued to be (Cobb, 2007; Efron, 2000; Ernst, 2004; Garfield
et al., 2012; Rossman, 2008 ):
 Conceptually simpler than Normal theory procedures
 Align well with student’s intuitions
 Principles generalize to many situations
 May reduce extraneous cognitive load, making it easier
to understand and connect concepts such as:
 Sample variability
 Sampling distributions
 p-values & Confidence intervals
 Connecting concepts promotes rich schema
CATALST: A SIMULATION-BASED
INFERENCE CURRICULUM
 Develops statistical thinking through bootstrapping and
randomization testing methods (Garfield et al., 2012)
 TinkerPlots™ used to build models and simulations
 Three Modules
 Chance Models: Develop understanding of chance
models and simulation methods; segue to use of p-values
to quantify evidence against a model
 Comparing Groups: Develop understanding of how to
build simulations to conduct randomization tests
 Estimate Effects: Develop understanding of how to build
simulations for bootstrap confidence intervals
THE STUDY
 Fall 2011 and Spring 2012
 729 students from 8 institutions
 Compared two implementations of CATALST course to nonCATALST introductory statistics courses
 CAT-MINN: CATALST course at the University of Minnesota (4
classes; N = 138)
 CAT-REPL: CATALST course at 5 other institutions (5 classes;
N = 151)
 NON-CAT: Non-CATALST courses at 7 other institutions (10
classes; N = 440; 5 institutions same as CAT-REPL institutions)
 Neither students nor classes were randomly assigned to
course
ASSESSMENT INSTRUMENT
 21 multiple-choice items
 16 items adapted from CAOS test (delMas et al., 2007)
 Five new items on randomization-based methods and
understanding statistical significance
 Topics: random sampling & assignment; generalization of
results; variability; sampling variability; informally compare
groups; understanding of null hypothesis; interpret p-values;
interpret confidence intervals; interpret statistical
significance
 Testlets: Items that shared a common context were treated
as a testlet (proportion correct); 8 single items and 5 testlets.
NEAR & FAR TRANSFER
 Each assessment item mapped to the frequency and
type of contact in CATALST curriculum
 Type of Contact:
 Direct Instruction: Concept specifically identified and
instruction designed to develop understanding of the
concept
 Indirect Instruction: Concept not identified in instruction,
but learning experience can lead to understanding of the
concept
 Practice: Number of times concept is required in in-class
activities or homework assignments
NEAR & FAR TRANSFER
Measured Learning Goal
Purpose of random
assignment
Factors that allow a sample
of data to be generalized to
the population
Correlation does not imply
causation
Match a scatterplot to a
verbal description of a
bivariate relationship
Expected patterns in
sampling variability
Direct
3
Indirect
3
Practice
3
Total
9
Transfer
Near
3
-
4
7
Near
-
5
2
7
Far
-
-
-
0
Far
2
3
-
5
Near
 Near Transfer: Direct instruction (9 scores)
 Far Transfer: No direct instruction (6 scores)
CONCEPTS by TRANSFER
NEAR
Random Assignment
Generalization to Population
Representative Sample
Sampling Variability
FAR
Correlation ≠ Causation
Trend in a Scatterplot
Variability (Repeated
Measurements)
Understanding of Statistical
Significance
Compare Groups
Understand Null Hypothesis
Interpret p-Values
Interpret Confidence Intervals
Interpret p-Values
Interpret Confidence Intervals
LME MODEL
 Linear mixed effects (LME) model with random
intercepts
 Fixed effects for curriculum group (CAT-UMINN, CATREPL, NON-CAT) and item transfer type (NEAR vs
FAR)
 Random effect for random intercepts within class (19
levels)
 2 contrasts for curriculum group
 CAT-UMINN – CAT-REPL
 CAT-REPL – NON-CAT
 Contrast for item transfer type: NEAR – FAR
MODEL ASSUMPTIONS
 Intra-Class Correlation (ICC): 0.225
 No evidence of extreme violations of assumptions
 Levene’s Test: (F(5, 1452) = 1.624, p = 0.150),
 Box’s M Test: M(6) = 10.633, p = 0.100
RESULTS
 Good precision in random effects estimates
Random Effect
Estimate [95% CI]
Class
0.069 [0.044, 0.093]
Student level standard deviation 0.172 [0.166, 0.178]
 All fixed effects statistically significant
Fixed Effect
Curriculum
Transfer Type
Interaction
df
2, 15.1
1, 1435.1
2, 1435.1
F
7.461
20.112
5.713
p
0.0056
< 0.0001
0.0034
CONTRASTS
Fixed Effect
Curriculum group: UMINN – REPL
Curriculum group: REPL – NON-CAT
Transfer item type: Near – Far
UMINN – REPL by Transfer
REPL – NON-CAT by Transfer
df
15
15
1435
1435
1435
t
3.030
3.541
4.485
1.373
3.380
p
0.008
0.003
< 0.001
0.170
< 0.001
DISCUSSION
 CATALST more effective than Non-CATALST courses
 CATALST students outperformed Non-CATALST
students on both Near and Far transfer items
 Limitation: Non-CATALST curriculum varied
 All items covered important topics in introductory
statistics
 On Far transfer items, CATALST effect most pronounced
for UMINN group
 Instructors may require more training and experience to
achieve Far transfer effects
REFERENCES
Bransford, J., Brown, A. L., & Cocking, R. R. (2000). How People Learn: Brain, Mind, Experience, and School: Expanded
Edition, Washington: National Academy Press.
Broers, N. J., Mur, M. C., & Bude, L. (2004). Directed self explanation in the study of statistics. In G. Burrill, & M. Camden
(Eds.), Curricular development in statistics education (pp. 21-35). Voorburg, The Netherlands: International Statistical
Institute.
Cobb, G. W. (2007). The introductory statistics course: A Ptolemaic curriculum. Technology Innovations in Statistics Education,
1(1), 1-15.
Cox, B. D. (1997). The rediscovery of the active learner in adaptive contexts: A developmental-historical analysis. Educational
Psychologist, 32(1), 41-55. doi:10.1207/s15326985ep3201_4
delMas, R., Garfield, J., Ooms, A., & Chance, B. (2007). Assessing students’ conceptual understanding after a first course in
statistics. Statistics Education Research Journal, 6(2), 28-58.
Efron, B. (2000). The bootstrap and modern statistics. Journal of the American Statistical Association, 95(452), 1293-1296.
Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science, 19(4), 676-685.
Garfield, J. (2002). The Challenge of Developing Statistical Reasoning. Journal of Statistics Education, 10(3).
Garfield, J., delMas, R., & Zieffler, A. (2012). Developing Statistical modelers and thinkers in an introductory, tertiary-level
statistics course. ZDM: The International Journal on Mathematics Education, 44(7), 883-898.
Lovett, M. C., & Greenhouse, J. B. (2000). Applying Cognitive Theory to Statistics Instruction. The American Statistician,
54(3), 1-11.
Paas, F. G. W. C. (1992). Training Strategies for Attaining Transfer of Problem Solving Skill in Statistics: A Cognitive-Load
Approach. Journal of Educational Psychology, 84(4), 429-434.
Perkins, D. N., & Salomon, G. (1988). Teaching for Transfer. Educational Leadership, 22-32.
Rossman, A. J. (2008). Reasoning about informal statistical inference: One statistician’s view. Statistics Education Research
Journal, 7(2), 5-19. Retrieved from http://iase-web.org/documents/SERJ/SERJ7(2)_Rossman.pdf
Singley, M. K., & Anderson, J. R. (1989). The transfer of cognitive skill. Cambridge, MA: Harvard University Press.
Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and instruction, 4(4), 295-312.
Robert delMas
University of Minnesota
250 Education Sciences Building
Minneapolis, MN 55455
[email protected]
612-625-2076