Psychometrics in LIBE
Bernard Veldkamp
Karel Kroeze
Twente University
PROJECT REF. NO. 543058-LLP-1-2013-1-IT-KA3-KA3MP
This project has been funded with support from the European Commission. This communication reflects the views only of the author,
and the Commission cannot be held responsible for any use which may be made of the information contained therein.
Rome, December 11th 2015
 LIBE project has a number of innovative features
LIBE project




Adaptive online learning system
Available in four languages
About 21st century skills
Designed for educational low-achievers
 Young adults
 Lowest 10%
Educational
low-achievers
 Often disappointed in the educational system
 Unmotivated
 Social media/internet literate
 Low chances of success on the Labour Market
Motivation
- To be moved to do something -
Educational
low-achievers
At their best
Pretty often
 Curious
 Apathic
 Inspired
 Alieniated
 Striving to learn
 Irresponsible
 Extend themselves
 Passive
 Master new skills
 Waiting for the weekend
 Apply their talents
 Reject growth
How to move them?
Selfdetermination
theory
Ryan & Deci (2000a, 2000b).
 Unmotivation
 Extrinsic motivation
 Intrinsic motivation
Self
determination
theory
 People will be motivated for activities that




Hold intrinsic interest for them
Have the appeal of novelty
Challenge them
Leave them in control.
 Competence
 Challenge them at their own level
Prerequisites
for learning
 Relatedness
 Within a stimulating environment
 Autonomy
 Locus of control
 A person might start with an activity because of an external
regulation.
Idea: Initiate
development
 This might allow the person to experience interesting properties
of the activity.
 This might result in an orientation shift.
 Until they are intrinsically motivated.
 This sense of value might be lost however.
In LIBE
1.
Carefully design the items, to be sure that they are not too
easy or too hard for the intended users.
2.
Apply adaptive testing, such that LIBE can adapt the difficulty
of the items to the level of the candidates.
3.
Offer an online environment.
4.
Student is in control.
Adaptive
testing
 Adapt the selection of the items to the information already
obtained, to be more efficient and more motivating.
 I am thinking of an object. You have 20 “yes-or-no”
questions to figure it out.
 Would you write out all your questions ahead of time?
Analogy: A
Game of 20
Questions
1) Is it an animal?
2) Is it a vegetable?
3) Is it blue?
4) Is it red?
5) Is it bigger than a car?
6) Etc.
 Isn’t it more effective to base your next question on
previous answers?
1) Is it an animal? NO.
2) Is it a vegetable? YES.
20 Questions,
Continued
3) Is it commonly found in a salad? YES.
4) Is it green? NO.
5) Would Bugs Bunny eat it? YES.
Algorithm for
CAT
1.
Initialize the ability estimate for a candidate
2.
Select the initial item
3.
Administer the item
4.
Estimate the ability level
5.
IF (stopping criterion is not met) GOTO STEP 2
Illustration
Several ways
to implement
CAT
 Fully automated CAT
 Multi-stage testing
 Self-adaptive testing
Psychometrics
of CAT
Results
Try-out of the
LIBE learning
units
 Six learning units were developed
 2 Norwegian LUs
 2 Portugese LUs
 2 Italian LUs
 Each of them pretested with 300 candidates
 Making ends meet (LUC, n = 158)
 Mountain biking (LUC, n = 128)
 Let’s eat (UPORTO, n = 236)
Sample sizes
 Save the world (UPORTO, n = 240)
 Everything you want (UNIROMA TRE, n = 209)
 How to write a resume (UNIROMA TRE, n = 214)
 Design of the try-out
Learning
results
Pre-test
 Intended effect
LIBE LU
Post-test
Learning unit
Pre-test
Post-test
Making ends meet
0,70
0,66
0,43
0,31
0.56
0.57
0.69
0.73
0.69
0.77
0.8
0.83
Mountain biking
Learning
results
Let’s eat
Save the world
Everything you want
How to write a resume
 Score on the post-test is lower than on the pre-test.
Preliminary
conclusions
 What can we learn from that?
 Was there no effect of the LIBE learning units?
 Or was the post-test more difficult?
More detailed analysis
Calibration of
the Rasch
model (LUC)
Making ends meet
Mountain biking
Calibration of
the Rasch
model
(UNIROMA 3)
Everything you want
How to write a resume
Calibration of
the Rasch
model
(UPORTO)
Let’s eat
Save the world
Testing the
model fit
 Comparision with 2PLM
 IRT analysis provides us with insight in the relative difficulty of the
pre-test and the post-test
Overall
conclusions
psychometric
analyses
 Traditional sum scores are not comparable, pre- and post-test are
not equally difficult.
 Model fit for part of the data was really good.
 Future research
 We did not apply a dynamic model yet.
 We can analyze the consequences of multiple attempts.
 Growth measurement
Implications
for LIBE
 Adaptive testing
 Quality control
Any questions?