Skills Diagnosis with
Latent Variable Models
Topic 1:
A New Diagnostic Paradigm
Introduction
• Assessments should aim to improve, and not
merely ascertain the status of student learning
• For test scores to facilitate learning, they need
to be interpretative, diagnostic, highly
informative, and potentially prescriptive
• Most large-scale assessments are based on
traditional unidimensional IRT models that
only provide single overall scores
• These scores are useful primarily for ordering
students along a continuum
• Alternative psychometric models that can
•
•
•
•
provide inferences more relevant to instruction
and learning currently exist
These models are called cognitive diagnosis
models (CDMs)
Alternatively, they are referred to as diagnostic
classification models (DCMs)
CDMs are multiple discrete latent variable
models
They are developed specifically for diagnosing
the presence or absence of multiple finegrained attributes (e.g. skills, cognitive
processes or problem-solving strategies)
• Fundamental difference between IRT and CDM:
A fraction subtraction example 2 124  127
• IRT: performance is based on a unidimensional
continuous latent trait 
• Students with higher latent traits have higher
probability of answering the question correctly
1.00
P( X  1|   1.2)  0.9
0.80
0.60
0.40
P( X  1|   0.8)  0.3
0.20
0.00
-3.5
  0.8
-2.5
-1.5
-0.5
  1.2
0.5
1.5
2.5
3.5
• Fundamental difference between IRT and CDM:
A fraction subtraction example 2 124  127
• IRT: performance is based on a unidimensional
continuous latent trait 
• Students with higher latent traits have higher
probability of answering the question correctly
• CDM: performance is based on binary latent
attribute vector   (1 , ,  K )
• Successful performance on the task requires a
series of successful implementations of the
attributes specified for the task
2 
4
12
7
12
• Required attributes:
(1) Borrowing from whole
(2) Basic fraction subtraction
(3) Reducing
• Other attributes:
(4) Separating whole from fraction
(5) Converting whole to fraction
 1 1612 
 1 129
 1 34
7
12
Basic Elements and Notations of CDM
• The response vector of examinee i will be
denoted by ,
• The response vector contains J items, as in,
• The attribute vector of examinee i will be
denoted by
• Each attribute vector or pattern defines a unique
latent class
• Thus, K attributes define
latent classes
• Example: When
, the total number of
latent classes is
• Although arbitrary, we can associate the
following attribute vectors with the following
latent classes:
Basic CDM Input
• Like IRT, CDM requires an
binary
response matrix as input
• Unlike IRT, CDM in addition requires a
binary matrix called the Q-matrix as input
• The rows of the Q-matrix pertain to the items,
whereas the columns the attributes
• The 1s in the jth row of the Q-matrix identifies
the attributes required for item j
Examples of Attribute Specification
(1)
Item
(2)
Borrow
Basic
from the
fraction
whole subtraction
Attribute
(3)
Reduce
(4)
Separate
whole
from
fraction
(5)
Convert
whole to
fraction
4 7
1. 2 
12 12
1
1
1
0
0
3
4
2. 7  2
5
5
1
1
0
1
0
Examples of Attribute Specification
(1)
Item
1
3. 2 
3
7
4. 3  2
8
(2)
Borrow
Basic
from the
fraction
whole subtraction
Attribute
(3)
Reduce
(4)
Separate
whole
from
fraction
(5)
Convert
whole to
fraction
Basic CDM Output
• The goal of CDM is to make inference about the
attribute vector
• The basic CDM output gives the (posterior)
probability the examinee has mastered each of
the attributes
• That is, we get
• For example,
, indicates
that we are quite certain that examinee has
already mastered attribute 1
• Each examinee gets a vector of posterior
probabilities
• For reporting purposes, we may want to
convert the probabilities into 0s and 1s
• We can use different rules for this conversion
• If
;
Otherwise,
Example:
• Each examinee gets a vector of posterior
•
•
•
•
probabilities
For reporting purposes, we may want to
convert the probabilities into 0s and 1s
We can use different rules for this conversion
If
;
Otherwise,
If
; or
If
;
Otherwise,
Example:
? – means we do not have sufficient evidence
to conclude one way or the other