Qualifying a cognition endpoint for use in
multiple sclerosis by combining data
from many clinical trials
Adam Jacobs
Associate Director, Biostatistics
Premier Research
The obligatory “big data” quote
Big data is like teenage sex: everyone talks about
it, nobody really knows how to do it, everyone
thinks everyone else is doing it, so everyone
claims they are doing it
Dan Ariely, January 2013
Outcomes in multiple sclerosis
•
•
•
•
Most widely used outcome measure in MS trials is the EDSS
Focused mainly on physical symptoms, particularly walking
But other outcomes are important in MS
4 main domains of MS-related disability:
• Ambulation
• Dexterity
• Visual acuity
• Cognition
Cognition measures
• 2 main cognition measures have been used in MS trials:
• Paced Auditory Serial Addition Test (PASAT)
• Symbol Digit Modalities Test (SDMT)
• Both mainly measure processing speed, but PASAT is also
affected by working memory capacity
• Neither is currently accepted by regulators as a valid outcome
measure for MS trials
FDA clinical outcome assessment qualification
program
“COA qualification is based on a review of the evidence to
support the conclusion that the COA is a well-defined and
reliable assessment of a specified concept of interest for use in
adequate and well-controlled (A&WC) studies in a specified
context of use. COA qualification represents a conclusion that
within the stated context of use, results of assessment can be
relied upon to measure a specific concept and have a specific
interpretation and application in drug development and
regulatory decision-making and labeling.”
Objectives of SDMT qualification
•
•
•
•
•
•
•
•
Floor or ceiling effects
Test-retest reliability
Changes in scores over time
Construct and convergent validity
Practice effects
Known group validity
Sensitivity to change
Minimum clinically important difference
MSOAC database
•
•
•
•
•
A database of 16 MS studies from various sponsors
N = 14,370
Most (11 studies, 11,843 patients) in RRMS
A variety of different treatments and study designs
Data lovingly crafted into standardised SDTM datasets by the
Critical Path Institute
• 14 studies (N=12,776) included in current analyses: 2 studies
did not measure cognition
Results
Number of patients
Analysis Population
Number of Patients Number of Studies
(N = 12,776)
(N = 14)
All studies set
12,776 (100.0%)
14 (100.0%)
PASAT set
11,702 (91.6%)
13 (92.9%)
SDMT and PASAT set
1,512 (11.8%)
1 (7.1%)
SDMT set
2,606 (20.2%)
2 (14.3%)
Age at baseline
All studies set
SDMT Set
12,727
2,586
Mean
39.5
38.6
Std. Deviation
9.92
9.43
17
18
40.0
39.0
72
61
N
Minimum
Median
Maximum
More demographics at baseline (all studies set)
Sex
Age
0.4%
31.1%
29.1%
32.5%
68.9%
38.1%
Male
Female
< 35
35-45
>45
Missing
Disease duration at baseline (years)
All studies set
SDMT Set
6641
2546
Mean
6.5
5.9
Std. Deviation
7.26
5.41
0
0
Median
4.0
5.0
Maximum
48
40
N
Minimum
Distribution of SDMT and PASAT pre-dose
SDMT
2,583
PASAT
11,630
Min
0.0
0.0
5th centile
23.0
25.0
1st quartile
37.0
42.0
Median
48.0
52.0
3rd quartile
57.0
57.0
95th centile
72.0
60.0
Max
110.0
60.0
N
Assessment of test-retest reliability
• Mixed model repeated measures analysis of SDMT or PASAT
• Measures taken only from period when EDSS was stable, and
no more than 6 months after baseline
• Included about 30% of all available data
• Statistical model included terms for test number and interval
since previous test to allow for practice effects
Measures of test-retest reliability
SDMT
PASAT
N (observations)
8,567
24,327
N (patients
2,094
7,962
Intraclass correlation coefficient
0.85
0.86
Within-subject residual SD
6.2
4.5
Regression coefficients for practice effects
Test number
SDMT
PASAT
2
-0.02 (-0.09 to 0.05)
0.10 (0.08 to 0.12)
3
0.05 (-0.02 to 0.13)
0.22 (0.20 to 0.24)
4
0.10 (0.03 to 0.18)
0.32 (0.30 to 0.34)
5
0.23 (0.15 to 0.31)
0.44 (0.41 to 0.47)
6
0.33 (0.25 to 0.40)
0.48 (0.45 to 0.51)
7
0.36 (0.28 to 0.44)
0.51 (-0.27 to 1.29)
Regression coefficients are relative to first test and expressed as effect sizes with 95% CI
Spearman correlation coefficients: baseline
values
Measure
SDMT
PASAT
CC
95% CI
EDSS
-0.34
-0.38 to -0.29
-0.21
-0.23 to -0.19
9-HPT
-0.47
-0.51 to -0.43
-0.32
-0.34 to -0.31
T25FW
-0.42
-0.46 to -0.38
-0.29
-0.30 to -0.27
0.34
0.30 to 0.39
0.20
0.18 to 0.23
-0.20
-0.24 to -0.15
-0.19
-0.23 to -0.15
0.54
0.50 to 0.57
LCVA
BDI
PASAT
CC
95% CI
Spearman correlation coefficients: change
from baseline at endpoint
Measure
SDMT
PASAT
CC
95% CI
EDSS
-0.12
-0.15 to -0.08
-0.02
-0.04 to 0.00
9-HPT
-0.22
-0.27 to -0.18
-0.04
-0.06 to -0.02
T25FW
-0.21
-0.25 to -0.16
-0.02
-0.04 to 0.00
0.19
0.14 to 0.23
0.01
-0.02 to 0.04
-0.30
-0.34 to -0.26
-0.03
-0.07 to 0.02
0.20
0.15 to 0.25
LCVA
BDI
PASAT
CC
95% CI
Change in SDMT and PASAT scores with age
2
0
-2
-4
-6
-8
-10
-12
< 25
25-29
30-34
35-39
SDMT
40-44
PASAT
45-49
50-54
≥ 55
Sensitivity to change of SDMT
N
Mean
change
95% CI
P value
Last score before relapse to first
score during relapse
185
-0.14
[ -1.20; 0.93]
0.8024
First score during relapse to first
score after end of relapse
144
2.16
[ 0.63; 3.69]
0.0060
Baseline to first score after EDSS
worsening
212
2.96
[ 1.43; 4.49]
0.0002
Baseline to first score after EDSS
improvement
224
5.35
[ 3.57; 7.13]
< 0.0001
Nature of change
Some challenges
• Big data means big datasets: slow to run!
• Create subsets of data for debugging programs
• Beware of P values! With 12776 patients, tiny, trivial effects
may be statistically significant
• Standardisation of data from many different trials with
different designs
• Data were not collected for the purpose for which we are
using them
• SDMT and SDTM!
Any questions?