American Journal of Epidemiology
© The Author 2014. Published by Oxford University Press on behalf of the Johns Hopkins Bloomberg School of Public Health.
All rights reserved. For permissions, please e-mail: [email protected].
Vol. 180, No. 8
DOI: 10.1093/aje/kwu202
Advance Access publication:
September 4, 2014
Practice of Epidemiology
Prognostic Score for Predicting Risk of Dementia Over 10 Years While Accounting
for Competing Risk of Death
Hélène Jacqmin-Gadda*, Paul Blanche, Emilie Chary, Lucie Loubère, Hélène Amieva, and
Jean-François Dartigues
* Correspondence to Dr. Hélène Jacqmin-Gadda, INSERM U897, Institut de Santé Publique, d’Épidémiologie et de Développement,
Université de Bordeaux, case 11, 146 rue Léo Saignat, 33076 Bordeaux, France (e-mail: [email protected]).
Initially submitted October 8, 2013; accepted for publication July 7, 2014.
Early detection of subjects at high risk of developing dementia is essential. By dealing with censoring and competing risk of death, we developed a score for predicting 10-year dementia risk by combining cognitive tests, and we
assessed whether inclusion of cognitive change over the previous year increased its discrimination. Data came from
the French prospective cohort study Personnes Agées QUID (PAQUID) and included 3,777 subjects aged 65 years
or older (1988–1998). The combined prediction score was estimated by means of an illness-death model handling
interval censoring and competing risk of death. Its predictive ability was measured using the receiver operating characteristic (ROC) curve, with 2 different definitions depending on the way subjects who died without a dementia
diagnosis were considered. To account for right-censoring and interval censoring, we estimated the ROC curves
by means of a weighting approach and a model-based imputation estimator. The combined score exhibited an
area under the ROC curve (AUROC) of 0.81 for discriminating future demented subjects from subjects alive and
nondemented 10 years later and an AUROC of 0.75 for discriminating future demented subjects from all other subjects (including deceased persons). Adjustment for cognitive change over the previous year did not improve
prediction.
competing risks; death; dementia; interval censoring; prediction; ROC curve
Abbreviations: AUROC, area under the ROC curve; BVRT, Benton Visual Retention Test; DSST, Digit Symbol Substitution Test;
IADL, Instrumental Activities of Daily Living; IPCW, inverse probability of censoring weighting; IST, Isaacs Set Test; MMSE, MiniMental State Examination; MMSE-EM, episodic memory subtest of the MMSE; PAQUID, Personnes Agées QUID; ROC, receiver
operating characteristic.
Prediction of dementia risk may be based on 3 types of markers: brain imaging, biomarkers, and neuropsychological tests.
Among them, the latter are clearly the least invasive and the
cheapest. Previous longitudinal studies have found that cognition is impaired long before the dementia diagnosis (3). Thus,
the main objective of this work was to develop and validate a
prognostic score based on cognitive evaluation for predicting a
person’s risk of developing dementia in the next 5 or 10 years.
Moreover, cognitive change between 2 assessments has been
shown to be predictive of dementia (4, 5), and an improvement
is often observed between the first 2 assessments in cohort studies (6). Thus, we first evaluated whether 2 cognitive assessments led to better prediction than a single one.
Dementia is a common disease in the elderly and is characterized by significant decline in several cognitive functions.
The cognitive decline is progressive and eventually leads to a
loss of autonomy. Early detection of subjects at high risk of
developing dementia in the future is essential, for 2 reasons.
First, current treatments given after dementia diagnosis have
demonstrated only modest efficiency, and research is now focused on the possibility of preventive treatment or nondrug
intervention that could reduce the rate of conversion to dementia among subjects at high risk (1). Thus, validated criteria for identifying this high-risk target population are required
(2). Secondly, early detection of subjects at the very beginning of the decline process may help in providing better care.
790
Am J Epidemiol. 2014;180(8):790–798
10-Year Dementia Prediction Accounting for Competing Death 791
The development and validation of prediction models for
dementia raises some methodological issues due to incomplete follow-up and competing risk of death that have not been
handled in previous work (1, 7–9). Studies of dementia predictors need cohorts of elderly subjects, which generally suffer
from right-censoring (loss to follow-up) and interval censoring
of age at dementia onset when cognitive functioning is assessed
only at the time of clinical visits with no possibility of identifying the exact date of disease onset. Moreover, death is a major competing risk in the prediction of dementia. Strictly
speaking, this is a semicompeting risk, since death may occur
before or after dementia, whereas dementia cannot occur after
death. In this context, the main problem of interval censoring
is that dementia status at death is unknown for subjects who
were dementia-free at the last clinical visit. Given that the risk
of death is much higher among demented subjects (10, 11),
the probability of developing dementia between the last clinical visit and death cannot be neglected when the planned interval between visits is 2 or 3 years and is frequently enlarged
because of missing visits. Moreover, many predictors of dementia are associated with the risk of death without dementia.
Thus, these issues must be accounted for in both the development and the validation of prediction scores.
In the development phase, a standard Cox model could be
used to estimate the cause-specific hazard of dementia only if
the data were not interval-censored. With interval censoring,
estimation of the cause-specific hazard requires fitting of the
illness-death model (shown in Figure 1) by maximum likelihood while accounting for interval censoring, as detailed
elsewhere (11, 12).
In the validation phase, definitions of sensitivity, specificity,
and the receiver operating characteristic (ROC) curve have
been extended for time-dependent outcomes with competing
risks (13, 14), and nonparametric estimators have been proposed for right-censored data according to an inverse probability of censoring weighting (IPCW) approach (15). Using a
crude imputation rule, these estimators may be applied to
interval-censored data with semicompeting risks. Alternatively, the predictive accuracy measures may be estimated by
means of an imputation approach based on the illness-death
model (16).
Our main objective in this paper is to propose a score combining cognitive tests, autonomy scales, and subjective memory
a01(t )
Health
Dementia
a02(t)
a12(t )
Death
Figure 1. The illness-death model of Joly et al. (11).
Am J Epidemiol. 2014;180(8):790–798
complaints to identify subjects at high risk of developing dementia in the next 5 or 10 years, accounting for the competing
risk of death and complex censoring schemes. We also aimed
to evaluate whether cognitive changes over the first year of
follow-up improved the score’s predictive ability after adjustment for current cognitive status.
METHODS
The PAQUID cohort
The Personnes Agées QUID (PAQUID) Study, a French
prospective cohort study, aims at studying cognitive aging
and loss of autonomy (17). The cohort was randomly selected
from electoral rolls and included 3,777 subjects aged 65 years
or older at baseline who were living at home in 2 departments
of southwestern France (Gironde and Dordogne). Subjects
were visited at home by a trained psychologist at baseline
in 1988/1989, and then again approximately 1 (T1), 3 (T3),
5 (T5), 8 (T8), and 10 (T10) years after the initial visit. Participants from Dordogne were not visited at T1. At each visit,
a questionnaire was administered that included information
about lifestyle and health characteristics, a battery of cognitive tests, and scales of disability. Dementia was assessed at
each visit using a 2-stage procedure: Subjects who met the
Diagnostic and Statistical Manual of Mental Disorders,
Third Edition, Revised, criteria for dementia (18) as assessed
by the psychologist underwent a clinical assessment by a neurologist, who ascertained the final diagnosis. Last, all cases
were reviewed by a group of experts. The PAQUID Study
protocol was approved by ethics committees, and all participants gave their informed consent.
Sample selection
A preliminary analysis was performed to evaluate whether
changes in cognitive function over the first year improved
prediction based on the cognitive and disability assessment
made at T1. For this analysis, the target sample (denoted sample A) included subjects visited at T1 who were free of dementia and not blind, deaf, or bedridden at T1 (n = 1,683).
Exclusion of subjects with missing data for the cognitive
and autonomy assessments at T1 or T0 led to a sample of
1,313 subjects. Among them, 121 subjects were diagnosed
with dementia by T10. Because the results showed that accounting for cognitive change between T0 and T1 did not improve prediction, the main analysis was performed using only
cognitive measures taken at T0.
The target sample for the main analysis (sample B) included
3,510 subjects who were dementia-free and not blind, deaf, or
bedridden at baseline (T0). After exclusion of persons with
missing data for the baseline cognitive/autonomy evaluation,
the final sample included 2,795 subjects. Up to the T10 visit
(between 1988 and 1998), 265 incident cases of dementia were
diagnosed. All subjects were censored after T10.
Prediction variables
Four cognitive tests from the neuropsychological battery of
the PAQUID Study—the Mini-Mental State Examination
792 Jacqmin-Gadda et al.
(MMSE), the Isaacs Set Test (IST), the Benton Visual Retention Test (BVRT), and the Digit Symbol Substitution Test
(DSST)—were selected as possible predictors for dementia
because scores on these tests were previously found to be decreased long before dementia diagnosis (3) or they were already
shown to be highly predictive of dementia (15) and because
scores were available at T0 and T1 with few missing data.
The MMSE is a global measure of cognitive abilities (19)
that evaluates various dimensions of cognition (memory, calculation, orientation in place and time, language and word
recognition), with a possible score ranging from 0 to 30.
The episodic memory subtest of the MMSE (MMSE-EM),
scored over a range of 8 points, includes the questions about
orientation to time and 3-word recall. Because this subtest
was previously found to be highly predictive of dementia,
it was included in the model in addition to the total MMSE
score (9, 20). The IST measures verbal fluency (21) and consists of generating a list of words (with a maximum of 10) belonging to 4 semantic categories (colors, animals, fruits, and
cities) in 15 seconds (range, 0–40). The multiple-choice form
of the BVRT measures short-term visual memory, with a
score ranging from 0 to 15 (22). The DSST measures attention and speed of information processing (23). Scores in the
PAQUID sample ranged from 0 to 76 points. For all of these
tests, a higher score indicates better performance.
Cognitive complaints were assessed by means of 4 questions requiring subjects to rate their current cognitive difficulties, such as forgetfulness in daily living, difficulty in retaining
new information, difficulty in calculation, and difficulty in spatial orientation. Each variable was coded 1 for “frequent difficulties” and 0 for “no or rare difficulties.”
Disability was assessed using the 4-item Instrumental Activities of Daily Living (4-IADL) Scale, which has been found
to have a high specificity for dementia diagnosis (24). This
score is the sum of 4 binary items from the IADL Scale (telephone, medication, budget, and transportation). Each item
was coded 0 if subjects were able to perform the task without
any help and 1 if they needed help or were unable to perform
the task.
Finally, standard predictors of dementia and death were
also considered: age (both as a continuous variable and as
a binary indicator for age ≥75 years), sex, and educational
level ( primary school diploma or higher level vs. no education or no diploma).
Statistical analysis
The illness-death model. Figure 1 displays the illnessdeath model. The transition intensities between 3 states—
healthy, demented, and deceased—are modeled by the
following proportional intensity models:
αijk ðtÞ ¼ α0jk ðtÞ expðβ jk Xijk Þ
for i ¼ 1; ; N; j ¼ 0; 1 and k ¼ 1; 2;
ð1Þ
where α0jk ðtÞ is the baseline transition intensity from state j to
state k and Xijk is the vector of predictors for subject i and transition jk associated with the regression parameters βjk. Assuming Weibull baseline transition intensities, the models
were fitted using the R package SmoothHazard (25) by
means of maximum likelihood, accounting for interval censoring (11). Use of semiparametric transition models fitted by
penalized likelihood (11) leads to very similar results.
Analysis strategy. For each sample (A and B), models
were chosen via step-by-step backward selection. The initial
models included all of the predictors listed in the “Prediction
variables” subsection above in the 3 transition intensity submodels. Then, for investigating the impact of learning in sample A, we successively tested in the selected model whether
the change in each cognitive measure between T0 and T1 was
significantly associated with the risk of dementia. In the main
analysis of sample B, we then tested the interactions between
educational level and all of the predictors retained after backward selection, because educational level was previously
found to be associated with differential prediagnosis decline
(26).
The proposed prediction score was the linear predictor
from the transition intensity model to dementia computed
with the estimated regression parameters:
Mi ¼ ^β01 Xi01 :
ð2Þ
Discrimination assessment. Sensitivity, specificity, the
ROC curve, and the area under the ROC curve (AUROC) were
estimated to evaluate the ability of the combined score M to
predict the risk of dementia in the next 10 years (t = 10) and
the next 5 years (t = 5). Denoting T as the time of the first
event and setting η = 1 if the first event is dementia and
η = 2 if it is death, the sensitivity (Se) for a cutpoint c and a
window of prediction t is defined as
Seðc; tÞ ¼ PðM > c j T t; η ¼ 1Þ:
We computed the specificity (Sp) using the 2 definitions previously proposed for the competing-risks setting (15):
Sp1ðc; tÞ ¼ PðM c j T > tÞ and
Sp2ðc; tÞ ¼ PðM c j fT > tg or fT t; η ¼ 2gÞ:
Thus, the ROC curve displaying Se(c,t) versus 1 − Sp2(c,t)
for all c (denoted ROC2) measures the discrimination between future demented subjects and all of the other subjects
(alive or dead without dementia), while the ROC curve displaying Se(c,t) versus 1 − Sp1(c,t) (denoted ROC1) evaluates
the discrimination between future demented subjects and
those who survive without dementia until time t.
Two estimators of the ROC curves were used. The nonparametric IPCW estimator, which accounts for right-censoring,
was computed using the R package timeROC (27). In this
approach, Se(c,t) and Sp(c,t) are computed in subjects uncensored at time t, but the contribution of each subject is
weighted by his/her probability of being uncensored. To
apply the IPCW estimator to interval-censored data, we used
the following simple imputation rules for subjects who died
without a dementia diagnosis: When the interval between the
last visit and death was less than 2 years, the subject was
Am J Epidemiol. 2014;180(8):790–798
10-Year Dementia Prediction Accounting for Competing Death 793
score, IST score, DSST score, and MMSE-EM score were
not associated with the risk of dementia (Table 2). After adjustment for total MMSE score and BVRT score, change in
MMSE score (P = 0.46) and change in BVRT score (P =
0.15) were not associated with the risk of dementia (not
shown). Associations were also nonsignificant when a binary
variable was used for the change (improvement in the score
between T0 and T1 vs. no improvement; results not shown).
Thus, after adjustment for current cognitive status, cognitive
change in the previous year did not appear to be associated
with the risk of dementia over the next 9 years.
considered deceased without dementia; otherwise, the subject was considered right-censored after the last visit. Even
if this strategy was shown to give good results with designs
similar to PAQUID, the ROC curves were also estimated by
using the illness-death model to impute the probability of
being a case or control for all of the subjects with unknown
status (16). Indeed, IPCW may be slightly biased for large
between-visit intervals and markers highly associated with
death risk, while model-based imputation may be slightly
biased in the case of a misspecified imputation model (16).
The 2 estimators are detailed in Web Appendix 1, available at
http://aje.oxfordjournals.org/.
The optimism due to the estimation of predictive accuracy
measures on the learning data set was corrected by means of a
pooled 10-fold cross-validation method (28).
Prediction score for 10-year risk of dementia
As a consequence, we performed the main analysis using
the whole data set and the cognitive assessment performed at
T0 to predict the risk of dementia up to T10. Table 1 gives the
characteristics and cognitive test scores of sample B. The mean
age at the initial cognitive assessment was 74 (standard deviation, 6.1) years, 55.8% of the subjects were women, and 71%
had a diploma from primary school or a higher educational
level. The mean MMSE score at baseline was 26.6 (standard
deviation, 2.7). Among the 2,795 subjects, 265 (9.5%) were
diagnosed as demented during the 10-year follow-up period,
1,015 (36.3%) died without a dementia diagnosis, and 120
(4.3%) died after a dementia diagnosis. The status of the
2,795 subjects at each visit is shown in Web Figure 1.
RESULTS
Impact of learning ability
The characteristics of sample A are provided in Table 1,
and estimates of the final illness-death model after backward
selection are displayed in Table 2. Five predictors of the risk
of dementia were retained: age as a continuous variable, complaints about forgetfulness in daily living, IST score, DSST
score, and MMSE-EM score. After adjustment for the above
variables, changes over the first year (T1 – T0) in 4-IADL
Table 1. Characteristics of 2 Participant Subsamples Chosen to Develop a Prediction Score for Identifying Subjects
at High Risk of Dementia, PAQUID Study, France, 1988–1998
Sample A (n = 1,313)
Characteristic
No.
Incident dementia
Death without dementia diagnosisa
a
Death after dementia diagnosis
%
High educational level
No.
121
265
424
1,015
59
%
Mean (SD)
120
Age, years
Female sex
Mean (SD)
Sample B (n = 2,795)
73.5 (5.9)
74.0 (6.1)
734
55.9
1,559
55.8
1,011
77.0
1,984
71.0
Cognitive test score
IST
28.3 (5.8)
27.5 (5.9)
DSST
30.2 (11.2)
27.5 (11.3)
BVRT
10.8 (2.4)
10.3 (2.5)
MMSE
26.9 (2.4)
26.6 (2.7)
6.5 (1.1)
6.3 (1.1)
MMSE-EM
Memory complaints
Forgetfulness in daily living
680
51.8
1,393
49.8
Retaining new information
406
30.9
913
32.7
Difficulty in calculation
261
19.9
640
22.9
Difficulty in orientation
50
3.8
119
4.3
Abbreviations: BVRT, Benton Visual Retention Test; DSST, Digit Symbol Substitution Test; IST, Isaacs Set Test;
MMSE, Mini-Mental State Examination; MMSE-EM, episodic memory subtest of the Mini-Mental State Examination;
PAQUID, Personnes Agées QUID; SD, standard deviation.
a
Because the 10-year follow-up visit could take place anytime between 9 and 11 years after baseline, models were
fitted using vital status at 11 years for everyone.
Am J Epidemiol. 2014;180(8):790–798
794 Jacqmin-Gadda et al.
Table 2. Adjusted Hazard Ratio for the 9-Year Risks of Dementia and Death in the Illness-Death Model According to
Cognitive Assessment at the 1-Year Visit (T1) and 1-Year Cognitive Changes (Sample A; n = 1,313), PAQUID Study,
France, 1988–1998
Risk of Dementia
Covariate
HR
95% CI
P Value
Risk of Death
HR
95% CI
Risk of Death After Dementia
P Value
HR
95% CI
P Value
Final model after backward
selection
Age, years
2.64 1.98, 3.54 <0.001 1.90 1.53, 2.37 <0.001 1.89 1.21, 2.94 0.005
Female sex
Forgetfulness in daily living
0.44 0.34, 0.56 <0.001
1.41 0.99, 2.01
0.060
4-IADL score
1.43 1.21, 1.68 <0.001
IST score
0.92 0.89, 0.96 <0.001
DSST score
0.97 0.95, 1.00
MMSE-EM score
0.72 0.62, 0.85 <0.001
0.017 0.99 0.97, 1.00
0.013
Association with 1-year cognitive
change (T1 minus T0)a
4-IADL
IST
1.01 0.97, 1.05
0.73
DSST
1.00 0.96, 1.04
0.94
MMSE-EM
1.13 0.96, 1.33
0.14
1.00 0.85, 1.18
0.99
0.99 0.97, 1.02
0.44
Abbreviations: CI, confidence interval; DSST, Digit Symbol Substitution Test; HR, hazard ratio; 4-IADL, 4-item
Instrumental Activities of Daily Living Scale; IST, Isaacs Set Test; MMSE-EM, episodic memory subtest of the
Mini-Mental State Examination; PAQUID, Personnes Agées QUID.
a
Adjusted for predictors from the final model.
Table 3 presents estimates from the final model after backward selection and inclusion of the significant interaction between sex and education in the risk of death. Seven predictors
remained associated with the 10-year risk of dementia. The 5
most significant predictors (P < 0.01) were the same as those
in the preliminary analysis: The risk of dementia was higher
for older subjects, those who complained about forgetfulness
in daily living, and those with poor performance on the IST,
the DSST, and the MMSE-EM. After adjustment for these
cognitive measures, a high educational level and a high global
MMSE score tended to be associated with a higher risk of dementia. Indeed, for a given current cognitive level, subjects
with high initial intellectual abilities had a greater risk of developing dementia. The risk of death among nondemented
Table 3. Adjusted Log Hazard Ratioa for the 10-Year Risks of Dementia and Death in the Final Illness-Death Model
(Sample B; n = 2,795) After Backward Selection, PAQUID Study, France, 1988–1998
Covariate
Age, years
Risk of Dementia
0.660 (0.075)
0.258 (0.145)
−0.142 (0.125)
0.333 (0.124)
−0.173 (0.084)
−1.07 (0.149)
Female sex
High educational level
Sex × education
Forgetfulness in daily living
0.735 (0.142)
−0.424 (0.153)
−0.210 (0.106)
4-IADL score
0.376 (0.051)
IST score
−0.060 (0.012)
DSST score
−0.044 (0.09)
MMSE-EM score
Risk of Death After Dementia
0.371 (0.174)
Difficulty in calculation
MMSE score
Risk of Death
0.912 (0.102)
−0.014 (0.005)
0.021 (0.008)
0.068 (0.029)
−0.318 (0.060)
Abbreviations: DSST, Digit Symbol Substitution Test; 4-IADL, 4-points score from the Instrumental Activities of Daily
Living; IST, Isaacs Set Test; MMSE, Mini-Mental State Examination; MMSE-EM, episodic memory subtest of the
Mini-Mental State Examination; PAQUID, Personnes Agées QUID.
a
Values are β coefficients with standard errors.
Am J Epidemiol. 2014;180(8):790–798
10-Year Dementia Prediction Accounting for Competing Death 795
Table 4. Area Under the Receiver Operating Characteristic Curve for the Proposed Dementia Prediction Score
Estimated in the Learning Data Set (Sample B; n = 2,795) and by Pooled 10-Fold Cross-Validation, Using an Inverse
Probability of Censoring Weighting Estimator and a Model-Based Imputation Estimator and 2 Different Definitions of
Specificity,a PAQUID Study, France, 1988–1998
Inverse Probability of Censoring Weighting
Model-Based Imputation
Parameter
Crude Estimate (SE)
10-Fold CV Estimate
Crude Estimate (SE)
10-Fold CV Estimate
AUROC1b
0.814 (0.015)
0.810
0.828 (0.014)
0.819
AUROC2c
0.750 (0.016)
0.746
0.770 (0.016)
0.761
AUROC1
0.844 (0.019)
0.838
0.851 (0.030)
0.843
AUROC2
0.817 (0.019)
0.811
0.830 (0.031)
0.822
10-year prediction
5-year prediction
Abbreviations: AUROC, area under the ROC curve; CV, cross-validation; PAQUID, Personnes Agées QUID; ROC,
receiver operating characteristic; SE, standard error.
a
For definitions, see text.
b
AUROC calculated using the first definition of specificity.
c
AUROC calculated using the second definition of specificity.
persons was lower for younger subjects, women, highly educated subjects, subjects without dependence on the IADL
Scale, and those with a high DSST score. After adjustment
for these factors, persons who complained about forgetfulness in daily living and about difficulties in calculation had
a lower risk of death without dementia. Finally, death risk
among demented persons was higher for older subjects,
men, and subjects with a high DSST score at baseline. Indeed, subjects who became demented despite a high initial
DSST score were declining more rapidly, a process that may
have led more quickly to death. Web Appendix 2 and Web
Table 1 show the good calibration of this model.
The dementia prediction score was computed by means of
equation 2, with the 7 predictors retained in the submodel for
the risk of dementia and the estimated parameters given in
Table 3. Table 4 gives the estimators of the AUROC at 5 and
10 years for this prediction score. Because of the large sample
size, the crude estimates and the 10-fold cross-validation estimates were very close. The nonparametric IPCW estimator and
the model-based imputation estimator led to similar conclusions. All of the AUROCs were significantly different from
0.5 (P < 0.001). The high values of AUROC1 meant that the
prediction score had a very good ability to discriminate between
subjects who would become demented in the next 5 or 10 years
and those who would still be free of dementia and alive at the
end of this period. The prediction score retained a very high
ability to distinguish subjects at high risk of developing dementia in the next 5 years from those who would remain dementiafree or die without dementia (AUROC2 > 0.80). For 10-year
prediction, the difference between AUROC1 and AUROC2
was larger because the number of deaths increased, and
AUROC2 dropped to about 0.75. To help users choose a cutpoint that is suitable for the objective of the screening, Table 5
displays estimated sensitivities and specificities for selected
prediction score cutpoints. The cutpoint −1.92 gives the highest value of the sum of sensitivity and specificity for both
10-year and 5-year predictions and the 2 definitions of specificity (similar to −1.55 for 10 years and Sp2) while maintaining
a good sensitivity; it is optimal when equal weights are given to
false-positive and false-negative subjects.
Figure 2 displays the ROC curves estimated by IPCW for
the proposed prediction score (cross-validation estimates)
and for the cognitive tests DSST, MMSE, and IST alone.
Whatever the definition of specificity, the predictive ability
of the combined score is better than that of each cognitive
test individually considered: AUROC2 values were 0.713
Table 5. Sensitivities and Specificities of Selected Dementia Prediction Score Cutpoints Estimated by Inverse
Probability of Censoring Weighting, Using 2 Different Definitions of Specificity,a PAQUID Study, France, 1988–1998
10-Year Prediction
Cutpoint
5-Year Prediction
Sensitivity, %
Specificity 1, %
Specificity 2, %
Sensitivity, %
Specificity 1, %
Specificity 2, %
−2.51
90.1
47.2
39.4
98.0
39.2
36.2
−1.92
79.9
67.5
58.8
92.9
59.4
55.6
−1.55
67.9
79.0
70.0
80.1
70.9
66.8
−1.15
53.0
88.7
80.0
70.4
81.6
77.7
−0.50
29.1
96.7
91.3
42.5
92.7
90.0
Abbreviation: PAQUID, Personnes Agées QUID.
a
For definitions, see text.
Am J Epidemiol. 2014;180(8):790–798
796 Jacqmin-Gadda et al.
B)
100
100
80
80
Sensitivity, %
Sensitivity, %
A)
60
40
60
40
20
20
0
0
0
20
40
60
80
100
1 − Specificity, %
0
20
40
60
80
100
1 − Specificity, %
Figure 2. Inverse probability of censoring weighting estimates of the receiver operating characteristic (ROC) curve for scores on the Digit Symbol
Substitution Test, the Isaacs Set Test, and the Mini-Mental State Examination and for the combined dementia prediction score M derived using
10-fold cross-validation, according to 2 different definitions of specificity, Personnes Agées QUID (PAQUID) Study, France, 1988–1998.
A) ROC curve among survivors (denoted ROC1 in the main text); B) ROC curve in the whole population (denoted ROC2 in the main text). Solid
line, combined score; dashed line, Digit Symbol Substitution Test; dotted line, Isaacs Set Test; dashed-dotted line, Mini-Mental State Examination.
for the DSST, 0.676 for the IST, and 0.656 for the MMSE as
compared with 0.746 for the combined score.
DISCUSSION
Using data from a large population-based cohort, this study
led to 2 main results. First, we found that 2 cognitive assessments administered 1 year apart did not lead to better prediction
of dementia risk than a single one. Especially, the improvement
in cognitive scores between the first 2 assessments was not predictive of a lower dementia risk when adjusted for the second
cognitive assessment. These results confirm those of 2 previous
studies showing that changes in MMSE score (5) and in
SIDAM [Structured Interview for the Diagnosis of Dementia
of the Alzheimer Type, Multi-Infarct Dementia and Dementias
of Other Aetiology] score (4) between 2 assessments given at
1- to 2-year intervals were predictive of future dementia only
when they were not adjusted for score at time 2. A possible explanation could be that this improvement reflects stress at T0
rather than true learning ability.
Secondly, we developed a simple prediction score based
on 3 cognitive tests (the IST, the DSST, and the MMSE) and
1 subjective memory complaint that exhibited a good predictive
ability to identify subjects at high risk of becoming demented in
the next 5 or 10 years. We emphasize that the subjective memory complaint remained a predictor of 10-year dementia after
adjustment for the objective cognitive assessment. Moreover,
this work confirms previous PAQUID results showing that
the MMSE-EM is more predictive than global MMSE score
(9, 20).
Important strengths of this study include the large sample
size and representativeness of the cohort and the quality of
the clinical diagnosis of dementia, including a clinical examination. Nevertheless, the main strengths of this work lie in the
careful handling of the semicompeting risk of death and interval censoring of dementia diagnosis in both the building and
the evaluation of the prediction score. Prediction scores previously proposed in the literature (2) have been built using a logistic model fitted in subjects with known dementia status at
the end of the prediction window (7, 9, 29–31) or using a
Cox model without dealing with interval censoring and
death (8). In both cases, the prediction scores were evaluated
in subjects with known dementia status at the end of the prediction window (excluding subjects who were censored or
who died without dementia). Although this strategy may be acceptable for short-term prediction, it presents several sources
of bias for long-term prediction. Because of attrition and
death, the selected sample is most often not representative of
the initial sample. Interval censoring is not handled, and thus
only demented subjects who survive long enough to be diagnosed are considered cases. Finally, it is generally not clearly
stated that the ROC curve estimated in such selected samples
evaluates the discrimination between future demented subjects
and subjects who are alive and nondemented (with possible
bias due to censoring). As illustrated here, the corresponding
AUROCs are often higher than the AUROCs measuring the
discrimination between demented subjects and all other subjects, because the predictors for dementia are also predictors
for death. In some studies, censored and deceased persons
were not excluded but the last observed disease status before
death or censoring was used (5), which can also lead to possible bias. With the above limitations, the previous studies led to
prediction scores with estimated AUROCs similar to or lower
than our estimated AUROC1: 0.77 for 10 years and 0.83 for
Am J Epidemiol. 2014;180(8):790–798
10-Year Dementia Prediction Accounting for Competing Death 797
5 years (28) or 0.79 for 4.5 years (8). A higher AUROC for
5-year prediction (0.88) was reached in the study by Nakata
et al. (31), but this estimate was not corrected for optimism,
whereas the sample size was small and highly selected by attrition (258 subjects followed among 465).
To our knowledge, this is the first proposed prediction
score for dementia that accounts for censoring and competing
risk of death. Even if external validation is found to be useful
in the future, this score, based on a simple cognitive assessment, exhibited good performance for prediction of 10-year
dementia risk. We emphasize that the method of considering
deceased subjects when evaluating the discriminatory ability
of a dementia prediction score must be clearly stated, since
discrimination among survivors may be largely better than
that in the whole population.
ACKNOWLEDGMENTS
Author affiliations: Institut National de la Santé et de la
Recherche Médicale (INSERM), Centre INSERM U897,
Bordeaux, France (Hélène Jacqmin-Gadda, Paul Blanche,
Emilie Chary, Lucie Loubère, Hélène Amieva, Jean-François
Dartigues); Centre INSERM U897, Institut de Santé Publique, d’Épidémiologie et de Développement, University of
Bordeaux, Bordeaux, France (Hélène Jacqmin-Gadda, Paul
Blanche, Emilie Chary, Lucie Loubère, Hélène Amieva,
Jean-François Dartigues); and Department of Neurology,
Bordeaux University Hospital, Bordeaux, France (JeanFrançois Dartigues).
This research was supported by a grant from France
Alzheimer awarded to H.J.-G. in 2009. The PAQUID
Study is supported by Ipsen (Paris, France), Novartis International AG (Basel, Switzerland), Caisse Nationale de Solidarité pour l’Autonomie (Paris, France), and Groupe AGRICA
(Paris, France).
Conflict of interest: none declared.
REFERENCES
1. Aisen PS, Andrieu S, Sampaio C, et al. Report of the task force
on designing clinical trials in early ( predementia) AD.
Neurology. 2011;76(3):280–286.
2. Stephan BC, Kurth T, Matthews FE, et al. Dementia risk
prediction in the population: Are screening models accurate?
Nat Rev Neurol. 2010;6(6):318–326.
3. Amieva H, Le Goff M, Millet X, et al. Prodromal Alzheimer’s
disease: successive emergence of the clinical symptoms. Ann
Neurol. 2008;64(5):492–498.
4. Hensel A, Angermeyer MC, Zaudig M, et al. Measuring
cognitive change in older adults: reliable change indices for the
SIDAM. J Neurol. 2007;254(1):91–98.
5. Hensel A, Luck T, Luppa M, et al. Does a reliable decline in
Mini Mental State Examination total score predict dementia?
Diagnostic accuracy of two reliable change indices. Dement
Geriatr Cogn Disord. 2009;27(1):50–58.
6. Jacqmin-Gadda H, Fabrigoule C, Commenges D, et al. A
5-year longitudinal study of the Mini-Mental State Examination
in normal aging. Am J Epidemiol. 1997;145(6):498–506.
Am J Epidemiol. 2014;180(8):790–798
7. Gomar JJ, Bobes-Bascaran MT, Conejero-Goldberg C, et al.
Utility of combinations of biomarkers, cognitive markers, and
risk factors to predict conversion from mild cognitive
impairment to Alzheimer disease in patients in the Alzheimer’s
Disease Neuroimaging Initiative. Arch Gen Psychiatry. 2011;
68(9):961–969.
8. Jessen F, Wiese B, Bickel H, et al. Prediction of dementia in
primary care patients. PLOS ONE. 2011;6(2):e16852.
9. Chary E, Amieva H, Pérès K, et al. Short- versus long-term
prediction of dementia among subjects with low and high
educational levels. Alzheimers Dement. 2013;9(5):562–571.
10. Ostbye T, Hill G, Steenhuis R. Mortality in elderly Canadians
with and without dementia: a 5-year follow-up. Neurology.
1999;53(3):521–526.
11. Joly P, Commenges D, Helmer C, et al. A penalized likelihood
approach for an illness-death model with interval-censored
data: application to age-specific incidence of dementia.
Biostatistics. 2002;3(3):433–443.
12. Leffondré K, Touraine C, Helmer C, et al. Interval-censored
time-to-event and competing risk with death: Is the
illness-death model more accurate than the Cox model? Int J
Epidemiol. 2013;42(4):1177–1186.
13. Saha P, Heagerty PJ. Time-dependent predictive accuracy in the
presence of competing risks. Biometrics. 2010;66(4):999–1011.
14. Zheng Y, Cai T, Jin Y, et al. Evaluating prognostic accuracy of
biomarkers under competing risk. Biometrics. 2012;68(2):
388–396.
15. Blanche P, Dartigues JF, Jacqmin-Gadda H. Estimating and
comparing time-dependent areas under receiver operating
characteristic curves for censored event times with competing
risks. Stat Med. 2013;32(30):5381–5397.
16. Jacqmin-Gadda H, Blanche P, Chary E, et al. Receiver
operating characteristic curve estimation for time to event
with semicompeting risks and interval censoring [ published
online ahead of print May 6, 2014]. Stat Methods Med Res.
(doi:10.1177/0962280214531691).
17. Dartigues JF, Gagnon M, Barberger-Gateau P, et al. The Paquid
epidemiological program on brain ageing. Neuroepidemiology.
1992;11(suppl 1):14–18.
18. American Psychiatric Association. Diagnostic and Statistical
Manual for Mental Disorders, Third Edition, Revised.
Washington, DC: American Psychiatric Association; 1987.
19. Folstein MF, Folstein SE, McHugh PR. “Mini-mental state”: a
practical method for grading the cognitive state of patients for
the clinician. J Psychiatr Res. 1975;12(3):189–198.
20. Carcaillon L, Amieva H, Auriacombe S, et al. A subtest of the
MMSE as a valid test of episodic memory? Comparison with
the Free and Cued Reminding Test. Dement Geriatr Cogn
Disord. 2009;27(5):429–438.
21. Isaacs B, Kennie AT. The Set Test as an aid to the detection of
dementia in old people. Br J Psychiatry. 1973;123(575):
467–470.
22. Benton A. Manuel pour l’Application du Test de Retention
Visuelle: Applications Cliniques et Experimentales. Paris,
France: Centre de Psychologie Appliquee; 1965.
23. Wechsler D. Wechsler Adult Intelligence Scale–Revised.
New York, NY: The Psychological Corporation; 1981.
24. Barberger-Gateau P, Commenges D, Gagnon M, et al.
Instrumental Activities of Daily Living as a screening tool for
cognitive impairment and dementia in elderly community
dwellers. J Am Geriatr Soc. 1992;40(11):1129–1134.
25. Touraine C, Joly P, Gerds TA. Package ‘SmoothHazard’: fitting
illness-death model for interval-censored data. Version 1.0.9.
Vienna, Austria: R Foundation for Statistical Computing; 2013.
http://cran.r-project.org/web/packages/SmoothHazard/
798 Jacqmin-Gadda et al.
SmoothHazard.pdf. Published March 8, 2013. Accessed
August 29, 2013.
26. Amieva H, Jacqmin-Gadda H, Orgogozo JM, et al. The 9 year
cognitive decline before dementia of the Alzheimer type: a
prospective population-based study. Brain. 2005;128(5):
1093–1101.
27. Blanche P. Package ‘timeROC’: time-dependent ROC curve
and AUC for censored survival data. Version 0.2. Vienna,
Austria: R Foundation for Statistical Computing; 2013. http://
cran.r-project.org/web/packages/timeROC/timeROC.pdf.
Published May 27, 2013. Accessed August 29, 2013.
28. Airola A, Pahikkala T, Waegeman W, et al. An experimental
comparison of cross-validation techniques for estimating the
area under the ROC curve. Comput Stat Data Anal. 2011;55(4):
1828–1844.
29. Tierney MC, Szalai JP, Snow WG, et al. Prediction of probable
Alzheimer’s disease in memory-impaired patients: a
prospective longitudinal study. Neurology. 1996;46(3):
661–665.
30. Tierney MC, Yao C, Kiss A, et al. Neuropsychological tests
accurately predict incident Alzheimer disease after 5 and 10
years. Neurology. 2005;64(11):1853–1859.
31. Nakata E, Kasai M, Kasuya M, et al. Combined memory and
executive function tests can screen mild cognitive impairment
and converters to dementia in a community: the Osaki-Tajiri
project. Neuroepidemiology. 2009;33(2):103–110.
Am J Epidemiol. 2014;180(8):790–798