OR Larner_Layout 1 29/01/2015 10:15 Page 1
Accuracy in cognitive assessment
z Original research
Speed versus accuracy in cognitive
assessment when using CSIs
Andrew J Larner MD, MRCP (UK)
When using cognitive screening instruments (CSIs) there is always a balance to be struck
between test sensitivity and specificity, with the chosen test cut-off being determined by
the needs of the particular clinical situation. Dr Larner investigates whether there is a
trade-off between CSI diagnostic accuracy and administration time, or, in other words,
whether shorter CSIs are less accurate than longer ones which may sample more
cognitive domains and/or in greater depth.
I
t has long been recognised that
there is a trade-off between speed
and accuracy in the performance
of voluntary movements, such that
more accurate movements are
performed more slowly. 1 This
speed/accuracy trade-off may perhaps apply to any task, and since
speed is inversely proportional to
time it may also be formulated as
a time/accuracy trade-off, longer
times being required for greater
accuracy.
The utility of cognitive screening instruments (CSIs) for the
diagnosis of dementia and lesser
degrees of cognitive impairment
may be indicated by a number of
summary parameters, probably the
most familiar of which are sensitivity and specificity. It is well recognised that there is always a balance
or trade-off to be struck between
test sensitivity and specificity, with
the chosen test cut-off being determined by the needs of the particular clinical situation. Distinction
may be drawn between case-finding, the identification of a target
condition with minimal false negatives, often measured by positive
predictive value, and screening, the
ruling out of a target condition with
minimal false positives, often measured by negative predictive value.
Likewise, the clinical setting
may also determine time constraints on CSI administration,
www.progressnp.com
CSI
Setting
No.
Dementia
prevalence
M:F
(% male)
Age range
(years)
Ref no.
MMSE,
MMP
Cognitive
function clinic
225
21%
130:95 (58)
20–86
(median 62)
14
ACE
Cognitive
function clinic
285
49%
147:138 (52)
N/A
15
ACE-R
Cognitive
function clinic
243
35%
135:108 (56)
24—85
(mean 60)
16
6CIT
Cognitive
function clinic
245
20%
124:121 (51)
16–94
(median 59)
17
DemTect
Cognitive
function clinic
111
52%
52:59 (47)
23–86
(median 63)
18
MoCA
Cognitive
function clinic
150
24%
93:57 (62)
20–87
(median 61)
19
TYM
Cognitive
function clinic
and old age
psychiatry
memory clinic
224
35%
130:94 (58)
20–90
(mean 63)
20
CSI = cognitive screening instrument; MMSE = Mini-Mental State Examination; MMP = MiniMental Parkinson; ACE = Addenbrooke’s Cognitive Examination; ACE-R = Addenbrooke’s
Cognitive Examination-Revised; 6CIT = Six-Item Cognitive Impairment Test; MoCA = Montreal
Cognitive Assessment; TYM = Test Your Memory test.
Table 1. Study demographics
which also need to be factored into
the decision as to which CSI(s) to
use. For example, in day-to-day
clinical practice in the secondary
care setting, instruments such as
the ADAS-Cog, 2 the Mattis
Dementia Rating Scale,3 and the
Neuropsychological Assessment
Battery4 are generally considered
too long for routine use, being
largely reserved for research settings. In primar y care settings,
instruments such as the MiniMental
State
Examination
Progress in Neurology and Psychiatry January/February 2015
21
OR Larner_Layout 1 29/01/2015 10:15 Page 2
Original research z
Accuracy in cognitive assessment
CSI
Approx.
administration
time (mins)
Total
score
No. of items/
questions
Accuracy for
dementia diagnosis
Ref no.
MMSE
5–10
30
21
0.86 (0.81–0.90)
14
MMP
5–10
32
23
0.86 (0.81–0.91)
14
ACE
15–20
100
52
0.84 (0.80–0.88)
15
ACE-R
15–20
100
66
0.89 (0.85–0.93)
16
6CIT
2–3
28
7
0.80 (0.75–0.85)
17
DemTect
8–10
18
13
0.78 (0.71–0.86)
18
MoCA
10–15
30
22
0.81 (0.75–0.88)
19
TYM
5–10 (selfadministered
under medical
supervision)
50
25
0.83 (0.78–0.88)
20
MMSE = Mini-Mental State Examination; MMP = Mini-Mental Parkinson; ACE = Addenbrooke’s
Cognitive Examination; ACE-R = Addenbrooke’s Cognitive Examination-Revised; 6CIT = Six-Item
Cognitive Impairment Test; MoCA = Montreal Cognitive Assessment; TYM = Test Your Memory
(TYM) test.
Table 2. Cognitive screening instrument (CSI) approximate administration time and surrogate measures thereof
(total test score, total number of test items/questions) with diagnostic accuracy for diagnosis of dementia
(MMSE) may be considered too
long, and briefer instruments have
been recommended for this setting such as the Six-item Cognitive
Impairment Test (6CIT), the
Memory Impairment Screen, and
the Mini-Cog.5 Patient self-administered tests such as the Test Your
Memory (TYM) test may be of particular value in situations where
clinician time is limited, precluding performance of clinicianadministered tests.6
A question arising from these
considerations is whether there is a
trade-off between CSI diagnostic
accuracy and administration time,
or, in other words, whether shorter
CSIs are less accurate than longer
ones which may sample more cognitive domains (eg attention, executive function, visuo-spatial
function) and/or in greater depth.
Conversely, another possibility is
that longer tests might be less
22
accurate than shorter ones, if
patients should become fatigued
and less cooperative with prolonged testing.
The speed/accuracy question
might be examined by comparing
test diagnostic accuracy against
duration of test administration.
Since the latter parameter is not
routinely measured in the clinical
setting, more easily accessible surrogate measures might be used,
such as the overall test score or by
counting the total number of
items/questions in the test (in
each case higher ≡ longer). Overall
test scores may be the same despite
different numbers of test items (eg
MMSE, Montreal Cognitive
Assessment [MoCA]) depending
on how each item is scored.
Study aim
The aim of the study presented
here was to examine the
Progress in Neurology and Psychiatry January/February 2015
relationship between overall test
accuracy for the diagnosis of
dementia versus no dementia of
several commonly used CSIs, and
the aforementioned surrogate
measures of test administration
time to see if there is a trade-off
between speed and accuracy. The
CSIs examined were: the MMSE;7
the Mini-Mental Parkinson
(MMP); 8 the Addenbrooke’s
Cognitive Examination (ACE); 9
the Addenbrooke’s Cognitive
Examination-Revised (ACE-R);10
the 6CIT; 11 DemTect; 12 the
MoCA;13 and the TYM test.6 None
of these CSIs is a timed test,
although some (MMP, ACE, ACER, DemTect and MoCA) include a
timed verbal fluency item.
Materials and methods
Data from seven previous pragmatic diagnostic accuracy studies
which examined the MMSE, 14
MMP (used in this instance as a
general cognitive screening instrument, not for cognitive impairments in Parkinson’s disease
patients), 14 ACE, 15 ACE-R, 16
6CIT,17 DemTect,18 MoCA,19 and
TYM20 were re-analysed. All studies were undertaken in dedicated
memory clinics based in secondary
care settings, and all used a standardised methodology. 21 Study
details are shown in Table 1. In
each of these studies, the criterion
diagnosis was established by the
judgement of an experienced clinician based on widely-accepted
clinical diagnostic criteria. Because
these were clinic-based pragmatic
studies, there was no normal control group; the non-demented
cases consisted of patients with at
minimum subjective memor y
impairment, as well as patients
with mild cognitive impairment
insufficient to mandate a
dementia diagnosis.
Overall test accuracy, defined as
the sum of true positives and true
www.progressnp.com
OR Larner_Layout 1 29/01/2015 10:15 Page 3
Accuracy in cognitive assessment
Results
Table 2 shows the approximate
administration time of the examined CSIs, along with the surrogate
measures of administration time
(total test score, total number of
test items/questions), along with
overall diagnostic accuracy of each
test obtained from the pragmatic
diagnostic accuracy studies. These
data are plotted in Figures 1 and 2
respectively.
There was a positive correlation
between accuracy and total test
score (r=0.61), a correlation which
may be classified as moderate but
which did not reach statistical significance (t=1.86, df=6, p>0.1).
Likewise, there was a positive
correlation between accuracy and
total number of items/questions
(r=0.72) which may be classified as
a high correlation and which did
reach statistical significance
(t=2.52, df=6, p<0.05).
Discussion
In this study, data from seven pragmatic diagnostic accuracy studies
of CSIs 14–20 were analysed to
examine the relationship between
overall test diagnostic accuracy and
surrogate measures of test administration time. Although overall
test accuracy for the diagnosis of
dementia versus no dementia was
similar for all tests (range
0.78–0.89), nonetheless there were
positive correlations between accuracy and the measures of administration time, though this was only
statistically significant for the total
www.progressnp.com
Accuracy vs total score
1
0.9
0.8
Accuracy
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
20
40
60
Total score
80
100
120
Figure 1. Scatter plot of test accuracy versus total test score (= surrogate measure of test
administration time)
Accuracy vs no. of items/questions
1
0.9
0.8
0.7
Accuracy
negatives divided by the total
number of patients tested (= output or effect, hence the dependent
variable, y axis), was plotted against
overall test score and against the
number of items/questions in the
test (inputs or causes, hence
independent variables, x axis).
Correlations between these parameters were calculated.
z Original research
0.6
0.5
0.4
0.3
0.2
0.1
0
0
20
40
No. of items/questions
60
80
Figure 2. Scatter plot of test accuracy versus total number of test items/questions (= surrogate measure of test administration time)
number of items/questions and
not for total test score. These
observations suggest that tests with
more items (ie longer tests) are
more accurate, or in other words
that there is a trade-off between
speed and accuracy, the speedier
(ie shorter) tests being less accurate. Incorrect diagnoses, the sum
of false positives and false negatives
(which if divided by the total number of patients tested = error rate
or inaccuracy or [1 – accuracy]),
may be twice as frequent in the
least accurate test (DemTect,
22/100) versus the most accurate
(ACE-R, 11/100).
Inter-study comparisons are
problematic, notwithstanding the
consistency of study protocols 21
and authorship of the studies
examined here. Potential shortcomings of the analysis were the
different sample characteristics for
each of the seven studies (Table 1),
particularly in terms of sample size
Progress in Neurology and Psychiatry January/February 2015
23
OR Larner_Layout 1 29/01/2015 10:15 Page 4
Original research z
Accuracy in cognitive assessment
(range 111–285) and dementia
prevalence (21–52%). Sample size
calculations were not performed,
but a pragmatic approach to sample size estimates has suggested
that normative ranges for sample
sizes may be calculated for common research designs, with anything in the range of 25–400 being
acceptable.22 A second limitation
relates to the progressive fall in
dementia prevalence that has been
seen in patients referred to the
clinic (and hence in the reported
studies) over the past decade, perhaps reflecting national dementia
directives which may have had the
(unintended?) effect of prompting
more individuals with subjective
memory impairment to attend for
assessment.23
In conclusion, the findings of
this study suggest that there is
indeed a trade-off for CSIs
between duration of administration and diagnostic accuracy, and
therefore that the traditional policy of long outpatient clinic
appointments (45–60 minutes) for
patients with cognitive complaints
is appropriate to permit adequate
time for the administration of
longer tests (if need be) and hence
accurate patient diagnosis.
Accordingly, any attempt to
shorten outpatient clinic appointment times in order to increase
patient throughput (ie total number of patients seen) would not
seem to be justified, unless and
until such time as shorter CSIs can
be shown to be as diagnostically
accurate as longer tests.
Dr Larner is Consultant Neurologist
at the Cognitive Function Clinic,
Walton Centre for Neurology and
Neurosurgery, Liverpool
Declaration of interests
There are no conflicts of interest
declared.
References
1. Woodworth RS. Accuracy of voluntary
movements. Psychol Rev 1899;3:1–101.
2. Rosen WG, Mohs RC, Davis KL. A new rating scale for Alzheimer’s disease. Am J
Psychiatry 1984;141:1356–64.
3. Mattis S. Dementia Rating Scale. Windsor:
NFER-Nelson, 1992.
4. Stern RA, White T. Neuropsychological
Assessment Battery. Lutz, FL: Psychological
Assessment Resources, Inc., 2003.
5. Brodaty H, Low-Lee F, Gibson L, et al. What
is the best dementia screening instrument for
general practitioners to use? Am J Geriatr
Psychiatry 2006;14:391–400.
6. Brown J, Pengas G, Dawson K, et al. Self
administered cognitive screening test (TYM)
for detection of Alzheimer’s disease: cross sectional study. BMJ 2009;338:b2030.
7. Folstein MF, Folstein SE, McHugh PR. ‘MiniMental State.’ A practical method for grading
the cognitive state of patients for the clinician.
J Psychiatr Res 1975;12:189–98.
8. Mahieux F, Michelet D, Manifacier M-J, et
al. Mini-Mental Parkinson: first validation
study of a new bedside test constructed
for Parkinson’s disease. Behav Neurol
1995;8:15–22.
9. Mathuranath PS, Nestor PJ, Berrios GE, et
al. A brief cognitive test battery to differentiate Alzheimer’s disease and frontotemporal
dementia. Neurology 2000;55:1613–20.
10. Mioshi E, Dawson K, Mitchell J, et al. The
Addenbrooke’s Cognitive Examination
Revised: a brief cognitive test battery for
dementia screening. Int J Geriatr Psychiatry
2006;21:1078–85.
11. Brooke P, Bullock R. Validation of a 6 item
cognitive impairment test with a view to primary care usage. Int J Geriatr Psychiatry
1999;14:936–40.
12. Kalbe E, Kessler J, Calabrese P, et al.
DemTect: a new, sensitive cognitive screening
test to support the diagnosis of mild cognitive
impairment and early dementia. Int J Geriatr
Psychiatry 2004;19:136–43.
13. Nasreddine ZS, Phillips NA, Bédirian V, et al.
The Montreal Cognitive Assessment, MoCA: a
brief screening tool for mild cognitive impairment. J Am Geriatr Soc 2005;53:695–9.
14. Larner AJ. Mini-Mental Parkinson (MMP)
as a dementia screening test: comparison with
the Mini-Mental State Examination (MMSE).
Curr Aging Sci 2012;5:136–9.
15. Larner AJ. Addenbrooke’s Cognitive
Examination (ACE) for the diagnosis and differential diagnosis of dementia. Clin Neurol
Neurosurg 2007;109:491–4.
16. Larner AJ. Addenbrooke’s Cognitive
Examination-Revised (ACE-R): pragmatic study
of cross-sectional use for assessment of cognitive complaints of unknown aetiology. Int J
Geriatr Psychiatry 2013;28:547–8.
17. Abdel-Aziz K, Larner AJ. Six-item Cognitive
Impairment Test (6CIT): pragmatic diagnostic
accuracy study for dementia and MCI. I nt
Psychogeriatr 2015;27:in press.
18. Larner AJ. DemTect: 1-year experience of a
neuropsychological screening test for dementia. Age Ageing 2007;36:326–7.
19. Larner AJ. Screening utility of the Montreal
Cognitive Assessment (MoCA): in place of – or
as well as – the MMSE? Int Psychogeriatr
2012;24:391–6.
20. Hancock P, Larner AJ. Test Your Memory
(TYM) test: diagnostic utility in a memory clinic
population. Int J Geriatr Psychiatry
2011;26:976–80.
21. Larner AJ. Dementia in clinical practice: a
neurological perspective. Pragmatic studies in
the Cognitive Function Clinic , 2nd edn.
London: Springer, 2014.
22. Norman G, Monteiro S, Salama S. Sample
size calculations: should the emperor’s clothes
be off the peg or made to measure? BMJ
2012;345:e5728.
23. Larner AJ. Impact of the National
Dementia Strategy in a neurology-led memory clinic: 5-year data. Clin Med 2014;14:216.
Visit us online
Progress in Neurology and Psychiatry’s dedicated
website www.progressnp.com features news,
events diary, CMHP reviews, journal supplements
as well as current and archived issues of Progress.
www.progressnp.com
24
Progress in Neurology and Psychiatry January/February 2015
www.progressnp.com