Factors Affecting Athletes'
Performance on Baseline and Post
Concussion Cognitive Testing
Philip Schatz, PhD
Saint Joseph’s University
Tracey Covassin, PhD, ATC
Michigan State University
Anthony Kontos PhD
Humboldt State University
Elizabeth Larson BS
Humboldt State University
RJ Elbin MA
Michigan State University
National Academy Of Neuropsychology - November 2009
Overview
•
•
•
•
•
Methodological Issues – Philip Schatz
History of Concussion – RJ Elbin
Learning Disabilities – Beth Larson
Depression and Anxiety – Anthony Kontos
Gender and Age – Tracey Covassin
Methodological Issues
(keeping the “psychology” in sports concussion assessment)
-Scheduling of baselines
-Individual vs. group effects
-Environmental distraction
-Feedback from test takers
Concussion Testing
(It all started with) Barth’s VA football study:
• within-subjects comparison
• changes from baseline performance
• between-groups comparison
• matched cohorts
• Model widely adopted (colleges, NFL, NHL)
Computer-based Assessment
Computer-based tests introduced (circa 2000)
ImPACT, CogSport, CRI, ANAM
Advantages:
• ease of administration
• automated scoring
• decreased practice effects
• increased test-retest reliability
Collie,et al. (2003). Br J Sports Med, 37, 556-559
Baseline Testing
Baseline pre-injury testing, serial follow-up is a
necessary component
CISG: Vienna (2001), Prague (2004), Zurich (2009)
AAN: (1997: Neurology, 48, 581-585):
…recommend neuropsych measures to detect
impairment associated w/concussion
NATA: (2004: Guskewicz et al., Neurosurg, 55, 891-895)
neuropsych data assists ATCs in
evaluating recovery
Scheduling of Baselines
Question: How often to update baselines?
After a concussion, start of academic year
Collie: (2001: BJSM, 35, 297-302)
Valovich: (2003: JAT, 38, 51-56)
More frequent for “youth” athletes
-compared to “older” athletes
-period of rapid cognitive maturation (8-15)
(Buzzini, 2006: Curr Opin Pediatr;
Maruff 2004: BJSM; McCrory: 2004: BJSM)
Scheduling of Baselines
How often to update?
Annually, independent of age, until long-term
test-retest reliability data are available
Randolph, McCrea, Barr: (2005: JAT, 40, 139-152)
Test-Retest Data: (ImPACT)
7-day (Iverson, Lovell, Collins: 2003: Clin Neuropsy)
45/50-day (Broglio, et al.: 2001: JAT)
4-month (Miller, et al.: 2007: AJSM)
Scheduling of Baselines
Schatz: 2009: AJSM
2-year test-retest reliability of ImPACT
117 Varsity Athletes, base-lined 2 years apart
-22 concussed between baselines
-7 excluded due to high Impulse Control
Resultant sample of 95 (no football players)
Scheduling of Baselines
Schatz: 2009: AJSM
Mean age Time 1: 18.8, Time 2: 20.8
54% male, 46% female
Mean difference of 1.9 years between
baselines
Scheduling of Baselines
Comparing Time 1 -> Time 2
-Pearson’s r, t-test
-Intraclass Correlation Coefficient
ICC can distinguish those sets of scores that are merely ranked in the same order from test to
retest from those that are not only ranked in the same order but are in low, moderate, or
complete agreement; also yields UER (consistency across baselines; measurement
instrument as source of error?)
-Reliable Change Indices (RCI)
Is change between repeated assessments reliable and meaningful; an estimate of the
probability that a score was not obtained as a result of measurement error
-Regression-based Measures
Scores from the first assessment are placed into a regression analysis, using the
score at Time 2 as the DV; the resulting equation provides an adjustment for the
effect of initial performance level, as well as controlling for any regression to the mean
Scheduling of Baselines
Time 1 -> Time 2
-Pearson’s r – little help (.27-.60)
-t-test – no significant differences
-Intraclass Correlation Coefficient
(ICC as well as UER)
Processing Speed (.75);
Reaction Time (.68),
Visual Memory (.66),
Verbal Memory (.47),
Symptom scores (.44).
Scheduling of Baselines
T1 -> T2: % athletes showing signif. Change
-Reliable Change Indices (RCI)
(“Traditional” /“Chelune” method – adjust for practice effects)
Verbal Memory
Visual Memory
Process Speed
Reaction Time
Symptoms
80% CI
8% / 12%
13% / 10%
7% / 10%
7% / 9%
8% / 8%
95% CI
5% / 5%
9% / 7%
4% / 2%
5% / 5%
6% / 6%
Nearly identical numbers of athletes showed improvement &
declines in performance from Time 1->Time 2
Scheduling of Baselines
T1 -> T2: % athletes within expect. limits
-Regression-based Measures (RMB)
Verbal Memory
Visual Memory
Process Speed
Reaction Time
Symptoms
80% CI
94.7%
94.7%
94.7%
96.8%
89.5%
95% CI
97.9%
97.9%
96.8%
97.9%
94.7%
Nearly identical numbers of athletes showed improvement &
declines in performance from Time 1->Time 2
Scheduling of Baselines: Summary
• 2-year test-retest data are stronger than
published data at 7-50 days
• 2-year test-retest data show stability
• Using RCI, only a small % of athletes show
significant change – Verb Mem, Symptoms
• Using RBM, no athletes showed signif change
• Stretching time between baselines from 1 to 2
years may have little effect on concussion
management in collegiate athletes
Individual vs. group effects
Benefits of computer-based testing:
• cost-benefit gains
• increased security
• savings in time, money
• rapid testing of an entire team
(French & Beaumont: 1987: Br. J Clin Psy;
Barak: 1999: Appl Prev Psych: Collie: 2001 BJSM; Collie &
Maruff: 2003: BJSM)
Individual vs. group effects
Role of distraction in group setting?
(Ligon: 1942: Educ and Psychol Measurement)
• Common errors in group tests:
• misunderstood instructions, carelessness,
• low motivation, dishonesty of subjects
• mental confusion due to too great excitement,
• size of group
• group inter-distraction.
To reduce errors, “training of group testers is
needed
Individual vs. group effects
Role of distraction in group setting?
• SATs, GREs, licensing exams=precedent?
• Ext. distraction may affect performance (McCrory 2005)
• Test should be administered in conditions without
noises or disruptions (Pierce & Pierce: 1963: J Consult Psychol)
• Wechsler (1955): Conducted in settings that are
“quiet and disciplined”
• Studies show no difference between individual
and group admin (PPVT: Morris: 1960:J Educ Psych; WAIS: Eme &
Walker: 1969, J Clin Psych; Bender-Gestalt:Brannigan:1995: Percep Mot
Skills; WAIS/IPET: Neva & Salo: 2003)
Individual vs. group effects
Role of distraction in group setting?
Schatz, Neidzwski, Moser, Karpf (in revision):
Evaluated subjective feedback from athletes taking
ImPACT
• Environmental: noise (talking, laughing, cell phones ringing), distractions
(keyboards clicking), discomfort (room temperature)
• Also recorded Computer-based and Instruction-based (problems) feedback
• 538 out of 1818 HS student provided subjective feedback (32.4%)
• 161 reported environmental distraction (9.7%)
Individual vs. group effects
Role of distraction in group setting?
Schatz, Neidzwski, Moser, Karpf (in revision):
Evaluated subjective feedback from athletes taking
ImPACT
• Students reporting environmental feedback endorsed signif. more
Symptoms (8.8 vs. 5.9; p<.001; d=.21)
• physical symptoms (headache, fatigue, difficulty falling asleep,
drowsiness, sleeping less),
• cognitive symptoms (feeling slowed down, mentally foggy, difficulty
concentrating, difficulty remembering)
• emotional symptoms (feeling irritable, sadness, nervousness,
feeling more emotional).
Individual vs. group effects
What to do about distraction in group setting?
• test individually
• seat athletes every-other computer (Broglio: 2007)
• make sure someone in room has “authority”
• Team captain, Coach, Head ATC
• explain the process, educate athletes
• recourse for inappropriate behavior
•Team is re-tested, not cleared to play until testing is complete
Individual vs. group effects
Role of distraction in group setting?
Need counterbalanced basic methodological design
• Group->Group
• Group->Individual
• Individual->Group
• Individual->Individual
Subjective Feedback
Benefits of computer-based testing:
“the freedom to increasingly focus on treatment or
qualitative assessment gained by the automation
of data collection…”
American Psychological Association. (1986). Guidelines for Computerbased Tests and Interpretations. Washington, DC: APA
Subjective Feedback
If athletes are tested in groups, where is the
“testing of the limits” during baseline exams?
• Are (neuro)psychologists even present?
• Do (neuro)psychologists need to be present?
• What qualitative data can be obtained?
Subjective Feedback
Who completes baseline assessments?
• team physician or web-based (CISG/Vienna, 2002)
• Neuropsychologists in best position to interpret
test data/batteries (CISG/Zurich, 2008)
• RTP decision is medical – results assist RTP
decisions (CISG/Zurich, 2008)
• Non-neuropsychologist personnel can administer
tests (Moser, et al/NAN PoP, 2007)
• Interpretation of results restricted to neuropsychologists (Moser, et al/NAN PoP, 2007)
Subjective Feedback
What qualitative data can be obtained?
• Subjective feedback from test takers (ImPACT)
• Environmental
• Computer
• Instructions
• Completed at the end of the examination
Subjective Feedback
Role of subjective feedback
Schatz, Neidzwski, Moser, Karpf (in revision):
Evaluated subjective feedback from athletes taking
ImPACT
• Computer-based feedback: mechanical problems with the mouse,
other issues (problems with the monitor, pop-up messages).
• Problems with test instructions: difficulty understanding specific
modules of the test, general issues (confusion, failure to read instructions,
not devoting enough time).
Subjective Feedback
Role of subjective feedback?
Schatz, Neidzwski, Moser, Karpf (in revision):
• 538 out of 1818 HS student provided subjective
feedback (32.4%)
• 199 reported computer problems (12%)
• 297 reported problems with instructions (17.9%)
Subjective Feedback
Students reporting computer problems had signif.
Faster reaction time (53 vs. 54; p<.004; d=.14)
(statistically but perhaps not clinically significant)
Baddaley and colleagues (1993): a form of attentional filtering
–allows subjects to concentrate on relevant items and exclude
irrelevant stimuli.
Relatively infrequent, arousing, and isolated problems may
actually serve to increase arousal levels, thus optimizing
performance (e.g., Yerkes-Dodson, 1908).
Subjective Feedback
Students reporting problems with instructions endorsed signif.
more symptoms (7.8 vs. 5.9; p<.004; d=.19)
• physical symptoms (balance problems, fatigue,
drowsiness, sleeping less),
• cognitive symptoms (feeling slowed down, difficulty
concentrating)
• emotional symptoms (feeling irritable, feeling more
emotional).
Related to history of concussion:
None: 16.7%
Previous: 23.6%
[X2(1)=7.9; p=.005]
Subjective Feedback
• Increased symptoms may reflect test-taking anxiety,
exacerbation of existing “issues” from testing
• Incorporating a question and answer session
following testing may enhance qualitative data.
• Findings reflect the importance of understanding
concussion-based symptoms and their relationship to
baseline symptoms and an athlete’s personal style
FACTORS THAT AFFECT
NEUROCOGNITIVE TESTING:
HISTORY OF CONCUSSION
Overview of Research: History of
Concussion(s)
• Numerous studies have investigated the
influence of concussion history on the risk and
recovery from future concussion
– Three questions have emerged…
• Are athletes with a Hx of concussion at a greater risk for
future concussion?
• Do athletes with a Hx of concussion take longer to
recovery from subsequent concussion?
• Are their any long-term effects associated with a Hx of
multiple concussions?
Are Athletes With a History of Concussion
at a Greater Risk for Future Concussion?
• Athletes with a history of concussion are at a four to six times
greater risk for incident concussion (Zemper, 2003; Wilberger, 1993; Gerberich
et al. 1983)
• Concussed H.S. and collegiate athletes are 3x times more likely to
sustain another concussion in the same season (Guskiewicz et al. 2000)
• Dose Response for # of previous concussions and risk for incident
concussion (Guskiewicz et al. 2003)
– 3+ (3.4x)
– 2 (2.8x)
– 1 (1.5x)
• Hx of 3+ and severity of future concussion (Collins et al. 2002)
– 9.3 times more likely to demonstrate 3-4 abnormal markers
– On-field LOC (6.7x), confusion (4.1x), anterograde amnesia (3.8x)
Do Athletes with a History of Concussion Take
Longer to Recover from Subsequent
Concussion?
• No NC differences between 1st and 2nd concussion
(Macciocchi et al. 2001)
• Prolonged recovery after two concussions?
– 2+ Hx of concussion
• Lingering memory impairment and slower reaction time up
to 5 days post-concussion (Covassin et al. 2008)
– 3+ Hx of concussion
• Significant memory impairment at 2 days post-concussion
(Iverson et al. 2004)
• Prolonged symptom resolution associated with 3+ Hx
(Guskiewicz et al. 2003)
Are There Long-Term Effects Associated
with a History of Multiple Concussions?
• Assessing NC function in athletes with prior
history of concussion
– Deficits in executive function and processing speed
observed in Hx of 2+ compared to 1 or zero (Collins et al.
1999)
– No differences on NC performance between H.S.
athletes w/Hx 2+ and recently concussed (w/in 2
weeks) (Moser et al. 2005)
• Support found in studies using paper-andpencil NC tests
Are There Long-Term Effects Associated
with a History of Multiple Concussions?
• No support for computerized NC tests
– Studies using ImPACT and CRI found no differences
between groups of athletes with and without hx of
concussions (0, 1, 2, 3, 4) (Collie et al. 2006; Iverson et al. 2006)
• Are paper-and-pencil better at detecting long-term
impairment than computerized tests?
– No differences between Hx vs. No Hx on computerized and
formal NC tests (Bruce & Echemendia, 2009)
• If long-term effects do exist, then NC testing may not be
sensitive enough to detect residual impairment (Broglio et al. 2006)
• Future study
– Neuro-imaging
Linking Research to Practice
• Current literature suggests that athletes with a
Hx of concussion are at a greater risk for injury
and may take longer to recover from
subsequent concussion
• Obtaining detailed information on previous
concussion history is important!
– Identify athletes who are at a higher risk for
incident concussion
– Educate them about concussion
What is the best practice to obtain this
information?
• Do we take the athletes word for it?
– Limitations of self-report/recall can be misleading
– Concussions are under-reported at all athletic levels (Kaut et
al. 2003; LaBotz et al. 2005)
• Pre-participation screening (PPS) forms?
– Lacks specific questions regarding concussion hx
(McCrory, 2004)
– Terminology issue (e.g., “ding” and/or “bell-ringer”)
• 8.5 % said yes when asked if they have had a
“concussion”
• 25% said yes when using lay terminology (ValovichMcLeod, 2008)
What is the best practice to obtain this
information?
• Concussion Symptom Scale (CSS)
– Higher potential for estimating concussion Hx than PPS
(Valovich-McLeod, 2008; LaBotz et al. 2005)
– The endorsement of one Sx does not imply concussion
• May over-report concussion Hx
– Interpretation is key!
– Headache and blacking out strongly associated with
concussion Hx on PPS
• Other sources and questions?
– Coaches and teammates are unreliable (McCrory, 2004)
Clinician Recommendations:
Detailed Concussion History
• Include…
– Specific questions about previous S/S of
concussion NOT just the perceived number
– Previous head, face, neck injuries
• Conconmitant concussion?
– Symptom severity vs frequency of impacts
• Indicate vulnerability?
– Consider “lay” terminology
• E.g., bell-ringer
– Use concussion symptom scale forms
• Inquire for both sport and non-sport injuries
Gender and Age Differences
Sport-related Concussion Research:
Exploring Gender
• The majority of concussion research in the past
has been conducted on males
• Female sport participation has increased at both
the collegiate and high school level
• Are there gender differences in:
– Risk
– Concussion symptoms
– Post-concussion outcomes
Gender Differences and Risk of
Concussion in Collegiate Athletes
• Females have a greater incidence of concussion than males
(Covassin et al. 2003; Gessel et al. 2007;Hootman et al. 2007; Colvin et al. 2009)
• Concussions represent 5 to 21% of all reported game injuries for NCAA
female athletes and 2 to 9% of all reported game injuries for NCAA male
athletes (Hootman et al. 2007)
Women’s ice hockey 21.6%
Men’s ice hockey
9.0%
Women’s lacrosse
Men’s lacrosse
9.8%
8.6%
Women’s soccer
Men’s Soccer
8.6%
5.8
Football
6.8%
Gender Differences and Risk of
Concussion in High School Athletes
• Rechel et. al. (2008) reported a greater
proportion of competition injuries 12.0%
compared to practice injuries 5.9%
Girls basketball
Girl’s soccer
Boy’s soccer
19.0%
18.8%
15.6%
Gender Differences and Concussion
Symptoms
• Female concussed athletes reported more
concussion symptoms than male concussed
athletes (Broshek et al. 2005)
–
–
–
–
Poor concentration
Increased fatigue
Lightheadedness
“Seeing stars”
Gender Differences and Concussion
Symptoms
• Recent study on high school and collegiate
soccer players found concussed female
athletes demonstrated more symptoms than
concussed male athletes (Colvin et al. 2009)
– Total concussion symptoms
– Headache
Gender Differences and Concussion
Outcome
• Female concussed athletes have
demonstrated longer recovery patterns than
concussed male athletes
– visual memory (Covassin et al. 2007)
– slower reaction time (Colvin et al. 2009; Broshek et al. 2005)
What may Account these Gender
Differences?
• Females have decreased head mass and
neck strength compared to males (Tierney et al.
2005;Manshell et al. 2005; Broglio et al. 2003)
• Females may report their concussion
symptoms with a greater frequency than
males
Age Difference and Concussion
• High school athletes have been found to
demonstrate a slower neurocognitive and
symptom recovery following concussion than
collegiate athletes (Field et al., 2003; Lovell et al.,
2003; McCrea et al., 2003; Sim et al. 2008)
Age Differences and Recovery from
Concussion
• High school athletes exhibited memory
impairment up to 14 days post-concussion
(Field et al. 2003; McClincy et al. 2006)
• Collegiate athletes demonstrate memory
impairment up to 5 days post-concussion
(Macciocchi et al. 1996;Field et al. 2003)
Age Differences and Concussion
• Immature or developing brain places high
school athletes at risk for more adverse
outcomes following concussion
• Larger head-to-body ratio
• Immature musculoskeletal systems (i.e.,
weaker and smaller)
• Larger subarachnoid space in the cranium
(e.g., allowing more room for brain
movement)
Clinical Implications
• Concussed high school athletes should be
removed from practice or competition (Guskiewicz
et al. 2004)
– Second impact syndrome