Inconsistencies across Panel Waves
and the Effectiveness of Dependent Interviewing
Annette Jäckle
UK Longitudinal Studies Centre
Institute for Social and Economic Research
University of Essex
[email protected]
http://www.iser.essex.ac.uk
Continuous Histories from Panel Interviews
• Continuous histories from
– Current situation at interview, plus
– Retrospective questions about events since previous interview
• BHPS examples: marital status, residential moves, labour market
activity, unearned income, full-time education
• Longitudinal inconsistencies:
– Causes are likely to differ for different types of items
– Some causes of inconsistencies are affected by length of time
between interviews
– Remedies (e.g. Dependent Interviewing) need to be adapted to
causes of problems
Data: ISMIE survey
‘Improving Survey Measurement of Income and Employment’
• Experimental survey, spring 2003
– Former UK-ECHP sample (1997-2001 with BHPS)
– Questionnaires based on BHPS
• Dependent Interviewing experiment:
– Random allocation to independent, proactive DI, reactive DI
• Validation study:
– Link to administrative records on benefits and tax credits
• Sample: 1033 respondents, 799 consented, 592 linked
• http://www.iser.essex.ac.uk/ulsc/methods/research/ismie/
Inconsistencies in Employment Transitions
Transition rate from Employment (%)
16
‘Seam Effect’
14
12
10
Reference period 1
Reference period 2
8
6
4
2
0
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
9
10 11
Month
Source: Jäckle & Lynn (in press) Journal of Official Statistics
Inconsistencies in Unearned Income Sources
18
16
Transition rate (%)
14
12
10
8
6
4
2
0
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
9
10
Month
Source: ‘Improving Survey Measurement of Income and Employment’
11
Outline
• Longitudinal inconsistencies
– Focus on ‘seam effects’
• Causes I:
– Underestimation of within wave change
– Over-estimation of seam change
• Causes II:
– Event characteristics and respondent situation
• Implications for:
– Length of interval between interviews
– Dependent interviewing (and other remedies?)
• Conclusions
Causes of Inconsistencies I:
Seam vs. Non-Seam Transitions
Interview 1
Interview 2
Interview 3
1. Spurious change
2. Under-reported
within wave change
3. Misplaced to seam:
under-reporting
4. Misplaced to seam:
misdating
Misclassification
Spurious seam
change
Under-reported
Over-reported
Under-reporting
Misplaced
seam change
Concentration of
seam transitions
Misdating
Under-reported within
wave change
Example: Benefit Income
• Please look at this card and tell me if, since <LAST
INTERVIEW>, you have received any of the types of
income or payments shown?
• And for which months since <LAST INTERVIEW> have
you received…?
Causes of Inconsistencies I:
Seam vs. Non-Seam Transitions (ctd.)
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Start 1
(N=200)
End 1
Start 2
(N=81)
End 2
Start 3
(N=464)
End 3
Number of waves spanned, by start/end date
no error
spurious seam change
under-reported within wave change
misplaced to seam: under-reported
misplaced to seam: misdated
under-reported seam change
misplaced seam to seam
Source: ‘Improving Survey Measurement of Income and Employment’
Causes of Inconsistencies I:
Seam vs. Non-Seam Transitions (ctd.)
• Items differ in likelihood of misclassification, underreporting and misdating
• Inconsistencies likely to be larger for event types
which are more likely to be misclassified, underreported or misdated
• Expect differences between items in
– Extent of inconsistencies
– Composition of inconsistencies (spurious seam change,
misplaced seam change, under-reported within wave change)
• Next: think about determinants of errors
Causes of Inconsistencies II: Event Characteristics
• Example: legal marital status histories in BHPS
– “What is your current legal marital status, are you Married,
Separated, Divorced, Widowed or have never been married?”
– “Has your marital status changed in the last year, that is since
Sept 1st <previous year>?”
Interview1
Date of
change?
Current marital
status1
Interview2
Date of
change?
Current marital
status2
Causes of Inconsistencies II: Event Characteristics (ctd.)
- Memorability
- Time since event
- Intervening events
- Regularity
-Sensitivity
- Mutually
exclusive categories
- Redefinition
- Memorability
- R ability
- R motivation
- Task difficulty
Misclassification
Under-reporting
Misdating
Respondent
Interviewer
Forgetting
Shortcutting
Deliberate
Forgetting
Shortcutting
Telescoping
Constant wave response
Spurious
seam change
Misplaced
seam change
Concentration of
seam transitions
Under-reported within
wave change
-Fuzzy dates
Example: Marital Histories
Event
Status of origin
Married
Separated
Divorced
Widowed
Never married
–
Negative
(fuzzy date)
Negative
(fuzzy date)
Negative
(clear date)
Impossible
(redefinition)
Separated
Same spouse
(fuzzy date)
–
Negative
(fuzzy date)
?
(clear date)
Impossible
(redefinition)
Divorced
Positive
(clear date)
Impossible
(2+ events?)
–
?
(clear date)
Impossible
(redefinition)
Widowed
Positive
(clear date)
Impossible
Impossible
(2+ events?) (2+ events?)
–
Impossible
(redefinition)
Never married
Positive
(clear date)
Impossible
Impossible
(2+ events?) (2+ events?)
Impossible
(2+ events?)
–
Married
• BHPS 2001-2002: % of transitions in 2002 reference period at 2001 seam (N)
Event
Status of origin
Married
Separated
Divorced
–
42.2
(83)
42.9
(28)
39.2
(74)
100.0
(9)
Separated
81.0
(21)
–
55.2
(96)
100.0
(3)
100.0
(2)
Divorced
26.9
(67)
100.0
(10)
–
93.8
(16)
100.0
(15)
Widowed
28.6
(7)
100.0
(1)
100.0
(12)
–
100.0
(4)
Never married
27.1
(155)
100.0
(5)
100.0
(14)
100.0
(4)
–
Married
Source: BHPS 2001/2002
Widowed Never married
Causes of Inconsistencies II: Event Characteristics (ctd.)
• Characteristics of events determine likelihood of
different types of errors
• Event characteristics likely to interact with respondent
characteristics
• Another example…
Causes of Inconsistencies III.
Event Characteristics and Respondent Situation
- Memorability
- Time since event
- Intervening events
- Regularity
-Sensitivity
- Mutually
exclusive categories
- Redefinition
- Memorability
- Coder experience
- R ability
- R motivation
- Task difficulty
Misclassification
Under-reporting
Misdating
Respondent
Interviewer
Coder
Editor
Forgetting
Shortcutting
Deliberate
Forgetting
Shortcutting
Telescoping
- Complexity of
coding frame
Constant wave response
Spurious
seam change
Misplaced
seam change
Concentration of
seam transitions
Under-reported within
wave change
-Fuzzy dates
Characteristics of Income Spells and Respondent
Situation Associated with Errors
• Misclassification
– Name changes
– Packaged sources
• Under-reporting
Forgetting
–
–
–
–
–
–
Shortcutting source
– Lower position on showcard
– Difficult task (see forgetting)
Deliberate
– Stigma
• Misdating (constant ‘yes’)
Receipt ended
– Current receipt
Short spell
– Start close to start of
Small proportion of total income
reference period
Reoccurring
Receipt of multiple sources
Packaged payments
Affected by length of recall period?
• Misclassification
– Name changes
– Packaged sources
• Under-reporting
Forgetting
–
–
–
–
–
–
Shortcutting source
– Lower position on showcard
– Difficult task (see forgetting)
Deliberate
– Stigma
• Misdating (constant ‘yes’)
Receipt ended
– Current receipt (-)
Short spell
– Start close to start of
Small proportion of total income
reference period
Reoccurring
Receipt of multiple sources
Packaged payments
Implications for Optimal Timing of Waves
1. Some of the problems increase with length of
interval between interviews
– How important?
– Would need to assess relative frequency of different
sources of error
2. Optimal interval might depend on length of
phenomena to be measured
– Length of interval relative to length of spells determines
nature (and magnitude?) of errors
1
1
Spell Durations: Benefit Receipt
Income related
0
0
Proportion ongoing spells
.2
.4
.6
.8
Proportion ongoing spells
.2
.4
.6
.8
Disability related
12
24
36
48
60
Month
Independent
12
24
36
48
Month
Records
Independent
Source: ISMIE
Records
60
Dependent Interviewing
• Use prior information for
– Routing and formulation of questions (proactive DI):
According to our records, when we last interviewed you, on
<INTDATE>, you were receiving <SOURCE>. For which months
since <INTDATE>…?
– Edit checks for apparent change (reactive DI):
Can I just check, according to our records you have in the past
received <SOURCE>. Have you received <SOURCE> at any time
since <INTDATE>? For which months since <INTDATE>…?
• Current applications
– Focus on reducing under-reporting, inconsistent classification
– Little attention to dating problems (asymmetric treatment…)
– Potential for dealing with other sources of errors?
Dependent Interviewing - Examples
• Reduces seam effects for labour market activity
histories, but not for income sources (why?)
– Reduces under-reporting of income sources
⇒Reduces constant wave ‘no’, but no effect on constant ‘yes’
• Can worsen estimates of spell durations
• Need to adapt cues to causes of errors
Transition rate from Employment (%)
Seam Effects in Activity Histories
16
14
12
10
8
6
4
2
0
-41 -32 -23
-1
4
05
16
27
38
94
5 11
6 7
11
10
12 8
13 914 10
15 16
Month
Independent Interviewing
Proactive DI
Source: Jäckle & Lynn (in press) Journal of Official Statistics
Seam Effects in Unearned Income Sources
18
Transition rate (%)
16
14
12
10
8
6
4
2
0
-41
-32
-23
-1
4
05
16
27
3
8
5 11
6 12
7 13
8 14
9 10
11
94 10
15 16
Month
Independent Interviewing
Reactive DI
Source: ISMIE
Proactive DI
Under-Reporting of Income Sources
Housing Benefit
Income Support
Family Credit
Child Benefit
Incapacity Benefit
0
Proactive DI
5
Reactive DI
10
15
20
25
30
Independent Interviewing
Source: Lynn et al. (2004) ISER Working Paper 2004-28
35
%
Dating of Transitions
• Expected rates: (1) 3/49=6.12% (2) 1/31=3.23%
Start at seam (%)
Disability
Income
(1) Independent
62.99
49.25
30.56
25.79
(2) Dependent
End at seam (%)
Disability
Income
33.07
26.88
6.94
7.14
1
1
Spell Durations: Benefit Receipt
Income related
0
0
Proportion ongoing spells
.2
.4
.6
.8
Proportion ongoing spells
.2
.4
.6
.8
Disability related
12
24
36
48
60
12
24
Month
36
48
60
Month
Records
Independent
Records
0
0
Proportion ongoing spells
.2
.4
.6
.8
Proportion ongoing spells
.2
.4
.6
.8
1
1
Independent
12
24
36
48
60
Dependent
12
24
36
48
Month
Month
Records
Source: ISMIE
Dependent
Records
60
To Reduce Inconsistencies across Panel Waves:
Ideally…
• Identify likely types of errors
– misclassification, under-rep, misdating, others?
– (specific to estimates to be derived from the histories)
• Identify likely predictors of errors
– memorability, sensitivity, fuzziness of dates, not mutually
exclusive categories, etc.
– including interaction between event characteristics and
respondent characteristics (and question format)
• Evaluate likely impact (frequency) of different error
sources, to decide where to focus efforts
– may depend on length of interval
• Adapt cues…