Collecting and Reporting
Follow-up Data
Or What’s Behind Table 5?
American Institutes for Research
February 2005
Table 5 and Follow-up
Table 5: Outcomes




2
Earned a High School Diploma or GED
Entered post-secondary School
Entered Employment
Retained Employment
Table 5 and Follow-up
Table 5: Outcomes

3
States submit six (6) data points for each
of the four outcomes:
1
2
3
4
5
6
Core Followup Outcome
Measures
#
Participants
With Main or
Secondary
Goal
(A)
(B)
#
Participants
Included in
Survey
# Participants
Responding
to Survey or
Used for Data
Matching
Response
Rate or %
Available
for Match
#
Participants
Achieving
Outcome
%
Achieving
Outcome
(C)
(D)
(E)
(F)
(G)
Table 5 and Follow-up
Table 5: Column (B)
Core Followup Outcome
Measures
#
Participants
With Main or
Secondary
Goal
(A)
(B)
#
Participants
Included in
Survey
# Participants
Responding
to Survey or
Used for Data
Matching
Response
Rate or %
Available
for Match
#
Participants
Achieving
Outcome
%
Achieving
Outcome
(C)
(D)
(E)
(F)
(G)
All participants who had the outcome as a goal
and who exited during the program year.
4
Table 5 and Follow-up
Table 5: Column (C)
Core Followup Outcome
Measures
#
Participants
With Main or
Secondary
Goal
(A)
(B)
#
Participant
s Included
in Survey
# Participants
Responding
to Survey or
Used for Data
Matching
Response
Rate or %
Available
for Match
#
Participants
Achieving
Outcome
%
Achieving
Outcome
(C)
(D)
(E)
(F)
(G)
The number of participants who were surveyed to
assess whether they had achieved the outcome.
Note: C should be equal to B unless sampling
was used, in which case C would be less than B. For
data matching C = [BLANK]
5
Table 5 and Follow-up
Table 5: Column (D)
Core Followup Outcome
Measures
#
Participants
With Main or
Secondary
Goal
(A)
(B)
#
Participants
Included in
Survey
# Participants
Responding
to Survey or
Used for Data
Matching
Response
Rate or %
Available
for Match
#
Participants
Achieving
Outcome
%
Achieving
Outcome
(C)
(D)
(E)
(F)
(G)
Of those surveyed or data matched, the number who
responded or were used for data match.
Note: D should be equal to or less than B and C each.
6
Table 5 and Follow-up
Table 5: Column (E)
Core Followup Outcome
Measures
#
Participants
With Main or
Secondary
Goal
(A)
(B)
#
Participants
Included in
Survey
# Participants
Responding
to Survey or
Used for Data
Matching
Response
Rate or %
Available
for Match
#
Participants
Achieving
Outcome
%
Achieving
Outcome
(C)
(D)
(E)
(F)
(G)
Percent of respondents or participants successfully
matched; column D divided by column B.
E = D/B * 100
Should be > 50%, but likely not 100%
7
Table 5 and Follow-up
Table 5: Column (F)
Core Followup Outcome
Measures
#
Participants
With Main or
Secondary
Goal
(A)
(B)
#
Participants
Included in
Survey
# Participants
Responding
to Survey or
Used for Data
Matching
Response
Rate or %
Available
for Match
#
Participants
Achieving
Outcome
%
Achieving
Outcome
(C)
(D)
(E)
(F)
(G)
Number of participants achieving outcome.
Note: F must be equal to or less than D.
8
Table 5 and Follow-up
Table 5: Column (G)
Core Followup Outcome
Measures
#
Participants
With Main or
Secondary
Goal
(A)
(B)
#
Participants
Included in
Survey
# Participants
Responding
to Survey or
Used for Data
Matching
Response
Rate or %
Available
for Match
#
Participants
Achieving
Outcome
%
Achieving
Outcome
(C)
(D)
(E)
(F)
(G)
Percent achieving outcome.
G= F/D * 100
9
Table 5 and Follow-up
Table 5: Example
# Participants
#
Responding
Participants
#
to Survey or Response
#
Core Follow- With Main or Participants
Used for
Rate or % Participants
%
up Outcome Secondary Included in
Data
Available Achieving Achieving
Measures
Goal
Survey
Matching
for Match
Outcome Outcome
10
(A)
(B)
(C)
(D)
(E)
(F)
(G)
Entered
Employment
1,000
1,000
800
80%
200
25%
Table 5 and Follow-up
Real Data – PY 2002–03 Examples
State
# Participants
#
Responding
Participants
#
to Survey or Response
#
With Main or Participants
Used for
Rate or % Participants
%
Secondary Included in
Data
Available Achieving Achieving
Goal
Survey
Matching
for Match
Outcome Outcome
(A)
(B)
(C)
(D)
(E)
(F)
(G)
State 1
1,056
0
972
0%
356
37%
What type of
follow-up was
used?
11
Table 5 and Follow-up
Real Data – PY 2002–03 Examples
State
# Participants
#
Responding
Participants
#
to Survey or Response
#
With Main or Participants
Used for
Rate or % Participants
%
Secondary Included in
Data
Available Achieving Achieving
Goal
Survey
Matching
for Match
Outcome Outcome
(A)
(B)
(C)
(D)
(E)
(F)
(G)
State 1
1,056
0
972
0%
356
37%
Why is this number
less than column B?
12
Table 5 and Follow-up
Real Data – PY 2002–03 Examples
State
# Participants
#
Responding
Participants
#
to Survey or Response
#
With Main or Participants
Used for
Rate or % Participants
%
Secondary Included in
Data
Available Achieving Achieving
Goal
Survey
Matching
for Match
Outcome Outcome
(A)
(B)
(C)
(D)
(E)
(F)
(G)
State 1
1,056
0
972
0%
356
37%
What data should
be reported here?
13
Table 5 and Follow-up
Real Data – PY 2002–03 Examples
State
# Participants
#
Responding
Participants
#
to Survey or Response
#
With Main or Participants
Used for
Rate or % Participants
%
Secondary Included in
Data
Available Achieving Achieving
Goal
Survey
Matching
for Match
Outcome Outcome
(A)
(B)
(C)
(D)
(E)
(F)
(G)
State 1
1,056
0
972
0%
356
37%
Is this percent
computed
correctly?
14
Table 5 and Follow-up
Exercise Questions
For states 2–6:




15
What method of follow-up did each
state use?
Which states, if any, filled in the table
correctly? Which did not?
Do you see any errors in the table?
Do you have confidence in these data?
Why or why not?
Table 5 and Follow-up
Exercise Example
State
16
# Participants
#
Responding
Participants
#
to Survey or Response
#
With Main or Participants
Used for
Rate or % Participants
%
Secondary Included in
Data
Available Achieving Achieving
Goal
Survey
Matching
for Match
Outcome Outcome
(A)
(B)
(C)
(D)
(E)
(F)
(G)
State 1
1,056
0
972
0%
356
37%
State 2
3,325
NOT
AVAIL.
0%
1,062
32%
State 3
8,644
0
0
0%
4,067
47%
State 4
31,363
0
31,363
100%
21,779
69%
State 5
9,633
3,853
1,156
30%
543
13%
State 6
224
224
164
73%
96
59%
Table 5 and Follow-up
Top Errors for Table 5

Missing Data



Unreasonably high response rates



17
Put “0” for # surveyed in Column (C)
Put “0” for # responded or matched in Column
(D)
Many 100% Unreasonably high data matching
Many 100% (DOL goal for matching = 90%)
Unreasonable matching of # who had goal to # who
responded
Column (D) = Column (B)
Table 5 and Follow-up
Top Errors for Table 5 (cont’d.)

Incorrectly computed percentages:

Response rates calculated as D/C instead of D/B
(B and C should be the same unless using random sampling, however)

18
Number achieving outcome computed as F/B
instead of F/D
Table 5 and Follow-up
19
Table 5 and Follow-up
2002–03
Data Collection Methods
Data Matching Methodology

Produces better quality data:







20
More student coverage
Better data validity
Better response rate
Matches Title I—better for OMB common measures
28 states use it for employment measures—we
need more states!
What are the reasons for not data matching?
How can we promote use of this methodology?
Table 5 and Follow-up
What’s Behind the Data?



State responsibility to ensure accuracy and
quality of follow-up measures
Requires monitoring of local program
procedures
Key steps:




21
Identify students—goal setting and tracking
Uniform data collection
Training
Reporting
Table 5 and Follow-up
Percent of Students with Goals
Measure
# with
Goal
Total #
Students
% with Goal
Entered Employment—Average ALL states
6,904
52,548
10%*
State 1
417
69,284
1%
State 2
2,624
138,184
2%
State 3
1,609
70,893
2%
2,919
52,548
6%
State 4
159
33,062
0%
State 5
333
31,574
1%
State 6
14,523
565,311
3%
7,207
52,548
20%**
State 7
7,192
114,008
6%
State 8
2,768
32,492
9%
State 9
33,118
387,710
9%
Entered post-secondary—Average ALL states
High School Completion—Average ALL states
* 38% Unemployed (Table 6)
** 18% Enrolled in ASE
22
Table 5 and Follow-up
Quality Control: Surveys

Identify students



Data Collection




23
Set student goals appropriately
By exit quarter for employment measures
Quarterly collection for employment
State survey instrument
Sufficient resources (staff, time, funds)
Sampling method (if applicable)
Table 5 and Follow-up
Quality Control: Surveys (cont’d.)

Training



Reporting


24
Trained staff on survey procedures
Improving response rate
Quarterly reporting
Database to link outcomes to programs and
students
Table 5 and Follow-up
Quality Control: Data Matching

Procedure for collecting and validating Social
Security numbers



25
Report all with goal, prior to eliminating invalid
or missing numbers (Table 5, Col. D)
Data have exit quarter and employment
records matched after correct exit quarter
Only students with goal matched and/or
reported—should not have 100% with
outcome
Table 5 and Follow-up
Discussion: Issues and Strategies



26
Improving reporting and collecting
Resources
Training on Monitoring
Table 5 and Follow-up