SUPPLEMENT ARTICLE
Do Estimates of Intervention Coverage Obtained
From Children at Immunization Clinics Provide
a Reasonable Approximation to Population
Values?
Richard E. Cibulskis,1 Samir Pujari,2 and Mac W. Otten1
1Global Malaria Programme, World Health Organization, Geneva, Switzerland; and 2Global Measles Branch, Global Immunization Division, Centers
for Disease Control and Prevention, Atlanta, Georgia
Objective. The purpose of this study was to determine whether the magnitude of selection bias incurred by
measuring child survival intervention coverage at convenient sampling opportunities (child immunization contacts) is
sufficiently small for the approach to be used as a management tool within country programs.
Methods. We estimated the magnitude of selection bias by calculating values of 13 health indicators for
31 countries using Demographic and Health Survey data for children immunized with the third dose of the
diphtheria-pertussis-tetanus vaccine (DPT3) and those who were immunized with measles vaccine, and comparing
their values to those obtained for the population as a whole.
Results. Estimates of intervention coverage derived from immunized children are close to population values
if immunization coverage exceeds 60%. Levels of bias were lower for interventions that were not delivered directly
by formal health services, such as use of mosquito nets among children and provision of more fluid for diarrhea.
Levels of bias were also lower when using results for measles vaccine than for DPT3, suggesting that the measles
vaccination contact may be the most opportune time to collect data on additional health indicators.
Conclusions. The coverage of immunization programs has reached 60% in 85% of African countries, so
selection bias does not appear to invalidate the measurement of intervention coverage at immunization contacts.
Regular monitoring of the coverage of child survival
interventions at district or health facility level is widely
promoted to make health programs more responsive.
Indeed, decentralized problem-solving and decisionmaking are considered essential for improving organizational performance [1–3]. At present, most health
systems have limited tools for measuring the coverage of
key interventions such as the use of oral rehydration
therapy (ORT), antibiotics for pneumonia, insecticidetreated nets (ITNs), or antimalarial therapy including
artemisin combination therapies (ACTs) at health
facility or district level.
Population-based household surveys, such as the
Demographic and Health Survey (DHS) and Multiple
Correspondence: Richard Cibulskis, BSc, MSc, MPH, PhD, World Health
Organization, 22 Avenue Appia, 1211, Geneva, Switzerland ([email protected]).
The Journal of Infectious Diseases 2012;205:S91–102
Published by Oxford University Press on behalf of the Infectious Diseases Society of
America 2012.
DOI: 10.1093/infdis/jir795
Indicator Cluster Survey (MICS), provide reliable information on the coverage of several child survival interventions at a national level. However, these surveys
cannot normally be conducted more frequently than
every 3–5 years for reasons of cost and organizational
capacity, and they require a high level of expertise,
training, and supervision to ensure high-quality data.
Moreover, they do not provide estimates of coverage
at the district or health facility level. Therefore, the information obtained from these surveys is of limited
value to decentralized decision-making.
A proxy of coverage can be obtained for some interventions by using information on the amount of
commodities supplied to an estimated target population. For example, the percentage of pregnant
women possessing an ITN can be estimated by dividing
the number of pregnant women receiving an ITN
through antenatal clinics by the number of women expected to be pregnant in the catchment area. However,
this ‘‘administrative’’ method does not work well in the
Measuring Coverage at Immunization Contacts
d
JID 2012:205 (Suppl 1)
d
S91
following circumstances: (1) where information on the numerator, the number of commodities distributed, is not accurately
captured by the government reporting system (eg, when a large
proportion of commodities, such as ACTs, are distributed through
the private sector); (2) where information on the denominator is
inaccurate (eg, when censuses are out of date or in settings with
much population movement such as rural populations that use
facilities in urban areas, conflict zones, or for unregistered populations such as street children or migrant labor) or difficult to
define in advance (eg, use of ORT for children with diarrhea);
and (3) where interventions require a behavioral change in addition to, or rather than, receipt of a commodity (eg, breastfeeding).
An alternative approach to estimating coverage of interventions has been to use clients of immunization clinics as
a sample from whom to ask questions about the use of another (unrelated) intervention, such as ORT for diarrhea or
antibiotics for respiratory infection. The approach was first
implemented in Benin in 2002, when a Ministry of Health
and Centers for Disease Control and Prevention project
used Expanded Programme on Immunization (EPI) contacts
to measure the use of ITNs for prevention of malaria
(F Onikpo, P Kple-Fuget, M Deming, unpublished report,
June 2006). In 2005, it was suggested that this method
could be used to measure several child survival indicators.
Zambia and Sierra Leone started using EPI contacts in some
health facility catchment areas to ask about 4 interventions:
ITN use, appropriate antimalarial use, exclusive breastfeeding up to 6 months, and appropriate ORT. In Sierra Leone,
Table 1. Median Values of Immunization Coverage, Child Survival Indicators and Levels of Selection Bias
No.
I
Xt
Xi
Xn
B
S
S .10%
Using DPT3: children aged 8–11 mo
Proportion of children with
Cough in past 2 weeks
30
61%
34%
37%
32%
1.11
1.7%
2
Diarrhea in past 2 weeks
30
61%
30%
30%
27%
1.04
1.7%
2
Fever in past 2 weeks
30
61%
38%
38%
37%
0.96
1.3%
–
Cough with rapid breathing in past 2 weeks
30
62%
51%
51%
50%
0.94
3.3%
4
1
Proportion of children
In households owning an ITN
26
58%
39%
43%
32%
1.17
2.6%
Sleeping under an ITN
20
61%
21%
23%
21%
1.14
1.9%
2
Receiving more fluid for diarrhea
32
60%
32%
36%
29%
1.24
3.2%
3
Receiving ORT for diarrhea
30
58%
30%
37%
24%
1.39
5.7%
8
Going to health facility for cough
30
62%
44%
53%
33%
1.37
4.0%
7
Going to health facility for rapid breathing
31
62%
50%
57%
42%
1.36
5.0%
4
Receiving antimalarial for fever
21
58%
40%
52%
34%
1.26
7.0%
2
Receiving ACT for fever
Receiving vitamin A
4
30
36%
61%
10%
64%
14%
74%
6%
52%
2.24
1.36
10.8%
6.8%
5
6
Using measles vaccine: children aged 12–17 mo
Proportion of children with
Cough in past 2 weeks
30
65%
32%
34%
31%
1.05
2.0%
–
Diarrhea in past 2 weeks
30
65%
27%
26%
29%
0.90
1.1%
–
Fever in past 2 weeks
30
65%
37%
35%
37%
0.95
1.1%
–
Cough with rapid breathing in past 2 weeks
30
69%
47%
47%
46%
0.97
1.7%
1
–
Proportion of children
In households owning an ITN
26
63%
38%
39%
35%
1.15
1.6%
Sleeping under an ITN
19
64%
21%
24%
18%
1.00
1.5%
–
Receiving more fluid for diarrhea
32
65%
38%
39%
33%
1.16
3.1%
1
Receiving ORT for diarrhea
30
63%
36%
43%
26%
1.63
4.5%
6
Going to health facility for cough
30
66%
43%
51%
35%
1.31
3.8%
3
Going to health facility for rapid breathing
31
69%
50%
56%
38%
1.32
3.7%
5
Receiving antimalarial for fever
21
64%
40%
46%
36%
1.24
4.6%
–
Receiving ACT for fever
Receiving vitamin A
5
30
65%
68%
5%
68%
5%
75%
2%
58%
0.90
1.29
1.8%
5.6%
2
8
Data represent median values of (1) immunization coverage, I; (2) child survival indicators among the population as a whole, Xt, and among immunized and
nonimmunized children, Xi and Xn; (3) the ratio of the child survival indicator between immunized and nonimmunized children, B; (4) the level of selection bias
observed, S, and the number of times the selection bias was observed to be .10%, S .10%. No. is the number of surveys with data on the indicators.
Abbreviations: ACT, artemisin combination therapy; DPT3, third dose of diphtheria-pertussis-tetanus vaccine; ITN, insecticide-treated net; ORT, oral
rehydration therapy.
S92
d
JID 2012:205 (Suppl 1)
d
Cibulskis et al
Figure 1. Different values of Xt (value of child survival indicator in the population), I (immunization coverage), and B (the ratio of the child survival
indicators in immunized and nonimmunized children) and their impact on selection bias, S.
an additional question was added about appropriate antibiotic
treatment of those with cough and difficulty breathing.
The advantage of this approachdwhich we call the EPI Contact Methoddis that a sample of children can be accessed at little
cost to the health service; data can be obtained using existing staff
at health facilities and even incorporated into routine monthly
reports. The method can therefore yield regular information on
intervention coverage at the health facility and district levels.
A potential drawback is that children receiving immunization
services may represent a biased sample of the population, who
generally have better access to services or come from wealthier
backgrounds, so that estimates of intervention coverage may be
unrepresentative of the population as a whole. Therefore, a key
question is whether the magnitude of this selection bias is large
enough to result in a misleading interpretation and invalidate the
EPI contact method as a management tool. In this article, we
examine the selection bias incurred by the EPI contact method in
2 ways. First, we identify factors that drive the magnitude of
selection bias, namely, the value of the health indicator of interest, the level of immunization coverage, and the extent to
which the health indicator differs in immunized children vs
nonimmunized children. We then devise formulae to predict the
magnitude of selection bias. Second, we use data from DHSs to
calculate values of health indicators in children who were immunized with the third dose of the diphtheria-pertussis-tetanus
vaccine (DPT3) and those who were immunized with measles
vaccine, and compare these values with those obtained for the
population as a whole.
METHODS
Theoretical Analysis
Let
Ni 5 the number of children receiving immunization,
Nn 5 the number of children not receiving immunization,
Xt 5 the value of a health indicator in all children; the
percentage of children with a health attribute, such as fever in
the past 2 weeks, or receiving a health intervention such as ORT
for diarrhea,
Xi 5 the value of a health indicator in children receiving
immunization,
Xn 5 the value of a health indicator in children not receiving
immunization,
B 5 a multiplier that describes the extent to which the value
of a health indicator differs in immunized children vs nonimmunized children such that Xi 5 B 3 Xn.
Measuring Coverage at Immunization Contacts
d
JID 2012:205 (Suppl 1)
d
S93
Table 2. Selection Bias for a Selected Indicator: Percentage of Children Receiving ORT for Diarrhea
Country
Year
No.
I
Xt
Xi
Xn
B
S
S .10%
Using DPT3: children aged 8–11 mo
Bangladesh
2007
64
86%
80%
82%
67%
1.24
2%
Burkina Faso
2003
209
56%
22%
24%
20%
1.25
2%
Benin
2006
179
64%
34%
38%
26%
1.49
4%
Bolivia
2003
202
56%
25%
29%
21%
1.41
4%
DRC
Congo
2007
2005
80
88
58%
67%
48%
24%
58%
32%
33%
7%
1.74
4.67
10%
8%
Yes
Cameroon
2004
59
58%
22%
35%
4%
8.82
13%
Yes
Ethiopia
2005
86
27%
16%
14%
17%
0.84
2%
Ghana
2003
74
63%
28%
32%
20%
1.62
5%
Guinea
2005
38
50%
33%
44%
22%
2.00
11%
Haiti
2005–06
80
41%
54%
61%
49%
1.25
7%
India
2005–06
544
53%
25%
32%
17%
1.86
7%
Kenya
Cambodia
2003
2005
121
85
61%
71%
33%
23%
35%
20%
31%
32%
1.12
0.61
1%
4%
Liberia
2007
134
45%
43%
57%
31%
1.84
14%
Lesotho
2004
50
85%
46%
44%
57%
0.76
2%
Madagascar
2003–04
57
57%
17%
9%
27%
0.33
8%
Mali
2006
168
61%
15%
19%
10%
1.98
4%
Malawi
2004
330
77%
66%
66%
65%
1.02
0%
Mozambique
2003
162
69%
58%
67%
39%
1.75
9%
Nigeria
Niger
2003
2006
103
72
14%
40%
18%
12%
29%
23%
16%
4%
1.76
5.25
11%
11%
Namibia
2006–07
63
93%
71%
70%
75%
0.94
0%
Nepal
2006
79
84%
35%
42%
0%
99.00
7%
Rwanda
2005
84
90%
17%
17%
13%
1.39
0%
Swaziland
2006–07
52
92%
91%
90%
100%
0.90
1%
Chad
2004
127
19%
26%
48%
21%
2.30
22%
Yes
Uganda
2006
98
9%
56%
44%
57%
0.78
12%
Yes
Zambia
Zimbabwe
2007
2005–06
133
76
19%
46%
65%
7%
73%
6%
64%
8%
1.15
0.79
8%
1%
Yes
Yes
Yes
Yes
Using measles vaccine: children aged 12–17 mo
Bangladesh
2007
89
84%
71%
71%
71%
0.99
0%
Burkina Faso
2003
315
42%
21%
28%
15%
1.82
7%
Benin
2006
252
62%
29%
35%
19%
1.83
6%
Bolivia
2003
384
51%
37%
42%
32%
1.35
5%
DRC
2007
126
53%
36%
44%
26%
1.69
9%
Congo
Cameroon
2005
2004
138
104
64%
65%
16%
36%
23%
38%
5%
32%
5.01
1.20
7%
2%
Ethiopia
2005
148
43%
27%
38%
19%
2.07
11%
Ghana
2003
112
81%
49%
52%
35%
1.50
3%
Guinea
2005
69
52%
35%
51%
18%
2.91
16%
Yes
Haiti
2005–06
104
48%
47%
70%
26%
2.70
23%
Yes
India
2005–06
722
58%
36%
43%
26%
1.64
7%
Kenya
2003
166
70%
35%
38%
29%
1.32
3%
Cambodia
Liberia
2005
2007
101
119
70%
59%
30%
45%
27%
52%
38%
36%
0.71
1.47
3%
7%
Lesotho
2004
51
79%
58%
60%
55%
1.09
1%
Madagascar
2003–04
93
40%
10%
13%
8%
1.53
3%
Yes
Mali
2006
280
61%
17%
20%
13%
1.62
3%
Malawi
2004
510
75%
65%
68%
58%
1.17
3%
Mozambique
2003
244
72%
51%
62%
24%
2.58
11%
Yes
Nigeria
2003
156
21%
21%
45%
15%
2.99
24%
Yes
S94
d
JID 2012:205 (Suppl 1)
d
Cibulskis et al
Table 2 continued.
Country
Year
No.
I
Xt
Xi
Xn
B
S
S .10%
Yes
Niger
2006
141
46%
21%
32%
10%
3.10
12%
Namibia
2006–07
114
81%
68%
74%
41%
1.81
6%
Nepal
Rwanda
2006
2005
114
117
85%
76%
28%
13%
31%
15%
7%
4%
4.71
4.31
4%
3%
Swaziland
2006–07
71
90%
87%
87%
86%
1.01
0%
Chad
2004
182
24%
16%
22%
14%
1.52
6%
Uganda
2006
135
57%
51%
53%
48%
1.10
2%
Zambia
2007
181
79%
63%
65%
55%
1.17
2%
Zimbabwe
2005–2006
122
64%
6%
8%
3%
3.38
2%
Values of (1) immunization coverage, I; (2) percentage of children receiving ORT for diarrhea among the population as a whole, Xt, and among immunized and
nonimmunized children, Xi and Xn; (3) the ratio of the percentage of children receiving ORT for diarrhea between immunized and nonimmunized children, B; and
(4) the level of selection bias observed, S, and whether or not the selection bias was observed to be .10%, S .10%. No. is the sample size for the comparison.
Abbreviations: DPT3, third dose of diphtheria-pertussis-tetanus vaccine; DRC, Democratic Republic of the Congo.
Then immunization coverage is given by I where
Ni
I5
Ni 1Nn
Note that the value of the health indicator in the entire
population, Xt, is a weighted average of the value in immunized
children and nonimmunized children. Hence, Xt can be expressed in terms of the value of the indicator in immunized
children and the magnitude of B:
Ni 3Xi 1Nn 3Xn
Xt 5
Ni 1Nn
Ni 3Xi 1Nn 3Xi =B
5
Ni 1Nn
Ni 3Xi Nn 3Xi =B
5
1
Ni 1Nn Ni 1Nn
5I3Xi 1ð12IÞ3Xi =B
The magnitude of selection bias, S, is given by the absolute
difference between the estimate of the health indicator in immunized children vs the whole population:
S5jXi 2Xt j
5jXi 2ð I3Xi 1ð12IÞ3Xi =B Þj
5jXi 3ð12I2ð12IÞ=BÞj
5jXi 3ð12IÞ3ð121=BÞj
Alternatively in terms of Xt:
S5jXt 3ð12IÞ3ð111=BÞj
Hence, the expected magnitude of selection bias, S, depends
on 3 factors: (1) the value of the health indicator in the entire
population, Xt; (2) the level of immunization coverage, I; and
(3) the extent to which the value of a child survival indicator
differs in immunized children vs nonimmunized children,
B. With this formulation, it is possible to explore how different
values of Xt, I, and B influence the magnitude of selection bias.
Empirical Analysis
We used data from 31 countries with DHSs undertaken between 2003 and 2007, and from the Eritrea DHS undertaken
in 2002. (Data were downloaded in August 2009; the full list
of countries was Bangladesh, Benin, Bolivia, Burkina Faso,
Cambodia, Cameroon, Chad, Colombia, Congo, Democratic
Republic of the Congo, Eritrea, Ethiopia, Ghana, Guinea,
Haiti, India, Kenya, Lesotho, Liberia, Madagascar, Malawi,
Mali, Mozambique, Namibia, Nepal, Niger, Nigeria, Rwanda,
Swaziland, Uganda, Zambia, and Zimbabwe). Since one intention of the analysis was to calculate indicators of malaria
program coverage, non-malaria-endemic countries [4] were
omitted. Of the 23 maternal and child survival interventions
from the Lancet child survival series [5], we identified those
that fit the following criteria: (1) delivered predominantly during
the first year of life, (2) operationally feasible to measure at the
time of immunization, and (3) adequately assessed using DHS
data. Five intervention areas were selected: ORT, antibiotics for
pneumonia, vitamin A supplementation, ITN utilization, and
antimalarial drugs. To measure these interventions, 9 specific
indicators were identified that could be analyzed using DHS data
(see Table 1). Although vitamin A supplementation is commonly delivered along with immunizations, we included it to
assess the magnitude of selection bias encountered with this
intervention. We also examined illness rates for diarrhea, fever,
cough, and cough with rapid breathing, as reported by caregivers.
A limitation of DHS data is that the coverage of an intervention, such as the use of an ITN, is estimated at the time
Measuring Coverage at Immunization Contacts
d
JID 2012:205 (Suppl 1)
d
S95
Figure 2. Values of child survival intervention coverage calculated in children immunized with a third dose of diphtheria-pertussis-tetanus vaccine
(DPT3) and measles vaccine, Xi, plotted against the value in the general population, Xt. Points represent country values for the indicator derived from
Demographic and Health Survey data for 31 countries, 2003–2007. Those surrounded by a gray circle are from countries with immunization coverage
,60%. Abbreviations: ITN, insecticide-treated net; ORT, oral rehydration therapy.
of the survey, whereas immunization of the child may have
occurred some months earlier. To minimize this problem, we
limited age groups to just after the age at which DPT3
(4–6 months) and measles (9–12 months) vaccinations are
normally received, and we made the age groups as narrow
as feasible given available sample sizes. We used children aged
8–11 months to evaluate intervention coverage among those
immunized with DPT3 vaccine, and children aged 12–17 months
to evaluate intervention coverage among those immunized
with measles vaccine. In keeping with standard practice in
analysis of survey data, children were regarded as immunized if
DPT3 or measles immunization was verified by immunization
card or by the child’s caretaker report of receiving that vaccine
S96
d
JID 2012:205 (Suppl 1)
d
Cibulskis et al
dose. We used SPSS version 16 and Microsoft Excel 2003
software for all analyses.
RESULTS
Theoretical Analysis
Hypothetical values of I (immunization coverage), Xt (child
survival indicator), and B (the ratio between immunized and
nonimmunized children in child survival indicators) were used to
determine the resulting magnitude of S (selection bias). The results are presented in Figure 1.
When the coverage of an intervention (Xt) is low in the
population (eg, 20%), then the absolute magnitude of selection
Figure 2.
Continued.
Measuring Coverage at Immunization Contacts
d
JID 2012:205 (Suppl 1)
d
S97
Figure 2.
bias, S, does not exceed 10% unless immunization coverage is
,50%, and immunized children are 3 times more likely to be
covered with the child survival intervention than nonimmunized
children. If immunized children are only 1.5 times more likely
to be covered by a child survival intervention, then the level of
selection bias does not exceed 10%, irrespective of the level of
immunization coverage. Therefore, if the coverage of a child
survival intervention is low, the level of selection bias is also low,
and the EPI contact method could provide a reasonably close
approximation to the true level of child survival intervention
coverage. This is because the selection bias considered is an absolute difference; if the value of a health indicator is low, it may
not produce a 10% difference in absolute values, even if the ratio
of a health indicator between immunized and unimmunized
children is large (B large).
As the coverage of the intervention increases, the magnitude
of selection bias increases for a given immunization coverage
level. However, the maximum potential bias is capped at the
percentage of children not vaccinated. For example, if immunization coverage is $80%, then selection bias cannot exceed
20%, and it does not exceed 10% unless coverage of the child
survival indicator is twice as high in immunized children as that
in nonimmunized children.
Empirical Analysis
For the data sets examined, Table 1 shows the median values of
(1) immunization coverage, I, for the DPT3 and measles vaccines; (2) values of health indicators for all children, Xt, and
among children immunized and nonimmunized, Xi and Xn;
(3) the ratio of the health indicator between immunized and
nonimmunized children, B; (4) the level of selection bias observed, S; and (5) the number of times the selection bias was
S98
d
JID 2012:205 (Suppl 1)
d
Cibulskis et al
Continued.
observed to be .10%, S .10%. Table 2 shows these values
by country for a selected indicatordpercentage of children
receiving ORT for diarrhea. Figure 2 shows values of 8 of the
9 health indicators among children immunized, Xi, plotted
against values for all children, Xn, as well as illness rates reported
by caregivers. Coverage of ACT was not plotted because only
4 surveys provided data.
The median proportion of children aged 8–11 months immunized with DPT3 was approximately 61%, whereas 65% of
children aged 12–17 months were immunized with measles
vaccine. The uptake of other health interventions among all
children ranged from approximately 20% for children sleeping under an ITN to .60% for children receiving vitamin A
supplementation.
The level of selection bias observed when indicators were
calculated solely from immunized children, S, varied by type of
indicator, being generally lower among morbidity indicators
(reported cough, diarrhea, fever, or cough with rapid breathing)
than among health intervention indicators. It was also usually
lower when indicators were derived for children immunized
with measles vaccine than DPT3. In total, approximately 15%
(38 of 247) of child survival indicators deviated by .10% of
their population values when calculated for children immunized
with DPT3, whereas 10% (25 of 247) of child survival indicators
deviated by .10% from their population values when calculated
for children immunized with measles vaccine.
The level of selection bias, S, is heavily dependent on the
value of immunization coverage, I (Table 3). When immunization coverage was ,60%, 29% of intervention coverage
indicators calculated from children immunized with DPT3
deviated by $10 percentage points of the true population
value, and 23% of child survival indicators calculated from
Table 3. Magnitude of Selection Bias, S, in Relation to the Level of Immunization Coverage, I, for 9 Child Survival Indicators Observed in
31 Household Surveys
Immunization Coverage
20%–39%
,20%
DPT
40%–59%
80%–100%
Total
Magnitude of selection bias, S
,10%
16
15
53
84
43
211
10%–19%
$20%
5
10
8
2
9
0
4
0
0
0
26
12
88
43
249
Total
31
25
62
% of indicators for which S exceeds
10%
48%
40%
15%
Measles
Magnitude of selection bias, S
% of indicators for which S exceeds
60%–79%
5%
0%
15%
Total
,10%
3
11
58
95
55
222
10%–19%
0
5
10
4
0
19
20%1
0
4
2
0
0
6
Total
10%
3
0%
20
45%
70
17%
99
4%
55
0%
247
10%
Abbreviation: DPT3, diphtheria-pertussis-tetanus vaccine.
children immunized with measles vaccine deviate by $10
percentage points; these points lie above the 610% bias
line in Figure 2. When immunization coverage was ,40%,
approximately 40% of measurements had .10% bias. However, when immunization coverage was .60%, only 3% of
intervention coverage indicators calculated from immunized
children deviated by $10 percentage points of the true population value (for both DPT3 and measles vaccine), with
a maximum deviation of 15 percentage points.
There was a ,1.5-fold difference between immunized and
nonimmunized children for most of the child survival indicators
calculated (0.67 , B , 1.5 for 274 of 432 indicators, 63%). B was
generally lower for morbidity indicators than for child survival
intervention indicators (Table 4). B is also lower for interventions that are delivered somewhat separately from the formal
health services (eg, provision of more fluid for diarrhea,
possession of an ITN, and sleeping under an ITN), and it is
higher for interventions that involve more contact with formal
health services (seeking treatment for acute respiratory infection [ARI], ORT for diarrhea, antimalarial treatment) or for
interventions that are normally delivered in conjunction with
immunization (vitamin A supplementation).
DISCUSSION
Our study found that estimates of intervention coverage derived
only from immunized children can provide a reasonable approximation to population values if levels of immunization
coverage exceed 60%. Less than 3% of 285 intervention coverage
indicators values calculated from immunized children deviated
by $10 percentage points from true population values when
immunization coverage exceeded 60%. A low level of selection
bias, S, was observed in a wide range of indicators, countries, and
indicator coverage levels. Whereas some surveys indicated that
selection bias could potentially be a problem when immunization
coverage is low, levels of immunization coverage have been increasing in recent years. By 2009, immunization coverage, as
estimated by the World Health Organization and the United
Nations Children’s Fund [5], exceeded 60% in 182 of 193
countries worldwide for DPT3 and for measles vaccine (and for
42 of 49 countries in sub-Saharan Africa, 85%). Thus, selection
bias may be a major factor for only a few countries today,
although care would be needed to ensure that immunization
coverage is adequate locally using available data on immunizations given and the size of target populations.
Low levels of bias in areas with high vaccination coverage
may be due to successful outreach programs, which facilitate
access to immunization services for children who are distant
from fixed health facilities, so that the population receiving
immunization may not be very different from those not receiving immunization. Analysis of the ratio of child survival
indicators in immunized children vs nonimmunized, B,
suggested that there is generally a ,1.5-fold difference between immunized and nonimmunized children. Theoretical
analysis indicated that, with such small differences between
immunized and nonimmunized children, the selection bias, S,
will not exceed 10% provided that immunization coverage is
$60% and the value of a child survival indicator is #60%,
a finding also borne out by our empirical analysis. (If the value
of a child survival indicator is 80%, then S would exceed 10%
if immunization coverage falls below 68%; such cases would
be expected to be comparatively rare, however.)
Measuring Coverage at Immunization Contacts
d
JID 2012:205 (Suppl 1)
d
S99
Table 4.
The Magnitude of B, the Ratio of Child Survival Indicators in Immunized and Non-Immunized Children, in 31 Household Surveys
SE of the Mean
No. of Surveys in Which the Value B Lay in the Range
No. Times
,0.67 0.67–1.0 1.0–1.5 1.5–2.0 2.5–2.5 2.5–3.0 3.0–3.5 4.01 B sig. diff.
No.
B
B
Cough in past 2 weeks
30
1.10
0.04
1
9
18
2
Diarrhea in past 2 weeks
30
1.09
0.06
1
12
13
4
Fever in past 2 weeks
30
1.00
0.03
0
16
14
Cough with rapid breathing in
past 2 weeks
30
1.06
0.10
3
17
6
3
1
7
5
54
51
9
1
20
1
10
10
Using DPT3: children aged 8–11 mo
Proportion of children with
Total for reported illness
120
5
4
4
Proportion of children
In households owning an ITN
Sleeping under an ITN
26
19
1.21
1.28
0.09
0.17
2
2
4
3
16
9
2
3
1
1
1
Receiving more fluid for diarrhea
30
1.37
0.12
1
6
14
4
2
2
Receiving ORT for diarrhea
29
1.82
0.32
2
6
9
8
1
Going to health facility for
respiratory infection
30
1.63
0.12
0
3
13
6
6
Going to health facility for
rapid breathing
30
1.57
0.27
2
7
10
8
1
Receiving antimalarial for fever
17
1.58
0.34
2
10
3
1
4
3.79
2.36
1
30
1.49
0.09
0
1
21
5
1
1
1
11
29
81
34
13
4
4
5
92
Receiving ACT for fever
Receiving vitamin A
Total for intervention indicators
181
1
1
5
3
2
1
1
9
14
1
11
1
9
1
3
21
Using measles vaccine: children
aged 12–17 mo
Proportion of children with
Cough in past 2 weeks
Diarrhea in past 2 weeks
30
30
1.07
0.91
0.04
0.03
1
2
12
19
16
9
1
0
0
0
0
0
0
0
0
0
5
3
Fever in past 2 weeks
30
0.99
0.04
1
18
10
1
0
0
0
0
6
Cough with rapid breathing in
past 2 weeks
30
1.03
0.05
1
16
11
2
0
0
0
0
5
5
65
46
4
Total for reported illness
120
0
19
Proportion of children
In households owning an ITN
26
1.21
0.05
0
6
15
5
0
0
0
0
Sleeping under an ITN
19
1.23
0.10
0
9
6
3
1
0
0
0
6
Receiving more fluid for diarrhea
30
1.40
0.16
2
6
14
5
1
0
1
1
4
Receiving ORT for diarrhea
Going to health facility for
respiratory infection
30
30
2.04
1.42
0.21
0.08
0
0
2
3
10
19
8
4
1
3
4
1
2
0
3
0
12
12
Going to health facility for rapid
breathing
30
1.44
0.10
0
5
15
6
2
2
0
0
17
Receiving antimalarial for fever
Receiving ACT for fever
19
3
1.42
0.87
0.14
0.39
1
1
2
1
9
0
2
1
4
0
1
0
0
0
0
0
8
1
30
1.55
0.13
Receiving vitamin A
Total for intervention indicators
187
8
0
1
19
7
0
2
0
1
23
4
34
88
34
12
8
3
4
91
The magnitude of B, the ratio of child survival indicators in immunized and nonimmunized children in 31 household surveys. No. indicates the number of surveys
with data on the indicators; it may differ from Table 1 because B could not be calculated if it involved a division of zero. The number of times that B differed
significantly from 1 is also shown.
Abbreviations: ACT, artemisin combination therapy; DPT3, third dose of diphtheria-pertussis-tetanus vaccine; ITN, insecticide-treated net; ORT, oral rehydration
therapy; SE, standard error; sig. diff., significantly different.
If immunization coverage is ,60%, then the EPI contact
method is likely to overestimate child survival intervention
coverage, although the selection bias was lower for children
S100
d
JID 2012:205 (Suppl 1)
d
Cibulskis et al
immunized against measles than those immunized with DPT3;
this is possibly because, in some countries, measles vaccine is
administered through mass campaigns and may reach a wider
range of children than DPT3, which is only achieved after 3 visits
to an immunization clinic or site. The selection bias was higher
for interventions in which formal health services play a significant role in delivery (eg, treatment of malaria, ARIs, or
treatment of diarrhea with ORT). These observations suggest
that if selection bias is a concern (eg, if implementing the EPI
contact method in a country where immunization coverage
is ,60% or where there is considerable variation in coverage
within a country), then more robust results may be obtained
by focusing on interventions not delivered exclusively by
formal health services (eg, use of ITNs in children ,5 years,
provision of more fluid for diarrhea) and asking questions
only at measles vaccination visits.
Selection bias was also observed to be smallest for morbidity
indicators such as fever, cough, or diarrhea. Although selfreported illness is of limited value in examining disease trends,
there is potential for formal examination of disease rates
at immunization clinics by using rapid diagnostic tests for
malaria parasites or testing for anemia. Indeed, integrating
disease testing at routine immunization visits may offer a particularly sensitive way of monitoring malaria trends because
children eligible for measles vaccine, aged 9–12 months, are
most prone to anemia and their malaria parasite infections
would have necessarily been acquired during the previous
year. Field studies in Malawi suggest that anemia (and maybe
parasitemia) measured at the time of routine immunizations
may be a good surrogate indicator for its measurement at the
household level [6].
There are several ways that the EPI contact method could be
incorporated into health service monitoring. For example,
questions could be asked by health workers at every measles
immunization, or periodically at health facility–based surveys,
or by community volunteers at health facility exit interviews.
Each scheme has advantages and disadvantages. Asking questions at each measles contact allows data collection and analysis
to be a continuous process, providing managers with information in a timely fashion and empowering them to take
decisions and monitor the results of their actions. For some
programs such as malaria control, this is a particular advantage
because indicators can be calculated separately for wet and dry
seasons; cross-sectional surveys such as the DHS and MICS
are often conducted only in the dry season to facilitate access,
which limits the representativeness of data. There is some advantage in data being reported, aggregated, and analyzed routinely in similar ways to immunization data; less regular data
collection that requires specific efforts at particular intervals
may not be implemented consistently nationwide and may
occur only intermittently over time. Routinely asked questions
may also provide benefits to health workers and clients: asking
about ITN use, for example, may serve as a reminder to the
client and result in greater uptake of the intervention.
Clients’ responses may inform health workers about barriers
to utilization of some interventions. The potential benefits of
asking questions at each measles immunization contact must be
weighed against the time costs to health workers, which will
be dependent on how clinics are organized and their workload; many immunization clinics include time for nutrition
advice and other health education messages at registration,
whereas some may not. Whatever scheme is chosen, it will
need to take into account local circumstances and be tested
on a small scale to ensure it is appropriate before wide-scale
implementation.
Our study had several limitations. First, we used fixed age
groups (8–11 months and 12–17 months) to estimate intervention coverage. In practice, measurement of intervention
coverage with the EPI contact method would be done at the
time of immunization and confined to children ,1 year,
which may be several months younger than the age group
that we evaluated. However, among the DHSs we examined in
this article, the vast majority of DPT3 (78%) and measles
(77%) immunizations were given before 1 year of age, and the
coverage of child survival interventions was largely invariant
by age in months at the time of the survey; it is also notable
in the analysis presented here that results for child survival
indicators obtained for children aged 8–11 months are similar
to those children aged 12–17 months.
Second, this study only examined selection bias and did not
address potential information bias from use in the field. Although
the format of the questions as listed in the EPI contact method
training manuals is nearly identical to the questions in the DHSs,
clinicians would be the interviewers in the EPI contact method
and may tend to provide answers that give their health facility
more favorable results. Likewise, respondents may be prone to
providing answers considered desirable when questioned by interviewers in an authoritative position. Skarbinski et al [7] concluded that information bias is a significant factor, but their study
only examined children attending fixed health facilities (as opposed to including children attending mobile clinics), which are
likely to be unrepresentative of the population as a whole, so
the bias they observed may have been selection bias. Issues of
information bias can be addressed to some extent by ensuring
adequate training and support for clinicians, including provision
of adequate time to ask questions, encouragement to seek truthful
answers, appropriate tools for data collection, and adequate
supervision of the process including community involvement in
use of results. Further research to examine the magnitude of the
information bias is needed.
We believe that the EPI contact has great potential as a management tool in terms of the decisions it can influence and the
extent to which it can promote better management of programs.
Whereas estimates could have some inherent bias and should be
interpreted cautiously, they do not necessarily need to be highly
precise to be useful for management purposes; estimates of intervention coverage within 10% of the true value are more than
Measuring Coverage at Immunization Contacts
d
JID 2012:205 (Suppl 1)
d
S101
adequate for identifying geographical areas at risk, or trends in
coverage that may require more attention. At present, managers
often work with little data on the coverage of child survival
interventions; the data that could be provided by the EPI contact
method would be a large step forward. Further work on assessing
the feasibility of incorporating relevant questions into routine
reporting systems, and potential levels of information bias incurred in field situations, would nevertheless be beneficial to
assessing the extent to which the method can provide timely and
accurate information to managers.
Notes
Disclaimer. The views expressed are solely those of the authors and
are not those of the World Health Organization.
Potential conflicts of interest. All authors: No reported conflicts.
All authors have submitted the ICMJE Form for Disclosure of Potential
Conflicts of Interest. Conflicts that the editors consider relevant to the
content of the manuscript have been disclosed.
S102
d
JID 2012:205 (Suppl 1)
d
Cibulskis et al
References
1. Devan T. Integrating Lean, Six Sigma, and high-performance organization: leading the change toward dramatic, rapid, and sustainable improvement. San Francisco, CA: John Wiley and Sons, 2004.
2. Hamel G. The why, what, and how of management innovation. Harvard
Business Review 2006; 84:72–84.
3. Harrington JH. Total improvement management. The next generation
in performance improvement. New York: McGraw-Hill, 1995.
4. World Health Organization. World malaria report 2009. Geneva,
Switzerland: World Health Organization, 2009.
5. World Health Organization. WHO/UNICEF estimates of national immunization coverage. http://www.who.int/immunization_monitoring/
en/globalsummary/timeseries/tswucoveragedtp3.htm. Accessed 14 April
2010.
6. Mathanga DP, Campbell CH, Vanden Eng J, et al. Comparison of
aneamia and parasiteamia as indicators of malaria control in household
and EPI-health facility surveys in Malawi. Malar J 2010; 9:107.
7. Skarbinski J, Winston CA, Massaga JJ, Kachur SP, Rowe AK. Assessing
the validity of health facility-based data on insecticide-treated bednet
possession and use: comparison of data collected via health facility and
household surveysdLindi region and Rufigi district, Tanzania 2005.
Trop Med Int Health 2008; 13:396–405.