APPENDIX I
The Affiliate Evaluation
Methodology
WORKING DRAFT -- NOT FOR DISTRIBUTION
The Affiliate Evaluation Methods
Design
Affiliate sites were selected to participate in Healthy Steps based on a minimal set of requirements.
These included: a commitment to implementing the program for three years and participating in
evaluation activities; establishment of a local advisory committee; identification of one or more local
funders for the program; and a large patient population to complete enrollment within six to nine
months. Affiliate sites did not utilize a comparison population. Six sites, comprising seven primary
pediatric practices, participated in the Affiliate Evaluation. The seven participating practices
represent a mix of organizational types including two group practices, a hospital-based clinic, a
community based clinic, two hospital-based residency training pediatric clinics, and a group practice
in partnership with the local health department.
Enrollment
Enrollment of children in the Affiliate Evaluation began in July 1997 when the first child enrolled at
the Garden City, Kansas site. The initiation of sites was staggered over a four-month period in part to
allow evaluation staff time to work with the sites to set up enrollment and other evaluation
procedures. Once enrollment was initiated, all sites but one consecutively enrolled newborns entering
the practice. At one site, program operations began before the implementation of the Affiliate
Evaluation. Families who had been receiving Healthy Steps services prior to the implementation of
the evaluation were “back-enrolled” into the evaluation. Enrollment took place in the hospital
following the child’s birth or at the first pediatric office visit. In cases of multiple births, one child
was randomly selected to be enrolled in the evaluation. Enrollment of families ended in September
1998 when the last child enrolled at the Houston, Texas site.
To be eligible for enrollment in the Healthy Steps Affiliate Evaluation, the newborn had to be less
than 4 weeks of age (from birth to 28 days of life, inclusive) and a patient at the Healthy Steps site.
Children were not eligible to participate if: 1) their parents expected to move from the area or change
site of care within 6 months; 2) their mothers (or custodial parent) did not speak English or Spanish
fluently; 3) they were to be adopted (hospital only) or placed in foster care; or 4) they were too ill to
make an office visit within the first 28 days of life. Eligibility was limited to the first 28 days of life
to ensure that each intervention family could be offered a home visit within the first few weeks
postpartum. Target enrollment at each site was 200 families (100 families per practice at ChicagoRavenswood, the site composed of two practices). On average, affiliate sites took seven months to
enroll 200 families.
Table A.1 provides enrollment information for each site. In total, 1,103 families enrolled in the
Affiliate Evaluation. As Table A.1 indicates, only 51 families (4%) approached to participate in the
program declined. Of the parents who declined to participate, most said they were “not interested,”
“did not have the time,” or they failed to give a reason. Other less commonly cited reasons included:
they lived too far from the practice, they were enrolled in a day care program, they felt they did not
need the support, or the father refused to participate. At one site, a number of families declined
because they chose a provider who was not participating in the Healthy Steps program. Ten percent
of families (n=126) deferred enrollment but failed to return during the eligible enrollment period.
31
WORKING DRAFT -- NOT FOR DISTRIBUTION
Table A.1
Enrollment of Families in the Affiliate Evaluation by Site
Site
Charlotte
ChicagoRavenswood
(Col. McNeil)
ChicagoRavenswood
(M/F Center)
ChicagoBethany
Garden City
Houston*
San Antonio
Enrollment
Start
End
Date
Date
9/97
3/98
9/97
2/98
Total
Eligible
Total
Enrolled
Total
Declined
Total
Deferred
N
240
124
N
212
94
%
88
76
N
18
6
%
8
5
N
10
24
%
4
19
9/97
5/98
100
96
96
0
0
4
4
9/97
7/98
251
192
76
15
6
44
18
7/97
10/97
10/97
1/98
10/98
5/98
192
109
264
186
107
216
97
98
82
3
2
7
2
2
3
3
0
41
2
0
16
1,103
86
51
4
126
10
Total:
*Enrollment at this site was slower than anticipated. Children enrolled through September 1998 were
included in the affiliate evaluation.
Informed Consent and Confidentiality of Participants
All enrollment procedures and study protocols were reviewed and approved by The Johns Hopkins
University Bloomberg School of Public Health’s Committee on Human Research. At the time of
enrollment, families were asked to sign a letter of consent (consent form).
In addition, at enrollment, each child was assigned a unique Healthy Steps identifier to be used rather
than the child’s name for data collection purposes. This number appeared on all questionnaires and
with all other references to the child. It identified the child, the Healthy Steps site, the child’s
evaluation sites and the site of enrollment (hospital or office). This unique identifier maintained the
confidentiality of the child and family, while making it possible to link information across affiliate
families and data instruments.
Characteristics of Families Participating in the Affiliate Evaluation
Information about characteristics of the infants and parents participating in the Affiliate Evaluation
was derived from enrollment forms and the Newborn Form. The Newborn Form was administered
either by an interviewer to the parent (or surrogate) or self-administered by the parent(s) as part of the
enrollment process. Questions on the Newborn Form covered characteristics of the infant and
demographic characteristics of the mother, father and family, prenatal utilization of services, health
behaviors of the mother and father, and parents’ choice of a pediatric provider for the newborn. It
was available in English and Spanish. The Newborn Form was completed by 1096 (99%) of the total
1,103 families enrolled.
The majority of Newborn Forms were filled out by mothers (84% by mothers alone and 14% by
mothers and fathers together). On average, mothers participating in the Healthy Steps Affiliate
32
WORKING DRAFT -- NOT FOR DISTRIBUTION
Evaluation tended to be young (more than one half were less than 25 years old), with limited
education (45% percent had not graduated from high school). Fathers tended to be slightly older and
better educated than mothers. About half of the mothers described their race as White; over half
reported that they were of Hispanic ethnic origin. Somewhat fewer than half of fathers described
themselves as White and one half reported that they were of Hispanic decent. Slightly more than half
of mothers (54%) were married at the time of enrollment into the Affiliate Evaluation. However,
more than two thirds reported that they were living with the baby’s father.
Approximately 40% of families enrolled in the Affiliate Evaluation were first-time parents. For more
than half of the families (54%), maternity care was paid for by Medicaid, with 25% of families
utilizing private insurance to cover the maternity care expenses. More than half of families reported
receiving assistance from the Women, Infants and Children (WIC) program during pregnancy.
Almost one quarter of families received food stamps during pregnancy. One quarter of families did
not receive any special services during pregnancy.
Slightly more than half the babies enrolled in the Affiliate Evaluation were boys. Fewer than 5
percent of infants enrolled weighed less than 2500 grams at birth. Almost 60% of babies stayed in the
hospital less than 48 hours after birth; nearly all (90%) of infants spent at least some time in the
mother’s hospital room. At the time of enrollment, less than 30% of infants were being exclusively
breastfed.
Very few mothers (fewer than 8 percent) reported smoking at the time of enrollment into the Affiliate
Evaluation. In contrast, more than 25% of fathers were smokers. Slightly more than one half of
mothers had 10-14 prenatal visits, with about 5 percent of mothers receiving very few (0-4) visits and
about 7 percent receiving 20 or more visits.
Almost 60% of families had selected their pediatric practice prior to the birth of their infant.
However, only 20% of families had actually visited the practice before the baby’s birth. The most
frequently cited means through which families knew of the pediatric practice was they had taken
other children there, or they learned about the practice from a prenatal care provider or a friend. The
two most frequently endorsed reasons for choosing the practice were that the practice provides good
care and that the practice had been recommended.
Data Collection
Both qualitative and quantitative data were collected as part of the affiliate evaluation. Data were
collected from providers, families, and medical records. Data instruments included: key informant
interviews; site questionnaires; provider knowledge, attitudes and practices surveys; HS Specialists’
logs of contacts with families; parent questionnaires; a parent telephone interview; and, medical
record reviews. A description of each follows.
Key Informant Interviews
Members of the Johns Hopkins University (JHU) evaluation team conducted interviews with key
informants (individuals identified as “stakeholders”) at each site at start-up and approximately 30
months into the program. The interviews were based on structured questionnaires that included openand close-ended questions. The questionnaires were mailed to informants so that they could complete
the objective portions of the questionnaire and return it to JHU prior to their interviews.
33
WORKING DRAFT -- NOT FOR DISTRIBUTION
At start-up informants included the head of pediatrics, the Healthy Steps lead physician(s), the site
administrator(s), and the HS Specialists. At some sites, representatives from local foundations
completed interviews. Interviews took place during the enrollment site visit (1997). They were
conducted face-to-face, in private and lasted between 30 minutes and 1 hour. Informants were
assured that all responses would be kept strictly confidential.
The interviews at start-up consisted of a series of questions related to the introduction of Healthy
Steps into the practice, and the decision-making process to participate in the program. Informants
were asked to comment on their previous experience with similar programs, understanding of the
Healthy Steps model, anticipated problems with implementation, the role of the HS Specialist, and
anticipated benefits of the program.
At 30 months, only the Healthy Steps lead physician(s), the site administrator(s), and the HS
Specialists participated in the key informant interviews. The majority of interviews were conducted
by telephone by a member of the evaluation team during the summer of 2000. One lead physician
was interviewed in person. Two lead physicians completed their interviews in writing. Telephone
interviews were scheduled at a convenient time for the informant and s/he was asked to find a private
place where s/he would feel free to talk openly about the program. Informants were assured that all
responses would be kept strictly confidential. Interviews at 30 months took on average 1 hour to
complete.
The 30-month key informant interviews contained objective questions related to the components of
the HS program, teamwork and communication patterns. Subjective, open-ended questions addressed
the practice context/environment and changes over time, factors that affected implementation, the
impact of HS in the practice, the role of the HS Specialist and the future of the Healthy Steps at the
site.
Sample
The exact number of interviews varied by site. Table A.2 gives the final sample by provider type and
time period. All informants invited to participate at start-up responded; at 30 months two lead
pediatricians failed to respond. Seventeen key informants completed interviews at both time periods.
Table A.2
Key Informant Interviews by Provider Type and Time Period
Start-up
4
8
8
11
4
Head of Pediatrics
Lead Healthy Steps Pediatrician
Site Administrator
HS Specialist
Local Foundation Representative
Total
35
30 Months
NA
5
6
12
NA
22
Numbers may vary from start-up to 30 months due to changes in staffing. At 30-months, interviews
with the Heads of Pediatrics and local foundation representatives were not requested.
Analysis
Separate techniques were used to process the objective and subjective data from the key informant
interviews. Similar techniques were used at start-up and 30 months.
34
WORKING DRAFT -- NOT FOR DISTRIBUTION
Prior to the interview, a member of the evaluation team checked the objective portions of the
interview for completeness. Missing or incomplete data were confirmed during the interview. The
objective portions of the interview were double-keyed to ensure accuracy using SPSS-DE. Following
verification of the data and data entry, analysis was conducted using the SAS programming package.
The quantitative data were explored using simple frequencies. Comparisons between responses at
start-up and 30 months were made when appropriate.
Within several days of conducting the interview, the interviewer summarized the subjective
information received from each informant using an interview guide. The guide combined responses
to one or more individual questions into larger topic areas.
The summaries by informants were subsequently organized across sites by informant category (HS
Specialist, lead physician, site administrator) and within sites across informant category. The
organization of responses by site and by informant type provided a framework for describing the
content of the interviews. This organization enabled evaluation staff to determine the consistency of
responses within each site as well as consistency and differences across sites.
The qualitative data were analyzed by members of the evaluation team, working independently. Each
reviewer identified topics or themes within the data. At start-up, topics and themes included: 1)
becoming a Healthy Steps site (characteristics of the decision-making process to participate; buy-in
by staff); 2) making Healthy Steps a reality (implementation of Healthy Steps, support for program
and evaluation activities, integration of the HS Specialist into the practice, and implementation of a
team approach); 3) expectations for the impact of Healthy Steps. At 30-months, topics were: 1) the
background and practice context into which Healthy Steps was placed; 2) the spectrum of the
implementation of Healthy Steps; 3) the Healthy Steps program and its components; and, 4) the
strengths, benefits and challenges of Healthy Steps. Two overarching themes, building relationships
and time, were identified.
Site Questionnaires
The first site questionnaire was administered at start-up (1997) to capture the practice context in 1996
before Healthy Steps. A second site questionnaire was administered at 30 months (2000) to address
changes in the practice since initiating Healthy Steps. Each site questionnaire was completed by the
site administrator, lead physician, or other designee. The questionnaires contained items regarding the
practice context and practice environment in which pediatric services were being delivered prior to
and towards the end of the Healthy Steps intervention.
Sample
Site questionnaires were returned by all practices comprising the six Healthy Steps affiliate sites at
both start-up and 30 months.
Analysis
Once the site questionnaires were returned to JHU, they were edited and coded for computer entry.
During this process, sites were consulted, as appropriate, to answer questions that arose during
coding, and provide missing data. Data were double-keyed. Following verification of the data and
data entry, analyses were conducted using the SAS programming package. Since the affiliate
evaluation did not include a comparison population, data were examined using simple frequencies.
Comparisons of responses between start-up and 30 months were made when appropriate using chi
square or t-test.
35
WORKING DRAFT -- NOT FOR DISTRIBUTION
Provider Knowledge, Attitudes & Practices Surveys
All providers who worked with Healthy Steps families at each affiliate site were asked to participate
in a survey at start-up and 30 months. These provider knowledge, attitudes, and practices (PKAP)
surveys provided information on a variety of topics, including providers’ perceptions of the barriers to
providing the best well child care to their patients, their opinions about the care they provided to their
clientele, their satisfaction with their ability to meet the needs of parents, their views of the HS
Specialists, and the topics they discussed with parents. The content of the provider KAP
questionnaires varied depending on the clinical or administrative position of the individual at the site.
Variations of the survey were administered to four groups of individuals: physicians and nurse
practitioners; nurses and other clinical staff; clerical and administrative staff; and Healthy Steps
Specialists.
Physicians and nurse practitioners (MDs/NPs) completed the longest questionnaire.
The
questionnaire at start-up contained questions related to their background, including their education,
number of years in pediatric care and working at the site, and any special training they had received in
child development or child behavior. Many of these questions were updated at 30 months. Questions
were asked about the amount of time they spent at well child visits in the first two months of life (at
start-up) and for two-year olds (at 30 months) performing these specific activities: physical exam,
anticipatory guidance, answering parents' questions, and other activities. The survey at start-up and
30 months also contained questions about barriers to providing well child care, topics they discuss
with families, their satisfaction with the time they spend discussing behavior and development with
parents, and their ability to meet the needs of parents. Similar questions were asked about their
satisfaction with the ability of clinical support staff to meet the needs of parents. An instrument was
included in the surveys that asked about physicians’ and nurse practitioners’ perceptions of the
services that the HS Specialist provided to families.
The questionnaires for nurses and other clinical staff included background educational information
along with questions about topics they discussed with parents, their perceptions of the care provided
at the practice, and their perceptions of the services provided by the HS Specialists. The
questionnaires for clerical and administrative staff included the same items except items relating to
topics discussed with parents. The HS Specialists responded to the questionnaire for nurses and
clinical staff.
A member of the evaluation staff, working with the HS Specialists, identified a list of eligible
providers at each site to determine the sample of providers to be surveyed. To maintain the
confidentiality of participants, each provider was assigned a unique identification number. This
number was the only identifying information on the questionnaire. At each site, the HS Specialists
hand delivered a copy of the questionnaire and an envelope to each provider. Providers were
instructed to complete the questionnaire and return it sealed in the envelope to the HS Specialist.
The name of the provider appeared on the envelope for tracking purposes only. Data collection was
to be completed within one month but was extended at all sites in order to improve response rates.
The HS Specialists returned the completed questionnaires to JHU for processing.
Sample
The exact number of completed questionnaires varied from site to site depending on staffing structure
and response rates. Tables A.3 provides the overall response rate and the response rate by site for the
PKAP Survey at start-up and 30 months. Please note that sites are presented in random order.
36
WORKING DRAFT -- NOT FOR DISTRIBUTION
Although the response rates include Resident physicians, they have been excluded from the sample
for analysis because only two affiliate sites had Resident physicians and their perceptions of the
program may not be representative of other primary care providers at the sites.
Accordingly, the sample at start-up included 24 MDs/NPs, 24 nurses and other clinical staff, 16
clerical and administrative staff, and 11 HS Specialists. The sample at 30 months comprised 22
MDs/NPs, 16 nurses and other clinical staff, 18 clerical and administrative staff, and 11 HS
Specialists. One important aspect of the samples is the number of respondents who completed
surveys at both start-up and 30 months.
Table A.3
Response to the Affiliate Provider KAP Survey at Start-up and 30 Months
Sites
MDs/NPs1
%2
N
Start-Up
Nurses/
HS
Other
Specialists
Clinical
N
%
N
%
Clerical/
Admin
Staff
N
%
Overall
N
%
1
2
3
4
5
6
TOTAL
0
6
10
0
8
0
24
1
2
2
2
2
2
11
0
4
6
3
1
2
16
1
21
27
5
16
5
75
Sites
0.0
85.7
66.7
0.0
100.0
0.0
63.2
MDs/NPs1
%2
N
50.0
100.0
100.0
100.0
100.0
100.0
91.7
0
9
9
0
5
1
24
0.0
81.2
75.0
0.0
100.0
100.0
68.6
30 Months
Nurses/
HS
Other
Specialists
Clinical
N
%
N
%
1
1
50.0
2
2
9
100.0
2
3
5
50.0
1
4
1
50.0
2
5
5
100.0
2
6
1
100.0
2
TOTAL
22
75.9 11
Sites are listed in random order.
100.0
100.0
50.0
100.0
100.0
100.0
91.7
1
7
4
0
4
0
16
100.0
70.0
50.0
0.0
66.7
0.0
64.0
0.0
57.1
85.7
42.9
100.0
66.7
59.3
14.3
77.8
75.0
31.3
100.0
62.5
66.9
Clerical/
Admin
Staff
N
%
Overall
N
%
3
10
2
1
1
1
18
7
28
12
4
12
4
67
100.0
90.9
40.0
100.0
100.0
100.0
81.8
87.5
87.5
48.0
80.0
85.7
100.0
76.1
1
At two sites, surveys were completed by resident physicians. They have been excluded from the analyses as their
disproportionate representation at only 2 sites would skew the results and due to the timing of the 30-month survey, the
response rate was comparatively low (30.2%).
2
Percentage based on number of eligible clinicians/staff.
Analysis
Upon receipt at JHU, the questionnaires were removed from the envelopes and the envelopes were
discarded to ensure the confidentiality of the respondent. The questionnaires were edited and coded
for computer entry. Quality assurance procedures were utilized. During this process, sites were
consulted, as appropriate, to answer coding questions and provide missing data. Coded
questionnaires were entered into the computer using SPSS-DE. Simple frequencies and binary
analyses were performed using the SAS-PC software package.
37
WORKING DRAFT -- NOT FOR DISTRIBUTION
Much of the information contained in the surveys included items that could be used to construct
scales that measured a broader concept. Examples of such scales include the satisfaction of clinicians
with their ability to meet the developmental needs of families with young children, or an index of
barriers to providing quality well child care related to managed care restrictions and policies. For
many of these scales, the process of identifying items to be combined together was straightforward.
For those for which the process was not straightforward, a factor analysis was performed on baseline
data to evaluate whether there might be some underlying linear structure in the data. This was the
case, for example, for the long instrument assessing perceptions of the care provided at the site.
Factor analysis was only moderately helpful in these instances, so the items were grouped primarily
based on conceptually similar content. Internal consistency was assessed for all scales using a
Cronbach Alpha. Only scales with an alpha value of at least 0.6, the generally accepted lower limit
below which the scale is assumed to not be internally consistent, are reported here.
Negative items were recoded so that the responses to all items in the subscales were in a positive
direction. Therefore, higher scores generally indicated better outcomes. We created mean scores for
the subscales to facilitate interpretation of the results. For questions with response categories based
on a 5-point Likert scale, a mean score of 4.5 or higher indicated that the respondent strongly agreed
or was very satisfied. For questions with response categories based on a 4-point Likert scale, a mean
score of 3.5 was equivalent to strongly agreed or very satisfied.
Because the Affiliate Evaluation did not include a comparison population, chi square analyses were
used to compare differences between groups of respondents at start-up and 30 months. Due to the
small sample sizes, analyses did not adjust for potential differences among sites. Although HS
Specialists completed the survey for nurses and other clinical staff, analyses were conducted
separately for HS Specialists due to their intimate involvement in the program.
Healthy Steps Specialists’ Log of Contacts
At the beginning of the HS evaluation, each HS Specialist was provided with data abstraction forms
and instructions for documenting contacts with the families receiving HS services. Each form was
labeled with the child’s name and HS identification number, and sent to the HS Specialist for
completion. HS Specialists were asked to document every contact they had with the family including
home visits, office visits, telephone calls from or to families, parent groups, mailings or other
contacts. Information requested about each contact included: the date of contact, whether the contact
was completed or not, the person(s) contacted, and actual subjects discussed with the person(s) during
the contact (see Figure A.1 for example of completed log). Up to 15 individual topics could be
recorded by the HS Specialist for any one encounter with a family.
Forms were returned to JHU on a quarterly basis. Trained coders at JHU edited and coded the logs.
During this process, Healthy Steps Specialists were consulted, as appropriate, to answer coding
questions and provide missing data. A comprehensive list of topics discussed with families was
developed; these topics were grouped into six larger categories by members of the team at Boston
University School of Medicine by whom the intervention was designed. The six categories included:
promoting development, nutrition, promoting health, providing family support, injury prevention, and
maternal health. Abstracted data were entered by Sosio Incorporated; 100% were double keyed.
Sample
Fifteen HS Specialists recorded contacts with affiliate families from birth to 32 months. Logs of
contacts were created by HS Specialists for 1,072 families, who comprised 97% of the 1,103 families
enrolled in the Affiliate Evaluation. The sample for analysis excludes 11 families who actively
withdrew from the evaluation, left the practice, or whose child died within 2 months of birth. An
38
WORKING DRAFT -- NOT FOR DISTRIBUTION
additional 49 families were excluded because they had not made a visit to the practice by 2 months
(the families who were back enrolled at the Garden City site were not excluded for this reason). The
final sample included 1,012 families (92% of the total enrolled). The distribution of families varied
across sites, reflecting the number of families enrolled.
Figure A.1
Sample Contact Log
Analysis
For purposes of the analysis, we truncated the sample of logs at 32 months. We split the data into two
time periods. The first period, birth to 14 months, reflected contacts made with families during the
child’s first year of life. We extended this period to 14 months in an attempt to capture the child’s 12month well child visit within the first time period. The second time period consisted of contacts with
the family between 15 and 32 months. As Figure A.2 demonstrates, 100% of families in the selected
sample had a contact with the HS Specialist within the first 2 months of the program. By 15 months
(60 weeks), approximately 77% of families were still actively participating in the program. For this
reason for some analyses, we limited the sample at 15-32 months to those 783 who had a contact on
or after 15 months. Finally, as the figure shows, only 29% of the original 1,012 families were still
participating in the program at 32 months.
Two units of analysis were considered. We conducted some analyses by contact in order to give a
snapshot of the information collected on the contact logs. Most analyses were conducted by family in
order to determine what services were provided to affiliate families. Simple frequencies and bivariate
analyses were performed using the SAS-PC software package.
39
WORKING DRAFT -- NOT FOR DISTRIBUTION
Figure A.2
Distribution of Affiliate Families who had a Recorded Contact with a
HS Specialist over Time
80
60
40
20
96
10
4
11
2
12
0
12
8
88
80
72
64
56
48
40
32
24
16
8
0
0
Percent before given
week
100
Weeks
Parent Questionnaires
As part of the Affiliate Evaluation, parents were asked to complete a brief, standardized form every 6
months beginning when they enrolled their child into the program and ending when the child reached
24 months of age (i.e., newborn, 6, 12, 18, and 24 months). These forms were designed to be selfadministered; however, the HS Specialist may have assisted parents who had difficulty completing
the form. Parents were eligible to complete the form during a window that depended on the child’s
birth date. For example, parents were eligible to complete the 12 month form when their child was 10
months to 16 months old. At some affiliate sites, forms were mailed to families or telephone
interviews were conducted with parents whose child did not make a visit during the window of
opportunity or who did not complete the form at the visit. At a few sites, parents could complete the
form at home before the office visit or during a home visit.
The standardized forms included questions about the parents’ perception of support they received for
child rearing activities from both formal and informal sources, and receipt of practice-based and other
services such as early intervention services and home visits. A series of questions focused on their
engagement in activities that promote their children’s health and learning, and on their use of safety
devices. Parental health behavior items included whether the mother or father smoked. Finally, there
were several questions focusing on the frequency of injuries sustained by the child, emergency
department visits, and hospitalizations in the last 6 months, the age(s) at which the event(s) occurred,
and the reason for the emergency department visit(s) or hospitalization(s).
HS Specialists collected and returned the parent questionnaires to JHU. Once at JHU, the front
identification sheet was removed and processed separately from the questionnaire to maintain the
respondent’s confidentiality. Trained coders at JHU edited and coded the questionnaires, checking
for missing or incomplete data. Sosio Incorporated entered the data. Data analysis was conducted
using SAS.
Sample
Table A.4 gives the total number of affiliate families who completed each parent questionnaire.
There are a number of reasons why questionnaires may not have been returned by parents: 1) families
may have left the practice or withdrawn form the evaluation; 2) site staff may have missed
opportunities to administer the questionnaire; 3) the parent may have declined to complete the
40
WORKING DRAFT -- NOT FOR DISTRIBUTION
questionnaire; or 4) the parent may not have brought the child in for an office visit during the
appropriate age window.
Table A.4
Response Rate to Parent Questionnaires
%
Newborn
6 Months
12 Months
18 Months
24 Months
N
1096
768
607
479
341
Total Enrolled
(n=1103)
99.4
68.7
46.0
43.4
30.9
Analysis
Data at each time period were explored using simple frequencies. However, in order to maximize the
sample size of the parent questionnaires, we conducted trend analyses. These analyses took into
consideration differences across sites and the correlation of individual responses.
A unique feature of the 24-month parent questionnaire was that The MacArthur Communicative
Development Inventories/Word & Sentences (CDI-WS) was incorporated into the questionnaire with
permission from the author1. The questionnaire was administered in the same manner as the other
parent questionnaires. Completed forms were returned to the evaluation team, where they were datestamped, edited, and coded. Approximately 27% of the dates were missing from the completed 24month questionnaires. Because the CDI-WS is age sensitive, missing dates were assigned as the date
of the nearest office visit within the window of opportunity for the questionnaire (23 – 26 months) as
documented on the HS contact logs. Comparison of children with imputed dates and reported dates
showed no significant differences in terms of the child’s age and sex. Age-specific and sex-specific
comparisons of outcomes were also performed. Only one statistically significant difference was
detected; a smaller percentages of boys with imputed dates than those with actual dates combined
words at 24 months. These results gave us confidence in the accuracy of the imputed dates.
Sample
Only those children who were determined to be clients at the practice during the window of
opportunity for the CDI-WS were included in the sample. To be eligible, a child must have visited the
practice or received a home visit from the HS Specialist between 23 and 26 months of age.
Of the 1,103 children originally enrolled in the evaluation at affiliate sites, 547 were determined to be
clients of the practice between 23 and 26 months of age. Response rates varied somewhat by site
(Table A.5). Overall, the parents of 341 (62.3%) of these children completed the CDI-WS. Of these
246 (72.1%) completed the English language version and 95 (27.9%) completed the Spanish
Language version.
1
Fenson L et al. The MacArthur Communicative Development Inventories: User’s guide and technical manual. San Diego, CA: Singular
Publishing Group. 1994.
41
WORKING DRAFT -- NOT FOR DISTRIBUTION
Table A.5
Percentage of Eligible Children with Completed MacArthur CDI-WS by Site
Site
1
2
3
4
5
6
Total
N
%
75
115
36
20
44
51
73.5
72.8
45.0
43.5
74.6
50.0
341
62.3
Sites are listed in random order
We compared the demographic characteristics of respondents to non-respondents (Table A.6).
Respondents differed significantly from non-respondents in ways that may affect results. Among
respondents, mothers had significantly more years of education than did non-respondents, were
generally older, and were less likely to have relied on Medicaid during pregnancy while fathers were
more likely to be employed. However, there were no significant differences in race, Hispanic
ethnicity, birth order, low birthweight, or mother’s employment. These differences suggest that
children in families that completed the MacArthur at affiliate sites may be at less risk for poor
developmental outcomes than families that did not complete it.
42
WORKING DRAFT -- NOT FOR DISTRIBUTION
Table A.6
Comparison of Affiliate Families who Responded to the MacArthur CDI-WS
to Families that Did Not Respond
Respondents
N = 341
%
Non-Respondents
N = 206
%
Mother’s Age*
19 or less
20 – 29
30 or more
20.2
42.8
37.0
26.2
48.5
25.2
Mother’s Education*
Some High School or Less
High School Graduate
Some College, Vocational
College Graduate
40.2
17.0
15.3
27.6
43.6
24.8
15.4
16.3
Mother’s Race
White
Black/African American
Asian/Native American
Other
61.5
10.3
0.6
27.7
51.2
18.5
1.0
29.3
Mother’s Ethnicity—Hispanic
54.0
57.8
Mother Married at Child’s
Birth**
67.5
51.0
First-time Parent
46.9
46.6
Used Medicaid to Cover
Pregnancy/Delivery***
37.7
63.5
Mother Employed Outside Home
32.4
24.4
Father Employed*
85.0
77.7
5.0
3.6
Child Low Birthweight
(<2500 grams)
*p<.05; ** p<.01; *** p<.001
Employment status was not reported for 4.6% of fathers
Parent Telephone Interview
Half way through the Healthy Steps program, when their children were approximately 18 months old,
affiliate families were asked to participate in a telephone interview. The telephone interview updated
demographic information on the family. It also contained questions about parenting practices related
to the child’s nutrition, development, safety, and health care utilization. In addition, parents were
43
WORKING DRAFT -- NOT FOR DISTRIBUTION
asked to comment on the Healthy Steps services they had received from their practice and their
satisfaction with these services. The interview also contained a limited number of questions
regarding the parents’ own health and health behaviors. The telephone interviews were conducted by
Battelle/Centers for Public Health Research and Evaluation using a computer assisted telephone
interview (CATI). CATI not only facilitated the interviewing process but it reduced data errors by
automatically skipping questions based on the respondent’s answers, refusing inconsistent or
“impossible” responses, and entering responses directly into the computer.
Sample
Of the 1,103 affiliate families originally enrolled in the Healthy Steps program, 30% were not eligible
to participate in this telephone survey, the majority because they had moved out of the area. Of the
784 families eligible to participate, 636 families (81%) completed an interview. Eighteen percent of
eligible families (141) could not be located. Only 7 eligible families refused to be interviewed.
Nearly all (99%) of interviews were completed by the study child’s biological mother; 73% of
interviews were conducted in English and 27% in Spanish. Interviews took on average 28 minutes to
complete. Most children were 17-18 months old at the time of the interview (92%) and 93% of study
children still were receiving care at the Healthy Steps site.
Of the mothers interviewed, 12% were less 20 years old; 33% were over 30 years old. At the time of
the interview, 12% of mothers interviewed were in school and nearly half (46%) were employed.
Sixty-two percent were married. Slightly over half (54%) of the mothers interviewed owned their
own homes.
When considering results from these interviews, it is important to keep in mind that a number of
families did not complete this interview; 81% of eligible families completed an interview but that
represents only 58% of the total families enrolled in the Affiliate Evaluation. We know that families
who were not eligible to participate in the interview or who did not complete the interview differed
from the eligible families who completed an interview. As Table A.7 indicates, a statistically
significant greater percentage of mothers who completed the interview at 18 months were older,
better educated and married than mothers who were not eligible to be interviewed or did not complete
an interview. A greater percentage of White mothers completed an interview, while fewer AfricanAmerican mothers did. Mothers did not differ in terms of their ethnicity. Families who completed an
interview tended to be wealthier than families who did not, as measured by the percentage of fathers
employed outside the home and the method of payment for pregnancy and delivery expenses. These
differences may bias the results.
44
WORKING DRAFT -- NOT FOR DISTRIBUTION
Table A.7
Comparison of Affiliate Families who were Not Eligible to be
Interviewed or Did Not Complete a Telephone Interview to Families who
Completed a Telephone Interview at 18 Months
Families Not Eligible/
Did Not Complete
Interview (n=467)
N
%
Families Completed
Interview (n=636)
N
%
Mother’s Age at Child’s Birth***
Less than 20
20-29
30 or older
117
256
87
25.4
55.7
18.9
140
298
196
22.1
47.0
30.9
Mother’s Education at Child’s Birth***
Less than HS
HS Graduate
Some college, vocational school
College graduate or higher
243
111
70
32
53.3
24.3
15.4
7.0
254
130
107
137
40.4
20.7
17.0
21.8
Mother’s Race***
White
African-American
Asian/Native American
Other
186
143
6
123
40.6
31.2
1.3
26.9
356
102
8
166
56.3
16.1
1.3
26.3
Mother’s Ethnicity – Hispanic
239
51.7
340
53.6
Mother Married at Child’s Birth***
202
44.1
380
60.5
Father Employed Outside the Home***
324
73.3
530
87.2
First-time Parent
180
39.0
278
43.8
Used Medicaid to Cover
Pregnancy/Delivery***
306
67.3
294
47.6
Child Low Birthweight (< 2500 gms)
*** p<.001
22
4.9
31
5.0
Analysis
Simple frequencies and bivariate analyses were performed using SAS-PC software package.
Medical Record Reviews
Medical record reviews were conducted for all children enrolled in the Affiliate Evaluation and for a
retrospective sample of children. For each site, the retrospective sample consisted of 100 children
who received care at the practice approximately 1 year prior to the Healthy Steps program. Sites
generated a list of all children who had attended the practice during the year prior to Healthy Steps.
From this list, the first 100 children were selected for the retrospective sample based on the same
criteria as the Healthy Steps sample; that is, they had to have made a visit to the pediatric practice
within the first 28 days of life.
45
WORKING DRAFT -- NOT FOR DISTRIBUTION
The same procedures were followed for abstracting data from a child’s medical record for the Healthy
Steps and retrospective samples. Data were abstracted using detailed protocols designed by the JHU
evaluation team. In order to ensure the consistency of data collection across sites, a member of the
JHU evaluation team provided on-site training in medical review.
For each visit, the reviewer recorded the following information: the date of visit, type of visit and
whether the visit was kept; the child’s height and weight; whether the child received any
immunizations or developmental assessments at the visit and if so, what kind(s); and whether the
child was referred for any services and if so, what kind and for what reason. All visits in the child’s
medical record were abstracted from birth until 14 months. We extended the abstraction through 14
months in an attempt to capture the child’s one-year well child visit.
Sample
In general, the sample used for the analyses of the medical record data consisted of all children who
made two or more visits to the practice, with one of the visits occurring after 6 months. We defined
the sample in this manner in an attempt to identify children who were still receiving care at the
practice at 14 months. The final sample used for the medical record analyses included 1,354 children
(890 children who participated in Healthy Steps and 464 children who had received care at the
practice one year prior to Healthy Steps. This represents 78% of eligible children.
Analysis
Simple frequencies and bivariate analyses were performed using SAS-PC software package.
Age-Appropriate Well Child Visits: We looked at the percentage of children who received a well
child visit at 1, 2, 4, 6, 9, and 12 months of age. The time periods were based on the American
Academy of Pediatrics’ recommended schedule of well child visits for the first year of life. For a
well child visit to be considered age-appropriate, the visit had to occur within a window around each
age. For example, the six-month well child visit had to occur between 5 to 7 months. For each time
period, we had to determine which children were actively seeking health care from the provider or in
other words, who were “at risk” for a well child visit. We defined the eligible sample as children who
had made a visit to the practice, as recorded in their medical record, during or after the previous ageappropriate well child visit window. For example, for a child to be eligible for the six-month ageappropriate well child visit, s/he had to have made a well child visit after 3 months.
Up-to-Date Immunizations: We created a variable to reflect whether a child had received all his/her
recommended immunizations by 12 months. By definition, a child was up-to-date if by 12 months
they had received 3 DTP vaccinations, 2 Polio vaccinations, 3 Hib vaccinations and 2 Hep-B
vaccinations. We did not include the MMR in the definition of up-to-date. The MMR should not be
given prior to 12 months. Because we stopped abstracting the medical record at 14 months, this did
not leave a very wide window for receipt of a MMR. If a child was missing data for any
vaccinations, they were categorized as missing the up-to-date immunization. At one site, a sizeable
number of children were missing the up-to-date report of their HIB vaccinations. We defined the
sample as all children who made 2 or more visits to the practice, with 1 visit occurring after 6 months.
46