!
!
!
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!!
!
Development!and!Testing!of!a!Locally!Adapted!
Psychosocial!Assessment!Instrument!
!
!
!
!
!
!
!
!
!
!
Conducted!through!the!Collaboration!of:!
!
Beyond!Borders!/!Fondasyon!Limyè!Lavi!
!
International!Office!on!Migration!
Centre!d’Action!pour!le!Développement!
Foyer!l’Escale!
Restavèk!Freedom!Foundation!
Ak!
Timoun!k!ap!Teke!Chans!
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!
Author:!Cara!L.!Kennedy,!PhD!
!
!
!
[email protected]! !
!
August!2012!
!
!
!
!
TABLE OF CONTENTS
EXECUTIVE SUMMARY ................................................................................................................................................... 4
PROJECT OBJECTIVES .......................................................................................................................................................... 4
METHODS ............................................................................................................................................................................. 4
RESULTS .............................................................................................................................................................................. 5
DISCUSSION ......................................................................................................................................................................... 5
CONCLUSIONS ...................................................................................................................................................................... 6
RECOMMENDATIONS ........................................................................................................................................................... 7
ACKNOWLEDGEMENTS .................................................................................................................................................. 8
INTRODUCTION ................................................................................................................................................................. 9
BACKGROUND ................................................................................................................................................................... 9
OBJECTIVES ..................................................................................................................................................................... 10
METHODS .......................................................................................................................................................................... 10
DEVELOPING THE DRAFT INSTRUMENT FOR FIELD TESTING ............................................................................................ 10
Instrument Selection....................................................................................................................................................... 10
Instrument Adaptation ................................................................................................................................................... 11
STUDY TEAM ..................................................................................................................................................................... 11
STUDY SITES AND PREPARATION ...................................................................................................................................... 12
PILOT STUDY ..................................................................................................................................................................... 12
RELIABILITY AND VALIDITY STUDY ................................................................................................................................. 13
ANALYSIS .......................................................................................................................................................................... 14
RESULTS............................................................................................................................................................................. 14
SAMPLE CHARACTERISTICS ............................................................................................................................................... 14
Table 1: Study sample characteristics .................................................................................................................................................................. 14
SCALE CHARACTERISTICS ................................................................................................................................................. 15
Table 2: Scale descriptive statistics ..................................................................................................................................................................... 15
INSTRUMENT RELIABILITY ................................................................................................................................................ 15
Internal Consistency Reliability ..................................................................................................................................... 15
Table 3: Cronbach’s alpha scores ........................................................................................................................................................................ 16
Test-Retest Reliability .................................................................................................................................................... 16
Table 4: Test-retest comparison ........................................................................................................................................................................... 17
Criterion Validity ........................................................................................................................................................... 17
ASSESSMENT ISSUES .......................................................................................................................................................... 17
Item Distributions .......................................................................................................................................................... 17
Outlier Analysis ............................................................................................................................................................. 18
Item Analysis .................................................................................................................................................................. 18
Table 7: Item analysis .......................................................................................................................................................................................... 18
Table 8: Descriptive statistics on subscales reduced based on item analysis ...................................................................................................... 19
Internal Consistency Reliability for Reduced Scales ..................................................................................................... 19
Table 9: Cronbach’s alpha scores on subscales reduced based on item analysis ................................................................................................ 19
Test Retest Reliability for Reduced Scales ..................................................................................................................... 20
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
2
!
!
!
!
Table 10: Test-retest comparison on subscales reduced based on item analysis ................................................................................................. 20
DISCUSSION ...................................................................................................................................................................... 20
CONCLUSIONS ................................................................................................................................................................. 22
RECOMMENDATIONS .................................................................................................................................................... 23
APPENDIX A: INFORMED CONSENT PROCEDURE ............................................................................................... 24
APPENDIX B: EXPLANATION OF RELIABILITY AND VALIDITY CONCEPTS ............................................... 28
APPENDIX C: OUTLIER ANALYSES ........................................................................................................................... 30
APPENDIX D: “YOUTH SELF-REPORT – HAITI” (ENGLISH TRANSLATION) ................................................ 32
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
3
!
!
!
!
Executive Summary
Project Objectives
This report describes the development and testing of an instrument to assess mental health and psychosocial problems
among survivors of child slavery (restavèk) living in the Port au Prince metropolitan area, Haiti. Development consisted
of generating a draft instrument that reflects the mental health and psychosocial problems that emerged in a previous
qualitative study among the same target population. Testing consisted of assessing the instrument’s local acceptability,
clarity, validity and reliability among the same population.
The objectives of the work described here are:
1. To develop a draft quantitative assessment instrument based on the psychosocial problems that emerged in the
previous qualitative study of the target population.
2. To test the acceptability, clarity, validity and reliability of this instrument among this same population and
finalize the instrument based on these results.
Methods
The first step was to develop a draft instrument that reflected the salient problems found in the previous qualitative study
among the target population of child survivors of restavèk living in the Port au Prince metropolitan area. We identified an
existing broad-based child measure that matched these criteria - The Youth Self-Report (YSR) which has been used and
studied in more than 80 societies. The YSR was adapted to more closely match the problems identified by our target
population in a previous qualitative study by adding items based on frequently mentioned issues in the qualitative study
that were not already captured in the YSR. No items were removed nor were any changes made in the administration of
the YSR, in order to enable future comparisons with data from other populations in other countries. The additional items
were analyzed as distinct local symptom and functioning scales, as well as in composite scales with YSR items. The
adapted YSR was translated into Haitian Creole with an emphasis on using the same vocabulary as that found in the
qualitative data.
In addition to the adapted instrument, study staff and the interviewers drafted an informed consent procedure to be read to
children prior to administering the instrument and prior to asking whether they agreed to be interviewed.
A pilot study of the instrument (now referred to as the YSR-Haiti or YSR-H) was conducted in order to detect any
problems with the interview procedure, the informed consent procedure and the instrument from the point of view of both
the interviewers and the interviewees, to determine whether the instrument was acceptable and understandable to our
target population, and to give the interviewers practice in interviewing. Based on feedback from interviewers and
interviewees the instrument and informed consent procedures were adjusted as necessary for use in the reliability and
validity study.
The study included assessing the YSR-H’s internal consistency, criterion validity, and test-retest reliability. Interviewees
were recruited from four sites (institutions) in the Port au Prince metropolitan area, all of which provide transitional care
(residential care and education) to children at risk for a period of months to years. Thirty-nine percent of the children were
re-interviewed 2-12 days (mean = 5.7 days) after their first interview in order to assess the YSR-H’s test retest reliability.
Internal consistency reliability was assessed using the Cronbach’s alpha measure. Criterion validity was assessed with the
help of center staff who work with the children on an ongoing basis. Staff and children participated in a process to identify
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
4
Assessment!Instrument!
!
!
!
!
!
children thought to have significant emotional or behavioral problems (Cases) and those thought not to have them (Noncases).
Results
Minor changes to the wording of some items and a more extensive protocol for responding to participants who endorsed
items related to self-harm and/or suicidal ideation were required based on the pilot study results. For the reliability and
validity study, a total of 73 children were assessed, with 67 of these completing the assessment.
The YSR problem based scales have good internal consistency (Cronbach’s alpha >.7 for internalizing, >.8 for
externalizing and total problems). The Haiti scales and Combined scales had alphas >.8 except for the Haiti function scale,
which was >.7. The YSR competence scales had low internal consistency reliability (.4-.5).
The results of test-retest reliability were generally lower. Using .6 as an indicator of adequate test-retest reliability, all
scales except YSR competence and Haiti function demonstrated adequate reliability. Internalizing scales demonstrated
higher test retest reliabilities (>.8) than externalizing scales (>.6). In contrast, the Haiti function scale and the YSR
activities, academic and total competence scales all performed poorly (.2-.5) in terms of test-retest reliability.
Though we attempted to assess criterion validity through sorting participants into those who have mental health problems
(“cases”) and those who don’t (“non-cases”) based on conversations with children and caregivers who know them well, in
many cases there was no agreement between the child report and the adult report in the sorting assessments.
Conventionally, all cases in which there was no agreement between child and caregiver would be excluded from the
study, so that only those where both parties agreed would be used. As a result, we were unable to proceed with a test of
criterion validity and thus cannot speak to the criterion validity of this instrument.
Additional analyses of distributions, outliers, and items suggested items that could be considered for removal from scale
scores based on either very low frequency of response (that may reflect cultural differences in behavior) or very low
correlations with the remaining items on the scale, suggesting that the item is not measuring the scale construct. Analyses
were repeated after removing items suggested by item analysis, and results indicate that the scales function similarly with
and without the indicated items.
In addition to the main study purpose, we also conducted preliminary epidemiological analyses of the study data to
explore scores of the sample on the YSR problem scales. Though norms will need to be established for a Haitian
population, a preliminary comparison has been made with US norms for each scale. For girls, the proportion of the sample
that scored in the borderline clinical to clinical ranges was 70% for internalizing problems and 30% for externalizing
problems. For boys, the proportion of the sample that scored in the borderline clinical to clinical ranges was 95% for
internalizing problems and 50% for externalizing problems. For total problems, 60% of girls and 80% of boys scored in
the borderline to clinical ranges.
Discussion
Reliability and validity testing revealed differences in the performance of YSR-H scales. On this study’s measures of
reliability and validity, we found that the YSR and Combined symptom-based scales (the YSR internalizing,
externalizing, and total problems scales, and the Combined internalizing, externalizing scales, and Combined total
problems scales) have solid psychometric properties in this population. The Haiti-specific scales performed well with the
exception of the Haiti function scale at test-retest. All of the YSR competence scales performed poorly except for the YSR
social competence scale at test retest.
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
5
!
!
!
!
Internal consistency reliability results were good for YSR externalizing, YSR total problems, Haiti symptoms, and the
Combined scales (internalizing, externalizing, and total problems), and acceptable for YSR internalizing problems and
Haiti function scales, indicating that these items on these scales perform well in terms of measuring the same underlying
concept. In contrast, the YSR competence scales (total competence, activities, social, and academic competence
subscales) demonstrated poor internal consistency reliability. As these scales have fewer items and were designed to
address a variety of competencies measured in different formats, the scales may not be assumed to tap a singular
construct.
Test-retest reliability results ranged from good (for the YSR internalizing, YSR total problems, Combined internalizing,
and Combined total problems scales), to desirable (for the YSR social competence subscale), and adequate (for the Haiti
symptoms, YSR externalizing, and Combined externalizing scales). In contrast, the Haiti function scale and the YSR
activities, academic and total competence scales all performed poorly in terms of test-retest reliability. It is important to
point out that while the YSR is a self-administered scale, for our purposes interviewers read the items and completed the
forms with the respondents. We were unable to ensure that the same interviewer was present for each re-interview, and
thus the test-retest results may be confounded by inter-interviewer differences as well. Analyses of distributions indicated
very low frequencies on a number of the rule-breaking behavior items, and it is possible that the externalizing scales were
more affected by social desirability as a function of their content, and/or that the strong cultural norms prohibiting certain
behaviors influenced responses when in the presence of different interviewers.
We were unable to test criterion validity due to the nature of our methods and the low agreement between children and
caregivers in this study regarding emotional and behavioral problems. Research on the use of multiple informants with
children/adolescents coupled with situational and cultural factors may help to explain low to modest levels of agreements
between pairs of informants. Other approaches that have used local mental health professionals’ assessment of children’s
significant emotional or behavioral problems may allow a future examination of criterion validity that may remedy the
problem of low agreement between multiple informants.
In addition to the main purpose of the study - developing an acceptable, reliable, and valid instrument - we also conducted
preliminary epidemiological analyses of the study data. We found that mean scores problem scales were elevated for boys
and girls, except for the externalizing scores for girls, suggesting difficulty across the range of problem categories
assessed by these scales: anxiety, depression, withdrawal, somatic complaints, aggression, social and cognitive problems
and attention problems. In general, boys and girls reported more internalizing problems than externalizing, which may be
explained by cultural and behavioral norms that preclude certain behaviors and/or social desirability effects. Most of the
children in our sample had been in contact with the centers where they were living for a period of months to years. We
would therefore expect that they have already been provided with some degree of support, care and stability and that the
scores of their peers who have been through restavèk and are either in the streets or in another setting in which they do not
have access to the services provided by these centers are likely to have substantially higher problem scores than those of
the children in our study.
Conclusions
Based on our study, the YSR-H is acceptable for use among children who have been in restavèk in Haiti. Overall, the
problem scales of the YSR-H show strong psychometric properties and therefore we believe them to be suitable for use
among this type of child population in Haiti.
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
6
!
!
!
!
These results suggest several options for use of the YSR-H within Haiti. The YSR-H could be used as a screener to help
identify children who need attention, based on the YSR and/or Combined total problems scale. The internalizing and
externalizing scales can be used to explore the nature and severity of the problems affecting each child, and therefore help
to tailor interventions to the child’s needs. The problem scales in the YSR-H can also be used to assess impact of
interventions provided within the centers where children are residing.
The YSR competence scales showed poor psychometric properties suggest that these scales may not be accurate.
Therefore, our results do not suggest that the YSR competence scores are useful either for screening children into
programs or assessing their progress. However, since this is the first time this instrument has been tested, and the study
was done among a population of children with prolonged exposure to transitional care settings, and who therefore may
have experienced marked improvement in their functioning skills due to exposure to these programs, we are not yet ready
to advocate removing the competent functioning items from the instrument. Instead, they should be retained for the time
being and their performance reassessed as part of analyses of future data.
Average problem scores were high and the distribution of scores shows a large proportion of the children falling in the
upper ranges of scores on internalizing, externalizing and total problems. This suggests that most children in our sample
have a wide range of significant psychosocial problems. Since most of our sample has been receiving services for several
months to years, we suspect that children who are not receiving services will have even higher scores.
Recommendations
It is recommended that transitional care settings in Haiti use the YSR-H as a screening tool to identify children in need of
tailored interventions, and to evaluate their progress over time on mental health dimensions.
The goals of these assessments may include:
a) Assessing the nature and severity of needs (by using the instrument as a survey tool).
b) Using this information to target resources and design appropriate interventions.
c) Assessing the impact of these interventions.
When the YSR-H is used in the future, analysis of the resulting data should include further characterization of the
accuracy of the instrument, particularly regarding criterion validity and the performance of the competence scales.
In future use of the YSR-H in community settings, if the length of the scale and time for administration is under
consideration, a number of items could be removed without diminishing the psychometrics of the scales on the
dimensions we assessed in the current study. The YSR Competence items, comprising two pages of the instrument, may
be removed, thus shortening the time of administration considerably. Though there are additional YSR problem-based
items, as well as Haiti symptoms items that could be removed without compromising the subscale psychometric
properties, these items may contribute to criterion validity, and the Haiti symptom items also contribute to content
validity. Thus, we recommend that all symptom-based items be retained pending an assessment of criterion validity.
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
7
!
!
!
!
Acknowledgements
Without the generous support of Equitas Group, this study would not have been possible. We would like to especially
thank Jonathan Scoonover, Program Manager at Equitas Group, for championing and advocating for the financing of this
project. We also thank Beyond Borders for welcoming the study and contributing staff time, organizational resources, and
project oversight. We are particularly grateful to the collaborating organizations, their site directors and staff, and the
interviewers who contributed to this study. Finally, we especially thank the children who provided their time and
participation to make this work possible.
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
8
!
!
!
!
Introduction
This report describes the development and testing of an instrument to assess mental health and psychosocial problems
among survivors of child slavery (restavèk) living in the Port au Prince metropolitan area, Haiti. Development consisted
of generating a draft instrument that reflects the mental health and psychosocial problems that emerged in a previous
qualitative study among the same target population. Testing consisted of assessing the instrument’s local acceptability,
clarity, reliability and validity among the same population.
In the current report, we describe the background to the activities described here, the methods used to develop and test the
instrument and the results of these tests. Conclusions based on the results are also included, as well as specific
recommendations for future activities. Finally, the report ends with three appendices containing the final instrument,
informed consent procedures, and an explanation of some of the technical terms used in the report referring to important
elements of reliability and validity.
Background
The activities described here (instrument development and testing) form part of a series of field-based activities to inform
the design, monitoring, and evaluation (DME) of services based an approach developed by the Johns Hopkins University
(JHU) Applied Mental Health Research (AMHR) Group.1
The DME process developed by the AMHR Group consists of the following stages:
1. Qualitative study of the problems affecting the target population and the tasks and activities that are important to
local people.
2. Development of a locally appropriate quantitative instrument (questionnaire) to assess the major psychosocial
problems emerging from the qualitative study.
3. Evaluation of the acceptability, clarity, validity and reliability of the instrument among the target population, with
subsequent revision of the instrument based on the results.
4. Identification and adaptation of an appropriate intervention to address the major psychosocial problems emerging
from the qualitative study.
5. Use of the final version of the instrument to conduct baseline assessments among individuals recruited into the
intervention.
6. Provision and monitoring of the intervention.
7. Follow-up assessment after participation in the intervention, including re-interview with the assessment
instrument to assess program impact.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1
"Applied!Mental!Health!Research!Group!(2011).!Design,"implementation,"monitoring,"and"evaluation"of"cross6cultural"HIV6related"mental"health"and"psychosocial"
assistance"programs:"A"user’s"manual"for"researchers"and"program"implementers"(Adult"Version)"Module"2:"Developing"quantitative"tools.""
Note:!The!instrument!development!and!validation!process!was!developed!by!the!AMHR!Group!with!the!support!of!the!United!States!Agency!for!International!
Development!(USAID)!Victims!of!Torture!Program.!"
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
9
!
!
!
!
This report describes the process and results of stages 2 and 3 described above and includes recommendations for future
activities. Full details of the qualitative study which constitutes stage 1 (and upon which the current work has been based)
are available at www.BeyondBorders.net.
Objectives
The objectives of the work described here are:
1. To develop a draft quantitative assessment instrument based on the psychosocial problems that emerged in the
previous qualitative study of the target population.
2.
To test the acceptability, clarity, validity and reliability of this instrument among this same population and
finalize the instrument based on these results.
Methods
Developing the Draft Instrument for Field Testing
Instrument Selection
In developing a quantitative instrument one important question is whether to adapt an existing instrument already used in
other populations, or produce an entirely new instrument for local use. Using an existing instrument is preferable if there
is one that adequately reflects the local situation, since use of an existing instrument allows for comparison with other
populations. In reviewing existing instruments, we based our choice on whether the instrument reflected those important
mental health and psychosocial problems that emerged from the previous qualitative study and were amenable to
interventions that are within the resources of the institutions involved in this collaboration who are already providing
services.
The qualitative study revealed that child survivors of restavèk in the Port au Prince metropolitan area experience a wide
range of mental health and psychosocial challenges, which we will refer to as “problems” in this report. These problems
can be categorized as emotional problems (also referred to as internalizing symptoms) such as sadness, crying,
rumination, remembering the bad moments, and being uncomfortable or nervous; behavioral problems (or externalizing
symptoms) such as insulting others, fighting, aggression, or being unruly; and relationship problems such as staying alone,
being subjected to insults or humiliation by others, and problems related to adaptation after being in restavèk. In addition,
a number of other social, economic, and community-level problems for children who have been in restavèk were revealed
in the qualitative study. Those problems are not addressed directly by the current study; however, the emotional and
behavioral symptoms that result from them are. The qualitative results suggested that no one particular problem was more
prominent than the others, and that most children had multiple problems. Therefore, it was decided that any appropriate
instrument would have to be a broadly-based measure that spanned the range of these problems, rather than one that
focuses on a particular symptom or group of symptoms, in order to assess the wide range of problems and symptoms
reported by children.
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
10
!
!
!
!
The Youth Self-Report (YSR), an existing instrument developed in the United States but suitable for use among our target
populations, was selected. The YSR assesses a broad range of psychosocial problems that closely matches those emerging
from the qualitative studies. The YSR is part of a set of instruments developed by the Achenbach System of Empirically
Based Assessment (ASEBA)TM. ASEBA instruments were designed to obtain standardized data on a broad range of
problems from multiple sources (e.g., parents, teachers, and youth ASEBA instruments have been translated into more
than 90 languages, and studied in over 80 societies.2 An entire supplemental manual now exists on the multi-cultural use
of ASEBA instruments, including variations in norms across different countries.3 The YSR is completed by youths
themselves to describe their own functioning and problems. The first 2 pages include demographic and competency
questions on the child’s interests, chores, social interactions, performance in academics, and open-ended questions about
illness, disabilities, problems and concerns. The following 2 pages contain 105 symptom and behavior specific items,
plus 14 positive qualities items, with responses rated on a 0-2 scale (0=not true, 1=somewhat or sometimes true, and 2
very true or often true).
Instrument Adaptation
Through communication with the developers of the YSR, we obtained a license agreement to translate the YSR and
underwent an iterative translation and back-translation process to generate the approved translation of the YSR. After
examining the items from the YSR in relation to the problems described in the qualitative study, all of the original YSR
items were retained in order to maintain the empirical base of the instrument and retain future comparability with results
from children in other countries. However, several additional items were added to the YSR based on frequently mentioned
issues in the qualitative studies that were not already well captured in the YSR. An additional 34 symptoms and 14
function items that were described in the qualitative study but not assessed by the YSR were added to the instrument.
The result was a draft version of the YSR-H (Youth Self-Report – Haiti), which contains sections on assessment of
functioning as well as an assessment of internalizing and externalizing symptoms. The team of Haitian staff conducting
the validation study reviewed each item of the draft instrument to ensure clarity, comprehensibility, and tolerability of the
items. Suggested changes to improve clarity or comprehension of the translation of the YSR items were only made if they
did not change the meaning of the item or its back-translation, so as to maintain fidelity to the approved translation.
During the review of the instrument, several interviewers objected to assessing children on an item assessing thoughts
about sex, as well as to items related to suicidal ideation and past attempt at self-injury or suicide. Interviewers wanted to
be sure that site supervisors were fully aware of the content of the interview before posing these questions. For all three
items, interviewers were assured that site supervisors were aware of the content of the questions and had authorized the
assessment procedures, child participants would be reminded that they would have the right to refuse to answer any
questions, and clear procedures for managing responses endorsing suicidal ideation and/or past attempts were developed
and practiced. Once this was confirmed, interviewers were comfortable asking these questions. A detailed procedure for
managing positive responses to the suicidal ideation/past attempt items was elaborated and interviewers role-played
executing the procedure.
Study Team
The study team consisted of the principal investigator, co-investigator, and five interviewers. In addition, Dr. Paul Bolton,
Associate Scientist at JHU, trained Cara Kennedy in the quantitative methodology and provided distance supervision
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2
"Berube,"R."L.,"&"Achenbach,"T."M."(2012)."Bibliography!of!published!studies!using!ASEBA!instruments:!2007!edition."Burlington,"VT:"University"of"Vermont,"Research"
Center"for"Children,"Youth"and"Families."
3
"Ivanova,"M."Y.,"Achenbach,"T."M.,"Rescorla,"L."A.,"et"al."(2007)."The"generalizability"of"the"Youth"Self6Report"syndrome"structure"in"23"societies.!Journal!of!Consulting!
and!Clinical!Psychology,!75"(5),"7296738."
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
11
!
!
!
!
periodically throughout data collection and analysis. Three of the interviewers were staff of the International Office on
Migration focused on psychosocial support and reinsertion for children who have been in restavèk. Of the remaining two,
one was an interviewer for the qualitative study, and the other is a professional trained in psychology with experience
working with both child and adult survivors of violence and restavèk.
Study Sites and Preparation
This study took place in four centers (institutions) in the Port au Prince metropolitan area. Two of these centers, Foyer
l’Escale and Restavèk Freedom Foundation, participated in the qualitative study. All four centers provide transitional care
to children at risk, providing housing and education for children for a period of months to years. Foyer l’Escale and
Restavèk Freedom Foundation work exclusively with children who have been in restavèk. The remaining two, Centre
d’Action pour le Développement (CAD) and Timoun K ap Teke Chans (Timkatec), work with children in difficulty for a
variety of reasons, with a portion of those children being children who have been in restavèk.
In the months leading up to the study, we worked with the four sites to build an understanding of the study and study
methods, and asked them to participate in a “sorting” process to help us to assess criterion validity. In previous studies
conducted by the AMHR Group in other parts of the world, part of the validity testing procedure has involved comparing
scores from the proposed instrument with the overall opinion of the child and caregivers as to whether or not they thought
they had a problem. Children who stated that they did have psychosocial problems should (if the instrument is valid)
report more symptoms and greater severity of symptoms compared with children who stated they did not have
psychosocial problems. In the study model used by the AMHR Group, a term or phrase generated in the qualitative study
is used to describe a state of having the psychosocial problem, and are used in the form of question to identify whether the
child had a psychosocial problem or not.
In the qualitative study in Haiti we did not identify any suitable, consistently-used or general term to describe a child with
the kinds of symptoms that emerged in the study. In discussions with field staff, no term could be agreed upon that would
accurately and consistently identify children who have mental health or psychosocial problems. Therefore, we decided to
use a general question asking whether the child’s emotional and behavioral functioning is in the normal range for a child
of his/her age or whether it is beyond the normal range, and the field staff provided the Haitian translation for this
question.
Interviewers were trained to conduct a short conversation with each child participant and the adult who was determined to
know the child best. This conversation was conducted as a means of generating lists of interviewees who were thought to
have the problems being assessed by the instrument (according to self-report and by report of an adult who knew them
well) and those not likely to have those same problems. These lists were generated prior to the study.
Pilot Study
Following the instrument translation and review, the draft YSR-H was ready for piloting in the field. The objectives of
the pilot study were to detect any problems with the interview procedure, the informed consent procedures and the
instrument from the point of view of both the interviewers and the interviewees, to determine whether the instrument was
acceptable and understandable to our target population, and to give the interviewers practice in interviewing.
The complete questionnaire consisted of two sections: the YSR in its entirety, which included a brief series of
demographic questions, items assessing the child’s competence in social, academic, and activity domains, and items
assessing mental health problems; and items assessing emotional/behavioral problems as well as positive aspects of
functioning that emerged from the qualitative study (See Appendix D for a copy of the finalized YSR-H).
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
12
Assessment!Instrument!
!
!
!
!
!
In addition, the study team translated a verbal informed consent procedure containing a description of the study, potential
risks and benefits of participation, a statement on confidentiality procedures, and contact information for any questions
that was read to children prior to administering the instrument and prior to asking whether they agreed to be interviewed.
(See Appendix A for a copy of the informed consent procedure).
On the first day of the pilot study each interviewer interviewed at least two children, not from the study sites, using the
draft instrument. The interviewing process was conducted as it would be during they study interview. Once this was
completed, the interviewer then asked additional questions about what the interviewee liked and did not like about the
interview process and whether they had difficulty understanding any questions. After conducting pilot interviews, the
interviewers returned to the training site to review the experience with the research team and each other, including a
review of the reactions of the interviewees and their responses to the additional questions. Any issues that emerged from
the pilot interviews were discussed. Based upon the pilot study, interviewers suggested minor modifications to the
instrument items where participants found them difficult to understand. Suggested changes to the translation of the YSR
items were only made if they did not change the meaning of the item or its back-translation, so as to maintain fidelity to
the approved translation. As expected, questions based on the qualitative data were mostly well understood. Only one
pilot study participant expressed discomfort with the item related to thoughts about sex. There were no instances of a
problem or concern occurring in multiple interviews, and thus there were no substantive changes made to the instrument.
Reliability and Validity Study
Following the pilot study interviewers commenced interviewing for the reliability and validity study, using the procedures
and instrument finalized at the end of the pilot study at all four study sites.
The purpose of the reliability and validity study was to determine if the YSR-H could accurately assess the presence and
severity of the mental health and psychosocial problems selected from the qualitative study. Reliability and validity
testing included assessment of the following instrument characteristics4:
1. Internal consistency reliability
2. Test retest reliability
3. Criterion validity
Test-retest reliability was evaluated by re-interview of approximately 39% (n=27) of the participants 2-12 days (mean =
5.7 days) after their first interview. For a portion of this sample, the instrument was administered by a different
interviewer at retest.5
One major focus of validity testing was to explore criterion validity. This refers to the extent to which the instrument
agrees with another method of assessing mental health problems that is known to be accurate. Since we have no such
method of known accuracy, we compared data from the instruments with the assessments by the survivors and adults
close to them as to whether they have problems related to their behavior and affect. While this is not a standard of known
accuracy, and both the survivor and other adult may be incorrect in many cases, we assume that a survivor is more likely
to have or not have each of these problems if the survivor and the other adult agree. Those said to have such problems
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
4
"Brief!explanations!of!each!of!these!parameters!are!provided!in!Appendix!B."
!Although!the!YSR!is!a!self^administered!instrument,!due!to!literacy!levels!the!items!were!read!to!all!children!except!those!who!demonstrated!adequate!literacy!and!
willingness!to!self^administer.!Following!ASEBA!administration!guidelines!(see!Manual"for"the"ASEBA"School6Age"Forms"&"Profiles),!interviewers!read!the!items!to!
respondents!and!wrote!down!their!answers.!If!respondents!could!read!at!all,!respondents!were!encouraged!to!read!from!the!form!while!the!interviewer!read!the!
questions.!!!
5
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
13
!
!
!
!
should therefore have higher scores than those said not to have such problems, if both the instrument and the opinions of
survivors and other adults are accurate assessment of the presence and severity of these syndromes.
Analysis
Based on extensive research,6 scales, or lists of problems that tend to co-occur, have been constructed for the YSR. We
have used two of these established YSR scales that most closely matched the conceptual distinction between symptoms
that emerged in the qualitative study: the YSR Internalizing scale includes symptoms of anxiety, depression, withdrawal,
and somatic complaints; the YSR Externalizing scale includes rule breaking and aggressive behaviors. Additional YSR
items in the total symptoms scale include social, attention and thought problems.
The reliability and validity analyses were based on scale scores on the instrument, both the standardized (YSR) scales and
the additional scales that emerged from the qualitative study that were not reflected in the YSR. Each of the symptom
items on the YSR and that emerged from the qualitative study was assessed on a 3-point Likert scale (0 = Not true; 1 =
Somewhat or Sometimes True; 2 = Very True or Often True).
All data analysis was conducted using STATA statistical software. Analysis of validity and reliability included
measurement of internal consistency reliability using Cronbach’s alpha and of test-retest correlation using the Pearson
correlation coefficient. Criterion validity was assessed by comparing scale scores for the survivors said by themselves and
an adult to have affective/behavioral problems with scores of survivors said by themselves and others to not have them.
RESULTS
Sample Characteristics
Interviewers assessed 75 children/youth and the adults close to them across the four study sites. Of the 75 children/youth,
one child left the center prior to the time of the validity study, one refused the assessment, three refused less than halfway
through the instrument, and the data for an additional three were deemed invalid based on the response patterns. The
characteristics of the remaining 67 participants are provided in Table 1.
Table 1: Study sample characteristics
Total N=67
Gender
Male
Female
Ages*
9-11
12-14
15-17
Sites
#1. Foyer L’Escale
N (%)
20 (30%)
47 (70%)
16 (24%)
34 (51%)
15 (22%)
29 (43%)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
6
"Achenbach,!T.!M.,!&!Rescorla,!L.!A.!(2001)."Manual"for"the"ASEBA"School6Age"Forms"&"Profiles.!Burlington,!VT:!University!of!Vermont,!Research!Center!for!Children,!
Youth,!and!Families.!!
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
14
!
!
!
!
#2. Restavèk Freedom
#3. Centre pour le Developpement et Action
#4. Timoun K ap Teke Chans (TIMKATEC)
6 (9%)
23 (34%)
9(13%)
*Missing age data for two participants
Scale Characteristics
The reliability and validity analyses were based on twelve scales. As the YSR is a highly regarded and widely-used
instrument, we first conducted analyses of the YSR Subscales (Internalizing Problems, Externalizing Problems, Activities,
Social, and Academic Competence) and Total scales (Total Problems and Total Competence). Subsequently, we analyzed
Haiti Symptom and Function scales. Finally, we combined analyses of YSR and Haiti items along Internalizing and
Externalizing dimensions as well as for Total Problems. The descriptive statistics for the twelve scales are provided in
Table 2.
Table 2: Scale descriptive statistics
#
of
Items
Mean
SD
Min
Max
YSR Subscales
YSR Internalizing Problems
YSR Externalizing Problems
YSR Activities Competence
YSR Social Competence
YSR Academic Competence
31
32
6
6
1
23.76
13.83
9.00
7.23
2.14
8.21
7.59
2.61
2.59
0.51
8
0
3.5
0
1
44
31
14.00
11.67
3
YSR Total Scales
YSR Total Problems
YSR Total Competence
105
13
68.90
18.38
24.42
3.98
22
8.83
133
25.67
Haiti Scales
Haiti Symptom*
Haiti Function*
34
14
21.14
22.56
8.77
4.41
2
13
41
28
Combined Scales
Combined YSR and Haiti Internalizing*
Combined YSR and Haiti Externalizing*
YSR-Haiti Total Problems*
48
40
139
35.45
17.23
89.85
12.81
9.65
31.30
10
0
28
63
39
165
*Not all respondents have complete data. Data presented for only those with complete data.
Instrument Reliability
Internal Consistency Reliability
Internal consistency reliability measures the extent of agreement among questions that assess the same underlying
concept. If questions believed to be measuring the same concept disagree this suggests that either the questions
themselves are unreliable, or they are not really measuring the same concept. Internal consistency reliability is measured
using Cronbach’s alpha, for which scores should be at least .7 and ideally between .08-.09.
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
15
!
!
!
!
Table 3 shows the Cronbach’s alpha scores on each of the scales for males, females, and the total sample.
Table 3: Cronbach’s alpha scores
Total Sample
Total Sample
(N=67)
Males
(N=20)
Females
(N=47)
YSR Subscales
YSR Internalizing Problems
YSR Externalizing Problems
YSR Activities Competence
YSR Social Competence
.753
.802
.584
.574
.740
.777
.723
.682
.776
.802
.526
.530
YSR Total Scales
YSR Total Problems
YSR Total Competence Scale
.834
.527
.866
.668
.853
.370
Haiti Scales
Haiti Symptoms*
Haiti Function Items*
.817
.768
.801
.785
.830
.770
Combined Scales
Combined YSR and Haiti Internalizing*
Combined YSR and Haiti Externalizing*
YSR-Haiti Total Problems*
.844
.846
.882
.830
.816
.902
.871
.844
.897
*Not all respondents have complete data. Data presented for only those with complete data.
Alpha scores for all problem-based scales are in the acceptable range, and are good for YSR externalizing problems, YSR
total problems, Haiti symptoms, and the Combined scales (internalizing, externalizing, and total problems). None of the
competence-based scales (total competence and subscales) are in the acceptable range.
Test-Retest Reliability
Table 4 shows test-retest reliability analysis results, based on the 27 (39%) interviews that were repeated 2-12 days
(mean=5.7 days) after the preliminary interview. In ten (37%) of the 27 cases, the instrument was administered by a
different interviewer at retest. Test-retest reliability is assessed using the Pearson correlation coefficient, which provides a
measure of how similar each scale score is on the first and second interviews. This provides an indicator of the extent to
which respondents tend to give the same answer to the questions constituting the scale when asked on different occasions.
For each comparison, a scatterplot of the scale scores on the first interview was compared with those on the second
interview in order to determine whether there was a linear relationship and therefore whether the Pearson correlation
coefficient was an appropriate measure. For all comparisons the scatterplot suggested a linear relationship, confirming that
the Pearson correlation co-efficient was an appropriate measure of test-retest reliability.
When assessing test-retest reliability, Pearson correlation coefficient scores of .7 are considered to be desirable, .6
adequate. On that basis, five of the twelve scales performed well, in the desirable range - YSR internalizing, YSR social
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
16
!
!
!
!
competence, YSR total problems, Combined internalizing, and Combined total problems- and an additional three scales
demonstrated adequate test-retest reliability (YSR externalizing, Combined externalizing, and Haiti symptoms).
Table 4: Test-retest comparison
Mean (sd)
First
Interview
Mean (sd)
Repeat
Interview
Correlation
YSR Subscales
YSR Internalizing Problems
24.15 (9.29)
19.67 (9.76)
.835
YSR Externalizing Problems
YSR Activities Competence
YSR Social Competence
YSR Academic Competence
13.33 (6.78)
10.72 (1.59)
7.67 (2.11)
2.20 (.45)
11.78 (8.93)
11.20 (1.40)
7.96 (1.47)
2.22 (.40)
.628
.371
.731
.213
YSR Total Scales
YSR Total Problems
YSR Total Competence
70.48 (23.86)
20.59 (3.19)
53.19 (27.40)
21.30 (1.83)
.834
.544
Haiti Scales
Haiti Symptoms
Haiti Function Items
20.52 (8.38)
22.67 (4.48)
20.15 (10.15)
23.81 (3.63)
.689
.410
YSR-Haiti Subscales
Combined YSR and Haiti Internalizing
Combined YSR and Haiti Externalizing
YSR-Haiti Total Problems
35.22 (14.10)
17.15 (8.84)
91.70 (30.51)
29.74(15.60)
16.00 (12.13)
78.85 (39.47)
.827
.617
.834
Criterion Validity
Though we attempted to assess criterion validity through sorting participants into those who have mental health problems
(“cases”) and those who don’t (“non-cases”) based on conversations with children and caregivers who know them well, in
many cases there was little agreement between the child report and the adult report in the sorting assessments.
Conventionally, all cases in which there was no agreement between child and caregiver would be excluded from the
study, so that only those where both parties agreed would be used. As a result, we were unable to proceed with a test of
criterion validity.
Assessment Issues
Item Distributions
Analysis of the distributions of scores on all items was completed. Items in which >= 90% of the sample responded on
one of the extremes of the response options (i.e., either not at all, or very true) were examined for content. Six YSR items
and one Haiti symptom item met criteria for examination
Five of the six YSR items were from the Rule-Breaking Behavior syndrome of the Externalizing subscale (items 72, 81,
82, 99, 105 – content pertaining to fire-setting, stealing, tobacco and drug use), and the remaining item related to picking
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
17
!
!
!
!
skin. In all six cases more than 90% of the sample responded “not true” to these items. The one Haiti item (item 19)
pertained to not wanting to attend school, and again over 90% of the sample responded “not true.” The content of these
items relates to behaviors that would be seen as unacceptable or shameful by Haitian cultural norms, which may explain
the lower rates of exhibiting and/or endorsing these items. However, it is not recommended that these items be removed as
their endorsement might indicate a severe degree of problem behavior that would merit attention. Thus, no items were
excluded on the basis of the distribution of responses.
Outlier Analysis
For each scale, box plots were created to identify outlying data points. For each scale the number and ID of the outlier
cases were noted at baseline (n=67) and retest (n=27). Based upon these analyses, there were no cases that were outliers at
both time points on a scale, nor on multiple scales, except when the scale for which they were outliers was also included
in a composite score. Without any other rationale that would explain their difference, none of these cases were removed
from analyses. Outlier analyses were also conducted on the difference scores between test and retest on each scale. Cases
with large differences in scores between test and retest raise questions as to what would explain these differences.
However, without observational data or other explanations as to situations that may have occurred between test and retest
that would explain the differences, and because there are no cases whose difference scores were outliers on multiple scales
that were not composites of one another, we cannot justify their removal from analyses. (See Appendix C for Outlier
Analyses of scale scores and difference scores).
Item Analysis
Item analysis was completed on all scales for the full sample to determine the reliability of each question by calculating
the correlation of the item with the entire scale, the correlation of the item with the rest of the items in the scale, and the
overall reliability of the scale without the item. Significant increases in alpha without the question and/or low correlation
between the item and the rest of the items in the scale would suggest that the item in question is not measuring the same
construct as the other items and should be removed. Items in which item-rest correlation was either negative or < .1 were
removed from their scales, and the reduced scales were analyzed. The YSR competence subscales were not subjected to
this type of item analysis due to the small number of items in each subscale. Additionally, the YSR total problems scale
was not analyzed in this way because it is by its nature a measure of different types of problems.
Table 7: Item analysis
Subscale
Item #
YSR Internalizing Problems
YSR Externalizing Problems
YSR Total Competence
# of Items
Removed
5
3
2
Haiti Symptoms*
Haiti Function Items*
3
0
17, 31, 34
Combined YSR and Haiti Internalizing*
Combined YSR and Haiti Externalizing*
6
3
Y5, Y14, Y30, Y32, Y42, H4
Y3, Y63, Y101
5, 14, 30, 32, 42
3, 63, 101
II. Sum Activities, VII. Mean Academic
*Not all respondents have complete data. Data presented for only those with complete data.
The descriptive statistics for the seven reduced scales are provided in Table 8.
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
18
!
!
!
!
Table 8: Descriptive statistics on subscales reduced based on item analysis
#
of Mean
SD
Items
YSR Subscales
YSR Internalizing Problems
26
19.33
7.96
YSR Externalizing Problems
29
12.10
7.49
YSR Total Competence
11
14.15
3.41
Min
Max
3
0
5.5
38
30
20.3
Haiti Scales
Haiti Symptoms*
Haiti Function Items*+
31
14
19.91
22.56
8.67
4.41
1
13
38
28
YSR-Haiti Subscales
Combined YSR and Haiti Internalizing*
Combined YSR and Haiti Externalizing*
42
37
30.65
15.48
12.50
9.44
6
0
57
38
*Not all respondents have complete data. Data presented for only those with complete data.
+Scale was not reduced.
Internal Consistency Reliability for Reduced Scales
Table 9 shows the Cronbach’s alpha scores on each of the scales for the total sample. All alphas were in the acceptable
range except for the YSR total competence scale. Alphas were good for the YSR externalizing, Haiti symptoms,
Combined internalizing and Combined externalizing scales. Relative to the full set of items, the alpha improved with the
removal of the items in question for all scales.
Table 9: Cronbach’s alpha scores on subscales reduced based on item analysis
Total
Sample
(N=67)
YSR Subscales
YSR Internalizing Problems
.784
YSR Externalizing Problems
.819
YSR Total Competence
.558
Haiti Scales
Haiti Symptoms*
Haiti Function Items*+
.826
.768
YSR-Haiti Subscales
Combined YSR and Haiti Internalizing
Combined YSR and Haiti Externalizing
.855
.861
*Not all respondents have complete data. Data presented for only those with complete data.
+Scale was not reduced.
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
19
!
!
!
!
Test Retest Reliability for Reduced Scales
Test Retest reliability was recalculated for the reduced scales. Correlations are good for the Combined internalizing scale
and acceptable for the YSR internalizing scale. Correlations are in the adequate range for YSR externalizing, Haiti
symptoms, and Combined externalizing scales, and were poor for YSR total competence and Haiti function scales. Testretest reliability improved by reducing the scales for the YSR externalizing, YSR total competence, and Combined
externalizing scales, but was not improved for the YSR internalizing, Haiti symptoms, or Combined internalizing scales.
Table 10: Test-retest comparison on subscales reduced based on item analysis
Mean (sd)
Mean (sd)
First
Repeat Interview
Interview
YSR Subscales
YSR Internalizing Problems
19.37 (9.25)
16.07 (8.20)
YSR Externalizing Problems
11.56 (6.66)
10.04 (8.46)
YSR Total Competence
15.93 (2.69)
16.46 (1.59)
Correlation
.792
.648
.577
Haiti Scales
Haiti Symptoms
Haiti Function Items+
19.67 (8.73)
22.67 (4.48)
18.78 (9.85)
23.81 (3.63)
.662
.410
YSR-H Subscales
Combined YSR and Haiti Internalizing
Combined YSR and Haiti Externalizing
30.22 (14.01)
15.33 (8.59)
25.74 (13.67)
14.56 (12.00)
.811
.626
+Scale was not reduced.
Discussion
Reliability and validity testing revealed differences in the performance of YSR-H scales. On this study’s measures of
reliability and validity, we found that the YSR and Combined YSR and Haiti symptom-based scales (the YSR
internalizing, externalizing, and total problems scales, and the Combined internalizing, externalizing scales, and
Combined total problems scales) have solid psychometric properties in this population. The Haiti-specific scales
performed well with the exception of the Haiti function scale at test-retest. All of the YSR competence scales performed
poorly except for the YSR social competence scale at test-retest.
Internal consistency reliability results were good for YSR externalizing, YSR total problems, Haiti symptoms, and the
Combined scales (internalizing, externalizing, and total problems), and adequate for YSR internalizing problems and Haiti
function scales, indicating that these items on these scales perform well in terms of measuring the same underlying
concept. In contrast, the YSR competence scales (total competence, activities, social, and academic competence
subscales) demonstrated poor internal consistency reliability. As these scales have fewer items and were designed to
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
20
!
!
!
!
address a variety of competencies measured in different formats, the scales may not be assumed to tap a singular
construct.
Test-retest reliability results ranged from good (for the YSR internalizing, YSR total problems, Combined internalizing,
and Combined total problems scales), to desirable (for the YSR social competence subscale), and adequate (for the Haiti
symptoms, YSR externalizing, and Combined externalizing scales). In contrast, the Haiti function scale and the YSR
activities, academic and total competence scales all performed poorly in terms of test-retest reliability. It is important to
point out that while the YSR is a self-administered scale, for our purposes interviewers read the items and completed the
forms with the respondents. We were unable to ensure that the same interviewer was present for each re-interview, and
thus the test-retest results may be confounded by inter-interviewer differences as well. Analyses of distributions indicated
very low frequencies on a number of the rule-breaking behavior items, and it is possible that the externalizing scales were
more affected by social desirability as a function of their content, and/or that the strong cultural norms prohibiting certain
behaviors influenced responses when in the presence of different interviewers.
Though we attempted to assess criterion validity through sorting participants into those who have mental health problems
(“cases”) and those who don’t (“non-cases”) based on conversations with children and caregivers who know them well, in
many cases there was no agreement between the child report and the adult report in the sorting assessments which
precluded our proceeding with the test of criterion validity. There are a number of reasons why this may have occurred in
the current study. First, research on the use of multiple informants with children/adolescents, including meta-analyses of
studies using the Achenbach scales for self, parent/caregiver, and teacher report have indicated low to modest levels of
agreements between pairs of informants, especially between self ratings and ratings of others, the case in our study.7,8
inverse).These results vary based role of the informants and situation in which they observe the child (self, mother, father,
teacher, etc.), the age of the child and type of behavioral/emotional problem. This may result from differences in
informants’ observations as well as differences in children’s behavior in different contexts (e.g., some children manifest
problems at home but not at school, some the The low to modest correlations across reporters have provided a strong basis
for the use of data that combines multiple informants, and/or assesses children on multiple axes that are designed to reflect
and capture the variations that surface across multiple informants. A sorting of cases and non-cases in our current study
could not capture these variations nor capitalize on the different perspectives of multiple informants.
Second, a good test of criterion validity using the approach in the current study is dependent upon respondents being
willing to provide a definite answer to the question being asked, and that the question being asked is fairly highly
correlated with the underlying concept being assessed by the instrument. In our case, because there was no commonly
known and used term for either internalizing or externalizing problems, nor for mental health or psychosocial problems
more generally, we had to use a phrase that described problems related to behavior and affect, without being more specific
about the nature of the problems. This is a limitation to our ability to assess criterion validity, as an unclear criterion, or a
criterion open to a wide range of interpretation, would be unlikely to yield to a strong test of criterion validity.
Third, situational and cultural factors also may have influenced both child and adult responses to the criterion validity
sorting process. In particular, interviewers noticed reluctance on the part of some staff to categorize children in what they
perceived to be negative terms, even when children themselves reported they had the problems being asked about. In
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
7
"Achenbach,"T."M.,"McConaughy,"S."H.,""&"Howell,"C."T."(1987)."Child/Adolescent"behavioral"and"emotional"problems:"Implications"of"cross6informant"correlations"for"
situational"specificity."Psychological!Bulletin,!101!"(2),"2136232.""
8
"Renk,"K.,"&"Phares,"V."(2003)."Cross6informant"ratings"of"social"competence"in"children"and"adolescents."Clinical!Psychology!Review,!24!"(2),"2396254."
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
21
!
!
!
!
some cases, caregivers wished to place children in an intermediate category, rather than selecting one of the two response
options, which interviewers interpreted as a means of politely categorizing the child as one with problems.
Though we were unable to test criterion validity due to the nature of our methods and the low agreement between children
and caregivers in this study regarding emotional and behavioral problems, other approaches that have used local mental
health professionals’ assessment of children’s significant emotional or behavioral problems may allow a future
examination of criterion validity. Research on the use of multiple informants with children/adolescents coupled with
situational and cultural factors may help to explain low to modest levels of agreements between pairs of informants which
may be remedied through the use of trained local professional assessment.
In addition to the main purpose of the study - developing an acceptable, reliable, and valid instrument - we also conducted
preliminary epidemiological analyses of the study data. We found that mean scores problem scales were elevated for boys
and girls, except for the externalizing scores for girls, suggesting difficulty across the range of problem categories
assessed by these scales: anxiety, depression, withdrawal, somatic complaints, aggression, social and cognitive problems
and attention problems. In general, boys and girls reported more internalizing problems than externalizing, which may be
partially explained by cultural and behavioral norms that preclude certain behaviors and/or social desirability effects.
Most of the children in our sample had been in contact with the centers where they were living for a period of months to
years. We would therefore expect that they have already been provided with some degree of support, care and stability
and that the scores of their peers who have been through restavèk and are either in the streets or in another setting in
which they do not have access to the services provided by these centers are likely to have substantially higher problem
scores than those of the children in our study.
Conclusions
Based on our pilot study, the YSR-H is acceptable for administration among children who have been in restavèk in Haiti.
Overall, the problem scales of the YSR-H show strong psychometric properties and therefore we believe them to be
suitable for use for this child population in Haiti.
These results suggest several options for use of the YSR-H within Haiti. The YSR-H could be used as a screener to help
identify children who need attention, based on the YSR and/or Combined total problems scale. The internalizing and
externalizing scales can be used to explore the nature and severity of the problems affecting each child, and therefore help
to tailor interventions to the child’s needs. The problem scales in the YSR-H can also be used to assess impact of
interventions provided within the centers where children are residing.
The YSR competence scales showed poor psychometric properties suggest that these scales may not be accurate.
Therefore, our results do not suggest that the YSR competence scores are useful either for screening children into
programs or assessing their progress.
Average problem scores were high and the distribution of scores shows a large proportion of the children falling in the
upper ranges of scores on internalizing, externalizing and total problems. This suggests that most children in our sample
have a wide range of significant psychosocial problems. Since most of our sample has been receiving services for several
months to years, we suspect that children who are not receiving services will have even higher scores.
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
22
!
!
!
!
Recommendations
It is recommended that transitional care settings in Haiti use the YSR-H as a screening tool to identify children in need of
tailored interventions, and to evaluate their progress over time on mental health dimensions.
The goals of these assessments may include:
a) Assessing the nature and severity of needs (by using the instrument as a survey tool).
b) Using this information to target resources and design appropriate interventions.
c) Assessing the impact of these interventions
When the YSR-H is used in the future, analysis of the resulting data should include further characterization of the
accuracy of the instrument, particularly regarding criterion validity and the performance of the competence scales.
In future use of the YSR-H in community settings, if the length of the scale and time for administration is under
consideration, a number of items could be removed without diminishing the psychometrics of the scales on the
dimensions we assessed in the current study. The YSR Competence items, comprising two pages of the instrument, may
be removed, thus shortening the time of administration considerably. Though there are additional YSR problem-based
items, as well as Haiti symptoms items that could be removed without compromising the subscale psychometric
properties, these items may contribute to criterion validity, and the Haiti symptom items also contribute to content
validity. Thus, we recommend that all symptom-based items be retained pending an assessment of criterion validity.
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
23
!
!
!
!
Appendix!A:!Informed!Consent!Procedure !!
9
Appendix A1: Haitian Creole Version
Konsantman Vèbal pou Etid ak Timoun!
Nòt pou moun k ap fè entèvyou:
Pati ki ekri an gra se pou nou li yo pou moun nou vle pale avèk yo anvan nou kòmanse fè entèvyou a. Si patisipan an
dakò li vle patisipe, fòk ou siyen sou liy ki make “Temwen pou Pwosedi Konsantman” an ba. Anplis, pa bliye note dat
entèvyou a anba.
Bi Etid la
N ap fè yon etid ki pral pèmèt nou aprann plis sou pwoblèm timoun epitou sou sa timoun an jeneral fè. Lè nou
aprann sou pwoblèm ak aktivite timoun, Beyond Borders/Limyè Lavi ak lòt òganizasyon k ap travay ak timoun
espere kreye pwogram ki kapab pi byen ede timoun ak fanmi yo. Beyond Borders / Limyè Lavi ap fè etid la, an
kolaborasyon ak OIM, Foyer L’escale, CAD, e ak Restavèk Freedom. Nou ta renmen envite ou patisipe.
Demach Etid la
Pou nou reyalize etid la, n ap pale ak timoun, e ak granmoun ki okipe timoun. Se poutèt sa nou te chwazi ou pou
patisipe. Si ou dakò pou patisipe, m ap poze ou kèk kesyon sou ou.
Risk ak Malèz Potansyèl
Chak entèvyou ap mande 30 al 60 minit. Gen de moun ki santi yo malalèz lè moun poze yo kalite kesyon sa yo.
Kapab gen kèk kesyon ou renmen oubyen ki fè ou malalèz. Ou gen dwa refize reponn kesyon sa yo, oubyen nenpòt
lòt kesyon si ou vle. Ou gen dwa kanpe entèvyou a nenpòt lè ou vle. Si ou santi ou malalèz apre entèvyou a fini, nou
gen yon moun nan ekip nou ki kapab pale avèk ou apre entèvyou a.
Benefis
Enfòmasyon sa a ap ede Beyond Borders / Fondasyon Limyè Lavi / OIM enfòme lòt pwogram ak sèvis ki kapab
amelyore sèvis pou timoun. Men, sa pa vle di ap genyen yon benefis dirèk pou ou pou patisipasyon ou.
Aspè Volontè
Ou pa oblije patisipe nan etid sa a. Pa gen yon benefis dirèk pou ou pou patisipasyon ou. Pa gen okenn konsekans
negatif si ou refize patisipe, ni si ou deside patisipe kounye a men ou deside sispann patisipe pi devan.
Konfidansyalite
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
9
!Adapted!with!permission!from!Applied!Mental!Health!Research!Group!(2011).!Design,"implementation,"monitoring,"and"evaluation"of"cross6cultural"HIV6related"
mental"health"and"psychosocial"assistance"programs:"A"user’s"manual"for"researchers"and"program"implementers"(Adult"Version)"Module"2:"Developing"quantitative"
tools.!
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
24
!
!
!
!
Pandan entèvyou a, m ap ekri enfòmasyon ou pataje avè m. Se enfòmasyon sa a n ap itilize pou etid la. Rejis
enfòmasyon an p ap gen anyen ladann li ki kapab idantifye ou. Se selman ekip etid la k ap gen dwa gade
enfòmasyon yo. Pa gen anyen nan sa ou pataje avèk nou ki ap pataje ak lòt moun ki pa fè pati ekip etid la. Nou p
ap pèmèt pèsonn konnen kiyès ki te bay yon repons, sòf o ka ou nou panse ou kapab an danje. Nan ka sa a, nou pa
kapab kenbe enfòmasyon sa a an konfidansyalite, epi n ap fè demach ki nesesè pou nou asire nou tout moun an
sekirite.
Pwen Kontak
Si ou gen kesyon kounye a, ou mèt poze yo kounye a. Si ou gen kesyon apre m kite, ou gen dwa mande [Driektris
Sant yo]. Epi ou kapab jwenn Cara Kennedy, reskonsab Etid la, nan biwo Beyond Borders sou [adrès] epi nan
nimewo telefòn [xxxx-xxxx].
Èske ou gen kesyon?
Èske ou ba nou otorizasyon pou ou patisipe nan etid la?
Wi [Kontinye]
Non [Kanpe]
___________________________________________
Non Timoun nan
[Mwen te eksplike etid la ak patisipan an]
________________________
Moun k ap fè entèvyou ki se temwen konsantman
_________________
Dat
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
25
!
!
!
!
Appendix A2: English Version
Verbal Consent Form for Research for Research Study Instructions for the Interviewer
Note to Interviewer: The following sections printed in bold are to be read to the subject prior to the interview. If
the subject then agrees to participate, you must sign on the line marked ‘Witness to Consent Procedures’ at the
end of this form. Also mark the date on the appropriate line.
Purpose of the Study
You are being asked to be part of a research study. We want to find out about the problems affecting
children/youth in this area and what children do in general. By learning about the problems of local people Beyond
Borders/Limyè Lavi along with other organizations that work with children hope to create programs that can
better support children and their families. Beyond Borders / Limyè Lavi are conducting this study with the
collaboration of OIM, Foyer L’escale, CAD, andk Restavèk Freedom. We would like to invite you to participate.
Procedures
To obtain this information we are talking with some children in the community as well as the adults who take care
of them. This is how we selected you. If you agree to help us, I will ask you some questions.
Risks and Discomfort
Each interview will take about 30-60 minutes. It is possible that you may not like some questions or that some
questions may upset you. You may refuse to answer these questions, or any questions, if you wish. You may stop
the interview at any time. If at the end of the interview you feel uncomfortable for any reason, we have a member
of our team who will be available to speak with you.
Benefits
This information will help Beyond Borders / Fondasyon Limyè Lavi / OIM to provide better programs to improve
the health of the people in this area. However, there may be no direct benefit to you personally.
Confidentiality
During the interview I will write down the information you tell me. This is the information we will use for our
study. The record of this information will not have any information which can be used to identify you. Only the
research team will be able to see this information. Nothing that you tell us will be shared with anyone outside the
research team. We will not allow anyone else to find out who gave a particular answer unless we think that you or
someone else might be in danger. In that case, we cannot keep that information private and we will take steps to
make sure that all people involved are kept safe.
Whom to Contact
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
26
!
!
!
!
If you have any questions now you can ask me. If you have any questions after I leave, you can ask (Site Directors).
And, you can also contact Cara Kennedy, director of this study. She can be contacted through (your center) in
[address], or by telephone at [xxxx-xxxx].
Do you have any questions?
Do you agree to participate in this study?
Yes [Continue]
No [Stop]
___________________________________________
Child’s Name
[I explained the study to the participant]
________________________
Name of Interviewer, Witness of Consent
_________________
Date
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
27
!
!
!
!
Appendix B: Explanation of Reliability and Validity Concepts
10
Reliability
Reliability refers to the extent to which different measures of the same concept agree with each other. It can refer to
measurements taken at the same time, or different times. To be useful an instrument must have good local reliability,
which must therefore be tested whenever a questionnaire is changed (including translation) or used among a new
population.
Test-Retest Reliability
Testing reliability over time is also useful. This is called test-retest reliability. The questionnaire is given to the same
subject on two different occasions. It is usually done at least a day later, to reduce the effect of memory on the responses,
but not too long because what is being measured may actually change (mood, for example). Therefore, the repeat
interview is usually done 1-7 days after the first interview. To make this comparison, a summary scale is first created
using all the questions on the same topic and calculated for both the first and second interview. Test-retest reliability
measures correlations between these scores. Opinions vary as to what is an acceptable score, although correlations above
0.7 are considered desirable for test-retest reliability. A problem arises in interpreting low scores. These may be due to a
poor instrument, or to using different interviewers, or because the concept being measured has changed.
Internal Consistency Reliability
This refers to how well questions measuring the same underlying concept on the same occasion agree with each other.
For example, two questions that measure different aspects of depression should agree with each other in that the same
individual should score high or low on both. Agreement is measured quantitatively by correlations. For questionnaires
with many questions measuring the same concept, a large number of correlations would be required to check the
agreement of every question with every other question, and some summary of these correlations would be needed.
Cronbach’s alpha is a statistical measure that provides this. It is a single figure that summarizes the average correlation
between all pairs of questions in a questionnaire. Cronbach’s alphas should be above 0.7 and ideally between 0.8-0.9. The
reliability of each question can be assessed by calculating the alpha with and without it. Significant increases in alpha
without the question would suggest that the question is not measuring the same construct as the other questions, and
should be removed. Studying the effect of each question in this way is called Item Analysis.
Validity
Validity refers to the extent to which the measurement provided by an instrument agrees with the correct measurement.
Instruments may be reliable but not valid, if they consistently give the same (but wrong) measurement and so both
reliability and validity must be measured to assess instrument accuracy. There are two aspects of validity to be considered
when testing a questionnaire:
Content validity
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
10
"Adapted!with!permission!from!Applied!Mental!Health!Research!Group!(2007)."Assessment"of"Urban"Street"Children"and"Children"living"in"Government"Institutions"in"
Georgia:"Development"and"testing"of"a"locally6adapted"psychosocial"assessment"instrument."
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
28
!
!
!
!
This refers to whether the instrument is considered by experts to be appropriate for measuring what it is supposed to
measure. Part of content validity is whether experts believe that the questionnaire covers all the important aspects of the
concept being studied. In the course of this study and the previous qualitative studies we consulted two groups of
‘experts.’ The first group consisted of Haitian and U.S. psychologists/clinicians who helped us choose the YSR as an
appropriate measure for this population. The second group was the local population, through the qualitative study; the
choice of these instruments was also based on trying to achieve as close a match as possible to the psychosocial issues that
emerged in that study.
Criterion Validity
This refers to the agreement between the questionnaire and an external measure (criterion) of the same construct known to
be accurate. In other words, comparing the questionnaire with a ‘gold standard.’ In this study the ‘gold standard’ was a
combination of the assessment of the staff of the institutions and the respondents themselves as to whether or not they had
a mental health problem.
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
29
!
!
!
!
Appendix C: Outlier Analyses
Appendix C1: Outlier analysis of scale scores
Scale
YSR Subscales
YSR Internalizing Problems
YSR Externalizing Problems
YSR Activities Competence
YSR Social Competence
YSR Academic Competence
YSR Total Scales
YSR Total Problems
YSR Total Competence
Haiti Scales
Haiti Symptoms*
Haiti Function Items*
YSR-Haiti Subscales
Combined YSR and Haiti Internalizing*
Combined YSR and Haiti Externalizing*
Time Point
#
of
Outliers
Baseline
Retest
Baseline
Retest
Baseline
Retest
Baseline
Retest
Baseline
Retest
0
0
0
2
0
0
1
1
0
0
Baseline
Retest
Baseline
Retest
1
0
0
0
Baseline
Retest
Baseline
Retest
0
0
0
0
Baseline
Retest
Baseline
Retest
0
0
0
1
Case ID
F003, F024
T002
F011
F003
F024
*Not all respondents have complete data. Data presented for only those with complete data.
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
30
!
!
!
!
Appendix C2: Outlier analysis of difference scores
Scale
#
of
Outliers
Case ID
YSR Subscales
YSR Internalizing
YSR Externalizing
YSR Activities Competence
YSR Social Competence
YSR Academic Competence
0
3
0
1
0
YSR Total Scales
YSR Total Competence
YSR Total Problems
3
3
F002, F005, F023
F024, C008, T001
Haiti Scales
Haiti Symptoms
Haiti Function
0
3
F001, F011, C008
YSR-Haiti Subscales
Combined YSR and Haiti Internalizing
Combined YSR and Haiti Externalizing
0
2
T001, F024
T001, F021, F024
F005
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
31
!
!
!
!
Appendix D: “Youth Self-Report – Haiti” (English Translation)
!
Assessment!of!Child!Survivors!of!Restavèk!in!Haiti:!Development!and!Testing!of!a!Locally!Adapted!Psychosocial!
Assessment!Instrument!
!
32
9:14 AM
Page 1
M
PL
E
4/9/09
SA
81945_CBCL_501
9:14 AM
Page 2
M
PL
E
4/9/09
SA
81945_CBCL_501
9:14 AM
Page 3
M
PL
E
4/9/09
SA
81945_CBCL_501
9:14 AM
Page 4
M
PL
E
4/9/09
SA
81945_CBCL_501
Local Symptoms and Functioning Items
0 = Not true
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
!
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
1 = Somewhat or Sometimes True
1. I am not comfortable
2. When other children are playing, I don't play with them
3. I stay by myself
4. I am always uncertain
5. I remember the bad moments
6. Sometimes I'm distracted
7. I feel distant from everything around me
8. Everything annoys me
9. I am sad
10. I am thievish
11. I am sorrowful
12. I have unruly behavior
13. I ruminate
14. I hit others
15. I don't trust people
16. I am always working and/or serving others
17. I get up early
18. I have difficulty learning in school
19. I don't want to go to school
20. I can be violent
21. I can be badly behaved
22. I give adults trouble
23. I feel I'm nothing
24. I am stressed
25. I don't listen
26. I don't take care of my hygiene
27. I am traumatized
28. I get angry
29. I can't express how I feel
30. I don't have hope in the future
31. I wet the bed
32. I feel ashamed
33. I am sad
34. I have emotional crises
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2 = Very True or Often True
35. I like to go to school
36. I play with other children
37. I learn well
38. I eat well
39. I respect my parents or other adults
40. I feel comfortable
41. I go to school
42. I take care of my hygiene
43. I do all of my work
44. I am healthy
45. I have a lot of friends
46. I have leisure activities
47. I am intelligent
48. I am comfortable with the people around me