Doing the right thing:
Ethical considerations when
conducting program evaluation
November 12, 2013
Today’s presenters
Tara McFadden
Research Assistant, Evaluation and Research
Ontario Centre of Excellence
for Child and Youth Mental Health
Lindsay Muir
Research Associate
Kinark Child and Family Services
Housekeeping notes
• This webinar will be recorded, and will be available on
the Centre’s website as an educational resource
• The slides have been sent to participants
• Log in with a phone whenever possible for optimal audio
quality
• We have staff online to assist with any technical
difficulties
• There will be a short evaluation survey for all webinar
attendees at the completion of the webinar.
Housekeeping notes
Addressing questions:
• This webinar will be followed by a question
and answer period, however questions are
encouraged throughout the presentation
• Questions can be submitted electronically or
verbally. Specifics around this process will be
clarified at the end of the webinar
About the Centre
We bring people and knowledge together to strengthen
the quality and effectiveness of mental health services for
children, youth and their families and caregivers.
Three strategic goals:
Learning
Foster a culture of
organizational learning
to support agencies in
using evidence to
improve client outcomes.
Collaboration
Build and develop
collaborative
partnerships to sustain
capacity within mental
health services.
Leadership
Be a true learning
organization and lead
by example.
Overview
• Overview of key ethical considerations for
program evaluation
• Description of different ethics review
processes
• Views from the field
• Suggestions for additional resources
What is ethics?
“Ethics refers to well-founded
standards of right and wrong that
prescribe what humans ought to do,
usually in terms of rights, obligations,
benefits to society, fairness or specific
virtues.”
(Velasquez et al., 2010)
Ethics and evaluation
• Risks are inherent in research and evaluation
• Ethical guidelines and principles help advance
knowledge while protecting participants
(Oakes, 2002; Tri-council Policy Statement, 2010)
Tri-Council Policy Statement
• Created by three research funding agencies
CIHR, NSERC, & SSHRC
• Outlines principles for research with human
participants
• REBs follow these principles when reviewing
research proposals
Core principles
• Respect for human dignity
o
o
o
Respect for persons
Concern for welfare
Justice
Tri-council Policy Statement, 2010
Source: B.S.Wise on Flickr
Free, informed and ongoing consent
Free, informed and ongoing consent
• Extent to which participants are aware:
o
o
o
participation is voluntary.
of the purpose of the evaluation.
of the risks and benefits associated with
participation.
Example – Tuskegee study
(CDC, 2011)
Practically speaking…
• You can ensure free and informed consent by
providing an information letter which explains:
•
•
•
•
•
purpose of evaluation
what is expected from the participant
benefits
risks
what has been done to ensure participant safety and
minimize risk and harm
Practically speaking…
• Include a voluntary participation statement
which explains
• participation in evaluation is voluntary.
• participants may withdraw at anytime.
• participating or not participating will not affect services
received
“Your participation in this evaluation is voluntary
and you may withdraw at any time. Your
participation will not affect the services you receive
in any way.”
Fairness and equity in participation
Consider fairness and equity in participation
Considering the scope
and purpose of your
evaluation you should be
inclusive in selecting
participants.
Example: Choosing participants
Source: Chris Lsys at freshspectrum.com
Privacy and confidentiality
Privacy and confidentiality
• Privacy refers to individual's right to control
information about oneself
• Confidentiality is an obligation to protect entrusted
information
• Not upholding confidentiality can lead to negative
consequences
Example: Deductive Disclosure
Fisher Folk: Two Communities on Chesapeake Bay, Carolyn Ellis
Source: Walt Hubis on Flickr
Practically speaking…
• Assign unique IDs to measures, interviews or
focus group data
• Remove identifying information from
qualitative data
• Assign pseudonyms
• Aggregate data
Practically speaking…
Sample Confidentiality Statement:
“You can be assured that your name will not be
disclosed and that the information you share with us
will be confidential. Your responses will be summarized
along with other responses. No names or identifying
information will be used.”
Practically Speaking…
• Who will have access to data collected?
• Where will data be stored?
• What will be done with data
following the study?
Conflicts of interest
Conflicts of interest
“…when activities or situations place an individual or
institution in a real, potential, or perceived conflict between
the duties or responsibilities related to research…”
(TCPS, 2010)
• Examples of conflicts of interest include:
•
•
Dual roles
Financial
Practically speaking…
• Addressing conflicts of interest will ensure:
•
•
•
high objectivity in evaluation procedures
rights, safety and well being of participants
high degree of integrity in your evaluation
Ethics process and review
Ethics in the evaluation process
•
•
•
•
•
•
•
•
•
•
Identified need
Organize stakeholder communications
Literature review
Finalize evaluation questions
Develop a methodology
Ethics submission (may be required)
Collect data
Analyze and interpret data
Communicate findings
Apply findings
thinking
about ethics
What are the options for ethics review?
• Ethics review committee (ERC)
• An independent committee that has been formally
designated to approve, monitor and review research or
evaluation involving humans with the aim to protect their
rights and welfare, e.g. an internal committee
• Ethics review board (ERB)
• An independent board established solely to review
research and determine if the rights and welfare of human
participants involved in research are adequately protected
Do I need ERC or REB review?
• agency policies
• research vs. evaluation
No review required, what can I do?
• ARECCI Ethical Guidelines for QI and Program Evaluation
o
o
Provides six questions to guide your ethical
thinking
e.g. How will you ensure that the participant (or
data) selection process is fair and appropriate?
Additional Considerations
• PHIPA (Personal Health Information Protection
Act)
• CES Ethical Guidelines
• AEA Guiding Principles for Evaluators
It sounds easy, but…
Lindsay Muir
Research Associate
Kinark Child and Family Services
Consent
Easier = Research projects Harder = Service as Usual
Wording:
It says we use this information for program evaluation, but we learned
something interesting that we now want to present at conferences and
publish…can we?
Format:
Clients are suppose to initial beside each point on our service agreement
including the research line…they didn’t…
•
What does that mean?
•
Can we use their data?
•
How do we track the no’s/withdrawals?
•
Can we access that information to clean future datasets?
Language/Understanding:
•
Reading level
•
Support for clarity and questions
Additional Consent?
Confidentiality
Challenges:
•Replacing names with ID’s when merging data
•BCFPI, CAFAS, Service Data
•When does data become identifying?
• Small unique programs
• Specific subject matter
Solutions:
•Security
•Paper and electronic storage
•Transporting data
•Transferring data
•When do I need even more protection?
An example
• Partnered with Michael Boyle from the Offord Centre to assist with the
Describing Feelings and Emotions Study (preliminary piece of Ontario
Child Health Study)
• Study approved through our Research Ethics Committee as well as through
McMaster University
• Objective: Test the equivalence of two measurement instruments to
classify 6 child psychiatric disorders.
• Method: Parent and child independently completed a self administered
measure and an interviewer administered measure on two occasions (1-2
weeks apart). The child’s teacher was also asked similar questions (paper
survey), so long as consent was received from the child & parent.
• Our role was to conduct the data collection interviews
Ethical Considerations:
• Participation completely voluntary and in no way affected
services with Kinark
• Consent obtained from parent / caregiver and children aged
12-17 (assent obtained from children under 12)
• Data kept confidential (unless there was risk to the child or
others)
• Although the teacher / school was involved, they were not
informed of any client data, nor were they informed that the
child was receiving services from Kinark
Safety:
• As staff would be entering client’s homes, a safety plan was
created
• Client profiles were screened to assess potential risks by noninterviewing staff –interviewing staff remained ‘blind’
• Interviews always conducted in pairs
• Interviewers were required to notify the project supervisor
upon completion of all interviews
Data Storage & Management:
• Data never left in a vehicle
• Data and consent forms are never stored in the same place
• All paper data stored in secure, locked location
• Electronic data stored on Kinark’s internal system, and only
accessible to the researchers specifically on that project
Data Transfer to Partner Agency:
• Data, consent forms, and audio files transferred separately
• Emailed data was password protected, encrypted file
• Consent transferred over password protected, secured FTP site
• Password information only communicated via telephone
Questions?
Questions or comments?
• To submit questions electronically, use the
question box located in your control panel
• To submit questions verbally, use the raised
hand icon also located in the control panel
Contact us
For more information on this webinar or topic, please
contact:
Tara McFadden
Research Assistant, Evaluation Support Service
Ontario Centre of Excellence for Child and Youth Mental
Health
[email protected]
Contact us (…cont’d)
For more information on this webinar or topic, please
contact:
Lindsay Muir
Research Associate
Kinark Child and Family Services
[email protected]
Upcoming webinars
Reviewing the literature
Ilana Smyth
Jenna Jones, Windsor Regional Children’s Centre
November 19 at 1 p.m.
Maximizing the lifespan and effectiveness of EIPs over time: A
webinar on sustainability
Sandra Huang Del Frari
Dr. Jonathan Golden, Kinark Child and Family Services
Dr. Zia Lakdawalla, Kinark Child and Family Services
November 27 at 1 p.m.
Evaluation
Please don’t forget to
complete the survey at the
end of this webinar.
Your feedback is very
important to us, so we thank
you for taking the time to
share your thoughts!