Designing a Survey to Assess the Health and Culture of a Residency Program
Rebecca Gladding, M.D., Dana Levy, M.D., Randall Espinoza, M.D., Ashley Dorin, M.D. and Peter Whybrow, M.D.
UCLA Department of Psychiatry and Biobehavioral Sciences
Study question: A common difficulty of psychiatry residency
training programs is determining how to address and quantify
culture or administration problems. Chief Residents are often
in the position to mediate discussion and formulate solutions,
but little information exists on experiences of how to
accomplish this task.
• Freeform responses highlighted issues that had never been overtly articulated, but were problematic
undercurrents within the residency. With regard to feedback, when asked “what are the major issues you faced as
an intern?” one person commented: “Have attendings tell us what their expectations for a call presentation [are].” In
a different section regarding possibly learning how to give feedback, a resident responded: “More useful to have a
culture where such feedback is expected and encouraged.” These sentiments were mirrored in faculty responses, as
shown below.
Methods: Beginning with data from our annual retreat and a
series of focus groups, we developed a 108-question survey for
residents that assessed the strengths, weaknesses and areas
for improvement in the following realms: goals and philosophy of
the program, professional development, well-being,
administration, call, feedback and supervision and PGY specific
issues. We sent the survey link out to residents via email more
than five times to achieve a response rate of 70%. We also
developed a companion study for faculty and administrators
mostly related to feedback to ensure that perspectives from all
sides were incorporated.
• Combining categorical and free-form responses yielded the most informative results, both within one question
and across similar questions. For example, asking categorical questions about feedback with an option to provide
free-form text helped identify a significant issue in the residency: lack of clear expectations regarding giving feedback
within the culture.
Results: The resident response rate to complete entire
survey was 56%, though 67% started the survey. PGY1-3
initial responses averaged 78%, which dropped to 71% for
completion. After the first week, the survey had to be divided
into two parts based on feedback from residents that it was
“too long” and taking “too much time” to complete. Residents
also negatively commented on having too many required
questions on each page of the Survey Monkey form. For
faculty, 37 started the survey and 27 completed it, totaling
73% of those who originally responded. Total faculty
response was low given that more than 100 faculty were
invited to respond to the survey; however, those who did
respond work closely with residents.
Sample Combined Categorical and Free-Form Questions
Attendings: 1) Do you feel like the culture is supportive of giving feedback to residents and backing up
attendings when they give that feedback? 2) In what circumstances, or for what reasons, would you not
give a resident feedback?
Attendings: “The culture does
Reasons attendings do not report problem behaviors in residents
not support giving feedback
It would create an awkward working situation
43%
Nothing would be done about it/the behavior would not be corrected or changed
30%
as there is no structural
No good comes of reporting problems with residents
23%
framework to do so regularly,
Somehow I would end up the bad guy
20%
and so when it is done, it
I would receive bad evaluations
20%
The resident would retaliate/ostracize me
13%
seems extraordinary.”
• Numerical/categorical data yielded helpful information to
validate or reject certain assertions about the program, such
as assessing the overall health of a program:
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Socially
Emotionally
Career
planning and
mentoring
Outstanding
Great
Good/Adequate
Fair
Not at all
Academically
Percent
How well the program meets resident
needs in the following categories:
Sample Categorical Questions
Residents: How often do you receive positive
feedback? Helpful/constructive feedback?
Attendings: How often do you give residents
positive feedback on their work? Constructive
feedback?
Comparison of Reported Amount of Feedback
Given by Attendings vs. Received by Residents
Positive feedback
Constructive feedback
Attendings
Residents
Attendings
Residents
Never (less than 25%)
0
21%
0
28%
Once in a while (26-50%)
7%
31%
27%
35%
Often (51-75%)
37%
27%
40%
35%
Always (>75%)
56%
21%
33%
2%
Conclusions:
•Free-form questions can highlight issues you do not know exist or themes that run across various groups (e.g.,
residents and faculty), especially when prompted by, or combined with, categorical questions
• Categorical questions can yield helpful overall state of the residency information and test assumptions/myths of the
program in an neutral/non-confrontational way
Survey Design:
• Must be anonymous when highly sensitive topics are being assessed
• Shorter, focused surveys are better received by busy participants
• Need to survey all stakeholders as appropriate (e.g., faculty, nursing)
• Define terms clearly to avoid confusion or bias (ex: what does “excellent” vs. “great” vs. “good” vs. “fair” mean)
• State questions neutrally and give respondents an opportunity to disagree with the statement or question; avoid
leading questions when possible
• Avoid too many free-form questions, as they take more time to complete and require more thought per response
• Do not require all questions be answered; rather, focus on requiring responses only to the most important ones.
Reference:
Fraenkel JR, Wallen , NE (2003). How to design and evaluate research in education, 5th ed. San Francisco , CA :
McGraw-Hill, page 395.