Providing Multi-Source Feedback
to Improve Learner Behavior
G DODD DENTON, MD MPH
PATRICIA SHORT, MD
UNIFORMED SERVICES UNIVERSITY
MICHAEL BATTISTONE
UNIVERSITY OF UTAH
Definitions of Terms
 Assessment
“the process of collecting information about the performance of a learner.” This may include
grade recommendations, multiple choice or other examination scores, Likert-scale
performance ranking,
ranking and/or narrative comments
comments.
 Evaluation



Closely related to assessment
Terms sometimes used interchangeably in US
Includes assigning value and structure to the information gathered, such as whether the
assessment is consistent with programmatic standards
 Feedback
“The process of communicating the information gathered via assessments and evaluations for
the purposes of improving future performance of the learner”
 360 degree evaluation

Evaluation from “all around” an employee (or learner) referring to the 360 degrees in a
circle
 Multisource Evaluation – subset of 360 degree evaluation
“A…process using several reviewer groups and self-assessment for formative assessment of
multiple performance domains”
Assessment
 We gather all this information about our learners
 Facilitated by today
today’ss electronic evaluation systems
 ACGME recommends 360 degree evaluations
 For house staff
 No current recommendation from LCME for students
 All this information can be overwhelming to the
clerkship director and confusing to the student
Evaluation
From Collecting Evaluations to
Providing Effective Feedback!
 Purpose of feedback is “to improve learner behavior”
 Many options:
 Provide access to online documents (i.e., Evalue or New
Innovations)
 Give paper documents to the learner


Both approaches allow learner interpretation
Facilitate feedback from the multisource evaluations
Talk to the learner and interpret the meaning of the assessments
 Facilitation is probably more likely to “improve learner behavior”

Historical Perspectives from Business
 First use of multiple source for feedback was German
military in WWII
 By early 90’s, most HR departments understood 360
degree feedback, using peer, supervisor, employee
and sometimes customer input.
 Dozens of online options to collect evaluations.

Because of this,, over 1/3
/3 of US companies
p
use Multi-source
feedback; 90% of Fortune 500 firms
 All business models depend on “coaches” to provide
context and support to recipients.

Trained outside consultants or HR department
Sample website
Perspectives from Medicine
 No data on medical student MSF
 In 2010
2010, Miller et al reviewed MSF for practicing
physicians

Mixed Results:
 Educational
value, but evidence for change was conflicting.
docs/surgeons little willingness to change
 Family physicians more prepared
 Junior

Performance change more likely when FB
 Was
credible and accurate,
 When coaching provided
Perspectives from Medicine
 In 2009, Overeem et al qualitatively examined
perceptions
p
of effectiveness of 3
360 degree
g
doctors p
evaluation.



Facilitation was essential.
Concrete goals made sessions more effective.
Follow-up assessment should be available.
 In 2010, Archer et al reviewed effective feedback
“Facilitative rather than directive feedback enhances learning
g for
high achievers”
“Feedback must be conceptualised as a supported sequential
process rather than a series of unrelated events”
Randomized, controlled trial
 In 2007, Brinkman et al randomized pediatric
residents to receive MSF with coaching
g or to receive
standard feedback




Residents in MSF group received self-assessment, a report
from parents/nurse evaluation, and had tailored coaching plus
“standard feedback”
Control group got “standard feedback”
5 months later, nurses subsequently scored MSF group
residents higher in communication skills,
skills in timeliness and in
demonstrating responsibility and accountability
Unclear whether coaching was principal reason for positive
results, but authors felt that was likely.
Based on industry
industry, resident and practicing
physician experiences, should we provide “coaches”
to help the learners interpret feedback; i.e.,
facilitation?
Facilitator Roles
 Facilitator can resolve disparate evaluations
 Facilitator can identify themes across evaluations
 Facilitator can align assessments with evaluation
framework and program goals
 Facilitator can weigh relative merits of sources
Facilitator Roles
 Facilitator can ensure format is accepted by learner
as accurate,, credible,, and useful
 Facilitator sessions provide formal structure to
feedback
 Facilitator can collaborate with learners to form
action
ti plans
l
 Facilitator can ensure inclusion of self-assessment
Introduction to Videos and Group Work
References









Hoffman, Robert. “Ten reasons you should be using 360-degree feedback” HR Magazine
April 1995 (Accessed 29 September 2010 at http://findarticles.com/p/articles/mi_
m3495/is_n4_v40/ai_16792797/?tag=content;col1)
360 DegreeFeedback com A Division of Team Builders Plus (Accessed 29 September
360-DegreeFeedback.com.
2010, at http://www.360-degreefeedback.com/coaching.html)
Advanced Informatics (Accessed 29 September 2010, at https://www.e-value.net/)
Myevaluations.com (Accessed 29 September 2010, at http://www.myevaluations.com/)
Assessment of Professional Behaviors. A Multi-competency, multi-source Feedback
Program. (Accessed 01 October 2010 at http://www.nbme.org/schools/apb/index.html)
360-degree feedback, From Wikipedia, the free encyclopedia. (Accessed 01 October 2010
at http://en.wikipedia.org/wiki/360_degree_evaluation)
Miller A., Archer J. Impact of workplace based assessment on doctors' education and
performance: a systematic review. BMJ. 2010 Sep 24;341:c5064.
Overeem K, et al. Doctors
Doctors' perceptions of why 360
360-degree
degree feedback does (not) work: a
qualitative study. Med Educ. 2009 Sep;43(9):874-82.
Brinkman WB, et al. Effect of multisource feedback on resident communication skills
and professionalism: a randomized controlled trial. Arch Pediatr Adolesc Med. 2007
Jan;161(1):44-9.