Feedback & Evaluation
Eva Aagaard, MD
Vice Chair for Education
Department of Medicine
Overview

Feedback





Why should you care about feedback?
What is feedback?
What’s the evidence that feedback works?
How do you give feedback effectively?
Case Discussion
Feedback
Why Should You Care?

Trainees WANT feedback

In a study of over 1500 residents, 96% believed
feedback was important for learning


Schultz, BMC Central 2004
Trainees feel they NEVER get enough feedback
Gil, J Med Educ 1984
Isaacson, J Gen Int Med 1995


Feedback is ESSENTIAL
to Teaching

Giving high-quality feedback is strongly
associated with teaching ratings


Torre, Acad Med 2003
Constructive and specific feedback can improve
learner knowledge and skills (More on this
later)…
Boehler, Med Ed 2006
Clay, Critical Care Med 2007


Definitions
Or What is Feedback?
Evaluation VS. Feedback

Evaluation:





Summative
Higher stakes
Generally standardized
Goal is to grade relative to
peers or a gold standard
Feedback:



Formative
Goal is to help people
improve
Can be formal or informal
Categories of Feedback

Brief feedback



Brief, specific notation of a positive or negative
behavior
Generally provided for directive teaching
Ex:
“I think that what you noted to be distension is actually
ascites. Let me show you some techniques to
distinguish these things.”
Branch, Acad Med 2002
Categories of Feedback

Formal Feedback



Set aside a period of time (5-20 min) to discuss
performance on a specific issue
Can be event or encounter specific
Exs:




Mistakes
Handling of a specific patient case
Feedback following a case presentation or a teaching
encounter
Still very specific and generally limited in scope
Branch, Acad Med 2002
Categories of Feedback

Major Feedback:



Scheduled sessions to review overall performance to
date (15-30 min)
Must be given in a timely enough fashion that trainee
has the opportunity to improve/ grow/ respond
Ex:

Midpoint feedback on a rotation
Branch, Acad Med 2002
The Evidence
Procedures Training
Boehler, Med ED 2006
Does Provision of Specific
Procedural Skill Feedback Improve
Performance in that Skill Compared
to General Comments?
Design






33 2nd and 3rd year Med Students at Southern Illinois
University
All given same instruction on 2-handed surgical knottying
Randomized to specific feedback or general
compliments
Videotaped performing the skill before and after
feedback
Videotapes scored by 3 blinded observers
Student satisfaction with instruction assessed via 7-point
global rating scale
Mean Performance Scores
25
P<0.001
P=0.2
20
15
Pre-test
Post-test
10
5
0
Specific Feedback
General Compliments
Total Possible Ratings 0-32; IRR >0.8
Mean Satisfaction Scores
6.2
6
5.8
5.6
5.4
5.2
P=0.005
Specific Feedback
General Compliments
5
4.8
4.6
4.4
7-Point Likert Scale (1=Very Poor; 7= Truly Exceptional)
Take Home Points

Specific feedback can improve performance on
procedural skills


Satisfaction with feedback is a poor marker of
the QUALITY of feedback


FEEDBACK  LEARNING
SATISFACTION ≠ QUALITY
Combination of specific feedback and
compliments may be best option
Nontechnical Skills
Savoldelli, Anesthesiology 2006
Does Feedback Improve Resident
Performance in Nontechnical Simulator
Skills?
Is Video-Assisted Oral Feedback Better
than Oral Feedback Alone?
Design



Forty-two 1st, 2nd and 4th year anesthesia residents at the
University of Toronto
Videotaped managing an intraoperative cardiac arrest on
a simulator with scripted operating room personnel
Randomized to 3 arms




No feedback
Oral feedback
Video-assisted oral feedback
Videotaped on a second (different) intraoperative cardiac
arrest scenario
Design Cont…

Videos scored by 2 blinded reviewers using a
validated behavioral marker system (ANTS)




Task management
Team working
Situational awareness
Decision making
Scenario order randomized
 Reviewers blinded to chronological order of
scenarios
 OUTCOME=Mean change in ANTS score

Key Feedback Points
Focused on nontechnical (cognitive and
behavioral) skills
 Resident reflection on encounter elicited
 Specific and constructive comments provided



Both positive and negative
Feedback limited to 4-6 main points
% of ANTS Score Change
P<0.05
16
14
P<0.01
12
10
8
Control
Oral Feedback
6
4
2
0
-2
Video-Assistaed Oral
Feedback
Take-Home Points
Oral feedback improves nontechnical skills in a
simulated crisis environment
 Specific, limited feedback along with resident
self-reflection were used in this study
 Videotape review was not better than oral
feedback alone

How Do I Do It Well?
Giving Feedback 101
Adapted from Ende, JAMA 1983
Key Principles
The Groundwork
 The Meeting
 Things to Remember

The Groundwork

Set clear objectives and goals upfront



What does the trainee hope to get out of your time
together?
What specific behaviors do you expect?
When will you give the trainee feedback?


SET AN APPOINTMENT NOW
When will you reassess their performance and reset
goals?
Preparation


Organize your thoughts and observations ahead of time
Setting:



Positive brief feedback can be given almost anywhere
Negative or major feedback should always be given in private
and without interruptions
Timing:




As close to event as possible
Enough time to make a change
Make an appointment
Make sure learner is ready to hear it



Not being paged
Not distracted
Not distraught
Opening the Encounter
 Describe
the Purpose
 Label it FEEDBACK
Sostock, Acad Med 2002
Conducting the Meeting

ASK


Elicit self-reflection
TELL


Give both positive and constructive feedback ALWAYS
Be SPECIFIC and use non-judgmental language




Behaviors not personality
Objective, observable and modifiable
Provide suggestions for how to improve
ASK


Elicit trainee understanding of feedback
Allow trainee to develop own suggestions for improvement plan
Closing the Meeting
 SUMMARIZE
Positives
 Areas for improvement
 Plan for improvement
 Plan for when to reassess/ meet again

REMEMBER….

Limit constructive feedback to NO MORE than 4
(and probably 2) areas
REMEMBER…

Listen to the persons perspective and feelings
PRACTICE
BRIEF FEEDBACK
The Consult Call
You ask the MS4 to call a consult on a patient
with brisk UGI bleeding. You listen as she
speaks to the fellow on call. The presentation
includes all pertinent material and is concise and
organized. She asks specifically if the fellow
would be willing to perform an emergent EGD,
and states that this is the reason for her call.
Your Feedback?
When?
Where?
What?
BRIEF FEEDBACK
 Timely-
in the moment
 Generally can be given anywhere (unless
of sensitive nature)
 Specific
 Objective
An Example
“I wanted to give you some quick feedback on
your consult call. You provided the patient
information clearly and concisely. More
importantly, you clearly stated the reason for the
consult. Nice work.”
FORMAL FEEDBACK
The Medical Student Case
Presentation
“Mr. Jones is a 50 year old man with cirrhosis
related to hep C and alcohol abuse who is admitted
with altered mental status. He was brought in by his
friend who found him confused and lethargic after
not showing up to work as expected. The patient is
unable to give a history of recent events due to
marked lethargy and confusion.
Past medical is as already noted.
Meds are unknown. He was discharged on
spironolactone and atenolol at his last admission.
Presentation Cont…
He works construction and lives alone. The friend
believes that he has quit drinking and drug use,
although he is not sure.
On PE he has normal vital signs.
Neuro exam is nonfocal.
Abdomen is distended with tense ascites, diffusely
tender without rebound.
Rectal is hemoccult negative.”
Your Feedback?
When?
Where?
What?
FORMAL FEEDBACK
Label It- “Let me give you some feedback on
your presentation on Mr. Jones”
 Set an appropriate time and place
 Timely
 Private place
 Include positive and negative
 Be specific
 Provide suggestions for improvement

An Example
+ Your identification of Mr. Jones as a cirrhotic due to
alcohol and Hep C helped give me an idea of what
the most likely causes of his altered mental status
are
- It would have been very helpful to have a better
understanding of the severity of his underlying
illness. It sounds as if that information may have
been available in the most recent discharge
summary
Suggest
Please make sure to include anything you
think might be important information from
the medical records in your oral
presentation next time.
MAJOR FEEDBACK
Mid-Point Feedback
You have been working with your medical student
for 2 weeks. His presentations are excellent and his
clinical acumen is strong.
Today one of your nurses pulled you aside. Per your
instructions, she had been calling him with the
results of labs and x-rays on the patients that he has
seen in clinic. She is upset because yesterday he
took a long time to return pages and when he finally
did call her back, he was abrupt.
Today is the day you are scheduled to give midpoint
feedback.
Your Feedback?
When?
Where?
What?
MAJOR FEEDBACK

Set the Stage



ASK—TELL---ASK




Private place
Enough time free of interruptions
Give both positive and constructive feedback
Be SPECIFIC and use non-judgmental language
 Behaviors not personality
 Objective, observable and modifiable
Provide suggestions for how to improve, and elicit a
self-improvement plan from the learner
Summarize
An Example

Set the Stage:




It’s time for our feedback session.
Let’s go to my office
Is now a good time?
ASK:



How do you think you are doing so far?
What things have gone particularly well?
Are there any areas that you think you need to
improve on?
Example Cont…
 Tell:
+ Your oral presentations and written notes are
clear, concise, and do an excellent job of
depicting our assessment and plan.
+ Your clinical decision-making has been
consistently right on such that I feel I can
completely count on your recommendations.
This has been true even in complicated cases
such as….
Constructive Feedback
One of the nurses approached me today
with some concerns…..
That really surprised me. Is something
going on that I should know about?
Example Cont.
 Suggest:
 If there are ever issues going on for you that make it
hard for you to do your work, it is extremely important
for you to tell someone. In this case, me or someone
else you feel comfortable talking to. We are here to
help you.

ASK:


What do you want to do at this point to rectify things
with Nancy?
Do you feel comfortable calling me or someone else if
something like this arises again?
Example Cont.

SUMMARIZE:





Overall your performance to date has been excellent
Specifically…..
 Oral presentations
 Written Notes
 Clinical judgment
You made a significant error in judgment by not letting
me know that you needed my help.
From now on you will call on me, or someone else
when there are other issues that interfere with your
duties
Let’s go talk to Nancy now to fix things….
Key Points
Set expectations UPFRONT
 Make the time and space to do it
 Be specific and describe observable behaviors
 Give suggestions for ways to accomplish your
expectations
 Practice, practice, practice!!!

QUESTIONS