PAR Evaluation Report:
Feedback from Physicians
1st Half 2015
Prepared by
101, 10155 114 Street
Edmonton, Alberta Canada
T5K 1R8
T 780.421.1199
F 780.425.2707
E [email protected]
Table of Contents
Introduction________________________________________________________________________________4
Participation Statistics_______________________________________________________________________4
Results _____________________________________________________________________________________6
Comments_______________________________________________________________________________ 13
Appendix A: Introductory Letter ___________________________________________________________ 24
Appendix B: Questionnaires _______________________________________________________________ 26
Appendix C: Frequency distribution _______________________________________________________ 29
Appendix D: Questions used for the Spearman Rho correlations _____________________________ 31
Appendix E: Physician comments on any aspect of participating in the PAR Program_________ 32
Physician Feedback About PAR –1st Half 2015
ii
List of Tables
Table 1. Respondent characteristics – 1st half, 2015 __________________________________________ 4
Table 2. Online completion rate ____________________________________________________________ 5
Table 3. Percent agreement with QI culture _________________________________________________ 6
Table 4. Percent agreement with ease of use _______________________________________________ 7
Table 5. Percent agreement with statements about the value of feedback ___________________ 8
Table 6. Percent agreement with contemplating practice change ___________________________ 9
Table 7. Percent agreement with the accuracy of PAR feedback ____________________________ 9
Table 8. Percent agreement with statements about practice change _______________________ 10
Table 9. Scope of practice change________________________________________________________ 11
Table 10. Percent of physicians who have made at least one practice change ______________ 12
Table 11. Spearman Rho correlations* _____________________________________________________ 12
Table 12. Changes made as a result of PAR program participation__________________________ 13
Table 13. Impact of changes made to practice ___________________________________________ 17
Table 14. Ways PAR could be more effective in supporting physicians _______________________ 19
Table 15. Number and percent of responses from all physicians _____________________________ 30
Table 16. Questions used for Spearman Rho correlations ____________________________________ 31
Table 17. Comments regarding participation in the PAR program ___________________________ 33
Physician Feedback About PAR –1st Half 2015
iii
Introduction
This report presents the results collected in the first half of 2015 of an ongoing survey conducted
with physicians who participated in the Physician Achievement Review (PAR) program. The
survey is conducted by Pivotal Research Inc. using a questionnaire developed by members of
the PAR-Committee on behalf of the College of Physicians and Surgeons of Alberta. The survey
was distributed with a cover letter from the Registrar, Dr. Trevor Theman, along with a certificate
of participation in the PAR program three months after a physician received his/her report. A
copy of the letter and questionnaires are included in Appendix A and B, respectively.
Participation Statistics
The physicians who responded between January 1st and June 30 th, 2015 included 42 medical
specialists; 26 surgical specialists; 21 general practitioners; 12 anesthesiologists; 11 episodic care
physicians; 10 pediatricians; 7 psychiatrists, 6 radiologists and 2 laboratory physicians. Nearly
eighty-five percent (84.1%) of physicians indicated they practice in an urban setting and nearly
half (45.7%) have been in practice for 16 years or more (Table 1).
Surveys were distributed to 283 physicians in the first half of 2015. The 138 respondents represent
a 49 percent response rate.
Table 1. Respondent characteristics – 1st half, 2015
Category
Practice Setting
Years of Practice
Number
Percent
Medical Specialist
42
30.4
Surgical Specialist
26
18.8
General Practitioner
21
15.2
Anesthesiologist
12
8.7
Episodic Care (Emergency Physician,
Walk-in type Family Practitioner,
Hospitalist and Locum)
11
8.0
Pediatrics
10
7.2
Psychiatrist
7
5.1
Radiologist
6
4.3
Laboratory Physician
2
1.4
Not specified
1
0.7
Urban
116
84.1
Rural
11
8.0
Not specified
11
8.0
1 – 5 years
20
14.5
6 – 10 years
24
17.4
11 – 15 years
19
13.8
16 years or more
63
45.7
Not specified
12
8.7
Physician Feedback About PAR –1st Half 2015
4
Beginning January 1, 2010, physicians had the option of completing the feedback survey online.
Of the 138 surveys included in this report, 36 were completed online (Table 2).
Table 2. Online completion rate
Time Period
2015
2014
2013
Percent
Completed
Online
First half
26.1
Second half
25.9
First half
30.4
Second half
21.6
First half
26.0
Physician Feedback About PAR –1st Half 2015
5
Results
Physicians were invited to evaluate various aspects of the PAR program by indicating their level
of agreement with statements regarding quality improvement, ease of use, value of stakeholder
feedback, accuracy of PAR feedback as well as contemplating and making practice change
based on their PAR report. To aide interpretation, we combined the percentage of respondents
who “strongly agree” or “agree” as are those who “strongly disagree” or “disagree”. The
highlighted data represents data collected from this reporting period (first half of 2015). We
structured the tables with the results separated into the first half of 2015 and the second half of
2014 for the reader to compare. Complete frequency distribution results for the first half of 2015
are displayed in Appendix C.
Quality Improvement (QI)
In total, 350 physicians provided feedback on the PAR program during the past 12 months. Of
this group, 64 percent indicated that the PAR program promotes quality improvement and 60
percent made a change in at least one aspect of their practice. According to their comments,
the most common changes relate to interacting and communicating with patients, co-workers
and colleagues, and better management of patient care. A detailed description of practice
changes is provided in Table 12.
Sixty-five percent of physicians in the first half of 2015 indicated that the PAR program promotes
quality improvement (Table 3). Figure 1 depicts these results.
Table 3. Percent agreement with QI culture
Statement
The PAR program promotes
quality improvement
Agree
%
Neutral
%
Disagree
%
n
1st Half 2015
65.0
19.7
15.3
137
2nd Half 2014
62.8
22.2
15.0
207
Total*
63.9
21.0
15.2
344
Period
* Un-weighted average
Figure 1. Percent agreement with QI culture
Physician Feedback About PAR –1st Half 2015
6
Ease of Use
Table 4 shows physicians’ agreement with statements about the ease of understanding PAR
reports and ease of participation in the PAR process. Similarly to the second half of 2014, over
half (52.2%) agree that participating in the PAR process was easy. The portion of physicians who
found their report easy to understand in the first half of 2015 remains similar to the second half of
2014. Two out of every three physicians who responded found their PAR report easy to
understand. Figure 2 displays these results graphically.
It is noteworthy to recall that a significant decrease was discovered in the portion of physicians
who found their PAR report easy to understand between the first and second half of 2014. In the
previous report it was suggested that this change may be due to the new PAR report format
introduced in early 2014 which displays physician scores as T-scores. As the results of the first half
of 2015 mirror that of the second half of 2014, the T-score report format is most likely responsible
for this change.
Table 4. Percent agreement with ease of use
Statement
Period
Agree
%
Neutral
%
Disagree
%
n
Participation in the PAR process
was easy
1st Half 2015
52.2
26.8
21.0
138
2nd Half 2014
56.2
21.9
21.9
204
My PAR report was easy to
understand
1st
66.4
16.8
16.8
137
66.8
21.0
12.2
205
Half 2015
2nd
Half 2014
Figure 2. Percent agreement with ease of use
Physician Feedback About PAR –1st Half 2015
7
Value of Feedback
Table 5 displays physician agreement with statements about the value of the feedback
provided by colleagues, co-workers, referring physicians and patients. Overall, approximately
two out of every three physicians (66.3%) felt that the feedback they received from colleagues,
co-workers and patients to be valuable. There is a directional decrease between the second
half of 2014 and the first half of 2015 in the portion of physicians who agree that feedback from
colleagues, co-workers and patients is valuable. A significant difference exists regarding
physicians’ agreement with the value of co-worker feedback1.
Similarly to the second half of 2014, most physicians indicated that comparisons between their
self-assessment and colleague scores are valuable (64.3%). Figure 3 displays these results
graphically.
Table 5. Percent agreement with statements about the value of feedback
Statement
Period
Agree
%
Neutral
%
Disagree
%
n
Feedback from medical
colleagues was valuable
1st Half 2015
66.2
17.6
16.2
136
2nd
72.0
16.2
11.8
204
Feedback from co-workers was
valuable
1st
61.8
19.9
18.4
136
76.8
12.3
10.9
203
Feedback from referring
physicians was valuable
1st
40.0
40.0
20.0
5
50.0
50.0
0.0
2
Feedback from patients was
valuable
1st
71.0
17.7
11.3
124
79.2
14.1
6.7
192
Comparison of self and medical
colleague ratings was valuable
1st
64.4
20.7
14.8
135
64.3
21.3
14.5
207
Half 2014
Half 2015
2nd
Half 2014
Half 2015
2nd
Half 2014
Half 2015
2nd
Half 2014
Half 2015
2nd
Half 2014
Figure 3. Percent agreement with statements about the value of feedback
1
Chi-square; p-value < 0.03
Physician Feedback About PAR –1st Half 2015
8
Contemplating practice change
Physicians were asked to specify their level of agreement with the statement “I have
contemplated practice changes as a result of my PAR report”. Similar to the second half of 2014,
most physicians indicated they had (Table 6). These results are displayed visually in Figure 4.
Table 6. Percent agreement with contemplating practice change
Statement
I have contemplated practice
changes as a result of my PAR
report
Period
Agree
%
Neutral
%
Disagree
%
n
1st Half 2015
55.8
23.2
21.0
138
2nd Half 2014
59.3
23.3
17.5
206
Figure 4. Percent agreement with contemplating practice change
Accuracy of feedback
A new question item was added to the physician feedback survey in early 2015 asking
physicians to indicate their level of agreement with the statement “Overall, I believe the
feedback accurately described my practice.” Table 7 demonstrates that while near 30 percent
(28.7%) are neutral regarding the representativeness of their PAR feedback over half (53.2%)
agree that their feedback accurately described their practice.
Table 7. Percent agreement with the accuracy of PAR feedback
Statement
Overall, I believe the feedback
accurately described my practice
Agree
%
Neutral
%
Disagree
%
n
53.2
28.7
18.1
94
Physician Feedback About PAR –1st Half 2015
9
Practice change
Table 8 shows physicians’ level of agreement with making a practice change in various aspects
of their practice as a result of their PAR report. There has been a directional decrease across all
items in the first half of 2015. A significant difference between the two time periods exists with
respect to change in interacting with medical colleagues 1 and aspects of patient care2. The
results are presented visually in Figure 5.
Table 8. Percent agreement with statements about practice change
Agree
%
Neutral
%
Disagree
%
n
1st Half 2015
31.6
32.3
36.1
133
2nd
40.7
37.3
22.1
204
33.6
29.9
36.6
132
38.2
37.7
24.0
204
25.0
25.0
50.0
8
33.3
33.3
33.3
3
40.2
29.1
30.7
127
43.0
31.0
26.0
200
28.0
34.4
37.6
125
44.0
33.0
23.0
200
Based on my PAR report I have
made changes in…
How I interact with medical
colleagues
How I work with co-workers
How I interact with referring
physicians
My communication with patients
Aspects of patient care (e.g.
investigations, education,
prevention)2
Period
1st
Half 2015
2nd
1st
Half 2014
Half 2015
2nd
1st
Half 2014
Half 2015
2nd
1st
Half 2014
Half 2014
Half 2015
2nd
Half 2014
Figure 5. Percent agreement with statements about practice change
1
Chi-square; p-value < 0.03
2 Chi-square;
p-value < 0.02
Physician Feedback About PAR –1st Half 2015
10
As shown previously in Table 8, physicians are asked if they have made a change to various
aspects of their practice as a result of their PAR report. Further analysis was conducted to
determine the scope of practice change. Do physicians tend to indicate change in one area of
their practice or in multiple?
Of physicians who provided feedback nearly 13 percent (12.5%) indicated that they made
practice change with peers, co-workers and patients (Table 9). In addition, nearly half (45.6%)
made a change in at least one aspect of their practice. A difference is observed between the
two time periods2. These results are displayed visually in Figure 6.
Table 9. Scope of practice change
Percent
1st
Response
Half 2015
2nd Half 2014
Made a change to all
12.5
24.2
Made at least one change
45.6
38.2
Neutral to all
13.2
15.9
Neutral or did not make a change at all
28.7
21.7
Total
100.0
100.0
Figure 6. Scope of practice change
Overall, fifty-eight percent of physicians indicated that they made a change in at least one
aspect of their practice (Table 10). This is a 4 percent decrease from physicians in the second
half of 2014.
1
Applies to all physicians who are not laboratory physicians or radiologists
2 Chi-square;
p< 0.04
Physician Feedback About PAR –1st Half 2015
11
Table 10. Percent of physicians who have made at least one practice change
Period
Have made a change in at least
one aspect of their practice
Agree
%
1st Half 2015
58
2nd Half 2014
62
Practice change and value of feedback
In other research we found a positive correlation between perceived value of a multi-source
feedback program and the likelihood of a practice change. This relationship was tested by
correlating the data from each feedback group (medical colleagues, co-workers and patients)
and the different practice changes reported. The results are shown in Table 11. The data
continues to suggest that physicians who value feedback are more likely to make a practice
change, or conversely, physicians who make practice change are more likely to value
feedback.
Table 11. Spearman Rho correlations*
Spearman
Rho
Significance
level
n
Medical Colleagues
0.41
0.01
131
Co-workers
0.36
0.01
132
Patients1
0.35
0.01
121
Feedback Group
*To see the questions used in the spearman rho correlations please see Appendix D.
Physician Feedback About PAR –1st Half 2015
12
Comments
Physicians were asked to comment on three questions regarding the impact of participating in
the PAR program. Tables 12 to 14 present the themes and sub-themes found in the responses,
the number of responses and specific sample comments that illustrate the themes. When a
physician’s comment addresses multiple themes or sub-themes only the pertinent comment is
shown in the sample comments column.
The question item asking physicians to “comment on any aspect of the process about
participating in the PAR Program” was removed from the PAR Feedback Survey in January of
2015. Appendix E contains the comments received by 25 physicians who completed a previous
version of the survey.
Full comments are provided as an Addendum. Some comments were edited for syntax and
grammar.
Changes Made
What changes, if any, have you made in your own practice as a result of participating in the
PAR program?
Ninety-six physicians offered comments in response to this question. See Table 12 for themes and
sample comments.
Table 12. Changes made as a result of PAR program participation
Theme A: Communication and Interaction (n = 45)
Sub-theme
Number
of
Responses
Sample Comments


Improve
communication/interaction
with co-workers/colleagues
18



Improve
communication/interaction
with patients and/or their
families
12


Improve relations with own office staff particularly!
I am ensuring that follow up on tests and investigations
are properly communicated so that my colleagues will
be informed before they see the patient.
Improve efforts in communication with colleagues and
co-workers.
I try to be more respectful to my co-workers + review
patient care with them so we are on the same page.
Explain myself more to patients and asking for their
feedback.
Improve patient communication, instructions, listening to
patients, slowing down.
Thought about how I communicate certain information
to patients/families.
Physician Feedback About PAR –1st Half 2015
13
Theme A: Communication and Interaction (n = 46) - continued
Sub-theme
Number
of
Responses
Sample Comments

Improve documentation
6




Reduce waiting
time/improved scheduling of
appointments
5


Improve communication
(general)
2
Cautious interactions with
colleagues and/or coworkers
Extensive documentation is sent along with referrals,
along with a more comprehensive referral letter.
Clearer discharge summaries.
Trying to make my handwriting more legible.
I have lessened any written communication + gone to
dictation wherever possible or have student/resident
write out my orders to lessen problems with my
handwriting.
I have had a discussion with the clinic manager
regarding improvement of the system for booking
appointments as one of the concerns noted was
difficulty accessing through the phone lines to book
appointments. The clinic has since then introduced
another system. Patients can send a text requesting
appointment and one of the clinic staff calls them within
24 hours regarding that.
Also focus on logistics of clinic (i.e. timing of patients and
communication about clinic running late).
Shorter waiting times for patients.


Always strive to communicate more effectively.
Focused on my communication style, slowing down,
taking more time.


More caution around nursing staff.
Owing to the subjective nature of the PAR process, I am
exquisitely careful about interactions with colleagues
and coworkers. I make more efforts to ensure they are
aware of my CME and other non-clinical activities.
Overall this has been a negative change, I feel... I am
trying to “sell” myself so that next time PAR comes
around, I might be seen more favourably. This does not
impact my patient care, which I believe remains
exemplary.
2
Theme B: Self-Management (n = 30)
Sub-theme
Better management of
patient care
Number
of
Responses
Sample Comments



14


Improved pt care and education.
Reviewed AHS guidelines on confidentiality.
Consultation letters more promptly arrive at the referring
physicians’ office.
Extensive documentation is sent along with referrals,
along with a more comprehensive referral letter.
Prompt return of insurance forms.
Physician Feedback About PAR –1st Half 2015
14
Theme B: Self-Management (n = 30) - continued
Sub-theme
Number
of
Responses
Improve stress management
or work/life balance
4
Improvements in practice
management
3
Pursue education or
professional development
Sample Comments



I am trying to manage my stress levels better.
Reviewed the impact of stressors from my job.
I have to work less.


Tried to make office more patient friendly.
I plan to decrease my administrative duties so as to be
accessible to patients

Planning to participate in more CME activities as
although my direct patient care was well rated, my
medical knowledge is what can most be improved.
Attend more continuous learning activities.
2


Increased self-awareness
2

I like the outside view of one's work and consider
perceptions verses realities of patient care. Very
interesting feedback and it is encouraging to see the
positive feedback from patients and the areas to work
on with coworkers. Thanks for your help.
Spent more time reflecting on my own behavior.


Participate in more committee work.
Will attempt to participate in committee in hospital more
frequently.
2


Time management/availability.
Better time management of visits.
1

I closed my practice.
Increase participation in
committee work
2
Improve time management
Closed practice
Theme C: Nothing (n = 35)
Sub-theme
None / minimal
Received favourable scores,
no significant changes
required
Number
of
Responses
25
5
Sample Comments


None. (n = 18)
Don't feel that it has impacted my practice in any
significant way.

Reviews were very positive so didn't need to make
changes. Nice validation.
No changes, continue with positive reinforcement or
good attributes noted in PAR.


Unable to make change
3
Nothing specific due to PAR
2



I didn't understand my report so I didn't make any
changes.
N/A, Currently on maternity leave, feedback received
whilst on leave
None based on results of PAR - I make changes based on
annual audits of my results.
I constantly try to be aware of issues within my practice.
Physician Feedback About PAR –1st Half 2015
15
Theme D: Other (n = 8)
Sub-theme
Number
of
responses
Sample Comments

Feedback is not
accurate/valuable
5
Difficult to get patient
feedback
1
Clinic manager should be
aware of communication
difficulties
1
Number of peer assessments
makes it arduous
1

I strive to improve, but I think colleagues confuse some
things. I prefer certain types of medical problems to look
after + others I do not. This does not mean I do not
manage both types of problems well.
Discussed results with individual co-workers to gain insight.
I wonder whether forms were filled in correctly as
interview very different than report for some questions.

It was difficult to get patient feedback for my ICU
practice.

Clinic manager should be aware of communication
problems regarding physicians & patients.

Number of peer assessments makes it arduous
Physician Feedback About PAR –1st Half 2015
16
Impact of Change
What is the impact of the changes made in your practice?
Of those physicians who reported making a practice change, forty-six physicians offered
comments regarding the impact of making their change(s). See Table 13 for themes and
sample comments.
Table 13. Impact of changes made to practice
Theme A: Specific Impact (n = 31)
Sub-theme
Number
of
responses
Sample Comments

Improvements for patients
Improved
communication/interaction
with colleagues and/or coworkers
8




6


Patients are more satisfied with the appointment booking
process now.
More handouts to improve quality of care SOGC + ACOS.
I think my patients have a better understanding of what
they are being treated for.
Better working relationships with my surgical colleagues.
Communication of above has reduced impatient and
demanding phone calls from referring physicians +
colleagues.
Improved communication and collaboration with
colleagues + co-workers.


Focus a bit more on just getting the job done. I spend an
inordinate amount of time on my work; the personal cost is
substantial.
Working towards self-improvement.
Hopefully to be the best physician I can be.
2


Better patient care and follow up.
Watch how my patients are triaged at our large clinic.
Improve communication
2


Improved efforts in written communication.
Thought about alternative ways of relaying the information.
Improvements to medical
practice
2


Positive impact in medical practice.
Overall better practice.
Receiving positive
feedback
2


Feedback from my patients and co-workers is positive.
However, in some aspects I am getting positive feedback.
Improve attention to
detail/decrease errors
2


Double check Fax numbers before faxing.
Hopefully less potential for errors to occur.
Increased care with
patient confidentiality
1

Pay closer attention as to how I discuss patients in situations
where I could be overheard.
Helped make decision to
close practice
1

It helped me decide to close my practice.
Improvements to selfmanagement
3
Better patient care
Physician Feedback About PAR –1st Half 2015
17
Theme A: Specific Impact (n = 31 ) - continued
Sub-theme
Number
of
responses
Increased awareness
1
Less enjoyment in work
1
Sample Comments

Higher awareness of how I run my practice and how it is
perceived by co-workers/colleagues.

Not as much fun to work.
Theme B: Impact not Specified (n = 20)
Sub-theme
Number
of
responses
Sample Comments
4



Too early to say.
Unsure yet.
None yet.
Positive/Beneficial
4



Positive impact.
Positive overall.
Beneficial
None/Minimal impact
2


None
Minimal impact, small tweaking; not all that helpful to be
honest; impact rate of 2 out of 10.
Neutral/Somewhat helpful
2


Neutral
Somewhat helpful but changes were already being made.
Uncertain
1

Uncertain
Various
1

Various
Changes made were due
to reflective visit
1

Nothing based on the PAR survey. Many changes were
made following my reflective visit by the PAR/CPSA team.
Changes have been time
intensive
1

The changes have been time intensive.
Unable to make changes
based on PAR feedback
1

It is difficult to justify wholescale changes based on
impressions: Real data is more useful.
Hopeful of impactful
changes made
1

I am hopeful that my co-workers see me in a better light as
we work together for our patients.

Unable to make changes
1
Unable to change parking (hospital based). Booking staff
are hired by hospital – difficult to change as it is not an
independent practice.
Will continue to make
improvements

1
I have always had great reviews so I will focus on continuing
to relate + communicate with all - patients, physicians,
coworkers.
Too early to tell/Not sure
yet
Physician Feedback About PAR –1st Half 2015
18
Ways PAR could better support physician practice
In what ways could PAR be more effective to support you in your practice (e.g., the PAR
process, report format, links to resources)?
Seventy respondents offered comments in response to this question. See Table 14 for themes
and sample comments.
Table 14. Ways PAR could be more effective in supporting physicians
Theme A: No Suggestions for Improvement Offered (n = 34)
Sub-theme
No improvement
required, continue as
is
Number of
Responses
8
Sample Comments



It is good enough as it is.
PAR was extremely supportive. Many thanks to the PAR team!
Not much - good to have feedback - it is like a golf swing - you
don't see yourself + can't be certain the ball is heading in the
right direction.

8

PAR was a total waste of my time. I will only complete it if it is
mandatory.
I consider PAR to be an imposition and a waste of valuable
time, money, resources. Program cessation would be my
recommendation.
No improvement
suggested
5



None (n = 2)
No recommendations.
Currently no active recommendations/suggestions.
Not sure
4


Not sure (n = 2)
I do not know.

PAR is a process that delivers only feedback that is obtained
from sources chosen by the MD thereby lessening feedback
input + effectiveness + need for support. Enormous effort + cost
considering the minimal gain; system issue?
I am not convinced its cost provides adequate value.
Discontinue
/Program not useful
Limited value
4


PAR was
stressful/detrimental
3


Already undergo a
similar review
2
The whole process was extremely stressful. Multiple times I
contemplated quitting the practice of medicine strictly based
on par. I dread the approaching process.
Not a single positive thing came out of this, but I sure felt
horribly for days after receiving my results + couldn't sleep well.
I am certified by the Amer. Board of Internal Medicine & have
to do recertification every 10 years. Each recertification
requires the same as PAR (evaluation from ~10 colleagues &
~25 patients). It would be nice if PAR & ABIM recognized this
mutually so that I, my colleagues and my patients do not have
to duplicate effort.
Physician Feedback About PAR –1st Half 2015
19
Theme B: PAR Process (n = 35)
Sub-theme
Number of
Responses
Sample Comments

Not
specialty/practice
specific enough
6



Patient feedback
requirements are
difficult/need
improvement
6




Speed up process
4


Reviewers biased/
randomize selection
of reviewers
Reviewers cannot
accurately assess
performance
3


3


Modify criteria for
subsequent
participation
2


Difficult to recruit
reviewers
2

Many questions on the patient questionnaire are not
appropriate to my practice. The patient questionnaire only
marginally applies in obstetrical anesthesia. Completing the
required patient questionnaires is difficult in a teaching
hospital. Patient interaction is through med students and
residents.
Separate private from university/hospital based practice.
Tailor questionnaires to pediatric practice
I may only care for one patient in a day = so takes a long time
to complete required # of questionnaires. Only 30% response
rate means, I need 75 questionnaires handed out to complete
the task. This took 7-8 months in my practice. I work part time
only.
Fewer questionnaires given to patients.
Consider lengthening the patient recruitment timeline.
Supply additional patient form initially as return rate ~50 – 60%.
COULD HAVE FORMS/INFO SHEET INSERTED INTO ENVELOPES
RATHER THAN HAVE PHYSICIAN DO IT. (i.e. PATIENT FORMS) = 60
IN MY CASE.
I have a low volume practice and as a result was sent
reminders to get my patient feedback reports back in a timely
fashion. However, the turnaround time on the feedback from
the PAR program took months. This seems unfair.
One aspect that can improve is the communication to people
who get the surveys and don’t fill them in a timely fashion.
More random selection of colleagues + co-workers. We ask
those who we get along well with in general.
People select colleagues who will say nice things about them
+ people generally reluctant to give bad reviews. Critical
feedback would be more useful to me.
You are asking others to comment on things that they have no
personal knowledge of. It has always been this way.
It is challenging for my colleagues to be a good judge of my
patient care and academic work, etc.
Different shorter format for physicians in practice >30 years. If
still a member of College of Family Physicians of Canada for
that long - should meet qualification for CME.
WHY DON'T YOU DO IT ONCE & THEN STOP IF WE ARE IN THE
SAME TYPE OF PRACTICE?
Impossible to get people willing to complete the survey and
obviously many of those that agreed did not complete the
survey.
For GP/FP in a solo small practice finding co-workers to do
report is VERY difficult.
Physician Feedback About PAR –1st Half 2015
20
Theme B: PAR Process (n = 35) - continued
Sub-theme
Number of
Responses
Sample Comments


I think the PAR program should call me first before they send
out the PAR package. The specialist package is not the most
appropriate since I do mostly general practice. This is reflected
in the data collected by the College but does not get to PAR
somehow.
Incorrect identification of my practice situation.
1

Have an online assessment for patients and coworkers
Collect feedback by
telephone
1

I’d like to suggest maybe telephone interviews be considered?
Improve online tool
1

Make the online form more user friendly!! Having to click on
each dot to get list of choices is painful and annoying!
Increase frequency
of participation
1

Consider PAR process to be done every 3 years instead of
current 5 year cycle.
Encountered
administrative
incompetence

1
I encountered incompetence in lost responses, incorrect
identification of my practice situation, phone calls from people
who could not explain the purpose of their call. etc.
Decrease number of
required reviewers
1

Less # of colleagues surveyed.
Feedback should not
be anonymous
1

Ask people to be responsible for their feedback - tell the
colleague in person, show their name.
Received incorrect
specialty package
Have all reviewers
provide feedback
online
2
Theme C: PAR Results or Feedback (n = 28)
Sub-theme
Number of
Responses
Sample Comments


Improve report
format
7



Provide concrete
suggestions in how to
improve
6
Improve links to
resources
3



The report might maybe be more understandable.
I realize the reporting has been changed over the years to
make it more understandable for more physicians. But I would
really like to see means + standard deviations for my rating
and the average ratings so I can understand my score better.
Report forward could be more descriptive, include outliers.
Wording within the report was occasionally terse and
intimidating.
Would be helpful to include some suggestions for changes
based on the results.
Perhaps more directive processes on how to fix certain issues if
they are identified.
Links to resources to help with identified areas of improvement
would be beneficial.
Links to resources around identified issues.
Physician Feedback About PAR –1st Half 2015
21
Theme C: PAR Results or Feedback (n = 28) - continued
Sub-theme
Number of
Responses
Sample Comments

Scores are not
accurate/meaningful
3

Improve
measurement scale
Concerns with
sample size/bias
Allow respondents to
provide comments
Improvements to
questionnaires
2
2
2


Score on 5 point L scale is limited in value
Going from a 5 item score to a 9 item score might help pull a
“spike”; into something closer to a bell curve which would
improve things considerably.

Small sample size means results can be swayed drastically by
outliers.
Reduce bias and comparison. Some physicians give feedback
forms to their favourite patients – not random.



Area on feedback form for free text would be useful.
I think verbal answers to questions from colleagues may be
more valuable. As a couple of colleagues said to me We
didn’t feel comfortable giving you all 4’s and 5’s so gave you a
couple of 3’s to make it look like I being honest!!!!

I don’t think all the questions asked of peers are helpful (e.g.) if
my family member were a patient would you feel comfortable
having this physician. Options for answers, do not suit question
i.e. “among the best”, “top half” etc.
Language Barrier. Impossible for patients to complete
questionnaire.
2

Provide further follow
up regarding issues
1
My report gave me quite a start. So many areas where I scored
in the lowest 10 percentile. It really had me quite worried. I
asked all my colleagues if they had any concerns all said no. I
had a long chat with Pivotal Research and discovered that
scores in my peer group were so closely grouped that one
person scoring me with a 3 rather than a 4 and everyone else
giving me 4 could drag me into the bottom 10%. I also
discovered that overall mine was one of the better reports and
I fell “more than 1 standard deviation above the mean”. A
relief but it made me feel the scores per item are rather
meaningless!
Look at outcomes, objective measures; too subjective

May a friendly reminder and follow about the particular issue
and some practical suggestions.
Theme D: Other (n = 6)
Sub-theme
Number of
Responses
Sample Comments

Prefer more extensive
evaluation/audit
2
What links to what
resources?
1
Do not minimize time
for completion
1
Problem is out
physician control
1

A more formal audit of practice with evaluation of results
would be much better than uninformed opinion.
Previously I underwent a more extensive eval. I think it should
be an option.

What links to what resources?????

DO NOT MINIMIZE TIME NEED TO COMPLETE PROCESS (TAKES
HOURS NOT MINUTES)

Often the problem is out of the reviewed physician's control
and is more related to provision of services by Alberta.
Physician Feedback About PAR –1st Half 2015
22
Theme D: Other (n = 6) - continued
Sub-theme
Will invite colleague
reviewers outside of
specialty next time
Number of
Responses
Sample Comments

1
It never occurred to me to ask colleagues outside of my
specialty to fill out my medical colleague form. I will definitely
do this next time.
Physician Feedback About PAR –1st Half 2015
23
Appendix A: Introductory Letter
Physician Feedback About PAR –1st Half 2015
24
Date
Dr. {Full Physician Name}
{Address}
{City, AB PC}
Dear Dr. {Physician Surname}:
SUBJECT: Help us make PAR a better tool for you – send us your feedback.
The College of Physicians & Surgeons of Alberta developed the PAR program as a purely educational
tool, independent of the College’s complaints and discipline functions.
As a checkup for medical practice, PAR has two major objectives: to promote quality improvement in
individual medical practice, and to foster a culture of quality improvement in the medical profession
that serves Albertans.
I hope your PAR results have given you insight into your practice that will help you identify and meet
quality improvement objectives.
Next steps:
1. Send us your feedback. To make PAR a better tool for you, we need to know about your experience.
Insert online and paper same as existing (a and b)
Your responses are completely anonymous. Results are released twice yearly in aggregate form for
review by the PAR Committee, and also reported in The Messenger newsletter and posted at
www.par-program.org.
2. Develop an action plan. To get the most out of your PAR, prioritize potential improvements based
on your unique situation and develop a plan to implement them. Be sure to tap into resources on
the PAR website. If you need help, email the Director of Practice Improvement at [email protected] or contact Deputy Registrar Dr. Karen Mazurek at [email protected].
3. Apply for CME credits. After your PAR, you can apply for credits through MAINPRO-C (College of
Family Physicians of Canada) or the MOC program section 3 (Royal College of Physicians and
Surgeons of Canada). Information is available on each of their websites. PAR is also a valuable tool
for helping you focus your Continuous Professional Development activities where it will benefit your
practice most.
Thank you for helping to improve the quality of medical practice in Alberta.
Sincerely,
Dr. Karen Mazurek, Deputy Registrar
College of Physicians & Surgeons of Alberta
Enc.
Appendix B: Questionnaires
Physician Feedback About PAR –1st Half 2015
26
Physician Review
of the PAR Program
Choose the categories that best describe you.
1. I mainly practice in:
an urban setting
2. I have practiced for:
1 - 5 years
a rural setting
6 - 10 years
11 - 15 years
Please rate your agreement with each statement using the scale
to the right.
16 years or more
Strongly Disagree
Disagree
1
2
Neutral
Agree
3
4
Strongly
Not
Agree Applicable
5
6
3. The PAR Program promotes quality improvement.
4. Participation in the PAR process was easy.
5. My PAR report was easy to understand.
6. Feedback from patients was valuable.
7. Feedback from co-workers was valuable.
8. Feedback from my medical colleagues was valuable.
9. Comparison of self and medical colleague ratings was valuable.
10. I have contemplated practice changes as a result of my PAR report.
11. Overall, I believe the feedback accurately described my practice.
Based on my PAR Report I have made changes in...
12. My communication with patients.
13. How I work with co-workers.
14. How I interact with medical colleagues.
15. Aspects of patient care (e.g. investigations, education, prevention).
16. What changes, if any, have you made in your own practice as a result of participating in the PAR Program?
17. What is the impact of the changes made in your practice?
18. In what ways could PAR be more effective to support you in your practice (i.e., the PAR process, report format, links to
resources)?
41065
123456789
Thank you for completing this survey!
Physician Review of the PAR Program
Radiologists
Choose the categories that best describe you.
1. I mainly practice in:
an urban setting
2. I have practiced for:
1 - 5 years
a rural setting
6 - 10 years
11 - 15 years
Please rate your agreement with each statement using the scale
to the right.
16 years or more
Strongly Disagree
Disagree
1
2
Neutral
Agree
3
4
Strongly
Not
Agree Applicable
5
6
3. The PAR Program promotes quality improvement.
4. Participation in the PAR process was easy.
5. My PAR report was easy to understand.
6. Feedback from medical colleagues was valuable.
7. Feedback from referring physicians was valuable.
8. Feedback from co-workers was valuable.
9. Comparison of self and medical colleague ratings was valuable.
10. I have contemplated practice changes as a result of my PAR report.
11. Overall, I believe the feedback accurately described my practice.
Based on my PAR report I have made changes in...
12. How I interact with referring physicians.
13. How I interact with medical colleagues.
14. How I work with co-workers.
15. What changes, if any, have you made in your own practice as a result of participating in the PAR Program?
16. What is the impact of the changes made in your practice?
17. In what ways could PAR be more effective to support you in your practice (e.g., the PAR process, report format, links to
resources)?
56576
123456789
Appendix C: Frequency distribution
Physician Feedback About PAR –1st Half 2015
29
Table 15. Number and percent of responses from all physicians
Strongly
Disagree
Disagree
#
%
#
%
#
%
#
%
#
%
#
%
The PAR Program promotes quality improvement.
5
3.6
16
11.7
27
19.7
73
53.3
16
11.7
137
100.0
Participation in the PAR process was easy.
10
7.2
19
13.8
37
26.8
56
40.6
16
11.6
138
100.0
My PAR report was easy to understand.
11
8.0
12
8.8
23
16.8
70
51.1
21
15.3
137
100.0
Feedback from my medical colleagues was valuable.
10
7.4
12
8.8
24
17.6
64
47.1
26
19.1
136
100.0
Feedback from my co-workers was valuable.
9
6.6
16
11.8
27
19.9
57
41.9
27
19.9
136
100.0
Feedback from my patients was valuable.2
7
5.6
7
5.6
22
17.7
60
48.4
28
22.6
124
100.0
Feedback from my referring physicians was valuable.1
1
20.0
0
0.0
2
40.0
2
40.0
0
0.0
5
100.0
Comparison of self and medical colleague ratings was
valuable.
11
8.1
9
6.7
28
20.7
71
52.6
16
11.9
135
100.0
I have contemplated practice changes as a result of my PAR
report.
9
6.5
20
14.5
32
23.2
59
42.8
18
13.0
138
100.0
Overall, I believe the feedback accurately described my
practice.
4
4.3
13
13.8
27
28.7
43
45.7
7
7.4
94
100.0
Based on my PAR report I have made changes in my
communication with patients.2
9
7.1
30
23.6
37
29.1
43
33.9
8
6.3
127
100.0
Based on my PAR report I have made changes in how I interact
with referring physicians.1
1
12.5
3
37.5
2
25.0
2
25.0
0
0.0
8
100.0
Based on my PAR report I have made changes in how I work
with co-workers.
10
7.5
39
29.1
40
29.9
37
27.6
8
6.0
134
100.0
Based on my PAR report I have made changes in how I interact
with medical colleagues.
11
8.3
37
27.8
43
32.3
35
26.3
7
5.3
133
100.0
Based on my PAR report I have made changes in aspects of
patient care (e.g., investigations, education, prevention).2
12
9.6
35
28.0
43
34.4
29
23.2
6
4.8
125
100.0
Statement
1
2
Neutral
Agree
Strongly
Agree
Total
Applies to laboratory physicians and radiologists only
Applies to all physicians who are not laboratory physicians or radiologists
Physician Feedback About PAR –1st Half 2015
30
Appendix D: Questions used for the Spearman Rho
correlations
Table 16. Questions used for Spearman Rho correlations
Questions
Feedback group
Feedback from…
Based on my PAR report I have made
changes in…
Medical Colleague
Medical colleagues was valuable.
How I interact with medical colleagues.
Co-worker
Co-workers was valuable.
How I work with co-workers.
Patient1
Patients was valuable.
My communication with patients.
1
Applies to all physicians who are not laboratory physicians or radiologists
Physician Feedback About PAR –1st Half 2015
31
Appendix E: Physician comments on any aspect of
participating in the PAR Program
Physician Feedback About PAR –1st Half 2015
32
Comments on the process of PAR participation
Please comment on any aspect of the process about participating in the PAR program.
This question item was removed from the PAR Feedback Survey in January of 2015. The following
table contains comments received by 25 physicians who completed a previous version of the
survey. See Table 17 for themes and sample comments.
Table 17. Comments regarding participation in the PAR program
Theme A: Ease of Participation (n=11)
Sub-theme
Process is difficult, time
consuming or stressful
Sub-theme
4
Sub-theme


Time consuming
Difficult/burdensome in addition to my clinical practice,
which has little clinical support already in rural setting.

Asking for evaluations from pts undergoing general
anesthesia is a waste of time.
Enrolling patients in the emergency department is at times
complicated given the complexity of the work
environment, complexity and severity of patients’
complaints. One then may be default selecting patients
with minor problems where the physician patient
interaction is minimal.

Difficult to recruit
appropriate patient
reviewers
2
Process takes too long
2
Difficult to recruit
colleague reviewers
1
Process is straightforward
but tedious
Online participation is
simple


Took too long to get feedback
Slow process to get results back; had to contact the
program

It is difficult to get a response from that many colleagues if
you haven't worked in various clinics that would allow you
to have met that many colleagues.
1

Process is straight forward albeit tedious.
1

On line participation is helpfully simple
Theme B: Concerns about Bias or Validity (n = 8)
Sub-theme
Number of
Responses
Sample Comment

PAR is not
specialty/practice
specific enough
2

In team-based and job sharing type of practices PAR does
not do well in evaluating individual physicians. The service
provided should be evaluated.
When you are the only specialist in the province doing your
type of practice it is hard to compare to peers when you
work in a University based practice group it is not possible
to control the "physiced plant" (sic) as evaluated by PAR.
Physician Feedback About PAR –1st Half 2015
33
Theme B: Concerns about Bias or Validity (n = 8) - continued
Sub-theme
Concerns about validity of
PAR
Number of
Responses
Sample Comment

For reasons I don't quite understand, the feedback from my
non-physician colleagues, nurses, unit clerks etc. on the
PAR was very different than my day to day feedback and
interactions. I may not have selected colleagues properly,
but I have never felt that my working relationship with them
was 'below average' at all. I think I am pretty self-aware
and unless my colleagues are telling me one thing and
writing down something very different, I am going to
choose to disregard that section. It does make me lose
faith in the whole process, however.

I talked to others who said I didn't play 'the game' properly
+ send it to friends who would give me all 5s as that’s what
they do.
There are rumours circulating that if you get any scores
below top marks your file will be flagged. I didn't
understand what "flagged" meant until I got my results so I
likely erred on the high side even on my self-evaluation to
avoid trouble.
2
Reviewer manipulation of
the results
2
Self-selection of reviewers
is biased
1
Reviewers cannot assess
performance
1


I think the process is a joke when you can choose your own
referees.

Often clinical/medical and referring colleagues are not
familiar with how the practice seems.
Theme C: Suggestions for Improving PAR (n = 7)
Sub-theme
Number of
Responses
Sample Comment


Improve report format
3

Allow physician to select
time within year to
participate
1
Develop different rating
systems based on years of
practice
1
Increase number of coworkers and colleagues
1
Provide practical
recommendations to
address concerns
1
Previous reporting was much easier to understand and
interpret.
The change in the Report distances the results from the
previous methodology - Now comparing me to a small
group of surgeons rather than given raw scores.
I'd like to see my absolute scores (1 - 5) rather than the
relative (percentile) score. If all physicians get 5 and I get
4, then I am below 10th percentile, but not too bad.

It would be good to be able to set up a convenient time of
year to do this.

I think you need a different rating system for physicians in
practice >30years

Increase to 25 the # of co-workers. Increase to 10 the # of
colleagues. Even in small centers is possible.

No practical suggestions from results - i.e.
recommendations of how to address concerns, what the
categories mean.
Physician Feedback About PAR –1st Half 2015
34
Theme D: Usefulness of PAR (n = 4)
Sub-theme
Number of
Responses
PAR is useful, valuable or
informative
1
PAR is not useful, a waste
of time and/or money
1
Usefulness depends on
choice of reviewers
1
Ensuring fair process
Sample Comment

I find PAR program is a very helpful educational tool
helping to improve the scope of my medical practice.

This was a complete waste of time + energy.

The main thing is to choose people that will provide you
with good and constructive feedback.

My thoughts are any responsible physician would and
should be aware of all that is included in the PAR program.
However I suppose there are some who are not and as
such a blanket approach is one way of ensuring a fair
process.
1
Theme E: Other (n = 1)
Sub-theme
Results were good
Number of
Responses
1
Sample Comment

Results were good no major changes required
Physician Feedback About PAR –1st Half 2015
35