Peer review process
Trottie Kirwan
Chair APA Peer Review Committee
SPAN Meeting 24 April 2009
Original process
Original Peer Review Process
• APA has run interdepartmental peer review since 1999
• Started in the big specialist children’s hospitals
• Aim
– raise standards of clinical practice within a paediatric anaesthetic
department. Allow for experiences to be shared, with the
dissemination of good or innovative practice (Review document)
– to assess departments in relation to standards which should
characterise anaesthetic departments throughout the country
(Peter Crean in Anaesthesia 2003)
• Based on Good Practice – a guide for departments of
anaesthesia. RCA and AAGBI. London, 1998
• 2 day visit by a team of four including a lay member
• All departments, visitors and visited, found it valuable
BUT
• The process is very demanding of both the visited
department and the visitors
– 2 day visit including interviews with 16 different staff members as
well as parents
– Detailed report averaging 30 pages
• APA decided to extend the process to be available to all
anaesthetic departments with paediatric practice
• Original process too great a commitment for smaller
departments with fewer paediatric anaesthetists,
particularly as visitors to write the reports
New Process
Simplified Peer Review Scheme
• New scheme developed by Peer Review Committee
– Alistair Cranston, Peter Crean, Trottie Kirwan, Madeleine Wang
• Intended as a tool for any department treating children to
– Check compliance with existing criteria of RCA, NSF, HCC/CQC
– Demonstrate that standards of practice are good
– Help in discussions with management and colleagues
• Simplify the process for departments with a smaller
paediatric workload and fewer paediatric anaesthetists
• Reduce the demands on the reviewing team
Simplified Peer Review Scheme
• Provide a structure for local self-appraisal to help
departments in raising standards of clinical practice.
– Self assessment
– Peer verification
• Based on
–
–
–
–
Good Practice Guide 2006 (JCGP RCA & AA),
RCA guidelines for the provision of anaesthetic services 2004
NSF for Children Standard for Hospital Services 2003
Criteria in the proposal come from RCA guidelines
Self-assessment
Self-assessment
•
Departments self check against criteria in a freely
available template
–
–
–
•
Department’s perceived strengths and weaknesses compiled by
the paediatric anaesthetists
Check-list of criteria for good practice (from RCA guidelines)
360° appraisal by colleagues and patients/families
Tool to monitor their own practice, without any
external input
Self-assessment Checklist
(from RCA guidelines)
•
•
•
•
•
•
•
•
•
•
•
•
Organisation
Theatres and day surgery facilities
Recovery
Equipment
Emergency department
Support services
Critical care
Transfer
Staff
Training
Clinical governance
Child and family experience
Self-assessment Checklist
RCoA guidelines
Checklist
Equipment
2.1 A full range of monitoring devices, paediatric anaesthetic
equipment and disposable items for general and regional
anaesthesia should be available in theatres and all other
areas where children are anaesthetised. This should include a
full range of disposable equipment including the following
which should be appropriate for use in children of all sizes and
ages:
blood pressure cuffs
intravenous cannulae
temperature probes
pulse oximetry.
2.2 Resuscitation drugs and equipment, including an
appropriate defibrillator, should be routinely available at all
sites where children are to be anaesthetised.
2.3 Anaesthetic machines should incorporate ventilators,
which have controls and bellows permitting their use over the
entire age range together with the facility to provide pressure
controlled ventilation.
2.4 There should be appropriate thermostatic control of the
operating room; temperature monitoring and patient warming
devices should be available in both the operating room and
recovery area.
2.5 Intravenous fluids should normally be administered only by
volumetric infusion pumps.
Wherever children are anaesthetised:
i. There is appropriate paediatric
anaesthetic equipment
ii. Monitoring equipment meets the
current standards of the
Association of Anaesthetists
iii. Ventilators are suitable for children
and can deliver pressure controlled
mode
iv. There are resuscitation drugs and
equipment
v. There are suitable temperature
monitoring, and warming devices
There are volumetric infusion pumps
for IV fluid administration
Strengths and Weaknesses
• The paediatric anaesthetists, separately and collectively, list
of those aspects of their work which they perceive as
good and those they think need improving
– The aspects which need improving are considered, any which can
be fixed from within the department are addressed
– The process is repeated until eventually the list consists only of
strengths, and of weaknesses over which the group has no control
• Found to be a useful tool for discussing problems people
have got used to but could actually deal with
360° Appraisal by Colleagues
• Colleagues are invited to help in the department’s self-review
–
–
–
–
–
–
–
Chief Executive
Medical Director
Head of Department of Anaesthesia, anaesthetic trainees
Consultant paediatricians, paediatric surgeons, ED, PICU & transfer
Theatre Manager, Senior ODP / anaesthetic nurse
Directorate Business Manager, Anaesthetic Secretary
Senior paediatric ward nurses, play specialist, pain team
• Explanatory letter to say that this is a voluntary process
• Process focussed on the department’s organisation and
practice – concerns about individuals should be reported
through the normal channels
360° Appraisal by Colleagues
• How the paediatric anaesthetic department is perceived
– Concerns with the management of the department or the service
• Anaesthetists working as a team
– Formal working relationships with colleagues
– Informal collaboration for lists and individual patients
• Effective management of the service
– Theatre time, emergency support
– Cover for absent colleagues
• Staffing
– Staff levels, recruitment and retention problems
– System to deal sensitively with colleagues below an acceptable level
• Effective planning for equipment, staffing and budget
Patient/Parent Appraisal
• Areas covered
–
–
–
–
–
Pre-admission
Pre-operative preparation
Anaesthetic room
Recovery
Analgesia
• Design of questionnaire
– Open questions elicit more information
– Closed questions give more consistency
• Timing
– At discharge easy and cheap
– After return home more informative
Peer review
Peer Review
•
•
•
•
•
–
–
–
–
–
–
–
Department completes self-assessment and volunteers
for peer verification
Visiting team
2 doctors and a lay visitor, one from outside the region
Lead reviewer with peer review experience
Could include an observer
Self-assessment paperwork sent to the visiting team
One day visit
Review the self-assessment evidence
meet paediatric anaesthetists
visit to clinical areas
debriefing session
Review team send a report within a month
Peer Review
•
External APA report
–
–
–
•
confirmation of good work
suggestions for consideration to improve organisation & practice
recommendations for use in negotiating plans with colleagues
and management
Problems arising at visit
–
–
visiting team cannot substantiate the self-assessment: this will be
stated at the debriefing meeting and no report sent
a potentially serious problem is unearthed during a review: the
review team leader will contact the regional co-ordinator and an
APA peer review committee member immediately
Peer review process
Organisation of the Process
• Peer review is organised on a regional basis
–
–
–
–
within local paediatric anaesthetic networks where they exist
regional co-ordinator arranges teams of reviewers
each team has an out-of-region member
national co-ordinator for hospitals not in a local network
• Extending the process nationally includes
– smaller departments of fewer paediatric anaesthetists
– less experience of review visiting and report writing
– more support is needed
• Standard job description documents
– task lists for regional co-ordinators, review teams, and
departments being reviewed
– criteria for eligibility for Peer Review Team membership,
anaesthetic and lay
Peer Review
Regional co-ordinator
Peer Review
1. Compile list of anaesthetists to be local review team members. Criteria for eligibility:
a. consultant anaesthetist
b. member of APA
c. experience of peer review
d. for team leaders, experience of 2 reviews
2. Recruit new team members. Criteria:
a. consultant anaesthetist
b. member of APA
People who are interested in getting involved but have no experience of peer review should attend one review
as an observer
3. List of local people to be lay visitors. Criteria for eligiblilty:
a. Experience of hospital visiting, receiving hospital paediatric and/or anaesthetic care, or working with
children and understanding their needs
b. Ability to be objective – should not work within the visited Trust, a neighbouring Trust, PCT or SHA
c. Willing to contribute to the report writing as a team member – so not to be visiting in the role of a
report writer for another institution
Suitable people might be found from voluntary groups listed with PALS, with former Public Patient Involvement
Forum administrators or from the new networks of voluntary organisations known as Local Involvement
Networks (LINKs). Local teachers or other workers with children who have experience with their own children
in hospital have been enlisted.
4. Receive self assessment from departments wanting to be reviewed, and set up a reviewing team consisting of:
a. a lead, who has at least experience as a reviewer,
b. another consultant anaesthetist who has at least experience as an observer
c. a lay member
At least one team member should be from outside the region and there may also be observers from other
regions and the APA peer review committee
5. The co-ordinator must ensure that the reviewing team members can write a fair report. Colleagues with a
professional interest in a good or bad report cannot serve as review team members for that visit
6. Be aware of communication and arrangements between the department to be visited and the review team, and
advise in case of problems
7. Receive the report from the review team, and the feedback on the process from the visited department and the
review team, and chase them up if they are not done in 6 weeks
8. Forward the report and feedback to the APA committee chair and discuss any problems in either the report or
the feedback
APA peer review version2009
Guidance for departments being reviewed
Peer Review
1. The department needs to agree that peer review will be useful, and carry out the self-evaluation, using it as
a tool for self-improvement (unless you are perfect already)
2. Carry out the SWOT process, getting all the paediatric anaesthetists to agree to a consensus document
3. Once the self-evaluation is completed and you are ready to be reviewed, contact the local paediatric
anaesthesia peer review co-ordinator, where there is one, or peer review committee chair.
4. Make contact with the review team leader
5. Inform relevant individuals, wards and departments in the Trust, emphasising that this is a voluntary process
undertaken by the anaesthetic department to recognise and improve the quality of the service, and not
because of a complaint.
6. Collate the short portfolio, self evaluation documents and supporting evidence paperwork and send a copy
to each member of the reviewing team, as much as possible in electronic format, 4 weeks before the visit to
send to the visiting team:
Paperwork to send to the review team members
• The short portfolio
• SWOT consensus document
• The results of the colleague appraisals
• The self-assessment check-list
Check-list criteria evidence which should be sent:
Reports of incident reviews
Department policies and guidelines
Summaries of completed audits from the last 2 years
Minutes of meetings which contain relevant information
Copies of patient information
Evidence of any research – facilities, support (ethics), trainee input
Critical incident reports and presentations to M&M meetings
List of any complaints and the management of them
Organisation of the Process
• Confidentiality
– Reports, and the information they are based on, are confidential
and will not be passed to anyone by the visiting team members
– The visited department may use the report as they choose
• Quality control
– individual performance monitoring by formal feedback after each
review by both the visitors and visited
– consistency monitoring by an external visitor
• Support for the process
– peer review committee
– e-network on APA website
– meetings at APA ASM and linkmen meetings
Feedback
Self-assessment
Surprisingly useful. We all thought that we knew the problems in our hospital but
when we discussed our thoughts we found some important concerns that I had not
considered before. Additionally it encouraged us to have a detailed meeting at which
I could disseminate useful information that others paed anaesthetists had been
unaware.
Arranging the review
Time consuming.Very difficult to arrange reviewers. People need a lot of notice.
Clear requirements for
evidence
I may not have read the documentation closely enough but I missed the requirement
for evidence. Would have been helpful to give a long list of possible sources.
Visit discussions useful and
professional
Visit discussion was very useful, there were no great surprises
Report received promptly, and
contents as expected from visit
discussions
Not received yet ( only 2 weeks ago)
Process helpful and fair
Process very helpful. Stimulated discussion and action within the hospital before the
review and added political muscle to help implement long overdue change in some
areas.
Other comments
Time consuming but completely worthwhile process. A continuous cycle of review
of the process will help to make it less time consuming and more rewarding
Process development
• Peer review intended to be a collaboration with users
– Proposals on APA website – colleagues invited to feed back
– SWACA pilot
• Changes to self-assessment
– drop case studies
– flexibility in training and experience criteria
• Changes to process
– better guidance on how to do it
– standard templates for questionnaires, reports, feedback
– addition of out-of-region review team member
• Good ideas
– children included in reviewing team
Benefits
• Visited department
– Incentive to look critically at local practice and make changes
before writing self-assessment
– Team building within the department and with other colleagues
– Reports have been useful in getting organisational change and
improvements in facilities and equipment
• Visitors
– Good ideas to manage common problems: meeting life support
criteria, managing scheduling of lists
• Region
– Clinical links reinforced
– Recognition that our colleagues in other hospitals are good
doctors and friendly humans
Misgivings
• Conspiracy
– HCC approach – choose inappropriate criteria and stop good
departments doing good work
– Predator approach – local big hospital stealing work
– Forced marriage approach – departments merge and one gets the
cream
• Cock-up
– Huge amount of work for no outcome
– Report which will be ignored
– Review will have no power to compel change
• May be overcome if some hospitals make a start while
others wait to see how it turns out
SPAN
• Some interest
• Some cynicism
• Risk/benefit balance in favour of giving peer
review a try
• Perhaps start with the psychopathically keen
References
•
Draft proposal for revised peer review scheme
http://www.apagbi.org.uk/docs/APA_PeerReview(Draft).pdf
•
Good Practice: A Guide for Departments of Anaesthesia. The Royal College
of Anaesthetists and Association of Anaesthetists of Great Britain and Ireland,
3rd edition 2006 http://www.rcoa.ac.uk/docs/goodpractice(oct2006).pdf
•
Department of Health (2003) Getting the right start: National Service Framework
for Children - A Standard for Hospital Services
http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndG
uidance/DH_4006182
•
Guidelines for the Provision of Anaesthetic Services. Royal College of
Anaesthetists. 2004 http://www.rcoa.ac.uk/docs/GPAS.pdf
•
Quality in paediatric anaesthesia: a pilot study of interdepartmental peer
review. Crean PM, Stokes MA, Williamson C, Hatch DJ Anaesthesia. 2003
Jun;58(6):543-8.