External Review
... a communications perspective
Anthony F. Camilleri
NCFHE Seminar
June 2015
www.kic-malta.com
What‘s in a Name?
Audit
Expert-Review
objective
Subjective
Precise standards
Broad guidelines
verifies account of client
Interprets account of client
standards-based
review
www.kic-malta.com
Principles of Auditing
• Integrity: the foundation of professionalism
• Fair presentation: the obligation to report
truthfully and accurately
• Due professional care: the application of diligence
and judgement in auditing
• Confidentiality: security of information
• Independence: the basis for impartiality of the
audit and objectivity of the audit conclusions
• Evidence-based approach: the rational method for
reaching reliable and reproducible conclusions in
a systematic audit process
ISO 19011
www.kic-malta.com
A reviewer is a communicator
University
Review Team
Agency
www.kic-malta.com
Communicating
with your team
www.kic-malta.com
What‘s in a name?
review team
a group of independent experts
vs
a unitary entity with its own mind
www.kic-malta.com
Inter-subjectivity
• different subjective perceptions, taken from
different viewpoints, give an objective
view of reality
• the quality basis of an external review
• requires consensus
consensus is not a diplomatic nicety
but an essential pre-requisite of quality
www.kic-malta.com
Before the Review
• Know your team
• Know their profiles
• Know their potential inputs
• Know their biases
www.kic-malta.com
Before the Review
• Discuss everything
• Divide responsibilities for research
• Do not divide responsibility for
analysis and decision
• Form a consensus
• Before
• During
• After
www.kic-malta.com
For each criterion list
• Issues, including features of good
practice, for further exploration
• Evidence provided
• Comments on sufficiency of evidence
• Further evidence to be requested
• Questions to be pursued at the EQA
visit (framework)
• Categories of people to meet during
the EQA visit
www.kic-malta.com
Practical Actions - Chair
Before every interview:
• Take 5 minutes break!
• Discuss for 5 minutes:
– What did we just learn?
– Do we need to change our interview strategy?
– Actively ask for team members‘ concerns
– Divide Responsibility for Questioning in the
next interview within a flexible framework
www.kic-malta.com
Practical Actions - Chair
During every interview:
• Introduce the team
• Make sure everyone in the team plays
an equal role (as per interview
framework)
• Google Docs!
www.kic-malta.com
Communicating
with the institution
www.kic-malta.com
Communication starts
with self-assessment!
• This is not a one-way exercise!
– Has the institution told you enough to allow a
successful review?
– What don‘t you understand, and who can
answer your questions?
• Be active in schedule-design
www.kic-malta.com
Running
a Conversation
• ‚Get to know each-other‘
• General discussion of role in quality
culture
• Specific questions based on report
• Test your assumptions
• Clarify all doubts
www.kic-malta.com
The Four Types of Interviewee
The
Show-Off
The
Politician
The
Grumpy
The
Bored
www.kic-malta.com
Dealing with: ‚The Show-Off‘
Main Characteristic:
„I want to talk about how great I am or we are“
Interview Strategy:
• refer to the self-assessment report
• don‘t ask person to self-evaluate
• ask for explanations of various aspects
www.kic-malta.com
Dealing with: ‚The Politician‘
Main Characteristic:
„This is a negotiation, and I need to manage the
interview“
Interview Strategy:
• win over the subject (in my university....)
• ask very specific questions
• demand direct answers
• surprise them
www.kic-malta.com
Dealing with: ‚The Grumpy‘
Main Characteristic:
„I am unhappy, and I want to complain“
Interview Strategy:
• do not focus on the complaint excessively
• ask for corroborating evidence
• focus on process for redress
• steer towards constructive self-assessment
www.kic-malta.com
Dealing with: ‚The Bored‘
Main Characteristic:
„I don‘t know why I‘m here, and I don‘t want to be“
Interview Strategy:
• explain your mission in relatable terms
• find out what subject is interested in
• focus on content, and extrapolate process
• ask for constructive evaluation
www.kic-malta.com
For all interviewees....
TAKE CONTROL OF THE CONVERSATION
AT THE BEGINNING
(or you‘ll never get it back)
• Introduce the team from a • Explain the objectives of the
position of authority.
discussion clearly
• Set out ground-rules for the • Play the host
conversation
• Have a proper chain of
command
www.kic-malta.com
NEVER
• get into conflict with your interviewee
• confuse the interviewee with the
institution
• Issue judgments or opinions
• leave the room with questions
unanswered
• breach confidentiality
www.kic-malta.com
Demand RESPECT
• Self-assessment should be complete
• All interviewees promised must attend
– even if review schedule changes
• All questions must be answered in full
• Any and all documents should be
provided
• Logistical support should be provided
www.kic-malta.com
Communicating
with the agency
www.kic-malta.com
„I don't know anything about
art, but I know what I like“
Gelett Burgess
www.kic-malta.com
„I don't know anything about
the standards, but I know
quality when I see it“
Unnamed reviewer
www.kic-malta.com
Standards of Proof
•
•
•
•
Some credible evidence
Preponderance of evidence
Clear and convincing evidence
Beyond reasonable doubt
www.kic-malta.com
In your Report
• Be yourself
Bad: The institution showed....
Good: The review team saw / found /
observed....
www.kic-malta.com
In your Report
• Be specific
Good: The institution showed....
Better: The review team found multiple and
consistent examples of
www.kic-malta.com
In your Report
• Say what you know
Bad: The institution lied....
Good: The review team found
inconsistencies between evidence (x) and
interview (y)
www.kic-malta.com
In your Report
• Give your opinion (where relevant)
Bad: The institution is... / or NOTHING
Good: We suspect, It seems likely that,
Given the evidence available, etc...
www.kic-malta.com
In your Report
• Give your reasoning
Bad: There is no quality.
Good: When we consider (X), (Y) and (Z),
we find it impossible to say there is quality
www.kic-malta.com
In your Report
• Link Effect with Cause
Bad: Quality systems are in place, but there
is no evidence of iterative improvement.
Good: Quality systems are not effective, due
to lack of iterative improvement procedures.
www.kic-malta.com
In your Report
• Be CLEAR
Bad: The physical conditions of classrooms are
in need of improvement.
Good: Classrooms are in a dismal state – no
heating, broken desks and no boards make it
impossible to learn
www.kic-malta.com
Grazzi!
Thank-you for your attention
Questions?
Anthony F. Camilleri ([email protected])
Presentation available from: http://www.slideshare.net/anthonycamilleri/
Released under a Creative Commons
Attribution ShareAlike 3.0 Slovenia License
You are free:
• to Share — to copy, distribute and transmit
the work
• to Remix — to adapt the work
Under the following conditions:
Attribution — You must attribute the work in the
manner specified by the author or licensor (but
not in any way that suggests that they endorse
you or your use of the work).
Share Alike — If you alter, transform, or build
upon this work, you may distribute the resulting
work only under the same or similar license to
this one.
www.kic-malta.com