Assessing competencies
evidence level, sufficiency, efficiency
Claire Hardiman
Michael Thomas
www.nshcs.org.uk
Professional Lead for Medical Physics
and Clinical Pharmaceutical Sciences
Professional Lead for Blood and Infection Sciences
@NSHCS
@NHS_HealthEdEng
Today’s aim
Assessing competencies
 What is evidence?
 Is it sufficient?
 Is it efficient
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Objectives
 What is competence and
competency?
 Recognise what traits we are
seeking to develop
 How assessment of competency
can be applied
 What is suitable evidence;
quantity versus quality?
 Answers to common questions
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
What is competence and competency?
 Competence: the ability (or
qualification) of an individual to
do a job properly
 Competency: a set of
knowledge, skills, experience
and other attributes necessary
to do the job properly
 Competence is the outcome
and competencies the inputs to
achieve the task
www.nshcs.org.uk
@NSHCS
Synonyms:
capability · ability · capacity ·
proficiency · accomplishment ·
adeptness · adroitness ·
knowledge · expertise ·
expertness · skill · skilfulness
prowess · mastery · resources
· faculties · facilities · talent ·
bent · aptitude · artistry ·
virtuosity · savvy · know-how
@NHS_HealthEdEng
Knowing how
to do the job
Understanding
policies &
procedures
@NSHCS
Being able
to apply
knowledge and
skills
consistently
Being
Competent
means
Fitting in with
others
In the
workplace
www.nshcs.org.uk
Understanding
why it should
be
done that way
Dealing with
every
day problems
Being able
to transfer
skills
to different
situation
Being able
to do different
tasks at the
same time
@NHS_HealthEdEng
Assessment:
Miller’s Pyramid
Expert
action
performance
competence
Novice
knowledge
KNOWS ABOUT
HEARD OF
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Assessment Hierarchy
• Awareness
Level 1
• The trainee has been introduced to the process/procedure associated
with the competency:
• Performance
Level 2
• The trainee has repeatedly performed the process/procedure (supervised)
with increasing confidence:
• Proficient
Level 3
• The trainee has demonstrated repeated successful performance of the
process/procedure (indirect supervision):
• Competent
Level 4
• The trainee performs the task(s) referring infrequently to their supervisor
as required:
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Assessment Hierarchy: Level 1
• Awareness
Level 1
• The trainee has been introduced to the process/procedure associated with
the competency:
•
The trainee has read all relevant SOP’s, COSHH and Health and Safety and other
recommended documents.
•
The trainee has an introductory level of knowledge and understanding of the
application of the process/procedure.
•
The trainee has been shown how the process/procedure is performed and
allowed to perform the task(s) under supervision.
•
The trainee requires direct supervision.
• This Level may be applied for competencies and assessments
undertaken on rotation in the first year.
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Assessment Hierarchy: Level 2
• Performance
Level 2
• The trainee has repeatedly performed the process/procedure (supervised)
with increasing confidence:
•
The trainee has a knowledge and understanding of the task(s) and is able to
identify situations when they should be applied or are relevant.
•
The trainee performs the task(s) with few or no errors and asks fewer questions
related to the task.
•
The trainee may only require indirect supervision.
• This Level may be applied for competencies and assessments
undertaken on rotation in the first year and when performing
tasks from Specialist Modules.
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Assessment Hierarchy: Level 3
• Proficient
Level 3
• The trainee has demonstrated repeated successful performance of the
process/procedure (indirect supervision):
•
Trainee has developed a level of knowledge & understanding of the competency
that allows them to critically analyse the task(s) and outcomes produced.
•
The trainee is able to identify potential sources of error and can correctly resolve
problems that may occur.
•
The trainee is able to successfully perform the task(s) without supervision.
• This Level is achievable for most procedures from the Specialist
Modules.
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Assessment Hierarchy: Level 4
• Competent
Level 4
• The trainee performs the task(s) referring infrequently to their supervisor
as required:
The trainee has repeatedly demonstrated a level of knowledge, skill and aptitude
of the competency to work with a level of independence but still recognising
their scope of practice. They are able to demonstrate or train other staff in the
processes and procedures relevant to the competency.
•
• This Level is expected for most procedures from the Specialist
Modules.
• This Level compliments the Good Scientific Practice Domain of Clinical
Leadership:
• Readiness for practice e.g. Shows competency at a level that is appropriate
for a newly registered clinical scientist, is a ‘safe pair of hands’, dependable,
trustworthy, efficient, knowledgeable about their specialism.
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Competencies and Portfolios
 The Learning Guides define
“Competencies”, or work activities,
to be Undertaken and Understood.
 Trainee needs to complete all of
these, and create a portfolio of
supporting evidence.
 The evidence supports the trainee’s
own learning as a record of what
was actually done for future
reference/e-portfolio for
CPD/Professional Regulation
purposes.
 Assessor needs to review, comment,
and sign off the evidence.
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Decisions on competency
 Consider the levels described and
how they may apply differently to
rotations and specialisms during the
course of training.
 Each module has a number of
assessments and these provide an
opportunity to demonstrate
competency.
 Use the detail given in the Learning
Guides. Next to each competence
there are knowledge and
understanding statements.
www.nshcs.org.uk
@NSHCS
• Awareness
Level 1
• The trainee has been introduced to the
process/procedure associated with the
competency
• Performance
Level 2
•The trainee has repeatedly performed the
process/procedure (supervised) with increasing
confidence
• Proficient
Level 3
• The trainee has demonstrated repeated
successful performance of the
process/procedure (indirect supervision)
• Competent
Level 4
• The trainee performs the task(s) referring
infrequently to their supervisor as required
@NHS_HealthEdEng
Finding Evidence for Competencies
 There is considerable variation in
the amount and type of evidence
and feedback being uploaded on
OLAT
 Providing evidence is not just a
‘tick box’ exercise!
 But, don’t overload the trainees
by asking for 10,000 word essays!
 Requires good communication
between the Training Officer and
trainees
 Why not ask other 2nd and 3rd
year trainees
www.nshcs.org.uk
@NSHCS
 What isn’t right:
 “My supervisor saw me do this”
 10 page extract from standard
textbook or SOP
 What about Plagiarism?
Demonstrates that trainee
undertook it, and understood it
 Supervisors have a responsibility
to make sure the evidence is the
trainee’s own work.
@NHS_HealthEdEng
Evidence for Competencies
Ensure:
 that a competency is completed for each
Learning Outcome that it applies to
 that there is supporting evidence for each
Learning Outcome
 that they relate to the work of clinical
scientists in that specialism
 Project work can be a very
motivational and efficient way to
complete competencies
 Discuss types of evidence with your
trainee
 Encourage them to be innovative
 Evidence should show that the trainee
 Avoid essays for every competence
Undertook and Understood the
activity
 Use evidence wisely
 Upload copies of work completed –
 Many competencies can be grouped
this may be local documentation
together, and single pieces of
evidence can be used to demonstrate  Ensure patient identifiable material is
not used.
their completion
Get them to do it as they go along and not all at the end…
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Looking for Evidence
 Documents that support the delivery of the training e.g. results and brief
summary documents that evidence knowledge skill and understanding
 If the evidence relates to a diagnostic test:
 use anonymised pre and post test results
 during the assessment ensure the trainee
understands the impact on the patient
 Good evidence includes:
 flowcharts, witness statements,
 trainee presentations,
 reference to regulations & supporting literature
 Use evidence prepared for equipment validation and ISO audits as evidence
on OLAT
 Consider a project based approach, and using one set of evidence to satisfy
multiple competencies
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Ideas for Evidence
“General competency”
“Clinical competency” “Professional competency”
the trainee could upload:
 a description of the
problem/clinical issue being
considered
 a case report, treatment
plans,
 analytic results etc.
 use references to academic
papers or guidance
documents about the
condition.
 Evidence prepared for other
purposes, e.g. routine
calibrations, audits etc. can
be used
the trainee could upload
anonymised information
such as:
 test results
 a management plan
 evidence they
understand the
impact on the
patient
www.nshcs.org.uk
@NSHCS
the trainee could show
engagement through upload of
evidence of
 Raising awareness
 Being inspirational to others
 Getting involved
 Becoming an ambassador
 Spreading the word
Signing Off Competencies:
Normally undertaken by the work place supervisor
but some professional competencies may appropriately be
signed off by the academic provider
@NHS_HealthEdEng
Level of knowledge expected
Appropriate to Master’s degree level
 A systematic understanding of
knowledge, including, where
appropriate, relevant knowledge
outside the field and/or discipline.
 A critical awareness of current
problems and/or new insights, much of
which is at, or informed by, the
forefront of their academic discipline,
field of study, or area of professional
practice.
www.nshcs.org.uk
@NSHCS
M
a
s
t
e
r
s
@NHS_HealthEdEng
Finding the level
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Finding the level
• Graduate:
PTP
• Perform quality control procedures for radiotherapy systems, including orthovoltage treatment
units, megavolatage units and other radiotherapy treatment units (e.g. high dose brachytherapy,
tomotherapy units).
• Masters:
STP
Radiotherapy
Module 1 : Dosimetry and Treatment Equipment
• At the end of this module the trainee will be able to perform a range of measurements associated
with a treatment beam and to ensure that the equipment is suitable and ready for clinical use. They
will understand and apply the relevant codes of practice and be able to perform limited patientspecific treatment and analysis.
• Perform required measurements to characterise a treatment beam.
• Perform required measurements to establish a treatment machine is suitable for clinical use.
• Interpret results and instigate corrective action where required.
• Doctorate:
HSST
Module 5 : Quality Control of Radiotherapy
Strategic Role in Managing Medical Equipment
• By the end of the module the Clinical Scientist in HSST will be able to analyse, synthesize, evaluate
and apply their expert knowledge of the medical equipment life cycle. They will also be able to
provide scientific and technical advice on appropriate medical equipment, its state of the art and
emerging trends and its strategic planning within the healthcare organisation. The Clinical Scientist
in HSST will be sensitive to the interest of various different….
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Examples
 The trainee needs to demonstrate the knowledge and/or
practical skills required for each competency but they do not
need to prove they are competent to do this task alone
 They upload evidence to OLAT that demonstrates this
 Large range of evidence can be used and it may be possible to use
one piece of evidence for several competencies or several pieces
of evidence for one competency may be needed
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Examples
 Evidence consisted of photos
of an experiment done with
HIFU equipment + a report
 Others have done literature
reviews for this competency –
purely theoretical
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Examples
 Performed QA on a general x-ray
set under supervision after being
trained with the QA equipment –
routine department work
 Wrote a report in university lab
report style (including a report of
results for the chief radiographer)
 Sufficient evidence for 7
competencies and 1 DOPS
www.nshcs.org.uk
@NSHCS
Change image acquisition
parameters
Undertake image quality
tests
Operate a basic range of
radiographic x-ray equipment
Measure the parameters of
an AEC
Assess image quality on a
plain x-ray machine
@NHS_HealthEdEng
Example feedback
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
But we don’t do that…
 Competences not done on site or ‘outdated’
 Understanding of pre and post
clinical/analytical processes and how
results affect the management of the
patient
 Try to visit other departments that may
do that test/procedure; consider
collaborative/consortia working
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Give feedback; encourage reflection
 Evidence
supports the trainee’s own learning as a
record of what was actually done for future
reference/e-portfolio for CPD/Professional
Regulation purposes
 Feedback
is crucial to supporting the trainee’s learning.
It needs to be supportive, constructive and
allow 2-way communication
I liked how you
explained the
expected learning
outcomes
You were clear and
precise when saying
the key new concepts
you learnt
Can you tell me about
the actions you used
to achieve this?
You were able to
evidence how your
knowledge has
changed
 Reflection
is also a very important aspect to the trainees
learning and can be used to add to evidence
already uploaded and accepted.
And, to cover learning and activities covered
in the Clinical Experiential Learning (start of
each module)
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Who’s watching who?
 Monitor the trainee’s progress
throughout
 The School will also be doing this
 Evidence of good progression will be
necessary
 Lack of evidence on OLAT could have
implications on a trainees ability to be
allowed to progress to the next stage of
the programme
e.g. rotational to specialism or to
access the OSFA’s.
 Progression rules were piloted in
Summer 2016 for HSST
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Summary
•
•
•
•
•
•
•
Roll out training on OLAT to rotational
Supervisors to ensure standardisation of
assessment of STP trainee
Use the detail in the Learning Guides. Next to
each competence there are knowledge and
understanding statements
Encourage the use of learning and assessment
from the degree (two-way)
Good constructive feedback is essential to
develop your trainee
Encourage reflective practice
Apply competence(y) appropriate for the level
of training
Remember the professional practice
competences!
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Questions
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Some Questions are guaranteed…
Some competencies ask for experience in techniques
that are now out-of-date and no longer used. Is it
sufficient to write about these techniques, or can we
just state that other techniques have now replaced this?
Who is suitable to sign off these competencies?
Many of the competencies are repetitive, sometimes
within the same specialism and sometimes between
specialisms. Is it okay to cut and paste information from
one competency to another, if they are essentially
discussing the same thing?
Can we use historical experiences as evidence for
professional practise competencies?
Answers are not!
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
Some Questions are guaranteed…
It is very difficult to get people to sign off competencies for
OLAT. This seems to be an on-going issue. Is there anything
the NSHCS can recommend to tackle this?
It would be useful to have more examples of competencies
that we and our supervisors could use as a guideline (what
can be used as acceptable form of evidence etc.).
One of the main issues about carrying out rotations in
different departments is that they don’t seem to know
what is required for our training, and sometimes haven’t
been informed that we are joining them. It would be better
if departments are given clearer guidelines (other than the
competency list) of what is expected of trainees and how
long we are expected to spend on different sections.
Answers are not!
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng
www.nshcs.org.uk
@NSHCS
@NHS_HealthEdEng