Influence of the Criteria Changes
on the Supervisory Rating
Márta Juhász
Budapest University of Technology and
Economics
Dep. of Ergonomics and Psychology
E-mail: [email protected]
15.05.2003. Lisboa
previous study1: Assessment process
Assessment process in 1996 at the Public Inquiry
Service of a Hungarian Telecom Organization.
Motivation of the organization to develop a new selection
process was:
•the introduction of the new information technology
•to have the more suitable employees who were able to
improve and keep their job performance high.
previous study1: Assessment process
Job and criteria analysis
•results of previous job analysis
•analysis of documents
•survey among supervisors, employees
•group discussions
•working day analysis
I. Cognitive ability (Memory, Attention,
Emphasing the importance, Comprehensihon)
II. Personality traits (Politeness, Self-discipline,
Tolerance)
previous study 1: Assessment process
Development of the assessment methods
We measured the cognitive ability with well-known
techniques and with our own instruments.
The personality traits were measured by a Big Five
personality questionnaire which consists of five
major domains (factors) of personality, as well as the
six facets (traits) that define each domain. (N=344)
I. Neuroticism (N)
II. Extroversion (E)
III. Openness to experience (O)
IV. Agreeableness (A)
V. Conscientiousness (C)
previous study 2: Validation process
The validation process was in 1997.
• According to the results from the assessment methods
the employees could be divided into 3 groups.
Suitable
SuitableWithDevelopment
Unsuitable
•Our hypothesis was that among these groups there
would be significant differences in the performance
measured.
•This was a validation process of our assessment
methods.
•(N=100)
Methods of the validation process
perf.analysis
based on
IBM system
perf.analysis
operators
based on
rating by
ACD system
each other
Supervisory
Methods of
rating
validation process
speaking style
evaluation
work-phase
analysis
categorising
and
analysing
the errors
analysing
occupational
status
The supervisory rating
For the evaluation of the employees’ performance by
their supervisors a scoring sheet was used which
contained the exact determination of the dimensions
to be used.
Supervisors used a 4-point scale for the evaluation.
The rating categories of the scoring sheet were
determined by the criteria of the selection.
1) Achievement, 2) Accuracy, 3) Software
Knowledge, 4) Endurance, 5) Quietness, 6)
Politeness, 7) Good communication skills, 8)
Pleasant voice, 9) Understanding of speech, 10)
Memory, 11) Good relationship with the colleagues.
Questions to consider
With the changes in the circumstances of the
organisation, requirements of fulfiling this job have
been changed, or not?
The supervisory rating about the employees’
performance is actually followed by criteria
changing, or not?.
Depending on the criteria changing the different
employees’personality will be preferred by
supervisors, or not?
Hypothesis and methods
In 1996 the whole organisation was in a learning
period. We supposed the role of the cognitive abilities
was more important than the personality traits.
The predictive validity of the Agreeableness (A),
Conscientiousness (C) and Extroversion (E)
personality factors are crucial in the service job.
Supervisory ratings applied in 1997 have been
repeated in 2000 and 2002.
We wanted to get information about:
•
the changes of criteria and their reflections in the
supervisory ratings during the 6 year period.
The differences of the supervisory rating were pointed
out through the operators’ personality.
Criteria analysis
NIT
Rating
1996
1. Emphasizing the
importance
2. Quickness
3. Attention
4. Software Knowledge
5. Politeness (A)
6. Helpfulness (A)
7. Memory
8. Flexibility (O)
9. Accuracy
10.Tolerance (N)
2002
1. Patience (N)
2. Politeness (A)
3. Accuracy (C)
4. Communication skills
5. Helpfulness (A)
6. Intelligence
7. Emphasizing the
importance
8. Quickness
9. Tolerance (N)
10. Dutifulness (C)
Characteristics of the supervisory
ratings
1997-2000-2002
Cognitive performance Affective performance
factor
factor
Understanding of speech
Memory
Software knowledge
Achievement
Good communication skills
Accuracy
Endurance
Quietness
Politeness
Pleasant voice
Self-discipline
Good relationship with
colleagues
Co-operation
0,25 0,25 0,22
0,2 0,22
0,3
0
0,2
-0,1
0,1
E
EEX
-0,3
OFA
OVA
O
E
EAC
-0,1
ATR
-0,2
0
AMO
OAC
O
NAN
EPE
EGR
EEX
E
ACO
C
0,28 0,26 0,29
0,26
0,25
0,3
O
0,23
0,2
G
0,1
N
0
I
-0,1
T
-0,2
-0,21
I 1 -0,3 -0,27
V 9
E 0,39 0,23
0,25
0,24
7
0,2
-0,4
-0,2
2
0
0
2
2 -0,3 -0,24
0
0 0,45 0,26
0,22 0,21 0,26
0,25
0
-0,32
-0,5
-0,35
NSC
NDE
EAC
E
AAL
NAH
EAC
AM
O
A
A
F
0,43
0,37 0,37
0,5
F
0,3
E
0,1
0,05
C
0,1
-0,1
-0,15
0
-0,3
T
-0,22 -0,2
-0,35
-0,31
ACO
ATM
NSC
-0,32 -0,34
-0,5
I
V
E
NAN: Anxiety, NSC: Self consciousness, NAH: Angry hostility, E: Extroversion,
-0,45
EEX: Excitement seeking, EGR: Gregariousness, EPE: Positive emotions, EAC:
Activity, O: Openness to experience, OAC: Actions, OVA: Value, OFA: Fantasy, A:
Agreeableness, ACO: Compliance, AMO: Modesty, ATR: Trust, ATM: Tender
Mindedness, AAL: Altruism.