Games are Better than Books: In-Situ Comparison of an
Interactive Job Interview Game with Conventional
Training
Ionut Damian1 , Tobias Baur1 , Birgit Lugrin1 , Patrick Gebhard2 , Gregor Mehlmann1 ,
and Elisabeth André1
1
Augsburg University, {damian,baur,lugrin,mehlmann,andre}@hcm-lab.de
2
DFKI GmbH - Saarbrücken, [email protected]
Abstract. Technology-enhanced learning environments are designed to help users
practise social skills. In this paper, we present and evaluate a virtual job interview training game which has been adapted to the special requirements of young
people with low chances on the job market. The evaluation spanned three days,
during which we compared the technology-enhanced training with a traditional
learning method usually practised in schools, i.e. reading a job interview guide.
The results are promising as professional career counsellors rated the pupils who
trained with the system significantly better than those who learned with the traditional method.
This is a paper draft. Final version to be made available at link.springer.com
Keywords: Technology-enhanced Training; Serious Game; Social Coaching; Job Interview; NEET; User Study; Virtual Agent
1
Introduction
As a consequence of worldwide economical and financial crisis throughout the last
decade, many countries are facing a rising number of people who are not in employment, education or training (NEETs). Especially young adults with low socio-emotional
and interaction skills [12, 8], such as a lack of self-confidence or sense of their own
strengths struggle to convince recruiters of their fit in a company during job interviews.
To address this issue, governments take action by introducing job interview training
early in graduation years of school as well as support various organisations which coach
youngsters in finding jobs and improving social skills pertinent for job interviews.
Compared to classical learning approaches (e.g. coaching), technology-enhanced
solutions such as serious games present themselves as viable and advantageous alternatives [16]. Their automated nature gives users access to personalized feedback without
the need for human coaches, improving scalability and repeatability. This reduces the
running costs of such systems making them a viable solution for a larger user group.
From a recruiters point of view, the goal of a job interview is to determine the fit
of the candidate to a particular position in the company by evaluating the candidate’s
verbal (i.e. content of utterance) and nonverbal behaviour (e.g. use of voice, gestures,
postures, facial expressions). Nonverbal behaviour is particularly critical as research
shows it takes a significant role during interpersonal interaction [4, 13]. Furthermore,
studies [5, 9] show that nonverbal behaviour has a large impact on the outcome of a job
interview. Training such behaviour can therefore be very beneficial to improving ones
chances for employment.
In this paper we investigate the potential of a virtual job interview game for training
young adults. The application was developed in scope of a larger research project [1]
that aims at creating a scenario-based simulation platform for young people to explore,
practise and improve their social skills in the domain of job interviews. We build upon
this application by adapting it to a narrower target group, i.e. 13 to 16 year old pupils
from a German school of lower education.
In the resulting system, the user takes part in a gamified job interview led by a
virtual character. Using social signal processing techniques, the system records and
analyses the user’s nonverbal behaviours which are used to trigger actions for the virtual
characters in realtime, but also as material for the debriefing phase.
The system is evaluated in a three day study during which we measured the impact of the system on the pupils’ job interview performance and compared it to a conventional learning method commonly used by the school (learning from a written job
interview guide). We were able to measure statistical significant improvements for the
pupils who interacted with the system but not for those who used the written job interview guide. Furthermore, after the final day, professional practitioners rated the overall
performance of the pupils who used the system significantly better than of those who
used the written job interview guide.
2
Related Work
A variety of methodologies have been developed for training social behaviour. Most
common techniques involve the learner memorizing certain behavioural patterns either
from a written source or from audio or video tutorials. More advanced forms rely on
human coaches who help the learner practise the learned behaviours through the use of
various exercises such as role-plays [7, 11] and video feedback.
Computerized social skill training tools have seen rapid evolution in the recent years
due to advances in the areas of social signal processing as well as improving virtual
characters. Such tools are meant to complement or even substitute traditional training
approaches [2, 10, 14].
The effectiveness of such technology-enhanced learning systems has been the focus of various studies in the recent years. Investigations conducted by Pan et al. [14],
for example, suggest that a party simulation involving a virtual female agent can help
reduce social anxiety in young adult males. Sapouna et al. [15] studied the effect of a
virtual learning system to reduce the bullying victimisation rate of children in schools.
Their results show that the system had a positive effect on the children’s abilities to
cope with bullying. Hoque et al. [10] explored the impact of a job interview training
environment on MIT students. They conclude that students who used the system to
train, experienced a larger performance increase than students who used conventional
methods. These results are encouraging for our research. However, while Hoque et al.
recruited MIT students as participants, our target group was job-seeking youths who
have been categorized as being at risk of exclusion. The ambition of our work was to
perform an in-situ study at a local school to investigate the impact of a job interview
training game on underprivileged youngsters. Furthermore, the study was embedded in
the existing curriculum of the school. This specific situation raised high expectations
from teachers and pupils which had to be met by the software.
To sum up, while related work points out the great potential of technology-enhanced
training, to the best of our knowledge none managed to ecologically validate the effectiveness of such tools in the domain of job interviews in-situ, i.e. in a school with real
job seekers at risk of exclusion.
3
User Group
The aim of the present contribution is to test a job interview training system, which
was developed as part of our previous work [1], in-situ and evaluate it with final year
pupils in the age range of 13 to 16 of a school in Bavaria, Germany. Unlike schools
in most other countries, the Bavarian school system foresees the same education for
all children only for the first four years of education. Afterwards, children are split up
to join different school types based on the grades in the fourth grade and their parents
intention, and are assigned to either Mittelschule (preparing for vocational education),
Realschule (providing a broader range of education for intermediate pupils) and Gymnasium (prepares pupils to study at a university). One of the effects of this structure
is that pupils who join a Mittelschule receive a lower level of education and thus have
fewer chances to find a qualified job, especially as they have to compete with pupils of
a higher education due to the current situation on the job market.
Regarding the overall aim of our project (to help youngsters that are in danger of
not finding a job), pupils in the final grades of the Mittelschule are best suited for our
target user group. Their lack of job interview pertinent skills is also known to their
teachers. To help their pupils prepare to find a job, eight-graders of a Mittelschule are
given lectures on writing applications and curricula vitae as well as instructions on job
interviews. “Many of our pupils are having a hard time preparing to find a job and many
have no assistance from their parents.” says the deputy headmaster of our cooperating
Mittelschule. “A technical system that puts pupils into prototypical situations in job
interviews would be very helpful for their preparation.”
4
Technology-enhanced Training System
The training system was developed as part of the European Project TARDIS [1]. The
aim of the project was to help young adults improve their nonverbal behaviour during
job interviews. The system enables users to take part in a job interview simulation where
a virtual character plays the role of the recruiter (Figure 1 (left)). The virtual character
is able to perform both proactive and reactive behaviour.
The proactive behaviour consists of a series of questions asked by the virtual character. After each question, the virtual character waits for the user to answer while displaying backchanneling behaviour, such as nodding, head tilting, maintaining or breaking
Fig. 1. The virtual character playing the role of an interviewer (left). The game cards give hints
to the pupils regarding appropriate behaviour for upcoming interview phases (right).
eye gaze. To facilitate such reactive behaviour the system analyses the nonverbal behaviour (gestures, postures, body expressivity, facial expressions and use of voice) of
the user in realtime during the interaction using social signal processing techniques and
various sensors [2, 6].
Besides impacting the behaviour of the virtual character, the results of the behaviour
analysis are also stored for later use in a semi-automatic debriefing phase. During this
phase, the user’s recorded behaviour is displayed side-by-side with the virtual character’s behaviour to facilitate an easy analysis of the interaction [3]. This supports the
identification of critical incidents which might have an impact on the outcome of a real
interview. Furthermore, various metrics are extracted from the behaviour of the user to
enable between-session comparison and improvement tracking.
To ensure the system was suitable for deployment in the learning environment of
our cooperating Mittelschule, we conducted workshops with the teachers at different
stages of the prototype. Due to their long experience in working with the pupils, the
teachers were able to provide us with detailed feedback regarding teaching pitfalls and
motivation techniques. Based on their recommendations, the system was incrementally
adapted to meet the requirements and abilities of the pupils. The largest concern the
teachers had regarded the system’s ability to keep the pupils engaged and motivated.
Thus, various game-like elements have been added to the system. More precisely, we
introduced physical game cards (Figure 1 (right)) which are similar in appearance to
those of classic board games and give hints on how to behave during the interview. A
scoring system keeps track of how well a pupil follows these hints. For example, if the
pupil smiles at an appropriate moment, the score will get incremented by one. The score
is meant to act as an incentive to replay the game in order to achieve a better score.
Further, we implemented two job interview scenarios which teachers characterized
as most appealing for the pupils of our cooperating Mittelschule: electro-mechanical
engineer and trained retail salesman. The interview simulation has been split up into
three phases (Welcome, Company Presentation, and Strengths and Weaknesses) to give
the training exercise more structure. For each phase, a specific game card has been
designed as illustrated in Figure 1 (right). Prior to each phase, the virtual character
instructs the pupil to pick up the corresponding game card, read it carefully and put it
back on the table. After each phase, the virtual character gives the user feedback on how
she or he performed.
Fig. 2. Mock job interview with professional career trainers (left) and interaction with job interview training system (right)
5
Evaluation
In order to test our adapted system with the target user group, we conducted a user
study in a cooperating Mittelschule (Parkschule in Stadtbergen/Germany). The objective of the user study was twofold: On the one hand we wanted to investigate whether
pupils’ skills are rated better by practitioners after using the interactive job training
game compared to before. On the other hand, we wanted to evaluate whether their skills
are rated better, or at least equally well, in comparison with pupils who trained using
conventional teaching methods.
Participants. In total, 20 pupils (10 male and 10 female) from the eight and ninth
grade (final and second to last years) have been recruited to take part in the study.
Participants were aged between 13 and 16 (mean = 14.37; SD = 0.94). The data of one
participant had to be removed due to extraordinary circumstances resulting in nervous
and unfocused behaviour (she accompanied her friend to the hospital after a minor
accident).
Additionally, two career counsellors participated in the study as professional practitioners. The career counsellors are employed full time at Career Service - Augsburg
University, where they advise students on choosing suitable jobs, preparing their application documents such as CV, and training for job interviews.
Procedure and Apparatus. The user study was conducted over the course of three days.
An overview of the procedure can be seen in Table 1. On the first day, all pupils participated in mock job interviews led by a practitioner (see Figure 2 (left)). The purpose
Table 1. Procedure of user study over three days.
experimental group
day 1 mock job interview
day 2 interaction with training system
day 3 mock job interview
control group
mock job interview
training with book
mock job interview
of these mock interviews on the first day was to establish a baseline regarding the job
interview performance of the pupils prior to additional training. Furthermore, as the
system’s goal is to help the users improve their nonverbal behaviour, the practitioners
were also asked to focus on the nonverbal behaviour, i.e. how the participants answer
rather than what they say. Two interviews were carried out in parallel in separate rooms
whilst each lasted for approximately 7 minutes. This duration was deemed sufficient by
the practitioners to get an objective measurement of the pupils’ job interview performance. After each mock interview, both pupils and practitioners filled in questionnaires
A and B respectively.
In Questionnaire A, practitioners rated 1) the pupil’s overall performance, 2) whether
they would recommend the pupil for employment, 3) appropriate usage of smiles, 4) appropriate usage of eye contact, 5) appropriate usage of gestures, as well as whether the
pupil seemed 6) nervous 7) interested and 8) focused. In Questionnaire B, pupils selfreported on whether they thought they 1) performed well in the interview, 2) were nervous, 3) used a lot of filler words such as “er” or “uhm”, 3) were focused, 4) were aware
of their non-verbal behaviour and 5) performed appropriate non-verbal behaviour. Both
Questionnaires used Likert scales ranging from 1 to 7, with a higher value indicating
a better performance. The only exception is the dimension nervousness, where a lower
score is considered being better.
On the second day, pupils were randomly divided into experimental group (EG) and
control group (CG), resulting in four females and six males for the EG and five females
and four males for the CG.
The EG interacted with the interactive training system adapted for the school. Figure 2 (right) shows a sample interaction with the system where a pupil analyses one
of the game cards, before starting the interaction phase. The participant was seated at
a school desk with a Microsoft Kinect and a webcam positioned to face her or him.
During the interaction, the participant was also wearing a close-talk microphone. Each
training lasted for about 15 minutes, split between game interaction and debriefing.
During the training session, the pupils’ nonverbal behaviour was recorded and analysed
by the system. In the debriefing phase, a researcher assisted the pupils in reviewing their
performance using the debriefing mode of the training system. However, the researcher
only provided technical support with the system and helped the pupils understand the
interface.
Pupils of the CG were reading a job interview guide3 for the same amount of time.
The written guide is published by a local youth advisory institution and regularly used
by our cooperating school to prepare their pupils for job trainings.
On the third day, a second round of mock job interviews was conducted with each
participant. Pupils of both groups (EG and CG) were brought to the practitioners in
random order, who were unaware of which condition the pupils have been assigned to
during the second day. After each mock interview, pupils and practitioners filled in the
same questionnaire they filled in during day one (questionnaires A and B respectively).
This allowed us to compare the pupils’ performance between day one and three.
3
https://www.aok-on.de/bayern/berufseinsteiger/beruf-zukunft/
koerpersprache-im-vorstellungsgespraech/
Day 1
CG
Day 3
EG
CG
7
7
6
6
5
5
4
4
3
3
2
2
1
1
EG
Fig. 3. Practitioners’ ratings of day one (left) and day three (right) comparing CG and EG. Dimensions marked with ∗ present significant differences between the two groups.
EG
D1
CG
D3
D1
7
7
6
6
5
5
4
4
3
3
2
2
1
D3
1
Fig. 4. Practitioners’ ratings of EG (left) and CG (right) across day one and three. Dimensions
marked with ∗ present significant differences between the two days.
5.1
Results
To determine the quality of the results we used independent two-tailed t-tests when comparing between groups, and paired two-tailed t-tests when comparing between days. In
both cases we apply the Bonferroni-Holm error correction method to adjust the significance levels. Analysing the first day of our experimental setup, no significant differences were found in questionnaires A and B comparing pupils that were later assigned
to either join EG or CG, using the independent two-tailed t-test (see Figure 3 (left)).
Comparing the two groups again after the third day (after either having used the system or the written guide on the second day and performing a second mock interview on
the third day) revealed interesting insights. We found statistically significant differences
for the practitioners’ ratings on overall performance (p = 0.004, α = 0.006), indicating
that pupils of the EG were rated better compared to pupils of the CG. A strong trend
was also found for the recommendation dimension (p = 0.012, α = 0.007). All other
dimensions were also rated better for the EG than for the CG, albeit not significant.
Figure 3 (right) illustrates these results.
Table 2. Mean values of control group (CG) and experimental group (EG) on first and third day.
Significant differences between groups on a particular day are written in bold and marked with ? .
Significant differences within groups between days are written in italic and marked with ‡ .
Questionnaire A
day 1
CG EG
overall performance 4.44 4.9‡
recommendation 4.55 4.7 ‡
smiles
4.44 4.2‡
eye contact
4.44 4.6‡
gestures
3.6 2.8
nervousness
4.33 4.0‡
interest
5.0 5.0
focus
5.0 5.1
day 3
Questionnaire B
CG EG
5.33? 6.2‡? overall performance
nervousness
5.33 6.2‡
use of filler words
5.33 5.7 ‡
focus
5.66 5.7 ‡
4.0 4.1 aware of n.v. behaviour
n.v. behaviour
3.55 2.7 ‡
5.55 5.8
5.55 5.9
day 1
CG EG
4.66 4.6
4.77 4.2‡
4.88 3.4
4.23 3.9
5.0 5.0
5.22 4.7
day 3
CG EG
5.33 5.2
4.33 2.2‡
4.22 3.0
4.56 4.6
5.77 5.4
5.77 5.1
In order to evaluate the improvement of performance for each group individually,
we compared the results within groups between day one and three. Our tests revealed
significant differences for the EG for the dimensions recommendation (p = 0.005,
α = 0.006), overall performance (p = 0.006, α = 0.007), nervousness (p = 0.006,
α = 0.007), eye contact (p = 0.007, α = 0.010) and smiles (p = 0.012, α = 0.013)
(Figure 4 (left)). No significant improvements in the practitioners’ ratings have been
found for the CG when comparing day three to day one (Figure 4 (right)). However,
trends have been found for smiles (p = 0.021, α = 0.006), overall performance (p =
0.035, α = 0.007) and eye contact (p = 0.047, α = 0.007).
Regarding the pupils’ self-assessment, no significant differences were found between the two groups on the first and third day. However on the third day, a strong trend
was found for the nervousness dimension (p = 0.030, α = 0.008), with pupils of the
EG rating themselves as less nervous than pupils of the CG. Comparing the two days
for each group separately, reveals a significant difference on the nervousness dimension
(p = 0.001, α = 0.008) for the EG only, with participants rating themselves being less
nervous on the third day compared to the first day.
Table 2 gives an overview of the mean ratings from both questionnaires on the first
and on the third day for both conditions.
5.2
Discussion
The analysis of the questionnaire data on the first day of our experiment revealed no
significant differences, suggesting that the pupils were performing equally well in their
job interviews. We can thus consider differences observed on the third day between
the groups to be caused by the training completed on the second day. In general, both
groups improved from the first day to the third. This is not surprising considering the
fact that all participants preoccupied themselves with the topic of job interviews over the
course of three days. However, only for the pupils of the EG were we able to observe
significant differences (for the dimensions overall performance, recommendation for
the job, smiles, eye contact and nervousness). Furthermore, when comparing the two
groups on the third day, practitioners rated the overall job interview performance of the
EG significantly better by than that of the CG.
This suggests that the technology-enhanced training had a greater effect on the
pupils’ job interview performance than the traditional method. We consider this very
encouraging, especially since the reading material that the CG was using on the second day is issued by a respectable local youth organisation and is regularly used by our
cooperating school.
The only statistical difference found in the pupils’ ratings was the self-reported
nervousness of the EG between day one and three. This is also interesting as it indicates
that the virtual job training environment might help users feel more comfortable during
job interviews.
The system also left a good impression on the school teachers who stated that “using the system, pupils seem to be highly motivated and able to learn how to improve
their behaviour [...] they usually lack such motivation during class.” As a possible reason for this they mentioned the technical nature of the system, which “transports the
experience into the youngster’s own world” and that the technology-enhanced debriefing phase “makes the feedback be much more believable.” Pupils also seemed to enjoy
interacting with the system. Most of them asked questions regarding how the score was
computed, and which of their behaviours contributed to the final score. This suggests
that the scoring functionality had a positive effect on the pupils’ engagement in the
exercise. Furthermore, the game cards were also received well. One participant even
asked for permission to photograph the game cards so she would be able to study them
at home.
6
Conclusion
In this paper we explored the use of a technology-enhanced training system to aid
youngsters at risk of exclusion in improving social skills pertinent to job interviews.
The system combines game elements with job interview simulation mechanics and debriefing techniques in an attempt to achieve a maximum impact on the youngsters.
Following a three day user study, we found that pupils who work with the training
system improve more than those who use traditional learning methods, i.e. reading a
written job interview guide. More precisely, professional practitioners rated the overall
performance of the pupils who trained with the system significantly better than of those
who did not. Further, only for the pupils who trained with the system were we able to
measure statistically significant improvements.
A major contribution of our work is the in-situ study that shows clear benefits of
computer-based job training systems for underprivileged pupils. To the best of our
knowledge, no other virtual job training system achieved such a result in a similar
ambitious setting. Despite the time- and resource-consuming nature of our three day
evaluation, the school was very supportive in embedding our study into the school curriculum. We are convinced that virtual environments have a great potential to be used as
job interview training instruments on a large scale, extending current teaching practices.
Currently, additional studies in other European countries with different school types are
being conducted.
Acknowledgements. This work was partially funded by the EC within FP7-ICT2011-7 (Project TARDIS, grant agreement no. 288578). We thank the teachers Bernhard Pietzowski and Richard Endraß from the Parkschule Stadtbergen for helping organize the study and the pupils for their participation. We also thank Julia Brombach
and Claudia Lange-Hetmann from the Career Service of the Augsburg University for
volunteering as practitioners.
References
1. Anderson, K., André, E., Baur, T., Bernardini, S., Chollet, M., Chryssafidou, E., Damian,
I., Ennis, C., Egges, A., Gebhard, P., Jones, H., Ochs, M., Pelachaud, C., Porayska-Pomsta,
K., Rizzo, P., Sabouret, N.: The TARDIS framework: Intelligent virtual agents for social
coaching in job interviews. In: Proc. ACE, LNCS, vol. 8253, pp. 476–491. Springer (2013)
2. Baur, T., Damian, I., Gebhard, P., Porayska-Pomsta, K., André, E.: A job interview simulation: Social cue-based interaction with a virtual character. In: Proc. SocialCom, pp. 220–227.
IEEE (2013)
3. Baur, T., Damian, I., Lingenfelser, F., Wagner, J., André, E.: Nova: Automated analysis of
nonverbal signals in social interactions. In: Human Behavior Understanding, LNCS, vol.
8212, pp. 160–171. Springer (2013)
4. Birdwhistell, R.L.: Kinesics and context: Essays on body motion communication. University
of Pennsylvania press (2011)
5. Carl, H.: Nonverbal communication during the employment interview. ABCA Bulletin 44(4),
14–19 (1980)
6. Gebhard, P., Baur, T., Damian, I., Mehlmann, G., Wagner, J., André, E.: Exploring interaction strategies for virtual characters to induce stress in simulated job interviews. In: Proc.
AAMAS. pp. 661–668 (2014)
7. Greene, J., Burleson, B.: Handbook of Communication and Social Interaction Skills. LEA’s
Communication Series, L. Erlbaum Associates (2003)
8. Hammer, T.: Mental health and social exclusion among unemployed youth in scandinavia. a
comparative study. Intl. Social Welfare 9(1), 53–63 (2000)
9. Hollandsworth, J.G., Kazelskis, R., Stevens, J., Dressel, M.E.: Relative contributions of verbal, articulative, and nonverbal communication to employment decisions in the job interview
setting. Personnel Psychology 32(2), 359–367 (1979)
10. Hoque, M.E., Courgeon, M., Martin, J., Mutlu, B., Picard, R.W.: Mach: My automated conversation coach. In: Proc. UbiComp. ACM (2013)
11. Kelly, J.A., Wildman, B.G., Berler, E.S.: Small group behavioral training to improve the job
interview skills repertoire of mildly retarded adolescents. Applied Behavior Analysis 13(3),
461–471 (1980)
12. MacDonald, R.: Disconnected youth? social exclusion, the underclass and economic
marginality. Social Work and Society 6(2), 236–247 (2008)
13. Mehrabian, A.: Silent messages: Implicit Communication of Emotions and Attitudes.
Wadsworth Publishing Co Inc, Belmont (1981)
14. Pan, X., Gillies, M., Barker, Clark, D.M.C.M., Slater, M.: Socially anxious and confident
men interact with a forward virtual woman: An experiment study. PLoS ONE 7(4) (2012)
15. Sapouna, M., Wolke, D., Vannini, N., Watson, S., Woods, S., Schneider, W., Enz, S., Hall, L.,
Paiva, A., André, E., et al.: Virtual learning intervention to reduce bullying victimization in
primary school: a controlled trial. Child Psychology and Psychiatry 51(1), 104–112 (2010)
16. Stapleton, A.J., Taylor, P.C.: Why videogames are cool & school sucks! In: Proc. AGDC
2003. vol. 20, p. 23 (2003)