CAN COMPUTER-BASED SIMULATION GAMES ENHANCE EMPLOYEE JOB
PERFORMANCE? A FIELD EXPERIMENT AT IBM
CHING-PING PATTY FARH
IBM China/Hong Kong Limited
[email protected]
JIING-LIH FARH
Hong Kong University of Science and Technology
To appear in Best Paper Proceedings of the 73rd Annual Meeting of Academy of Management, Orlando,
Florida, USA, August, 2013
INTRODUCTION
Capitalizing on the widespread experience of playing computer games in today’s
workforce, computer-based simulation games (CBSGs) are increasingly becoming the
preferred instructional delivery method for corporate training. According to the
Entertainment Software Association, the average computer gamer is 34 years old, has been at
it a dozen years, and 40% are women. In an attempt to leverage computer gaming to enhance
employees’ work performance, large corporations such as Cisco, IBM, Microsoft and even
the Army now use computer games to provide training and services (Penenberg, 2010) – and
for good reason. People have the innate need to play, and the positive benefits of play at work
– such as bringing excitement and exploration to the job, promoting mastery and creativity
(Brown, 2009) – may explain why, beyond entertainment, learning is reported to be the
second largest category of applications of computer games (Cai, Sun & Farh, 2008).
According to Morariu of IBM (2012), “Playing computer-based games is not merely a global
phenomenon for entertainment, but also a strategic direction to more effectively engage and
prepare the emerging workforce…Since early 2000, IBM has made significant investments in
research and development in the efficacy and benefit of [these] games to address 21st century
global challenges to not only accelerate deep competencies and skill development, but to
enable immersion in game-based scenarios … to solve real world problems” (p. 1-2).
But to what extent do CBSGs actually promote learning? Defined as “instruction
delivered via personal computer that immerses trainees in a decision-making exercise in an
artificial environment in order to learn the consequences of their decisions” (Sitzmann, 2011,
p. 492), a recent meta-analysis found that those trained using CBSGs not only demonstrated
higher declarative and procedural knowledge but also higher post-trainings self-efficacy,
compared to control groups. Even with this initial evidence, there are still empirical
uncertainties in the literature. First, most field research on CBSGs evaluated training
outcomes in terms of trainees’ satisfaction and knowledge improvement, with few studies
examining on-the-job behavior or performance (DeRouin et al., 2005; Sitzmann, 2011). Thus,
it is unclear whether the learning gained through playing simulation games can transfer to job
situations and result in higher job performance. Second, few extant studies were conducted in
corporate settings. In Sitzmann’s (2011) study, only 5% out of the 65 samples involved
employees. Thus, it is unclear whether the salutary effects of CBSGs are limited to university
students or can be generalized to the workforce context. Finally, in the few studies in which
simulation games were studied in corporate settings, participants were not randomly assigned
to experimental conditions and were without pretest measures (e.g., DeRouin, Fritzsche, &
Salas, 2005). As a result, the efficacy of effective CBSGs remains an unanswered question.
In the present study, we extend on prior work by conducting a field experiment with
random assignment to experimental conditions and pretest-posttest measurements at IBM.
We test the efficacy of Innov8 PM, a newly developed CBSG for training junior project
managers and developing their project management skills on the job. Following Kirkpatrick’s
(1976) taxonomy of training evaluation, we assess the effectiveness of Innov8 PM on
multiple levels: user reaction and satisfaction, knowledge gain, and behavioral transfer on the
job.
HYPOTHESES
Technology plays an important role in corporate training today. 95% of companies
use some form of e-learning in their companies according to a survey conducted by American
Society for Training and Development (ASTD) (Ellis, 2003). CBSGs represent one type of
e-learning using interactive role play to immerse players to make decisions and practice and
perfect their skills in a three dimensional (3D) virtual reality environment, before taking on
real life challenges. Several theories of learning explain why CBSGs may be effective
vehicles for training:
1) Malone’s (1981) intrinsic motivation theory emphasizes that the intrinsic
motivation and play-type activities can enhance trainees’ deep learning and competency.
When trainees are intrinsically motivated, they immerse and dive deep into the learning
contents, enjoy the learning experience, put in more effort and stay in the learning longer and
are more confident that they can apply what they learned to their jobs.
2) According to the input-process-outcome model, Garris, Ahlers, and Driskell (2002)
proposed that CBSGs enables a cyclical relationship among user judgments, user behavior,
and system feedback – which in turn promotes an optimal experience where learners are so
immersed and engaged in the computer-based simulation game that they totally ignore the
world outside of the games (Csikszentmihalyi, 1990).
3) According to the interactive cognitive complexity model, Tennyson and Jorczak
(2008) proposed that learning is the outcome of an interaction between internal and external
variables to the learners’ cognitive systems. Trainees’ affective structures (e.g., motivation
and attitudes) and cognitive structures (e.g., memory, knowledge base, and execution ability)
interact with each other in CBSGs to enhance learners’ learning outcomes. Thus, simulation
games are more effective than other training methods because they tend to engage learners’
affective and cognitive systems simultaneously.
4) According to the Dale’s learning pyramid model, different levels of learning
experience and learning retention are associated with different instructional methods, where
the practice-by-doing method leads to a 75% retention rate (Lalley & Miller, 2007). CBSGs
can be characterized as such a method, and is expected to achieve higher learning
effectiveness than traditional lecturing, reading, demonstrating, and discussing.
As such, we offer the following hypotheses:
Hypothesis 1: Participants in the CBSG group will have higher knowledge gain than those in
the comparison group.
Hypothesis 2: Participants in the CBSG group will have higher level of post-training job
performance than those in the comparison group.
Hypothesis 3: Participants in the CBSG will report a high level of motivation and satisfaction
with the game.
METHOD
Company Situation
Participants in this study were recruited from the China branch of IBM Corporation to
play the in-house designed Innov8 PM CBSG course. The objective of Innov8 PM is to train
fundamental project management skills for part-time project managers (defined as the junior
project managers or project manager trainees or project leaders or technical leaders who are
not experienced enough as professional full-time project managers but have business needs to
manage small-scale projects and then grow into a full-time project manager). IBM relies
heavily on IT project teams to deliver services to customers globally and must train thousands
of project managers globally on a yearly basis. In the recent decade, the training demand is
most acute in growth markets. Because of geographic distance, time zone differences, and the
general lack of time available for junior project managers to attend traditional training
sessions, the high accessibility, high scalability, high reusability and speedy delivery of
Innov8 PM presented an attractive solution. Further, the option to practice project
management skills in Innov8 PM enables junior project managers to accelerate their growth
from knowledge to action.
Aligned with these corporate needs, Innov8 PM is a highly interactive three
dimensional virtual reality game that simulates the project management experience. Using an
animated character, the participant role plays the project manager through a project
management cycle, from initiating the project, to planning the project, to executing the
project, then to closing the project. There are many challenges and risks throughout the
entire project cycle, and the participant needs to apply their project management and control
skills to drive the project to the end with high satisfaction from project customer and
executives and complete the project on time within the pre-allocated budget. As the trainee
navigates these challenges, he or she learns to communicate with different stakeholders,
collect information, work out options, provide analysis, and solve different challenges
associated with the process. A short YouTube video on the IBM Innov8 game series website
may be found at http://www.youtube.com/watch?v=SjiQTWZ_MK8 and more information on
the IBM Innov8 game series may be found at http://www.ibm.com/innov8.
Study Design and Sample
HR Learning identified 100 part-time project managers to participate in the study
using three criteria: a) currently undertake a project management role in a small project, b)
have not previously attended the four-day company-run Project Management Fundamental
face-to-face course, and c) have started the Project Management Orientation e-learning
course. Sixty of the 100 selected part-time project managers were randomly assigned to the
experimental group (the simulation game group) and 40 to the control group (no-game
comparison group).
All participants’ baseline knowledge of project management (pretest knowledge
measure) was assessed in an initial survey two weeks before the game was opened to
participants in the experimental group, and their baseline on-the-job project management
skills (or job performance) were assessed by their direct managers at the same time (pretest
on-the-job skills measure) . The participants in the experimental group were then given a
period of four weeks to download and play the game individually on personal time.
Immediately after completing the game, the players filled out a short feedback survey. The
participants in the control group were not invited to play the game during the experiment but
were given the opportunity to play the game when the study was over. Two weeks after the
players completed Innov8 PM, participants in both experimental and control groups
completed the same knowledge test for a second time (posttest knowledge measure). Six
weeks after the players completed Innov8 PM, the direct managers of all participants were
asked to evaluate their on-the-job project management skills for the second time (posttest
on-the-job skills measure).
Forty-two of the 60 (70%) participants in the experimental group accepted the invitation,
completed the game, and both knowledge surveys; 31 of the 40 (78%) participants in the
control group accepted the invitation and completed both knowledge surveys.
Measures
Knowledge Test. With the aid of SMEs (subject matter experts) in project
management profession, a pool of 12 multiple choice items was developed for testing basic
project management knowledge. The knowledge test score is the sum of the number of
correct answers by the participants (ranging from 0 to 12). The 1st knowledge test was
embedded in the initial survey sent to all participants in the study. The 2nd knowledge test
was administered two weeks after participants in the experimental group completed the game
(about six weeks after they completed the initial survey). The test-retest reliability with six
week separation was .65 (p < .01) for the Control group and .30 (p < .06) for the
Experimental group, which is considered satisfactory.
Performance Evaluation. The direct managers of all participants provided on-the-job
project management skills evaluation. Managers were blind to the experimental conditions,
and had no knowledge of when and whether their direct reports had played Innov8 PM.
Participants’ on-the-job project management skills (or job performance) were evaluated on
nine items developed by subject matter experts of HR Learning on a six-point Likert scale (1
= strongly disagree, 6 = strongly agree).
RESULTS
We first checked whether the two groups were equivalent on whether they had
completed the Project Management Orientation e-learning course provided by the company,
whether they had served as a project leader before, and how often they played video games
during their leisure time as a child, and their knowledge test scores and job performance
ratings at Time1 (baseline). T-tests of the mean differences showed that the two groups did
not differ significantly on completion of e-learning course, project leader experience and
childhood interest in playing video games. Moreover, the two groups did not differ
significantly on knowledge test, managers’ ratings of on-the-job project management skills at
Time1 (baseline assessment). The above findings suggested that random assignment had
generated statistically equivalent groups on these measures.
Hypothesis Testing
A 2 Pretest-Posttest (Time1 vs Time2) X 2 Training (Experimental vs Control group)
General Linear Model was performed on knowledge test and on-the-job project management
skills ratings respectively, with Training as a between-subjects factor and Pretest-Posttest as a
within subjects factor. For project management knowledge test, the main effect for
Pretest-Posttest was significant, F(1,71) = 5.33, p < .05, ƞ2p = .07, and the interaction effect
between Training and Pretest-Posttest was also marginally significant, F(1,71) = 3.58, p
< .063, ƞ2p = .05. These results indicated that while there was a general improvement in
knowledge test scores between Time 1 and Time 2, the improvement in knowledge score for
experimental group was significantly higher than for the control group. Thus, hypothesis 1
was supported.
For on-the-job project management skills, the main effect for Pretest-Posttest was
significant, F(1,71) = 4.31, p < .05, ƞ2p = .06, and the interaction effect between Training and
Pretest-Posttest was also highly significant, F(1,71) = 7.92, p < .01, ƞ2p = .10. These results
indicated that while there was a general increase in on-the-job project management skills
ratings between Time 1 and Time 2, the skill increase over the two assessments for
experimental group was significantly higher than for the control group. Thus, hypothesis 2
was supported.
Twenty-five of the 42 players completed the feedback questionnaire after the game.
The participants reported that Innov8 PM was much more effective than e-learning (mean =
4.64) (5 “completely agree”, 1 “not at all”); the learning from the game had practical benefits
to their current jobs (mean 4.12); they were interested in playing another simulation game to
learn new skills (mean 4.68); and they were happy to recommend this game to their friends
(mean 4.56). The overall satisfaction level with the game was 4.44, indicating that
participants were fairly satisfied with their game experience. Hypothesis 3 was supported.
DISCUSSION
CBSGs are a new trend in corporate training – however, few studies have examined
the effectiveness of such games in corporate settings, and none have examined whether skills
learned in the game can transfer to job settings and result in heightened performance. The
current study represents a rare field study conducted in multinational giant IBM and sheds
valuable insights on this issue. The results show that a well-designed CBSG like Innov8 PM
can improve trainees’ knowledge improvement even with a 2-week delay in assessment,
increase the trainees’ job performance (i.e., on-the-job skill development) 6 weeks after the
training, and result in high user satisfaction.
Our results further validate the following design recommendations for future CBSGs.
Games should a) align the training objective with corporate strategy and business needs, b)
feature instructional design that enables trainees to learn actively instead of passively, c)
allow trainees to practice instructional material without any limitation of time and location, d)
allow trainees to role play in the first person and interact with other animated key
stakeholders in a game scenario which simulates the real job experience, e) submit trainees to
instant coaching and guidance from a built-in animated coach, and f) ensure trainees receive
instant feedback and report on their game performance (DeRouin et al., 2005; Sitzmann,
2011). Innov8 PM was designed according to these principles with exceptional results.
Finally, our results corroborate Sitzmann’s (2011) recommendation that maximizing
the potential of CBSGs require a blended approach. As in Innov8 PM, the blended mode
involves trainees being first exposed to concepts and principles from passive e-learning
before practicing and applying that knowledge to actively role play and make decisions in
virtual reality. Thus, CBSGs are perhaps best leveraged for interactive practice to close the
gap between knowledge and experience.
CONCLUSION
In conclusion, while prior work suggests that CBSGs generally increase knowledge,
self-efficacy, and retention (Sitzmann, 2011), the results of our field experiment suggest that
the effectiveness of CBSGs extends to promote actual on-the-job performance outcomes. We
demonstrate this effect among junior project managers in IBM, a world-leading IT service
corporation with over 433,000 employees globally in 2011. Our findings suggest that the
global trend towards utilizing CBSGs as training solutions is warranted and worthwhile. We
hope that our work spurs practitioners and researchers alike to further understand when and
why CBSGs represent the future of training and development in the 21st century.
REFERENCES AVAILABLE FROM THE AUTHOR(S)