A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS
FOR DEGREE OF LICENTIATE OF PHILOSOPHY.
Game-Based Instruction
within IT Security Education
K JE LL NÄ C K RO S
Department of Computer and Systems Sciences,
Stockholm University & Royal Institute of Technology
Stockholm, Sweden 2001
Game-Based Instruction within IT Security Education
K JE L L N Ä C K RO S
Copyright © Kjell Näckros, 2001.
Report series no. 01-018
ISSN: 1101-8526
ISRN SU-KTH/DSV/R--01/18--SE
Department of Computer and Systems Sciences,
Stockholm University / Royal Institute of Technology
Forum 100
SE-164 40 Kista
Contact information:
Kjell Näckros
Stockholm University/KTH
Forum 100
SE-164 40 Kista
Sweden
Phone: +46 (8) 16 17 65
Fax: +46 (8) 703 90 25
Email: [email protected]
URL: http//www.dsv.su.se/~kjellna
Game-Based Instruction within
IT Security Education
K JE LL NÄ C K RO S
Department of Computer and Systems Sciences,
Stockholm University & Royal Institute of Technology
ABSTRACT
This thesis will show that computer games can be an efficient teaching
method when learning to understand IT security. Through series of
experiments carried out in Sweden during 1999-2000 with 76 undergraduates
and 24 professionals data have been collected and subjected for quantitative
and qualitative analyses. In the experiments learning preferences and teaching
methods were matched and compared with acquired understanding of
IT security related issues. A framework for Game-Based Instruction and
Design, grounded in existing learning theories is presented and discussed.
Based on this framework an applicable teaching method, a computer game to
increase awareness/understanding of IT security related issues was developed
and evaluated.
Keywords: IT security education,
Instruction, learning styles.
security informatics,
Game-Based
This work was in part funded by the Swedish National Board for Industrial
and Technical Development (NUTEK), project P10535-3. The work was also
conducted in collaboration with SEIS and Säkdata Sert AB1. During the
writing of this thesis SEIS2 became integrated with GEA3, (The Swedish
Alliance for Electronic Commerce) and Nexus AB4 bought Säkdata.
1
2
Securitas Säkdata Sert AB, a company that implements IT security solutions e.g. e-signatures.
SEIS, “Säker Elektronisk Information i Samhället” - a Swedish non-profitable association that
promotes secure electronic communication in Sweden.
3
GEA-Gemenskapen för Elektroniska Affärer (The Swedish Alliance for Electronic Commerce).
4
Nexus AB is a Swedish company that markets e-commerce solutions.
i
TABLE OF CONTENTS
List of figures and tables.................................................................................................. iv
Preface and acknowledgments ........................................................................................ v
Glossary .............................................................................................................................. vi
C HA PT ER 1
Introduction........................................................................................................................ 1
1.1 Background to the research................................................................................ 1
1.2 Purpose of the research....................................................................................... 3
1.3 Research methodology ........................................................................................ 4
1.4 Research hypotheses............................................................................................ 6
1.5 Research contribution.......................................................................................... 7
1.6 Overview of the research.................................................................................... 7
1.7 Structure of the thesis.......................................................................................... 9
C HA PT ER 2
Theoretical background..................................................................................................11
2.1 Chapter introduction .........................................................................................11
2.2 Learning styles.....................................................................................................12
2.3 Towards a theory on Game-Based Instructional Design ...........................19
2.4 The theoretical framework for Game-Based Instructional Design ..........26
2.5 Quality of knowledge.........................................................................................29
2.6 Applicability within IT security education.....................................................31
2.7 Chapter conclusion ............................................................................................33
C HA PT ER 3
Game-Based Instruction ................................................................................................35
3.1 Chapter introduction .........................................................................................35
3.2 Characteristics of a computer game................................................................36
3.3 People and games ...............................................................................................37
3.4 Similar approaches..............................................................................................41
3.5 Model for computer game design ...................................................................45
3.6 Chapter conclusion ............................................................................................46
C HA PT ER 4
The Paradise model .........................................................................................................47
4.1 Chapter introduction .........................................................................................47
4.2 Development of the GBI model .....................................................................48
4.3 Chapter conclusion ............................................................................................58
ii
C HA PT ER 5
Research design and methodology............................................................................... 59
5.1 Chapter introduction ......................................................................................... 59
5.2 Methodological considerations........................................................................ 59
5.3 Experiment design ............................................................................................. 60
5.4 This experiment design ..................................................................................... 61
5.5 Instrumentation .................................................................................................. 64
5.6 Procedures and collection of data ................................................................... 68
5.7 Data analyses ....................................................................................................... 69
5.8 Evaluation phases and preliminary investigations........................................ 73
5.9 Validity and reliability of the experiment....................................................... 78
5.10 Ethical considerations ..................................................................................... 79
5.11 Limitations in the study................................................................................... 80
5.12 Chapter conclusion .......................................................................................... 80
C HA PT ER 6
Findings and limitations ................................................................................................. 81
6.1 Chapter introduction ......................................................................................... 81
6.2 Evaluation of phase 3 – professionals............................................................ 81
6.3 Evaluation of phase 4 – students .................................................................... 82
6.4 Limitations in the results................................................................................... 92
6.5 Chapter conclusion ............................................................................................ 92
C HA PT ER 7
Concluding discussion and suggestions for future research.................................... 95
Bibliography...................................................................................................................... 99
Appendices .....................................................................................................................105
Appendix A – Material to participants
Appendix B – Data Analyses with Anova
Appendix C – Quantitative Results, Pretest and Post-test
Appendix D – Qualitative Results
iii
LIST OF FIGURES AND TABLES
Number
Page
Figure 1. Graphical overview of the research. ....................................................9
Figure 2. Experiential Learning Cycle ................................................................22
Figure 3. Bloom’s Taxonomy of Educational Objectives ..............................29
Figure 4. The SBC Model .....................................................................................32
Figure 5. The Systemic Holistic Model ..............................................................32
Figure 6. Classification of learning preferences and disciplines taught........33
Figure 7. An example view of SecSim game. ....................................................50
Figure 8. An example of the Authorware Environment ................................56
Figure 9. The Paradise Game...............................................................................57
Figure 10. Experiment design (I).........................................................................63
Figure 11. Experiment design (II).......................................................................63
Figure 12. Output from Keirsey’s Temperament II sorter ............................65
Figure 13. Evaluation phases ...............................................................................75
Figure 14. Distributions of Individuals in knowledge quadrants ..................84
Figure 15. Results – Learning preference and Instruction .............................86
Figure 16. Results – Learning preference or Instruction ................................87
Table 1. Distributions of participants ................................................................77
Table 2. Age & Gender distribution...................................................................77
Table 3. Results – Relative difference as percentage units .............................83
Table 4. Distribution of Groups .........................................................................85
iv
PREFACE AND ACKNOWLEDGMENTS
This thesis is a compound of three refereed papers [Näckros 1999a; Näckros
2000; Näckros 2001] and a number of reports for Nutek, which partly funded
this study together with our industry partners.
The author wishes to thank docent Robert Ramberg for help with the design
of the experiments.
Furthermore, the author wishes to thank Peter Lönnqvist, Fabian von Schéele
and Fredrik Björck for the inspiring conversations, and of course all the
participants in the experiments.
At last, I wish to thank all of my friends whom I constantly forced to listen to
my frustrations.
v
GLOSSARY
The following terms are used throughout this study.
CBI (Computer-Based Instruction) – The concept of utilising computers
as instruction, used to distinguish CBI from conventional instructional
theories which do not take today’s instructional setting into account e.g.
hyper-media, distance learning. When discussing learning in CBI it is called
Computer-Based Learning (CBL).
Cognitive Style – Specific categories that describe ways in which individuals
process information such as field dependence/independence, visual/haptic,
serialistic/holistic and reflexivity/impulsivity [Summerville 1999].
Constructivism – Conceptions of learning assume that knowledge is
individually constructed and socially co-constructed by learners based on their
interactions with the world. The meaning that learners construct depends on
their needs, beliefs and prior knowledge [Jonassen 1997].
GBI (Game-Based Instruction) – A sub domain to Computer-Based
Instruction (CBI). The concept of utilising non-linear computer games as
instruction. In this thesis a suggestion of a framework for GBI is proposed to
be used when designing effective instructional computer games. When
discussing learning in GBI it is called Game-based Learning (GBL).
Inquisitivism – A descriptive approach designing effective CBI for adults.
One of the key ideas in the inquisitivist approach is the removal of the
paralysing fear that many adults have with learning technology [Harapnuik
1998].
Instructional Environment – Characteristics of the way teaching is taught.
Learning Preference – Specific categories that describe an individual’s
learning strategy or cognitive style. In this study the polar extremes
holistic/serialistic are used. Also called “Conceptual competence” [Pask and
Scott 1972].
Learning Styles – An individual’s preference for a particular environmental
characteristic such as amount of light or quiet [Summerville 1999].
Matching Instructional Environments – Instructional environments
designed to match a student's cognitive style [Summerville 1999].
vi
MBTI (Mayer Briggs Type Indicator) – An attempt to measure and
identify a person’s psychological profile/’type’ [McCaulley 1979]. MBTI
differentiates between 16 psychological types and is based on Jung’s theory of
psychological types [Jung 1923].
PKI – Public Key Infrastructure. All the different components together, that
is needed to ensure secure electronic transactions based on asymmetric
encryption.
Satisfaction – The degree to which students are satisfied with the learning
environment, are comfortable with the learning environment, and perceived
that they learned from the experience [Summerville 1999].
SPQ (Study Preference Questionnaire) – An attempt to identify a person’s
learning preference i.e. holist, serialist, or versatile. The questionnaire is a
simplified version of Pasks extensive tests [Pask 1976a] and consists of 18
questions [Ford 1985].
Teaching Strategy – Characteristics of instructional environments i.e. topdown or bottom-up approach.
Holist – also called bricoleurs, non-linear, global, relational – use a top-down
approach when learning, remembering and recapitulate a body of information
– as a whole, in terms of ‘higher order relation’.
Serialist – also called linear, analytical – use a bottom-up approach when
learning, remembering and recapitulate a body of information in small well
defined and sequentially ordered segments.
vii
viii
Chapter 1
INTRODUCTION
This thesis will show that computer games can be an efficient teaching
method when learning to understand IT security.
Through series of experiments carried out in Sweden during 1999-2000 data
have been collected and subjected for quantitative and qualitative analyses.
In this thesis a framework for Game-Based Instruction and Design grounded
in existing learning theories is presented and discussed.
Based on this framework an applicable teaching method – a computer game
to increase awareness/understanding of IT security related issues – has been
developed and evaluated.
1.1
Background to the research
In the modern information technological intense society, where more and
more responsibility and demands are placed in the hands of the individual
[Fillery-James 1999a], a general understanding of IT security is more
important than ever. Not only for the individual in the role as a professional
but also in the role as a citizen.
Current instructional methods in understanding of IT security have a
tendency to fit certain individuals better than others. Which in turn increases
the feeling of insecurity for those who do not understand and therefore also
increase the IT vulnerabilities in the systems they are using.
Therefore it is important to find alternative/complementary methods that will
stimulate learning/understanding of IT security issues also for these
individuals, in order to strengthen the viability of the system as a whole.
1
Chapter 1
When teaching IT security, no matter of educational setting, it is impossible to
meet every individual’s personal needs, although, it is substantiated that
people learn and understand in several different ways of which some tend to
be more efficient. One reason for this may be that most didactic methods e.g.
books, multimedia, and front-end teaching, are structured and performed in a
linear fashion i.e. teaching small pieces of information in a predetermined
sequence in hope that the learner eventually will understand the overall
picture. Unfortunately, the linear teaching strategy does not meet a large
number of individuals’ learning capabilities5. This leads to unnecessary efforts
for the unmatched group of people in the learning process [Pask 1976c;
Turkle 1990; Yngström 1996] with a possible dissatisfaction and a low degree
of understanding as a consequence.
In order to deal with the complexity and practical nature of IT security,
several teachers and researchers have underlined the need of practical
applications within IT security education and also begun to experiment and
investigate the possibilities of including this in the curriculum [Fillery-James
1999b; Irvine 1999; White and Sward 1999]. A common approach is to strive
for teaching both soft and technical aspects of IT security, instead of either
or, to make the learners understand the whole picture of IT security.
Although, depending on the teacher and the teaching aids this can still be
taught in a linear or a non-linear mode and so far there have been a lack of
discussions about meeting the learners’ learning style within IT security
education.
The combination of identifying the learning style and the instructional
treatment designed to meet the learning style characteristics, may prove to
have significant impact on learner’s achievement and/or satisfaction with
acquiring IT security issues.
This researcher believes that IT security is too important not to consider
meeting the students’ individual cognitive learning styles. There is still a lack
of alternative non-linear teaching methods that will stimulate holistic learning
within IT security education.
5
2
Also called cognitive styles
Introduction
This thesis investigates if computer games can be such a method.
1.2
Purpose of the research
The purpose of this thesis was to investigate if computer games are a suitable
teaching method to stimulate holistic learning of IT security issues and if
computer games as a teaching method also would stimulate other aspects of
leaning about IT security related issues, such as the amount of acquired
understanding6. Given such a method people with holistic learning preferences7
should be able to acquire awareness of IT security issues, in terms of
understanding, more efficiently than through conventional, linear teaching
methods. This is based on the idea that computer games are an appropriate
contrast to linear instruction i.e. non-linear, and therefore also ought to be
especially suitable for people with holistic learning preferences. A tentative
hypothesis was proposed in [Näckros 2000] wording the problem as; people
with holistic learning preferences acquire IT security awareness, in terms of
understanding, more efficiently when learning through playing computer
games instead of using conventional, linear instruction.
To investigate this the following activities were chosen.
1. To determine whether matching or mismatching subjects’ learning
preference with teaching strategy had any effect on acquired
understanding of IT security issues.
2. To determine whether matching or mismatching subjects’ learning
preference with teaching strategy had any effect on subjects’
satisfaction with the teaching strategy and the learning experience.
3. Moreover, to be able to discuss efficiency in a particular learning
experience we also investigated the time spent on learning.
6
Since we want to investigate the amount of acquired understanding, we will use Bloom’s taxonomy of
educational objectives [Bloom, Engelhart et al. 1956] throughout this thesis. He distinguishes between
‘knowledge’ and ‘comprehension’ where ‘comprehension’ has a higher degree of understanding than
‘knowledge’. We will discuss this further in chapter 2.
7
In this study we will investigate one aspect of cognitive styles, learning preference. Gordon Pask called this
conceptual competence [Pask and Scott 1972] and distinguishes between serialists and holists – serialists
learn from details to wholes and holists from wholes to details.
3
Chapter 1
1.3
Research methodology
Since a research hypothesis have been proposed in combination with suitable
independent variables – teaching strategy and learning preference – and a
dependent variable – the amount of acquired understanding in IT security –
we have chosen to conduct two experiments with pretest – post-test within a
control group design [Cohen, Manion et al. 2000].
This researcher has compared and evaluated the acquired understanding from
two types of instructions by using experiments. The conventional linear
instruction was a text, explaining PKI8 crypto systems and IT security to
management in organisations, presented on a computer screen. The
experimental instruction was a computer game explaining PKI. The
instructions were evaluated and revised in order to be comparable in content
and quality.
To be able to develop a computer game as instruction a framework for the
design of Game-Based Instruction (GBI) grounded in existing theories of
learning was proposed. Based on this framework the computer game was
developed by this researcher for the solitary purpose to evaluate the research
hypothesis.
Two identical simultaneous instructional experiments have been conducted
on sixty-three undergraduates at the department of computer and systems
sciences. The students’ learning style decided which experiment to be part of.
The learning style was decided through Keirsey’s Temperament Sorter II
[Keirsey and Bates 1984] that is based Mayer-Briggs Type Indicator (MBTI)
and Study Preference questionnaire [Ford 1985]. The result indicated if the
student had serialistic or holistic learning preferences. In this thesis these will
be referred to as serialists or holists.
8
4
PKI, Public Key Infrastructure.
Introduction
The subjects were divided into an experimental group that was exposed to the
experimental instruction and a control group that was not exposed to the
experimental instruction but instead the conventional instruction9.
The amount of acquired understanding was evaluated by using two types of
questions in the pretest and post-test. These questions were designed to either
match ‘knowledge’ or ‘comprehension’, according to Bloom’s taxonomy
[Bloom, Engelhart et al. 1956].
Before the subjects started their learning the subjects filled in a pretest and
after they filled in a post-test. The results from the pretest and the post-test
were evaluated and the improvements compared between the groups.
The collected data were analysed quantitatively to measure mean-differences
in improvements between the four groups. The means were compared using
Analysis of Variance (ANOVA).
We also analysed each individual’s characteristics with his/her amount of
improvement using qualitative methods.
Thus, our experimental design will account for:
Quantitative results through the pre- and post-tests, and time.
Qualitative results through general questionnaire, the results from the
quantitative analyses, post-test and the evaluation form.
We decided that the intended target groups would include company
management, ordinary employees and the public (students).
A second evaluation has also been conducted to include professionals. The
purpose was to evaluate user acceptance of the game as a teaching strategy
against age, gender and previous knowledge in PKI. The evaluation took
place at three different occasions with three different user groups and
consisted of twenty-four subjects. Only qualitative data were collected.
9
In this thesis the linear instruction was regarded as ‘normal’, therefore the control groups received this
type of instructional treatment.
5
Chapter 1
1.4
Research hypotheses
The research design with two experiments and altogether four groups of
subjects made it possible to extend the activities mentioned in section 1.2 and
to discuss possible connections between learning preference, teaching
strategy, satisfaction with the teaching strategy and the learning experience,
and acquired understanding by also investigating all players, all readers, all
holists, all serialist.
Therefore, the following questions and working hypotheses were guiding the
data collection and analyses.
1. Subjects who receive treatments matched with learning preference
will have higher amount of acquired understanding about IT security
than those who are mismatched.
2. Subjects who receive treatments mismatched with learning preference
will have lower amount of acquired understanding about IT security
than those who are matched.
3. Subjects who receive treatments matched with learning preference
will have higher amount of satisfaction than those who are
mismatched.
4. Subjects who receive treatments mismatched with learning preference
will have lower amount of satisfaction than those who are matched.
5. Subjects who receive treatments with a non-linear teaching strategy
will have higher amount of ‘comprehension’ than those who received
treatments with a linear teaching strategy.
6. Subjects who receive treatments with a linear teaching strategy will
have higher amount of ‘knowledge’ than those who received
treatments with a non-linear teaching strategy.
7. To what extent does learning preference (serialist/holist), teaching
strategy (linear/non-linear), matching/mismatching learning
preference and teaching strategy, age, gender and satisfaction with the
instruction singly or in combination, affect achievement in acquiring
understanding of IT security.
6
Introduction
1.5
Research contribution
This thesis’ research shows that computer games can be an efficient teaching
method when learning to understand IT security.
We also show that this counts especially for people with little or no initial
knowledge in the area.
In this thesis a framework for Game-Based Instruction and Design grounded
in existing learning theories is presented and discussed.
Based on this framework an applicable teaching method – a computer game
to increase awareness/understanding of IT security related issues – has been
developed and evaluated.
Furthermore, we will demonstrate that the type of knowledge acquired
depends on the teaching method used i.e. the subjects who learnt through the
computer game acquired more ‘comprehension’ and the subjects who learnt
through reading acquired more ‘knowledge’.
1.6
Overview of the research
The approach will essentially be pragmatic.
The research was carried out in four stages: literature review and related work,
development of a framework for GBI, development of an educational model
and evaluation of the model as instruction.
1) During the first stage general literature and publications within
IT security, IT security education, learning theories, teaching
strategies, computer based instruction(CBI)/ learning(CBL),
computer games, game development and educational research were
studied and investigated.
The author of this thesis did not find at the time (Aug.1998) any current or
previous practical attempts within IT security education, to match teaching
strategy with learning preference, or any research that investigating relations
between teaching strategy and type of acquired knowledge, or any attempts to
7
Chapter 1
investigate computer games as instruction10. However, within IT security,
Yngström [Yngström 1996] p.160 has discussed that “...educators will have to
pay attention to developing new educational tools, which will stimulate and
support various learning preferences...” Fred Cohen’s [Cohen 1998] work
within scenario-based training resulted in the development of security games,
but in the opinion of this author, the games do not have the non-linearity
dimension and were not developed with learning preferences in mind.
2) Development of a framework for Game-Based Instruction (GBI)
based on Inquisitivism. To investigate the requirements on a
computer game as instruction.
3) The findings from the first and second stage initiated the third stage
in which a computer game based on GBI framework was developed
as the non-linear teaching strategy11. We also decided to use Garefelt
and Westerlund’s book on cryptology and PKI [Garefelt and
Westerlund 1997] as the linear teaching strategy. The two teaching
strategies were tuned to match each other in terminology and
contents.
4) The fourth stage included the evaluation of the research hypotheses
through experiments. The sampling strategy used included both
professionals and students, but for practical reasons12, we needed to
divide the evaluation into one approach for the professionals –
uncontrolled environment – and one approach for students –
controlled environment. For the first group we only evaluated the
teaching strategy with qualitative data. The second group’s evaluation
was conducted as two experiments with pretest – post-test and
control group design. The collected data were analysed with
qualitative and quantitative methods. Consequently, the research
hypotheses were evaluated using both quantitative and qualitative
analyses.
This research is multi-disciplinary, containing theories within IT security,
learning, knowledge, instruction and game design. Figure 1 is an informal
10
Since this thesis’ research started two game-based instructions have been developed, these will be
discussed in chapter 3. Two adjacent studies on meeting students’ cognitive style have also been
accomplished, although not within IT security. Will be discussed in chapter 2.
11
During the development of the computer game we also developed several models to illustrate how the
graphical objects were related on different conceptual levels as described in reports for NUTEK
[Näckros 1999c; Näckros 1999b]
12
which will be elaborated in chapter 5
8
Introduction
mindmap, on different conceptual levels, illustrating how the different
keywords and concepts in the thesis relate to each other. The intention is to
keep the figure as simple and holistic as possible and therefore the type of
relations are not discussed. Marked area indicates scope of thesis’ research.
Lines indicate relations.
Figure 1. Graphical overview of the research.
1.7
Structure of the thesis
This thesis is divided into seven chapters. Chapter 2 discusses common trends
and strategies within learning and understanding in general, and how
computer games as instruction can be grounded in existing theories of
learning by developing a framework on GBI. Chapter 3 continues the
discussion on the characteristics of GBI by investigating the game-playing
community and discussing similar approaches within teaching that utilises
multimedia techniques.
Chapter 4 presents the non-linear model, the computer game used in the
experiments, developed to stimulate holistic learning. We will also discuss our
experiences during the development process.
Chapter 5 describes the research method and discusses the methodologies
used.
Chapter 6 presents the findings of the thesis’ experiments and discusses the
limitations in the used process.
9
Chapter 1
Chapter 7 summarises the thesis’ research as a whole and discusses possible
further research.
10
Chapter 2
THEORETICAL BACKGROUND
This chapter consists of five sections. The first section discusses learning
styles, the theories behind them and how these can be measured. The second
section presents theories for learning that can be used to develop a
framework for Game-Based Instructional design. Section three will also
suggest such a framework. The forth section discusses quality of knowledge
and how to measure it. The chapter ends with a discussion on the applicability
within IT security education.
2.1
Chapter introduction
According to a survey distributed among IT security managers in Sweden
during 1994 [Johansson and Kager 1995], one of the main threat against
computer-stored information was regarded to be un-intentional and
un-correct changes and erases on existing information made by personnel.
This opinion is also supported by more recent surveys [Björck 1998a;
Lidholm 1999]. In an effort to map IT related crimes and incidents in
Sweden, BRÅ13 found that 63% of the reported events involved insiders14
next to viruses and intrusions made by outsiders [BRÅ 2000]. The report
stressed the importance of increasing the IT awareness and suggested more
resources for education and training. Furthermore, there is also a need for a
more efficient/applied education within IT security awareness. Björck [Björck
and Yngström 2001] discusses the gap between ‘what’ is currently being
taught (meaning abstract technical information security research) and the
increasing demands from industry of ‘what is needed’. Industry would like to
see more focus on organisational context oriented IT security awareness that
13
BRÅ – Brottsförebyggande Rådet (National Council for Crime Prevention in Sweden).
14
In the report, the following categories of IT related crimes were investigated; intrusion, virus, fraud,
blackmail, illegal gambling, threats, prostitution, privacy, youth-related crimes, copyright, smuggling,
economical crimes, money laundry, organised crime and information warfare.
11
Chapter 2
is useful. Perhaps it is not the wrong issues being taught, but more about
‘how’ it is taught? The lack of a holistic/interdisciplinary view during the
instruction may lead to unsuccessful attempts for the learner, to put the issue
in its real context. The learner will fail to understand the overall picture.
Since all people learn and all educators teach according to their own personal
references, the most efficient educational setting to raise IT security awareness
ought to be individualised teaching, where teaching strategy and learning style
are matched with each other [Felder 1995; Smith 1995].
It is unrealistic, however, to expect teachers in all educational settings to alter
educational environments in order to meet each student’s educational needs
such as differences in cognitive style [Summerville 1999].
In spite of the fact that the idea of using computer games as educational
setting is neither new nor unique (as will be explained in chapter 3), there are
very few discussions about their effect on the learner, especially in contrast to
‘conventional’ instruction. One of the reasons for this, maybe the lack of
theories supporting such a technology based instructional design. Instead
designers have ‘borrowed’ elements from existing learning theories, most
notably from behaviourists and cognitive psychologists. The focus on
knowledge construction and knowledge presentation perspectives makes the
theory of ‘experiential learning’ (also called ‘discovery learning’) most useful in
the instructional design [Nelson and Palumbo 1992]. In this research the
theory of inquisitivism [Harapnuik 1998], which borrows important elements
from experiential learning, are used as a base for the design of a computer
game-based instruction (GBI).
2.2
Learning styles
“Everyone thinks and learns in different ways, and implicitly accepts different
thinking and learning ‘styles’ as a fact of our birth” [Smith 1995] p.7.
In 1970, Marton distinguished, in 16 different experiments, two different
approaches how to remember [Marton 1970]. The participants were supposed
to remember 48 names of famous persons, written as a list on a paper. Some
12
Theoretical background
of the participants started by analysing the different groups of people (e.g.
movie stars, authors, politicians). The other simply tried to remember as many
names as possible, starting from top. These two groups had two completely
different procedures to accomplish the task. The first group tried to
understand the content of the list and then advanced to the details – the
names – while the second group started with the details – the names – to the
whole – the list – by collecting as many names as possible. At first, the first
group spent a lot more time and used a more flexible structure in categorising,
but in the end they could repeat the names fast and fluid. The second group
was inferior in the long run, regarding time, structure and result.
There are apparently, at least, two different approaches to learn and process a
body of information. Learning styles are the preferences for which types of
information enter an individual’s brain in the most efficient way and how that
information will be processed. Several researchers have tried to categorise
these differences and there is a lot of literature on the topic. However, it is
difficult to distinguish and compare the content of the studies since they all
use a different terminology, depending on each one’s specific approach.
Cohen [Cohen 1969] uses the categories ‘analytical’ versus ‘relational’. Pask
and Scott [Pask and Scott 1972] use the term ‘conceptual competence’ and
call their categories ‘serialist’ versus ‘holist’ (or ‘sequential’/’local’ versus
‘global’). Some use the term ‘learning style’ to refer to whether people are
‘visual’, ‘auditory’, or ‘tactile/haptic’ learners. Others call these differences
‘modalities’. Witkin [Witkin and Goodenough 1981] use the term ‘cognitive
style’ and group people roughly in two categories ‘field independent’ versus
‘field dependent’, based on the way they perceive the world around them and
process this information. Dunn and Dunn [Dunn and Dunn 1978] divides a
person’s preferred ‘learning style’ into 18 factors, referring to environmental,
emotional, sociological, and physical stimuli that affect a person’s ability to
absorb and retain information. Kindell and Hollman [Kindell and Hollman
1989] use the term ‘learning style’ and call their two categories ‘linear’ versus
‘global’. McCarthy [McCarthy 1981] combines Kolb’s experimental learning
model (will be discussed later) [Kolb 1984] and Jung’s four types [Jung 1923]
with left/right hemisphericity and receives 8 ‘learning styles’. Cornett [Cornett
1983] has a list of ‘learning style’ parameters several pages long. Some
researchers use the term ‘learning style’ synonymously with ‘cognitive style’.
13
Chapter 2
Felder [Felder 1993] suggests that a student’s learning style may be defined in
part by the answers to five questions:
What type of information does the student preferentially perceive:
sensory – sights, sounds, physical sensations, or intuitive – memories,
ideas, insights?
Through which modality is sensory information most effectively
perceived: visual – pictures, diagrams, graphs, demonstrations, or verbal
– sounds, written and spoken words and formulas?
With which organisation of information is the student most
comfortable: inductive – facts and observations are given, underlying
principles are inferred, or deductive – principles are given, consequences
and applications are deduced?
How does the student prefer to process information: actively – through
engagement in physical activity or discussion, or reflectively – through
introspection?
How does the student progress toward understanding: sequentially – in
a logical progression of small incremental steps, or globally – in large
jumps, holistically?
[Felder 1993] p.286
In this research we have investigated the last of these questions; concerning
how a learner prefers to approach a learning task, sequentially or globally.
2.2.1 Serialist versus Holist
Learners who prefer a sequential learning approach are referred to as serialists
and learners who prefer a global learning approach are referred to as holists.
Serialists prefer to focus on the details, which they eventually connect to bring
the whole picture together – bottom-up approach. Holists prefer to have an
overall understanding before they fit the details within the whole picture –
top-down approach.
One of the first to study this learning difference was Pask [Pask and Scott
1972], who through series of experiments with extreme forms of ‘global’ and
14
Theoretical background
‘local’ approaches [Pask and Scott 1972; Pask 1973; Pask 1976c; Pask 1976b]
identified two different learning strategies, holists and serialists.
Serialists learn, remember and recapitulate a body of information in
terms of string-like cognitive structures where items are related by
simple data links... Holists, on the other hand, learn, remember and
recapitulate as a whole. [Pask and Scott 1972] p.218
Based on [Pask 1988], Felder elucidates this further:
Sequential learners absorb information and acquire understanding of
material in small connected chunks; global learners 15 take in information
in seemingly unconnected fragments and achieve understanding in
large holistic leaps. Sequential learners can solve problems with
incomplete understanding of the material and their solutions are
generally orderly and easy to follow, but they may lack a grasp of the
big picture – the broad context of a body of knowledge and its
interrelationships with other subjects and disciplines. Global learners
work in a more all-or-nothing fashion and may appear slow and do
poorly on homework and tests until they grasp the total picture, but
once they have it they can often see connections to other subjects that
escape sequential learners.
[Felder 1993] p.288
As mentioned in the chapter introduction, it is impossible to meet each of the
learner’s personal learning style. But, the fact is that most of today’s
instruction are only designed to meet the serialists’ preferred way of learning.
This may lead to unnecessary learning effort for the holists. “Before holists
can master the details of a subject they need to understand how the material
being presented relates to their prior knowledge and experience, but only
exceptional teachers routinely provide such broad perspectives on their
subjects. In consequence, many holists who have the potential to become
outstanding creative researchers fall by the wayside because their mental
15
Instead of using the terms ‘global’ and ‘sequential’ learners this thesis will use the two categories
serialist and holist as proposed in [Pask and Scott 1972].
15
Chapter 2
processes do not allow them to keep up with the sequential pace of their
science courses” [Felder 1993] p.28816.
2.2.2 Matching instruction
Regardless of learning style inventory used, many researchers assert that it is
important to match instruction with the preferred way of learning of the
individuals for any relevant learning to take place. “Student with learning
styles that are different to the teacher tends to perform less well” [Davison,
Bryan et al. 1999] p.13. Previous studies on matching/mismatching
instruction and learning style show that “matched instruction favours learning
and mismatched instruction completely disrupts it (specifically, for
explanation eliciting questions) and leads to specific types of misconception”
[Pask 1976c] p.138.
Most of the learners do not know their own learning style or their preferred
way of learning. The most dominant method of instructional setting today, is
based on the concept that students are passive containers for information that
the instructor (teacher or instructional media) fills with contents [Harapnuik
1998]. As learner, one simply has to adopt the instructional setting offered
and therefore suppress the otherwise chosen way of learning, with less quality
of learning as a result.
In this thesis’ experiments (explained in chapter 5) two learning style models
have been used in order to decide the students’ learning style, the
Myers-Briggs Type Indicator and Study Preference Questionnaire.
2.2.2.1 The Myers-Briggs Type Indicator
One of the earliest attempts to categorise different styles of information
gathering and decision-making, was illustrated in Jung’s theory of
Psychological Types [Jung 1923]. His work is a model for many of today’s
cognitive theories. He proposed separate categories characterising the basic
preferences affecting not only what people attend to, but also how they draw
conclusions about what they perceive.
16
Felder have based this reasoning on an article in Chemical Engineering Education, Winter 1990, p.7,
‘Meet your students:2. Susan and Glenda”, which the author failed to find.
16
Theoretical background
One of the most known and widely used analyses based on Jung’s typology,
which has been applied to education, is the Myers-Briggs Type Indicator
(MBTI) [McCaulley 1979; McCaulley, Godleski et al. 1983; Felder and
Silverman 1988; Davison, Bryan et al. 1999]. It attempts to measure and
identify a person’s ‘type’, in this case, a psychological or personality profile
based on Jung’s typology of conscious functioning (archetypes). MBTI
differentiates between 16 psychological types based on the preferred
orientation on each of the four axes or dimensions. [Myers and McCaulley
1985]:
Extroversion-Introversion (EI) – This dimension identifies a person
as an extrovert (oriented primarily toward the outer world making
judgments on people and objects) or an introvert (making judgments
on concepts and ideas).
Sensing-Intuition (SN) – This dimension has to do with perceiving.
One may perceive through the senses or intuition.
Thinking-Feeling (TF) – This dimension differentiates between a
person’s judgment. A person may think impersonally on logical
consequences or feel decisions based on personal or social values.
Judgment-Perception (JP) – This dimension focuses on a person’s
extroverted side, how s/he deals with the outside world. A person can
either use judgment or a thinking process. Or, s/he might use a
perceptive process.
The MBTI emphasise the interpersonal and affective dimensions of a
person’s psyche. In order to distinguish serialists from holists we based the
interpretation of the psychological types on the findings made by Ehrman.
She applied MBTI to learning second language acquisition [Ehrman 1989;
Ehrman and Oxford 1990], with the following indications:
Extraversion – ‘Extraversion’ makes conversational risk-taking
possible, which is an important for trying out new vocabulary and
syntax, or just simply jumping in to a conversation even though one is
not 100% sure of what is going on. The extravert depends on outside
stimulation and interaction to engage in learning.
Introversion – ‘Introversion’ provides for concentration and
independence. The introvert weighs his/her utterances slow and
17
Chapter 2
carefully. S/he does not take any chances and is therefore not a
risk-taker.
Sensing – ‘Sensing’ allows for systematically process information and
observing details. The sensing person needs a clear sequence of
materials, clearly stated goals, and lots of structure.
Intuition – ‘Intuition’ makes inferences and guesses from a context
possible. S/he can structure his/her own training and conceptualises
well. S/he also builds good models from which to work. The intuitive
person is not concerned with accuracy and may overlook important
details. S/he may also seek excessive complexity in discourse.
Thinking – ‘Thinking’ facilitates analysis, self-discipline and
instrumental motivation. A thinker may be overly anxious because
self-esteem is connected with achievement and may need to have a lot
of control.
Feeling – ‘Feeling’ provides for an integrative motivation, which
leads to a good self-esteem through good relations with teachers and
classmates. A feeling person needs appreciation and must avoid tense
social situations.
Judging – ‘Judging’ is needed to engage in systemic work and to get
the job done. A judge is rigid and intolerant of ambiguity.
Perceiving – ‘Perceiving’ allows for an openness and flexibility,
which implies adaptability and new experiences. However, lack of
persistence and/or consistency may obstruct performance.
Holists are differentiated from serialists by being focused on concepts rather
than facts (‘intuition’ in contrast to ‘sensing’). Holists also want to understand
fully without necessarily reaching a conclusion rather than to take in just
enough information to arrive at a decision (‘perception’ in contrast to
‘judging’) [Felder and Silverman 1988; Davison, Bryan et al. 1999]. This was
also tested through a conducted preliminary study of the experiment
explained in chapter 5.
2.2.2.2 The Study Preference Questionnaire
The ‘Study Preference Questionnaire’ (SPQ), developed empirically by Ford
[Ford 1985], is based on Gordon Pask’s work on knowledge structuring and
learning styles. This is a questionnaire with eighteen questions. Each one of
18
Theoretical background
these has five possible answers. The outcome of this analysis shows the
participants learning strategy, if s/he uses a holistic, a serialistic or both
strategies. The latter is called versatile learners, which means that the person uses
holist and serialist strategies as appropriate and in an effective sequence. The
questionnaire was developed because of the complexity of the existing
analyses e.g. Pask’s ‘Spy Ring History’ and ‘Smugglers’ tests. SPQ has
efficiently been proved useful in a recent study investigating connections
between navigation in hypermedia and learning styles [Lönnqvist 2000].
2.3
Towards a theory on Game-Based Instructional Design
“Tell me, and I will forget. Show me, and I may remember.
Involve me, and I will understand.”
Confucius (about 450 BC)
As mentioned in the introduction of this chapter, there is a lack of learning
theories about learning through computer games. There is also a lack of a
corresponding technological framework on the design of such a computer
game. One reason could be the many ways in which computer games’ design
varies. A second reason could be the lack of game-based instructions (GBI).
A third reason may be that computer games are considered to be designed for
children, and consequently not for adults. This is important, because most of
the evolving learning theories do not uniquely take into account adult
learners, and there are differences such as adults need a valid reason to learn.
While children have few experiences, their brains are able to create new
neurological structures when learning occurs. In contrast, adults tend to
have existing neurological structures already due to their vast
experiences. New learning for adults, then, tends to require new
connections between existing neurological structures. For this to occur,
higher-order brain functions must take place.
[Conner 2000] 17
17
Web source: http://www.learnativity.com/
19
Chapter 2
Several researchers are questioning current learning theories’ applicability to
today’s instruction [Jonassen 1991; Dryden and Vos 1994; Harapnuik 1998].
They argue that the theories do not account for modern IT in education such
as distance learning and hypertext-media. Instead, new forms of theories have
started to evolve such as inquisitivism.
Although this theory does not distinctively have GBI in mind, but rather the
more general computer-based instruction (CBI), this is the theoretical
framework chosen as a base towards a model for GBI.
The ten key concepts of the inquisitivistic approach are: fear removal,
stimulation of inquisitiveness, using the system to learn the system, getting
started fast, discovery learning, modules can be completed in any order,
supporting error recognition and recovery, forum for discussions and
exploiting prior knowledge, real world assignments, and developing optimal
training designs. None of the concepts are particularly unique for
Inquisitivism instead they are reflecting a synthesis of constructivism,
discovery learning, active learning, functional context, situated learning and
minimalism.
2.3.1 Constructivism
Learning is an active process in which learners construct new ideas or
concepts based upon their current/past knowledge [Kearsley 1997]. In the
theory of andragogy the instructional designer has to consider that adults need
valid reasons to learn, often approach learning as problem-solving and they
learn best when their acquired knowledge can be applied immediately.
Knowles put forward that adults are self-directed and expect to take
responsibility for their own actions and decisions [Knowles 1975; Knowles
1984a; Knowles 1984b].
20
Theoretical background
2.3.2 Discovery/Experiential Learning
“There is an intimate and necessary relation between the processes of
actual experience and education.”
[Dewey 1938]
Learners should obtain their own knowledge with an instructor (teacher or
instructional media) as an assistant to either solve a problem or gather
information and develop a hypothesis. Learning depends on the way we
process experience or, more important, our critical reflection of experience.
In ‘Kolb’s Experiential Learning Model’ [Kolb, Rubin et al. 1974], Kolb
describes learning as a four-step process, identified as: watching and thinking
(mind), feeling (emotion), and doing (muscle). The concept explores the cyclical
pattern of all learning from experience through reflection and
conceptualisation to action and to further experience. His work is primarily
based on works from Dewey (emphasising the need for learning to be
grounded by experience) [Dewey 1938], Lewin (emphasising the importance
of the learner being active in the learning process) [Lewin 1951], and Piaget
(describing intelligence as the result of the learner and the environment)
[Piaget 1952].
2.3.2.1 Kolb’s Experiential Learning Model
According to Kolb, a learner is forced to select a ‘mode’ to manage incoming
information. This idea is based on Piaget’s figurative and operative aspects of
thought18; with this in mind he defines learning style on a two-dimensional
scale based on how a person perceives and processes information. The way a
person perceives information is categorised as ‘concrete experience’ or
‘abstract conceptualisation’ and the processing of information is classified as
‘active experimentation’ or ‘reflective observation’ as illustrated in Figure 2.
Learners usually begin a concrete experience, which involves them in the
topic or material. They then engage themselves in reflective observations,
considering the subject from different angels. This process leads to abstract
conceptualisation in which they develop theories or generalisations about the
18
Through experiments with children, Piaget investigated how knowledge and insight develops [Piaget
and Inhelder 1971].
21
Chapter 2
topic. At last, they use active experimentation to apply their theory to related
material.
Figure 2. Experiential Learning Cycle [Kolb 1984]
When combining the two dimensions Kolb gets four types of learners.
1. Diverger (concrete, reflective) – experience is gathered through
immediate experience and is turned to thought by internal reflection
on the external world. This type of learner usually ask ‘Why?’
2. Assimilators (abstract, reflective) – understand information best when
it is presented symbolically and conceptually and time is given for
internal reflection. This type of learner usually asks ‘What?’
3. Convergers (abstract/active) – understand and perceive information
best through concepts and symbols but need to work actively
(external manipulation) in order to process the information. This type
of learner usually asks ‘How?’
4. Accommodators (concrete/active) – gather information best by
immediate experience and need to process information through active
action. This type of learner likes to solve real problems and usually
asks the question ‘what if ?’
22
Theoretical background
Many researchers have questioned the validity and reliability of this model as a
result Kolb revisited the primary model in 1984 [Kolb 1984]. Although, many
researchers still question this, the model is widely spread and used.
2.3.3 Activity Theory
To require ‘understanding’ learning is mediated through practical activity;
activity is mediated by cultural signs i.e. language, tools, media, and
conventions. As the products of learning change, activity changes along with
the consciousness of the participants in a continuous, evolving cycle of
learning [Harapnuik 1998].
2.3.4 Functional Context
The Functional Context Theory was developed for adults, training technical
and literacy skills. The emphasis is to use instructional material similar to
those that are used in the ‘real world’ [Sticht 1975]. Functional context
approach is based on the following:
Instruction should be made as meaningful as possible to the learner
with regards to the learner’s prior knowledge.
The learning material used should reflect the ones the learner will
actually use after training.
Literacy can be improved by: improving content knowledge,
information processing skills, or the design of the learning materials.
Evaluation of acquired knowledge in the learning requires
context/content specific measurement.
2.3.5 Situated Learning
The Situated Learning Theory stresses the importance of social interaction.
Learners become engaged in a ‘community of practice’ and adopt the beliefs
and behaviours of that community. Novices interact with the experts and
become an expert of their own. This ‘mentorship’ is called ‘cognitive
apprenticeship’ and is supposed to occur unintentionally. “Cognitive
apprenticeship supports learning in a domain by enabling students to acquire,
23
Chapter 2
develop and use cognitive tools in authentic domain activity. Learning, both
outside and inside school, advances through collaborative social interaction
and the social construction of knowledge” [Brown, Collins et al. 1989]. This
pedagogic framework has successfully been applied to CBI [Tholander 2001].
2.3.6 Minimalism
“One of the key ideas in the minimalist approach is to present the smallest
possible obstacles to learners’ efforts, to accommodate, even exploit, the
learning strategies that cause problems for learners using systematic
instructional materials.” [Caroll 1990].
In an experiment at IBM Watson Research Center, Caroll gave a control
group of employers the traditional 94 pages systems-style manual to learn. An
experimental group were given a reduced version – 25 cards providing only
general information for completing a task. The cards were intentionally
created incomplete so that the learner had to fill in the missing details for
him/herself. When evaluating, the participants were required to complete a
real task. The results from the experiment showed that learning through the
cards allowed for much faster learning and more successful performance in
the task.
According to [Harapnuik 1998], the minimalist approach offers one of the
best theoretical foundations from which adult instruction can be designed.
2.3.7 Inquisitivism
“...when learning computer games. Within minutes of handing a computer
program or game to a child they have it installed, clicked on every menu item
or button on screen, and are well on their way to racking up a score that most
adults could never dream to achieve” [Harapnuik 1998].
Inquisitivism is about removing fear towards instructional tools and
stimulating the natural curiosity. The key concepts include [Harapnuik 1998]:
1. Fear removal. Dealing with the paralysing fear that many adult
learners experience must precede the stimulation of one natural
inquisitiveness. Demonstrating that the computer or other piece of
technology is not easily broken, providing explanations, examples and
24
Theoretical background
solutions for common errors and problems and the application of
data backup will help quell the adult learners fear.
2. Stimulation of inquisitiveness. With the fear abated, encouraging
adult learners to become like children and enjoy the pleasure of
inquisitiveness can be easily facilitated. Encourage the use of the
‘HHHMMM??? What does this button do?’ approach.
3. Using the system to learn the system. All training must take place
on the actual system that is being learnt.
4. Getting started fast. Adult learners often have other interests than
learning a new system. The learning they undertake is normally done
to complement their existing work. The ‘welcome to the system’
prefaces and other non-essential layers in an introduction are simply a
waste of the learners’ time.
5. Discovery Learning. There is no single correct method or
procedure. Allowing for self directed reasoning and improvising
through the learning experience will require the adult learner take full
responsibility for his/her learning.
6. Modules can be completed in any order. Materials must be
designed to be read or completed in any order. Students impose their
own hierarchy of knowledge with is often born of necessity and
bolstered previous experience. This will eliminate the common
problems that arise from material read or completed out of sequence.
7. Supporting Error Recognition and Recovery. Much of what
learner does is ‘errors’. Since there is such a pervasiveness of errors in
most learning it is unrealistic to imagine that errors can be ignored.
Error recognition and recovery strategies need to be implemented to
enable learners to learn from their mistakes instead of being trapped
by them. Use of Frequently asked Question lists (FAQ’s) Help
Forums and other help strategies must be implemented to deal the
errors and problems that arise.
8. Forum for Discussions and Exploiting Prior Knowledge. Adult
education dealing with technology is often conducted through
alternative delivery. Distance education, web-based instruction (WBI)
and other alternative delivery methods can isolate students. Providing
a conferencing system or similar computer mediated communication
system for the replacement of face2face interaction is a crucial
component of any alternative delivery program. Most adult learners of
technology are experts in other areas or domains. Understanding the
learners’ prior knowledge and motivation and finding ways to exploit
25
Chapter 2
it is one of the keys to effective adult training. In addition, adult
learners can share their expertise or assist each other and should be
encouraged to use conferencing system or other forms of computer
mediated communication to facilitate social interaction.
9. Real World Assignments. ‘Make-work’ (purposeless) projects are
simply useless. All assignments must have a real world application.
Adult learners are often undertaking training to be able to work in
their own area of expertise more effectively. If possible the
assignments should be tied directly to the learner’s personal or
professional interests.
10. Developing Optimal Training Designs. Feedback facilities like
online surveys or email should be used to allow learners to
immediately provide feedback on any aspect of a program. Problems
with instructions, assignments, wording or other problems should be
immediately addressed corrected. Instructional models are not
deductive or prescriptive theories they are descriptive processes. The
design process should involve the actual learner through empirical
analysis so that adjustment can be made to suit the learners’ needs.
‘Develop the best pedagogy that you can. See how well you can do.
Then analyse the nature of what you did that worked.’[Bruner 1960]19.
2.4
The theoretical framework for Game-Based Instructional Design
A well-designed computer game ought to be an ideal learning environment
because the learner will face real problem solving situations, on the
assumption that all tools and pieces of information are available to solve a
problem/concept. Additionally issues are:
Owned knowledge; greater transfer of training and greater retention,
because the discovered knowledge is ‘owned’ by the learner and then
organised in terms of the learner’s interests and cognitive structures
[Travers 1977].
Learner control; it is important that the learner has control over the
learning situation [Goforth 1994], where control is defined as the
power to manage the interaction between learner and learning
material.
Time; it is reasonably to assume that learning through a discovery
learning process takes longer to accomplish than linear teaching
19
Bruner according to [Harapnuik 1998].
26
Theoretical background
methods. Additional time used in discovery learning and its effects
must be taken into account when discussing its effectiveness as a
method of instruction.
By using elements of the previous mentioned learning theories the
applicability, and suitability of using these in a framework for Game-Based
Instructional Design based on Inquisitivism will be discussed. This framework
is not complete and need to be revised to include the findings from the
experiments presented in chapter 5. The following are the qualifications to
use the ten concepts from inquisitivism:
1. Fear removal
By learning through playing a game, the learner can explore the menus,
buttons and objects on the screen in his/her own pace, without pressure.
Since the learner already knows it is only a game, and therefore not a reality,
s/he can be relaxed in a surrounding already familiar – the computer.
2. Stimulation of inquisitiveness
In a computer game, exploration is necessary – there is no game without it.
By presenting a holistic view over the constituent parts and concepts
involved, the learner can be stimulated to explore.
3. Using the system to learn the system
A computer game can be played on the learner’s own computer (the actual
system), simulating tools and concepts partly familiar. In a well-defined
simulation, you can expose and exploit the actual computer’s resources
without any fear. But, it has to be clear that it is only a simulation and nothing
will happen to the system or to the data used.
4. Getting started fast
The computer game needs to be constructed in a way that the player can start
play immediately and interact with the objects. The goal has to be clear and
the interface navigation easy and intuitive. Depending on the computer game
an introduction may be needed.
27
Chapter 2
5. Discovery Learning
Through a computer game the player will need to cycle the experiential
learning model by Watching, Thinking, Feeling and Doing – ‘Trial and Error’.
The computer game needs to offer all the tools and information needed to
solve a problem or learn a concept in a format that the learner ‘makes sense’
of.
6. Modules can be completed in any order
Through a computer game the learner has to be able to interact with all
objects on the screen, in any order and by ’trial and error’ manners drawing
conclusions about the behaviours.
7. Supporting Error Recognition and Recovery
Error recognition and recovery strategies need to be implemented to enable
learners to learn from their mistakes instead of being trapped by them. Help
forums and other help strategies must be implemented to deal with the errors
and problems that arise. There has to be a possibility to ‘start from scratch’. In
order to find out what failed and to learn from mistakes the learners need a
guide to assist them in what has happened so far – a sort of a history
description.
8. Forum for Discussions and Exploiting Prior Knowledge
A forum for discussions is a good help in getting answers. However, in a
single user computer game this is not possible, although there has to be some
kind of help within the game – perhaps a vocabulary or links to further
reading/discussions about the subject being taught.
9. Real World Assignments
All problems/assignments must have a real world application. The simulation
needs to take account for different kind of learners, with different kind of
previous knowledge.
10. Developing Optimal Training Designs
In order to improve the training design there has to be a possibility for the
learners to give feedback.
28
Theoretical background
2.5
Quality of knowledge
Traditionally, questions in tests are still to a large extent presupposing one
correct answer, without considering the reasoning behind. This is also
reflected in the positivistic pedagogical view of knowledge, as being able to
recall facts, variable names etc. In an attempt to structure the domain of
knowledge Benjamin Bloom and his colleagues designed a learning taxonomic
classification of cognitive, affective and psychomotor behaviours where
learning was divided into cognitive, affective and psychomotor domains
[Bloom, Engelhart et al. 1956].
Cognitive Domain – The recall or recognition of specific facts,
procedures, concepts, and universals that serve in the development of
intellectual abilities and skills.
Affective Domain – The way in which we deal with things
emotionally – feelings, values, appreciation, enthusiasms, motivations,
and attitudes.
Psychomotor Domain – Involves physical movement, coordination,
and the use of motor skill areas.
In an attempt to establish a hierarchy of educational objectives, the cognitive
domain, which involves knowledge and the development of intellectual
attitudes and skills, was divided into subdivisions ranging from the most
simple to the most complex behaviour, illustrated in Figure 3.
Figure 3. Bloom’s Taxonomy of Educational Objectives
[Bloom, Engelhart et al. 1956]
In this thesis’ experiments we compared the level of understanding acquired
during different instructional settings. The level of acquired understanding
was determined by a test with 12 questions designed to match ‘knowledge’
and 10 questions designed to match ‘comprehension’ using Bloom’s
taxonomy of educational objectives. The following are the educational
objectives with typical questions from the test.
29
Chapter 2
Knowledge
The level of ‘knowledge’ simply requires recalling acquired knowledge –
memorisation. Words typically used are: define, recall, recognise, remember,
who, what, where, when. For example:
A symmetric encryption key is used to?
1. encrypt a message
2. decrypt a message
3. both of the above
Comprehension
The level of ‘Comprehension’ requires a sufficient understanding to organise
and arrange material mentally – understanding. Words typically used are:
describe, compare, contrast, rephrase, put in your own words, explain the
main idea. For example:
Explain in your own words how Adam can send a message to Eve, and be sure that no one
else except Eve can read it?
The common interpretation is that ‘knowledge’ is required to achieve
‘comprehension’, but it is possible to ‘comprehend’ how to use asymmetric
encryption in practice without ‘knowing’ the constituent parts such as how
the mathematical algorithms are constructed. It is also possible to ‘know’ how
one particular mathematical algorithm for asymmetric encryption is
constructed without ‘comprehend’ how to use it in practice.
These two are the only levels used in this thesis’ experiments in order to
discuss how the quality of acquired knowledge depends on the type of
educational instruction. Therefore no examples are given here. Nevertheless,
the other four levels, which will not be discussed further in this thesis, are:
Application
Implies the use of knowledge to solve problems.
30
Theoretical background
Analysis
Examining material or relationships of information of constituent parts and
arriving at some solution or response.
Synthesis
Combining elements and parts into a unified entity.
Evaluation
Involves making judgements, appraising, measuring, and critically inspecting
some idea or object and determining its relative value or worth.
2.6
Applicability within IT security education
The complexity of the information and communication security field makes
instruction challengeable.
Most of the IT security models today, include not only technical but also
managerial and social issues. Kowalski suggests in his proposed SBC-Model,
illustrated in Figure 4, the importance of including ethics, politics and law,
operations and management, and technology when protecting information
handled by IT [Kowalski 1994]. Yngström stresses the importance of relating
IT security problems to its surrounding by delimiting the problem to existing
physical entities, on specific abstract levels, in specific contexts. She has
summarised the ideas in the Systemic-Holistic Model, illustrated in Figure 5,
and proposes a systemic-holistic approach when trying to understand
IT security [Yngström 1996]. Both models are based on general systems
theories.
Because of the extremely wide span of issues to consider and the
multi-disciplinary nature of these issues, versatile approaches are necessary
when teaching IT security. Today’s instruction, which is still, to a large extent,
linear, does not fit when teaching IT security.
Since learners have different learning preferences and IT security is multidisciplinary we need to be open-minded to alternative/complementary types
31
Chapter 2
of instructions in order to improve understanding of IT security. There is a
lack of instructions supporting learning in quadrant 3 and 4, Figure 6.
Figure 4. The SBC Model [Kowalski 1994] p.19
Figure 5. The Systemic Holistic Model [Yngström 1996] p.31
32
Theoretical background
Figure 6. Classification of learning preferences
and disciplines taught. [Yngström 1996] p.158.
There is a lack of instructions supporting
quadrant 3 and 4.
2.7
Chapter conclusion
In this chapter we have presented:
1. Different learning styles and how these can be measured.
2. Theories for learning that can be used to develop a theory/framework
for GBI design.
3. We have also suggested such a framework for GBI design.
4. We have also discussed and decided how to measure quality of
knowledge.
5. Furthermore, we have discussed the applicability of using GBI within
IT security education.
We have discussed the importance of not applying learning theories for
children to adults. They think and learn differently. We will also discuss this in
the next chapter when discussing the players of computer games.
33
Chapter 2
In the next chapter we will continue discussing the design of GBI from a
user/developer perspective.
34
Chapter 3
GAME-BASED INSTRUCTION
This chapter starts with a discussion about what characterises a game.
Thereafter, comes a presentation of a selection of existing or planned
game-based teaching methods within IT security and adjacent domains.
3.1
Chapter introduction
The usefulness of using computer games as instruction in order to acquire a
higher level of awareness within IT security is an unexplored educational field.
Although, there exist adjacent game-approaches to learning and some of these
will be presented. But, so far this researcher has not encountered a serious
evaluation of their effect on the quality of learning, especially not in contrast
to conventional instructions. In the previous chapter was discussed that
learning through discovery usually takes longer time than conventional, linear
instructions and, as we will discuss in this chapter, GBI are more expensive to
develop. So, the question is; are GBI motivated to develop?
I will start by exploring the nature of computer games in terms of:
The characteristics
The people involved
o The players
o The developers
35
Chapter 3
3.2
Characteristics of a computer game
According to Chris Crawford, a game is a closed formal system that
subjectively represents a subset of reality [Crawford 1984]. By that he refers
to:
‘Closed’ – The model world created by the game is internally complete
and self sufficient as a structure.
‘Formal’ – The mainstream of game play has explicit rules.
‘System’ –A game consists of a collection of parts that interact with each
other in complex ways – a system.
‘Subjectively represents’ – A game creates a fantasy representation, not a
scientific model. Crawford means that the distinction between a
simulation and a game is that “a simulation is a serious attempt to
accurately represent a real phenomenon in another, more malleable
form. A game is an artistically simplified representation of a
phenomenon” [Crawford 1984] p.8. A player’s fantasy is the key agent
in making the game psychologically real.
‘Subset of reality’ – First, a game is only a model of reality, and thus a
subset of reality. Second, a game provides ‘focus’ to the choice of
matter and therefore only presents a subset of reality. “...too large a
subset of reality defies the player’s comprehension and becomes
almost indistinguishable from life itself...” [Crawford 1984] p.9.
Because of each computer game’s uniqueness, it is difficult to strictly
categorise a certain game. But it seems that the following categories can be
recognised as distinguishable:
Action
In this category, the player is represented as the main person. The story is
usually about beating the opponents, with cunning and pure force.
For example; Tomb Raider, Doom, Quake.
Adventure
The player is placed in another world. The story is about solving mysteries
and/or exploring the world. For example; Safecracker, Grim Fandango.
36
Game-Based Instruction
Recreation and sports
The game is based on actual /or fantasy/ recreations. For example; footballand golf games.
Role-playing
The player is one of a company/group, and all involved players create part of
the intrigues and story as time passes. For example; Final Fantasy, Ultima
Online.
Simulation
The game is a model of reality e.g. in action and/or behaviour. For example;
flight simulators as well as SimCity.
Strategy
The player has with cunning to overcome his/her opponent in different
scenarios. For example; Command and Conquer, Warcraft.
Puzzle
The game is usually a computerised representation of simple board-games.
For example; cards, Yatzee, Trivial Pursuit.
Most of the games listed above have more than one aspect attached to it and
could easily fall under several categories.
In some categorisations, ‘Children games’ and ‘Educational games’ are added
to the list of game categories. Interestingly is that games for children and
educational games, usually falls under the same category.
3.3
People and games
Why are not educational computer games professionally developed? This
question will be discussed by answering some more questions.
What do the typical player and developer look like? Are they tuned with each
other? Do they have something in common? Is there a difference in
producing a computer game in contrast to other similar products?
37
Chapter 3
3.3.1 The players
One thing that is probably unique with game players, in contrast to users of
other entertainment, is the possibility of becoming famous or even rich, as a
player [Frank 2001]. Nowadays, there are competitions all over the world, to
survive the longest, to receive the highest score etc. The carrot is money and
glory, perhaps even outside that particular ‘game-community’20.
Though the computer game industry is relative young (about two decades),
people have used traditional games for recreation, amusement, learning and
competition for as long as we know.
A general opinion is that computer games are only for children but this is not
the case. All groups in the society play games. There are some differences
however, in how and why people play. The independent variables –
determining factors – seems to be based on age, gender and how pressed for
time one feels [Malekani 1998].
Age:
This relates to the familiarity with computers and new media a person has.
The young people of today are not used to slow mediums such as books or
conventional instruction [Keirsey and Bates 1984; Highland 1992; Näckros
1999a; Prensky 2001]. Marc Prensky calls this group ‘the under-30 generation’
[Prensky 2001]. They are used to fast media such as MTV, cellular telephones
and Internet where everyone (of importance!) is reachable at all times. They
have developed a new way of communication. The conventional teaching
strategies are simply too boring. Characteristics for this game-playing group of
people (19>35) are:
How: They can afford buying expensive computer games, and,
Why: The reason for playing is usually to relax [Malekani 1998].
20
All players sharing the same interest in a particular game usually creates a community together in
which they exchange ideas and game strategies with each other.
38
Game-Based Instruction
Younger people (13>22) tend to use computers as a medium for social
activities. They often play together in net-based games21 [Frank 2001].
Older people (limited familiarity and/or not grown up with computers) play
for the same reasons but prefer using more traditional games, not requiring
computers or games that are similar to these [Frank 2001].
Kids (3>16), usually play to learn something, solve problems and/or feel to
have control and create their own worlds [Östfeldt 1999]. This thesis does not
cover this group of people. But, it is important to mention that kids and
adults learn and think differently [Knowles 1975; Knowles 1984a; Harapnuik
1998; Conner 2000], have different views and reasons to play, as presented
above. It is therefore not possible to develop/evaluate games and instructions
for the two groups according the same models.
Gender:
According to [Malekani 1998], a significant majority of the adult players are
males. This is however not the case when it comes to children, where the
difference between the genders is minor [Östfeldt 1999]. Today’s children are
used to modern technology and therefore integrate it in their everyday life.
Characteristics between genders are [Malekani 1998]:
How: About 41% of the males played regularly, mostly ‘action’ games
for about four hours a week. Only 11% of the females played
regularly, preferring ‘strategy’ and ‘adventure’ games, playing in
average two hours a week.
Why: The male players stated that they play for fun, and a good way
to pass time and to escape from reality. The females did not state any
particular reason. They simply disliked it. Too time-consuming and
they prefer to meet friends in person instead.
Lack of time:
Stressed people with much lack of time do not play. That is also the main
reason from the participants in Malekani’s investigation who answered that
21
Net-Based games are played by several players, at different computers and locations, connected by a
network, sharing the same game.
39
Chapter 3
they do not play. They simply do not have the time to spare. However, males
tend to a larger degree to play, as a safety-valve, to relax.
Summary for players:
The common adult player is a male between 25 and 30 years old and plays
mostly action games for fun to relax and escape reality for about 4 hours a
week.
3.3.2 The developers
The author of this thesis attended at a seminar in Stockholm, 27 April, about
current trends and the future of computer games [REPLAY 2001]. Among
the participants were computer game developers, producers, journalists,
scriptwriters, artists and educational institutions represented.
Subjectively, this researcher realised that the developers’ target group is male,
between 13 and 35, that is, the same group of people that actually produces
the majority of games. The author of this thesis suggested at the seminar,
expanding the target group by producing other kinds of games, but the
participants at the seminar had no plan of doing this. The author continued
by suggesting educational computer games for adults, but still very little
response.
My impression was that it seems most of the games are produced by
enthusiasts developed for the same enthusiastic target group. Peter Molyneux,
a computer game developer, has also verified this phenomenon at a later
occasion [Heimbach 2001]. It also seems that educational computer games are
apprehended as childish by definition, and something they do not want to be
related to. Instead, they want to impress and astonish each other with the
latest technology, in the hunt for the perfect virtual world. Developing
educational tools will make them feel restricted in the development process.
This researcher hopes that this reflection was wrong; otherwise the
technology will lead us (our creativity), instead of the other way around.
40
Game-Based Instruction
In Germany, Molyneux notice a new trend of computer games where moral
and ethics are vital [Heimbach 2001] such as; feed your chicken otherwise it
will become hungry and cruel and may hunt you down.
Anders Frank has identified four major problems with the computer game
industry today [Frank 2001]:
1. Game development is regarded as a high-risk project. A modern
computer game is expensive to develop, which makes the publisher
/producer/ only to invest in already tested games. This eliminates
new ideas and new form of games to be developed.
2. The cost of developing a working demo. It is very difficult to get
investors to the project. Often they demand a working demo of the
actual game. Who is paying for the development of this?
3. The pressure on the developers. A computer game has a time-span
for about five moths. As soon as the project is initiated, the
advertisement of the product begins. So, it is very important to keep
milestones and deadline for distribution in order to get return on
investments. This can restrain the actual developers.
4. The dependence of technology. It is not uncommon that during the
development process the technology changes, and the developers
have to redesign the whole game.
To develop a modern computer game is a time-consuming and an expensive
task. Today, a standard computer game requires between ten and fifty
developers, working fulltime for about two years [Fjellman and Sjögren 2000].
So, this researcher repeats the question; are they motivated to develop?
3.4
Similar approaches
At the time this research was initiated (1998) most of the following games
within IT security were not developed i.e. Cyberprotect and Warning for
Virus. During 2000, Telia22 and ÖCB23 together with ‘Dataföreningen i
Sverige’24 decided to develop a computer game to increase the players’
22
Telia - a major phone company in Sweden.
23
ÖCB – Överskyddstyrelsen för Civil Beredskap (the Swedish Agency for Civil Emergency planning).
24
Dataföreningen i Sverige (the Swedish Information Processing Society).
41
Chapter 3
awareness of basic IT security. The author of this thesis was also involved
through active participation in the reference group.
Cyberprotect is an interesting product because it is a professionally developed
product that utilises action in such a way that the user maintains the overview
of how the different objects in the system interact together.
It is the author’s belief that these are both good examples of introducing
computer games as instruction in the society. Unfortunately, this researcher
has so far not seen any attempts to evaluate these as instructions.
3.4.1 IT security
Cyberprotect from SAIC
This is a graphical arcade-like game produced by US Defence Information
Systems Agency and Carney Interactive. The idea is to protect your assets
from network intrusions.
As player, you start with a budget and invest in protective means with regards
to malicious code, viruses and users, both insiders and outsiders. When you
are ready, it is time to test your solution. The network and the system are then
exposed to three randomised attacks. You will then get feedback on the
outcome.
The player will learn that securing your network is a continuing task, and that
it is impossible to secure the system against everything.
Warning for Virus from SBA-education25
The player will be introduced to a collection of scenarios, and has to choose
one of these to play. Each scenario is designed to match a security problem to
solve. The general process of doing this is the following; a problem has
occurred and something went wrong. By interviewing different roles in the
organisation the player is now supposed to find out how the problem could
have been avoided and safeguarded.
The purpose is to make employees understand basic IT security.
42
Game-Based Instruction
The Cracking Game from Fred Cohen & Associates26
The player’s goal is to get into the mystery site, find out what they are doing
in there, act accordingly, and get out without being tracked back to your real
identity.
The purpose is to understand how crackers thinks.
Network Security Simulator from Fred Cohen & Associates
The purpose is to make employers understand basic IT security vulnerabilities
in a computer network.
SecGo Tutor from Instrumentointi Oy27
This is not really a game but a teaching aid that utilises sound and images to
navigate and read about IT security. Through guided tours, dictionary, lessons
and tests, the player acquires awareness in IT security.
The purpose is to make employees understand basic IT security.
3.4.2 Economics
Hotcalc from Adactor28
Hotcalc is a role-playing, simulation game that takes place in a fictitious
Swedish town and is played over a network. Up to six groups of people can
participate at the same time and a role-playing ‘leader’ is needed to control the
events. The players run companies within hotel and restaurant businesses and
have the ability to cooperate or compete with each other. As in real life they
must plan their actions and are forced to come to a decision according to
their prerequisites.
The purpose is to make people within hotel and restaurants businesses to
learn how to run a business.
25
http://www.dfs.se/sba/vvhs/
26
http://all.net/
27
http://www.secgo.com/
28
http://www.hotcalc.se
43
Chapter 3
Moneymaker from Intermezzon29
MoneyMaker, is an interactive, scenario-based simulation training designed to
practice complex B2B30 sales processes. Through realistic scenarios the player
is supposed to win a huge contract from the customers.
The purpose is to teach the player sales, customer service and relationship
management.
3.4.3 Law, Criminology, Journalistic
VR-reporter from James Madison University, Virginia3132
This is an investigating game. The player is a reporter, on a crime scene and is
supposed to find out what has been happening. By turning himself 360
degrees on the screen, the player can see everything on the scene and
interview the people he wants. Thereafter, he can supplement his notes with
additional phone interviews.
The purpose is to learn how to observe and investigate crimes.
Ongoing project – Multimedia as jurisprudential teaching aid33 from IRI34
The purpose is to increase the awareness and knowledge within law of
procedure.
3.4.4 Drilling tools
Straight Shooter from Games2train35
This game reminds of the existing 3-dimensional action games on the market
e.g. Quake, where instead of shooting monsters the player ‘shoots’ ideas with
his cellular phone towards demanding customers. The goal is to obtain new
customers to the bank.
29
http://www.intermezzon.com
30
B2B (Business to Business)
31
http://maccentral.macworld.com/news/0102/09.reporter.shtml
32
http://www.jmu.com
33
http://www.juridicum.su.se/iri/pewa/bevisproj.htm
34
IRI – Institutionen för RättsInformatik (the Institution for Law and Crime, Stockholm University).
35
http://www.games2train.com
44
Game-Based Instruction
The purpose is to make the players remember a given material e.g. a company
policy.
3.4.5 Design
The Monkey Wrench Conspiracy from Games2train
Through making and repairing tools e.g. weapons, the player is going to
rescue a space station from being hijacked by aliens.
The purpose is to make the player learn how to use a certain tool for
CAD/CAM i.e. design tool.
3.4.6 Simulators
The purpose with a simulator is to simulate real-time events. It is important
that the surrounding is similar to reality e.g. flying an aircraft, car, nuclear
power plants etc.
The purpose is to make the player experienced and familiar with the simulated
scenario without actually being there (without any risk).
3.5
Model for computer game design
When producing an educational game we need to be able to evaluate its
effectiveness and usefulness e.g. in order to make improvements. As any
other software this also includes the design. The fact that there is a lack of
formalised models for evaluating computer game design makes evaluation
complicated.
To be able to evaluate the quality of game design we needed a number of
critical parameters that could illustrate the game’s constituent elements. We
decided to use Geof Howland’s model that separates the game design into
five design elements [Howland 1998], graphics, sound, interface, gameplay
and story.
1. Graphics; is all images that are visual and all effects that are applied
on these. It also includes 3D-objects, textures, 2D-objects, video and
everything else that the player is going to see.
45
Chapter 3
2. Sound; is music, sound effects and everything else that the player is
going to hear.
3. Interface; is everything that the player needs to interact with to be
able to play the game. It includes the clickable graphics, menus that
the player has to navigate through and the game’s own possibilities to
adjust or control the game’s different parts.
4. Gameplay – how fun it is to play. How involved or engaged the
player becomes. Could be measured in time that the player plays.
5. Story; is about all background information that are necessary to know
before the game starts, all information that the player receives while
playing and when the game is over and all information that the player
learns about or through the characters in the game.
3.6
Chapter conclusion
In this chapter we have identified that the framework of a GBI for
IT security, suitable for adults, should additionally meet the following
requirements:
1. Use the latest technology regarding sound and graphics
2. Reflect a ’real system’, simulation.
3. Be result oriented
4. Not be childish
5. Not be too obviously educational, but give the feeling of making
progress
6. Be possible to evaluate
We have also presented an evaluation model for the design of a computer
game.
46
Chapter 4
THE PARADISE MODEL
The two previous chapters discussed the importance of matching learning
and teaching strategies within IT security, and presented a selection of GBI
within adjacent domains. However, they were not specifically developed in
the context to contrast linear instruction, and especially not within the field of
IT security. In this chapter we want to discuss and present the developed
GBI, the process and our experiences from the development.
4.1
Chapter introduction
We did not find a GBI within IT security, at the time (1998), suitable for
evaluating its usefulness as a non-linear instruction, instead we needed to
develop a GBI that would match the requirements/framework presented in
the two previous chapters.
To design an interactive computer game is a task differing from designing
regular software. Even in its simplest form, there are many complex
relationships on different conceptual levels [Yngström 1999].
Because the lack of existing methodologies and documentation regarding the
development of GBI, we found it suitable to present some of our evolved
experiences and ideas.
47
Chapter 4
4.2
Development of the GBI model
The game design sequence follows Chris Crawford’s empirical suggestions on
how to approach this kind of production [Crawford 1984].
1. Choose a goal and a topic
2. Research and preparation
3. Design phase
a. I/O structure
b. Game structure
c. Program structure
d. Evaluation and design
4. Pre-programming phase
5. Programming phase
6. Play-testing phase
7. Post-mortem
This is not a step-by-step procedure, “...game design is far too complex an
activity to be reducible to a formal procedure...formal reliance on procedures
is inimical to the creative imperative of game design” [Crawford 1984] p.49.
We will explain the different sequential steps as they come.
4.2.1 Goal and Topic
It is vital that a game must have a clearly defined goal, expressed in terms of
its effect on the player. Since this is an educational game, the goal will be to
make the player learn/understand the intended message.
The means, in which the goal is expressed, is the topic i.e. the environment in
which the game will be played. After some brainstorming we ended up with
four different topics, initially called scenarios:
48
The Paradise model
1. Communication between two individuals via an electronic media, to
show the need of security mechanisms in ensuring confidentiality,
authenticity, integrity and non-repudiation. The solution could be
digital signatures and encryption of messages.
2. Contingency planning – to show the balance between time for/extent
of/ recovery from loss of data/information and the cost/extent of
doing this. What can be done? The need for backup etc.
3. Classification of documents/information. Different kinds of
documents that need to be categorised in a rapid speed, with a
classification scheme as a model. The model is first presented to the
player and when the documents start popping up on the screen, the
player needs to classify it according to the model, as fast and correct
as possible. If s/he fails doing this, the document becomes a potential
threat and an event randomly occurs e.g. confidential information is
being exposed to unauthorised users, modified or deleted.
4. Take the role of an IT-security manager in a simulated
information-intense company. Different events happen all the time
e.g. new threats, new employees, un-recoverable backups,
interceptions. The balance between the effectiveness of the company
and the implementation of different safeguarding mechanisms will be
visualised by economic consequences and sound working conditions
as for instance happy and loyal employees.
At first we decided to continue working with topic number 4. This was the
most challenging one and would offer good opportunities to instruct how to
look upon a system as a whole by viewing security on different conceptual
levels. Hopefully, the learners will understand that security needs to be
integrated into the system instead of just being viewed as an additional
module. This approach had the holistic view and the simulation aspects we
wanted to achieve. The idea was to create a technological framework with a
set of modules. One module would correspond to one security threat or a
safeguarding method. Students could then create additional modules and
successively expand the possibilities of the game, see Figure 7 – the SecSim
game.
49
Chapter 4
Figure 7. An example view of SecSim game.
Unfortunately, we did not, at the time, have the resources to go through with
this. We also saw difficulties in evaluating the results against our hypothesis.
We chose together with our industry partners SEIS and Säkdata, both
working within PKI (Public Key Infrastructure), topic number 1.
The goal became therefore to make the players of the game to understand
PKI. How it works, what and how it is used.
The topic was decided to use a holistic view, in a non-linear fashion, on the
communication process i.e. two participants communicating electronically –
(1) a sender and (2) a receiver of (3) a message, via (4) a network, exposed to
50
The Paradise model
(5) a malicious user and (6) a common third part upon which all participants
trust and rely. The player should at all times be able to view the actual
contents of the message in order to understand if it had been modified, i.e.
understand integrity. By playing, the player should also be able to understand
the concepts of authenticity, confidentiality and non-repudiation, the meaning
with and use of a reliable third part, certificates, electronic signatures,
symmetric- and asymmetric encryption.
4.2.2 Research and preparation
During this step we wanted to know about previous and current attempts to
make ordinary people understand PKI.
Within SEIS there had been an attempt to explain the use of electronic IDcards to the general public using scenario techniques [Bjärbo, Boström et al.
1996]. Scenarios were taken from everyday life and were initially presented in
a comic-like fashion. In a paper by [Bell, Thimberly et al. 1999] a traditional
game sending around boxes containing secret messages in a classroom was
described. The boxes were armament with different padlocks, keys kept by
different participants, this way presenting (parts of) a PKI in a gaming
situation to students.
Another strategy was used by [Whitten 1999] where it was concluded that
regular undergraduate computer science students had difficulties in using a
PKI solution in their regular life situations.
We found the scenario-based comic-like approach most rewarding and this
thesis is partly a spin-off from their work.
In order to determine how the scenarios of the GBI would look we started to
investigate the requirements for such an instruction from the perspective of
users/players/learners.
We conducted an extensive research on the connection between the use of
computer games and target groups [Malekani 1998; Östfeldt 1999] (see the
previous chapter), which resulted in a study of children’s use of computer
51
Chapter 4
games [Östfeldt 1999] and a study of grown-ups’ use of computer games
[Malekani 1998].
We also analysed organisations’ IT security needs and discussed how
IT security awareness may be measured [Björck 1998b; Lidholm 1999].
Furthermore, we discussed the value of IT security education [Yngström and
Björck 1999].
We wanted to know how a security related computer game for adults with no
or little initial knowledge about IT security would look like.
We also investigated existing linear and non-linear instructions (see the
previous chapter) within PKI and related issues.
We discussed a variety of detailed ideas on what we wanted the player to learn
and different ways of doing this. A frequent dilemma was the conceptual and
technical level of the issues we wanted to teach, still preserving the simple and
holistic view. The result was a list of issues to be implemented in the design.
The final outcomes were the requirements discussed and concluded in
chapter three. One thing we found, when investigating computer games and
target groups, was that if there is a use of multi-media technology in the game
it has to be professionally made in looks and feeling. Otherwise, the target
group will not accept it. The easiest way to get around this is to not make a
claim of doing it i.e. not to use multi-media facilities unless professionals can
make them. Today’s players are used to extremely well-designed games and if
the game is not fulfilling their expectancies they will just throw it away,
regarding it as bad and/or childish. Because of this, we emphasised that this
should only be regarded as a game-prototype and not a professional computer
game, and it would mainly be developed to evaluate the effects of the
instruction.
52
The Paradise model
4.2.3 Design phase
We chose to develop the prototype of the game in Macromedia Authorware36,
a high-level multimedia-authoring tool. The choice was based on several
reasons:
We wanted the game to function on both Windows and Macintosh.
We wanted the game to function both as a stand-alone application as
well as on the Internet through a common web-browser.
Because of the project objectives and limited resources, we decided to
use a high-level environment, supporting rapid evolutionary
programming [Sommerville 1996], focusing more on content than on
programming.
According to an analysis of authoring tools available for CBI [Dalgarno 1998],
it seemed that Authorware was suitable to fulfil our requirements.
The Design phase was divided into three interdependent structures: The I/O
structure, the game structure and the program structure. The I/O structure is
the interface between the user and the system. The game structure is the
internal architecture of the game i.e. the relationships between all the
obstacles the player has to overcome. The program structure is the
organisation of the actual code, subroutines and data that the program
consists of.
4.2.3.1 I/O structure
We wanted to achieve a non-linear instruction so we did not use a storyboard
or any traditional linear preparations. Instead we concentrated on the earlier
identified objects; how to communicate with these and how they in turn
should react upon the player’s choices.
The learners’ prerequisites, regarding learning speed and perception, are so
different that we decided to build the game symmetric i.e. the system will wait
until the player has finished his/her turn.
36
Authorware ver.5 is made by Macromedia. web address: http://www.macromedia.com
53
Chapter 4
The interface is completely based on graphical representations on the
computer screen. The user interacts with the system through clicking on
objects choosing a suitable action related to that particular object.
The system responds to the user by changing behaviour and presumptions of
the game.
4.2.3.2 Game structure
At first, we concentrated entirely on a game that gave no actual feedback to
the user. The user had to find out everything for him/herself, investigating
how the use of certain security mechanisms related to/effected
confidentiality, integrity, authenticity and non-repudiation. During a pilot
study (discussed in chapter 5, Research design and methodology) we found
that the learners needed a more clear problem to solve and direct feedback
from the system. Therefore three different game-modes were designed.
Demo – The system will present different ways of sending a message.
Practise – This mode is completely non-linear with little feedback and
mostly a simulation of a system. The idea is that the learner can
interact with the objects on the screen. Create messages, apply certain
security mechanisms and send the message to a recipient. The
recipient can read the message, decrypt and verify it and its sender.
Test – This is basically the same as the Practise mode except that the
user receives a specific problem to solve and feedback of what went
wrong and a suggestion on how to correct it.
4.2.3.3 Program structure
Because of our choice of programming environment, we do not have to pay
so much attention to this stage. Most of the memory handling is delegated to
the Authorware system. However, it is important to keep all resources (e.g.
images), as small as possible when planning on streaming the game via the
Internet.
4.2.3.4 Evaluation and design
We now felt that we had a fairly good idea of what to create in order to use
the game as instruction of IT security awareness. During the research and
54
The Paradise model
preparation step we found a good linear teaching method, a book, about PKI
[Garefelt and Westerlund 1997]. When designing the game we tried to match
it with the book in contents and terminology.
In order to separate the PKI area from reality, we chose to place the game in
a surrealistic cartoon-like environment – the Paradise. The actors are Adam,
Eve and the Snake37 and above them, there is a cloud in which all of the
participants trust. Adam and Eve need to communicate with each other in a
secure way, without the Snake finding out the contents. The intention is to
have the player understand and become aware of how to achieve
confidentiality, integrity, authenticity and non-repudiation within a
computerised publicly available environment, and therefore a network was
introduced into the Paradise.
4.2.4 Programming phase
The programming was made in Macromedia Authorware version 4.
Authorware uses a flowchart metaphor (see Figure 8), with symbols in the
flowchart indicating where the resources (information, interactions, decisions
and media) can be found. Every object is represented as a visual symbol with
its own methods and variables. There is a limited scripting language provided,
but most of the symbols can interact and use lots of variables and functions
with logic allowing for most of the tasks needed to be carried out. By using
Macromedia Director files as resources, it is possible to use a more extensive
scripting language called Lingo if necessary.
37
Instead of Alice, Bob and Eve(sdropping) that is common in security education.
55
Chapter 4
Figure 8. An example of the Authorware Environment
Authorware reassembles existing resources as images, movie clips, sound and
text files. To create these resources we used the following programs:
Digital Movies – Macromedia Director 6.5,
Bitmapped images – Macromedia xRes, Painter, Adobe Photoshop 4,
Vector images – Adobe Illustrator 8,
Image processing – Adobe Photoshop 4,
3D-rendering – RayDreamStudio 5,
Images of People – Poser 2,
Film converting – Adobe Premiere 3.
When handling so many resources a consistent name convention has been
proved to be most important. The first step was to design all the graphical
and text objects that are visible on the screen. Thereafter, we reassembled the
objects and implemented the design we had made, illustrated in Figure 9.
56
The Paradise model
Figure 9. The Paradise Game
4.2.5 Play-testing phase
To be able to evaluate the quality of the game design we needed a number of
critical parameters that could illustrate the game’s different parts. We decided
to use Geof Howland’s model that separates the game design into five design
elements [Howland 1998], graphics, sound, interface, gameplay and story (see
chapter 3, section Model for Computer Game Design).
As mentioned above in section ‘game structure’ we conducted a pilot study
where several changes were made. As a game, it was too boring. As an
instruction it received encouraging positive feedback. Some of the symbols
were too small and there was a need for a vocabulary. We concentrated on
improving the game as instruction instead of making it more fun. We will
discuss the outcome in more detail in the next chapter.
57
Chapter 4
4.2.6 Post-Mortem
The future. This is only a prototype with the sole purpose of making ordinary
users understand how PKI works in practise, and so far it has no economic
ambitions or claims to be an exciting game.
4.2.7 Experiences from the development of the model
To develop a GBI is very time-consuming in contrast to a linear instruction.
Using Crawfords’ model for the game design sequence turned out to be a
good help in the developing process. Because of the non-linear nature of a
well-designed computer game, ordinary models for system development were
incomplete and unsatisfying. With so many different relations that a computer
game consists of, on different conceptual levels, it is important to have a good
documentation with a terminology and language, richly illustrated that is easily
understood by all participants. We could not find a suitable holistic/visual
model for documentation so initially in the development process we created
such a model [Näckros 1999c; Näckros 1999b].
4.3
Chapter conclusion
In this chapter we have presented the experimental teaching model used in
the experiment discussed in the next chapter. We have also discussed the
development process and our experiences from this work. The next chapter
will present and discuss the design of this thesis’ experiments.
58
Chapter 5
RESEARCH DESIGN AND METHODOLOGY
This chapter describes the methodology used in this study by discussing the
methodological considerations, data collection and how the data were
analysed.
5.1
Chapter introduction
The purpose with this study was to investigate if computer games are a
suitable teaching method to stimulate holistic learning of IT security issues.
By comparing a computer game with a conventional linear instruction in two
experiments, data were collected and subjected to quantitative as well as
qualitative analyses. The analyses were also made to catch possible side effects
in the learning experience such as satisfaction, gender and age differences, and
efficiency in terms of time-consumption versus acquired knowledge. This
chapter will describe the methodology used.
5.2
Methodological considerations
This study will mainly use a quantitative approach. The reason is that we are
investigating new forms of instruction (alternative/complement) within
IT security education and not new forms of measurements. Most of today’s
courses still use quantitative measurements of the learners acquired
knowledge in terms of tests, exams etc. We wanted to use a similar method.
So, in this test we have intentionally used two kinds of questions that require
different levels of understanding according to the discussion in chapter 2,
section 2.5.
The collected data were analysed quantitatively to measure differences in
improvements. However, we also analysed the improvements using qualitative
methods as a complement. The reasons for this are:
59
Chapter 5
We are investigating what people understand e.g. how they relate and
use the content of details to a larger context. This can be difficult to
measure in standardised forms.
We want to study if there is a connection between acquired
understanding and the satisfaction the person felt with the learning
experience.
In the computer game there are many graphical objects symbolising
concepts. How are these processed/interpreted?
To conclude, we are also interested in subjective values, which can be difficult
to catch when only using quantitative measurements.
5.3
Experiment design
An experiment is based on assumptions about a certain connection between
variables. Independent variables can be systematically varied and the result on
the dependent variables measured. An experiment therefore requires a
hypothesis about connections that can be tested. To test a hypothesis
statistically it has to be formulated so it can be falsified. It is therefore also
common to design a counter-hypothesis [Halvorsen 1992].
An ideal experimental design uses a ‘pretest – post-test control group design’.
That is, randomly assigning subjects to separate groups (R) of treatment into
an experiment group that will be exposed to an experimental variable (X) –
the effects of which to be measured – and a control group that will not be
exposed to the experimental variable. An experiment also needs
measurements/observations (O) before and after the exposure, which can be
compared and analysed in order to determine the effect of the experimental
variable i.e. pretest and post-test.
The ideal experimental design can be represented as [Campbell and Stanley
1963]:
60
Research design and methodology
Experimental
Control
RO1
RO3
X
O2
O4
time
An experimental design relies upon certain key factors [Cohen, Manion et al.
2000]:
1. Random allocation of participants into matched groups (control
group and experimental group) and ensure the initial measurement is
the same for all groups (i.e. pretest);
2. Identification, isolation and control of key variables;
3. Exclusion of any other variables;
4. Final measurement (post-test);
5. Comparison of groups
6. Stage of generalisation.
A general problem with experimental approaches is that the experiment
situation differs from real life, which makes generalisation outside the
experiment environment difficult [Halvorsen 1992].
5.4
This experiment design
The following is a description of the design of the investigation, conducted to
meet the research problem and the hypothesis from chapter 1. The purpose
with this thesis was to investigate if computer games are a suitable teaching
method to stimulate holistic learning of IT security issues. The hypothesis is
that people with holistic learning preferences acquires IT security awareness,
in terms of understanding, more efficiently when learning through playing
computer games instead of using conventional, linear instruction.
Without the ambitions to generalise our possible findings, we decided to
conduct an experiment on a number of undergraduate students by comparing
61
Chapter 5
two different types of instructions. Is it possible to evaluate the hypothesis?
We believe it is. Because we have;
1. a dependent variable that is measurable –
The experiment can be measured. This is facilitated by defining
‘efficient’ as a combination of meeting the intended goal of having the
learners understand PKI with regards to IT security with the time
spent on learning. This can be measured through pretest – post-test,
2. a suitable independent variable –
There is at least one suitable independent variable – the type of
instruction. This can be varied as the instruction may be performed
through a computer game or a linear text,
3. randomised treatment groups – It is possible to form two randomised
groups where one acts as the experiment group and the other as the
control group.
In addition to the three points above we also want to include evaluations of
the satisfaction the subjects felt with the learning experience and its possible
effects on learning. This will be possible through adding an evaluation
accounting for qualitative values.
The hypothesis also includes a second independent variable – learning
preference. A person has either a holistic or a serialistic learning preference.
Because of this we chose to isolate and identify the impact of learning
preference; therefore we also conducted an identical experiment with subjects
of serialistic learning preferences.
Thus, in the experiments there were four groups of subjects:
A - Holists learning by playing
B - Holists learning by reading
C - Serialists learning by playing
D - Serialists learning by reading
Group A was the primary experimental group and group B was the primary
control group. In the second experiment, group C was the experimental
62
Research design and methodology
group and group D control group, see Figure 10 and Figure 11 using the
symbols adopted earlier. The findings from these experiments would be
compared and analysed.
Figure 10. Experiment design (I)
Experimental
Control
Experimental
Control
ROA
ROB
X
Y1
OE
OF
ROC
ROD
X
Y
OG
OH
time
1
Y is regarded as traditional, linear instruction
Figure 11. Experiment design (II)
Individuals to investigate
The intended target groups were students, company management as well as
ordinary employees within Sweden. Common knowledge in sending emails
was a prerequisite.
This study investigated 78 students at the Department of Computer and
Systems Sciences (DSV) at Stockholm University (SU) and Royal Institute of
Technology (KTH). The students consisted of two groups; the first group
consisted of 48 first-year undergraduates at the department and the second
consisted of 30 third-year undergraduates, beginning their specialisation in
security informatics.
63
Chapter 5
The professionals were 24 selected employers and customers to our industry
partners SEIS and Nexus.
All subjects involved in the study participated voluntarily.
5.5
Instrumentation
The instrumentation includes instruments used in the accomplishment of the
experiments i.e. deciding the participants learning preferences, the forms, the
variables and the test environment. This section will briefly describe these.
5.5.1 Learning preference
Two different tests were used to determine the participants learning
preference, MBTI and SPQ. The main instrument was the MBTI. Ford’s SPQ
was only used as a reference.
Although, SPQ is less complex than MBTI several participants felt
uncomfortable with it, they did not know what to answer on the questions.
SPQ was primary developed by Ford for subjects already familiar with studies
on a postgraduate level that already have an opinion of their most effective
learning environment. The undergraduate students in our study did perhaps
not have that kind of opinion, which led to several contradictions in their
answers.
The outcome of Keirsey’s online MBTI test – Temperament Sorter II
[Keirsey and Bates 1984] – is of qualitative nature with eight variables
grouped in four pairs as presented in Figure 12.
64
Research design and methodology
Figure 12. Output from Keirsey’s Temperament II sorter
The four pairs of variables are:
1. Attentive – Expressive
2. Introspective – Observant
3. Tender – Tough
4. Probing – Scheduled
The participants were divided as follows; serialists are those that show higher
value for the variables ‘Observant’ and ‘Scheduled’ than their corresponding
pair-variables ‘Introspective’ and ‘Probing’. People with such values are called
‘guardians’ (SJ) according Keirsey’s terminology and considered here as
serialists. The three others, ‘idealists’ (NF), ‘rationals’ (NT) and ‘artisans’ (SP)
are considered as holists of varying degree. This division also corresponds
with previous studies [Davison, Bryan et al. 1999].
5.5.2 Parameters of measurement
In order to decide the degree of acquired understanding, the pretest –
post-test consisted of two parts; part 1 with twelve closed multiple-choice
questions designed to match ‘knowledge’ according to Bloom’s taxonomy,
and part 2 with ten open questions designed to match ‘comprehension’
according to Bloom’s taxonomy. Each correct answer in the two parts gives
the subject one point.
65
Chapter 5
When evaluating against the research hypothesis the target of measurements
will be the questions in part 2 – the level of comprehension.
The following parameters were measured. The full questionnaires are attached
in appendix A (in Swedish).
o General questionnaire – age, gender, occupation, previous courses,
interests, familiarity with computer games, previous knowledge of
PKI.
o Knowledge questionnaire (pre- and post-test, 22 questions). The
questionnaire is divided in two parts:
o Part 1 – 12 closed multiple-choice questions (requires low
degree of understanding)
o Part 2 – 10 open questions (requires high degree of
understanding)
o Evaluation questionnaire – 8 closed questions and 9 open questions
about the learning experience such as satisfaction, if they feel they
learnt something, what they think about the instruction in general and
suggestions for improving the design.
o Time spent learning
These data ought to be sufficient in order to make correct quantitative and
qualitative analyses, comparing the two instructions. Both the parameters of
measurement and the way data was collected were tested and evaluated prior
to the actual tests in a pilot study.
5.5.3 The teaching strategies
These are the teaching methods that were compared in the experiment.
The two teaching strategies – a short book and a computer game on PKI –
were evaluated and revised – in order to be comparable in content and
quality.
5.5.3.1 The Book
The linear teaching method – the book [Garefelt and Westerlund 1997] – was
in electronic format shown on the computer screen and included the ability to
66
Research design and methodology
turn pages back and forth. The electronic version consists of 20 pages and is
richly illustrated. The original purpose of the book was to explain PKI crypto
systems and IT security to management in organisations.
In the experiments, this teaching method was regarded as the conventional
linear instruction, and offered to the control groups.
5.5.3.2 The Computer Game
We have designed a prototype of /parts of/ a computer game which we used
as the experimental teaching method in the experiments (presented in
chapter 4).
The prototype was delimited to the area of PKI (Public Key Infrastructure)
and rests on background research [Björck 1998c; Malekani 1998; Lidholm
1999; Näckros 1999a; Östfeldt 1999].
5.5.4 Test environment
The experiments were conducted at the Department of Computer and
Systems Sciences on the Intranet. One reason for this was that the researcher
wanted to be available for questions and to prevent possible errors. A second
reason was that by running on the Intranet at the department we could
monitor the participants in a more reliable way.
At the department, there were at the time approximately 120 PCs running
Windows NT 4.0, situated in 12 different rooms. The department installed
the compulsory Authorware pluggin, required to run Authorware applications
over the network in the local web browsers, on all of these machines and also
made the necessary security configurations.
When all the above-mentioned software was implemented, tested and
possible errors had been accounted for; we regarded the DSV Intranet as a
controlled environment, even though the researcher was not physically
present in the rooms.
Thereafter, the participant could choose his or her own favourite workstation
to run the test from – a surrounding already familiar (all students had at the
67
Chapter 5
time for the experiment worked on the PCs, during previous courses), at a
time of his or her own choice.
5.6
Procedures and collection of data
The data was collected from the participants by software applications on the
Intranet. The routine was as follows.
1. The participant used Keirsey’s online MBTI test (fetched from a
website http://www.keirsey.com/swd.html) and ‘Study Preference
Questionnaire’ developed by Ford [Ford 1985].
2. The participant received information 1) to decide whether to receive a
ticket for cinema or lunch and to notify Kjell Näckros about the
decision, 2) with a request to go to a web page on the site
“http://L247.dsv.su.se/projects/PKIdemo/participate.asp” together
with a private unique username (pseudonym) and password.
3. On the site. The participant was requested to test their computers
with a test application. If that failed, s/he was instructed on how to
reinstall and reconfigure the computer with necessary configurations.
4. On the site. The participant logged on to the site.
5. On the site. The participant was welcomed and received instructions
on how the test was to be carried out. S/he was also notified and
encouraged to contact the research leader (electronically or live) in
case of questions.
6. On the site. The participant filled in a general questionnaire (age,
gender etc.), appendix A.
7. On the site. The participant filled in a knowledge questionnaire
(pretest), appendix A.
8. On the site. Depending on the group belonging the participant was
now expected to learn PKI with either a text material as learning
method or a computer game as learning method. The participant did
not know about the existence of two different teaching methods and
therefore did not know which version to receive.
9. On the site. Whenever the participant felt ready to fill in the post-test
(appendix A, same as in 7), s/he ended the instruction and started to
fill in the second knowledge questionnaire.
68
Research design and methodology
10. On the site. The participant was requested to fill in an evaluation
form.
11. On the site. A “goodbye and thank you for participating” message.
All the data were collected electronically in a Microsoft Access Database on
the Intranet.
5.7
Data analyses
The research hypothesis is that people with holistic learning preferences
acquires awareness of IT security issues, in terms of understanding, more
efficiently when learning through playing computer games instead of using
conventional, linear instruction.
The hypothesis is based on the idea that computer games are an appropriate
contrast to linear instruction i.e. non-linear, and therefore also ought to be
especially suitable for people with holistic learning preferences. Based on this
idea the following questions and working hypotheses were guiding the
analyses.
1. Subjects who receive treatments matched with learning preference
will have higher amount of acquired understanding about IT security
than those who are mismatched.
2. Subjects who receive treatments mismatched with learning preference
will have lower amount of acquired understanding about IT security
than those who are matched.
3. Subjects who receive treatments matched with learning preference
will have higher amount of satisfaction than those who are
mismatched.
4. Subjects who receive treatments mismatched with learning preference
will have lower amount of satisfaction than those who are matched.
5. Subjects who receive treatments with a non-linear teaching strategy
will have higher scores on ‘comprehension’ tests than those who
received treatments with a linear teaching strategy.
69
Chapter 5
6. Subjects who receive treatments with a linear teaching strategy will
have higher scores on ‘knowledge’ tests than those who received
treatments with a non-linear teaching strategy.
7. To what extent does learning preference (serialist/holist), teaching
strategy (linear/non-linear), matching/mismatching learning
preference and teaching strategy, age, gender and satisfaction with the
instruction singly or in combination, affect achievement in acquiring
understanding of IT security.
In order to analyse the collected data according to the working hypotheses we
used both quantitative and qualitative methods.
5.7.1 Quantitative analyses
The quantitative analyses were mainly based on the subject’s acquired
knowledge i.e. recorded differences between post-test and pretest, both in
part 1 and part 2. This knowledge test had the same appearance in the pretest
as in the post-test and consisted of two parts:
Part 1, the first 12 questions were closed multiple-choice questions,
with one or more correct answer, designed to match ‘knowledge’.
Part 2, the last 10 questions were open questions, designed to match
‘comprehension’. The examiner of the test marked against a template.
The terminology was not necessary to receive full score. Context and
order of events were important.
The subject received 1 point for each correct answer i.e. the max score was
12 in part 1 and 10 in part 2. Subject’s score was measured in percentage of
correct answers of max score. The percentage score in the post-test was
subtracted by the percentage score in the pretest. The remaining ‘percentage
unit’ is then regarded as the ‘improvement’38 of the subject. Only
improvements were thus subjected for quantitative analyses. The reason for
measuring improvements by percentage units is an attempt to minimise the
effect that “subjects scoring highest on a pretest are likely to score relatively
lower on a post-test; conversely, those scoring lowest on pretest are likely to
score relatively on a post-test” [Cohen, Manion et al. 2000] p.126.
38
Some subjects show negative ‘improvements’.
70
Research design and methodology
The average improvement for each of the four groups (serialist/holist using
game/text) was compared between the groups. The comparisons were based
on p-values one-way ANOVA (method for analysis of variation for
comparison of means) with a significance level of 1%39. Statistical
calculations were made using SPSS for Windows version 10.
An example:
1) Subject s1 has serialistic learning preferences.
2) When filling in the pretest s1 scored 5 correct answers in part 1 and
2 correct answers in part 2. This implies that s1 scored 5/12≈42% in
part 1 and 2/10≈20% in part 2.
3) After being exposed to the independent variable (the teaching
method – in this example the game and therefore he belongs to
group C), s1 scored in the post-test, 7 correct answers in part 1 and
5 correct answers in part 2. This implies that s1 scored 7/12≈58% in
part 1 and 5/10≈50% in part 2, in the post-test.
4) The improvement of s1 in part 1 is 58-42≈16 percentage units and
58-20≈30 percentage units in part 2. The subjects average
improvements for this group C were 32,69 percentage units in part 1
and 24,62 percentage units in part 2. The corresponding
improvements for group D were 35,56for part 1 and 10,67 for part 2.
5) We now want to compare the means for group C and D in part 1 to
investigate if the difference is statistically significant or not by using
ANOVA at significant level 1% (confidence level 99%).
The result from ANOVA is a p-value of 0,7450. Since the
significance level is smaller than the p-value there is no statistically
significant difference.
39
Recommended reading about ANOVA [Araï 1999]
71
Chapter 5
Subjects for investigation were:
Differences in improvements of acquired knowledge between users
(serialist and/or holists) of linear method and users of the computer
game.
Differences in time used for learning between users of linear method
and users of the computer game. Also, in relation to acquired
knowledge.
Age and gender discrepancies.
The results were collected in a table and also presented in a coordinate
diagram positioning each of the subject’s improvement in part 1 and part 2.
When evaluating against the research hypothesis the main target of
measurements was the improvements in part 2, the level of acquired
‘comprehension’.
5.7.2 Qualitative analyses
The main intention with the qualitative analyses of the collected material was
to investigate user acceptance of the game as an instruction and to find
possible relations between user’s degree of improvement in the learning
experience and user’s characteristics and experiences like:
1. Way of expressing oneself, extent and content
2. Interpretations of symbols
3. Perceived acquired knowledge/understanding
4. Satisfaction with the instruction
Simplified, what are the characteristics of the individuals that acquired
high/low improvement in ‘knowledge’ and/or ‘comprehension’?
The data, upon which the analyses were made, were collected from the
general questionnaire, the result from the quantitative analyses, the post-test
and the evaluation form.
The analyses were made by; 1) systematically analysing the material to map the
different qualities of the above mentioned phenomena. When going through
72
Research design and methodology
the material answer-categories gradually appeared and gave us a possibility to
classify the different phenomena. 2) The researcher divided the individuals in
four categories:
A - Those who acquired low improvements in ‘knowledge’ and high
improvements in ‘comprehension’.
B - Those who acquired high improvements in ‘knowledge’ and
‘comprehension’.
C - Those who acquired low improvements in ‘knowledge’ and
‘comprehension’.
D - Those who acquired high improvements in ‘knowledge’ and low
improvements in ‘comprehension’.
3) The characteristics within each category were compared with the other
categories.
5.8
Evaluation phases and preliminary investigations
Unfortunately, we could not carry out the evaluation in the same way for the
professionals as for the students, so we approached that evaluation in two
ways; 1) evaluating only the acceptance of the computer game as an
instruction, using a qualitative approach in an uncontrolled environment, and
2) evaluating the effect of using a computer game as an instruction in a
comparable study with a linear instruction using both quantitative and
qualitative approaches in a controlled environment.
We conducted two pilot studies to meet the two approaches by dividing the
study into four different phases40 (see Figure 13). Phase 1 was a pilot study
with the intention to improve the game design, the knowledge questionnaires
(pretest and post-test) and the test environment. Phase 1 was primarily a
pre-stage to phase 4, which was our main data acquisition phase. Phase 1 and
4 were performed with students within a controllable environment.
40
The research design was decided after extensive discussions with docent Robert Ramberg at the
department of Computer and Systems Sciences.
73
Chapter 5
Phase 2 was a preliminary stage to both phase 3 and 4, and was carried out
together with our industry partners. The purpose of phase 2 was to prepare
the game for Internet. This implied some redesigning of the game and the test
environment.
In phase 3, the customers of our industry partners were given a possibility to
give feedback on the game through the Internet. It is important to mention
that this phase was not evaluated, in that respect that we neither demanded a
learning preference test nor filling in a questionnaire – and therefore not
offering the text version of the PKI concept – but only asking for feedback of
utilising the game as a general learning method, through the evaluation form.
Phase 4 was our main data acquisition for evaluating the study. The
experiments were carried out on the Intranet of the university, within a
controllable environment.
74
Research design and methodology
Figure 13. Evaluation phases
The preliminary phase 1 was carried out in a laboratory environment with
thirteen students that had been chosen by the project leader with equally
distributed pre-conditions. During this phase, the instructions ran as
stand-alone applications on the computers. All the data were also manually
collected from forms on paper. The outcome of this experiment lead to
extensive improvements of the game and some redesigning of the tests in
order to map the questions according to Bloom’s taxonomy. One important
finding was that the students needed a better feedback on their
correct/incorrect activities.
75
Chapter 5
A second important finding was rather disappointing. It turned out that we
could not teach about all the aspects of security (integrity, confidentiality,
authenticity, non-repudiation) at the same time. The learning was much more
effective if students were given a very specific question and had to answer
each one individually.
A third interesting finding was, in combination with the positive feedback we
received from the holists, a tentative hypothesis; “a person with holistic
learning preferences will increase his/her awareness within IT security related
issues more efficiently when using interactive computer games instead of
traditional learning methods” [Näckros 2000].
The preliminary phase 2 was carried out in an informal manner with students
and employers at DSV with the purpose of managing ‘streaming the game’
over Internet. We also evaluated and improved the navigation hierarchy in the
game.
Phase 3 was conducted together with our industry partners (SEIS –
10 persons, Nexus AB – 14 persons), where the selection of participants was
out of our control. The purpose was therefore to evaluate user acceptance of
the game as a teaching strategy against age, gender and previous knowledge.
Three different user groups took part in this phase; users with no knowledge
in PKI, with some knowledge in PKI and experts in PKI. This took place at
three different occasions.
In phase 4, we investigated two student groups at the Department of
Computer and Systems Sciences (DSV) at Stockholm University (SU) and the
Royal Institute of Technology (KTH). The first group included 48 first-year
undergraduates at the department41. The second group included 30 third-year
undergraduates42, beginning their specialisation in security informatics. All in
all, seventy-eight students participated in the evaluation. But, fifteen of these
tests were incomplete and had to be removed from the study, leaving
sixty-three participants as final research subjects. The method of initial
selection was based on a voluntarily participation.
41
Approximately 60 students were given the possibility of a total of 107.
76
Research design and methodology
According to the results from the learning preference tests, the participants
were divided into serialists and holists. Each of the groups was thereafter
randomly distributed into two additional groups43. After the division we had
four groups of subjects as shown in table 1.
Number of Participants
Game version
Text version
18
17
13
15
Holist
Serialist
n = 63
females=38
males =25
Table 1. Distributions of participants
In the study twenty-four men and thirty-eight women participated, their ages
varying between twenty and forty-four, see Table 2.
Table 2. Age & Gender distribution
42
The total number of students were 37.
43
We controlled the randomised division by comparing the mean scores on the initial pretest and the
covariance between the groups in order to have comparable groups initially (see appendix C).
77
Chapter 5
5.9
Validity and reliability of the experiment
In an experiment, threats against the internal validity need to be considered
[Cohen, Manion et al. 2000]. The considered threats in phase 4 of the
experiment are, using the terminology from Cohen et al.:
History – Between any two observations events other than the
intended treatment from the experiment may occur.
Maturation – Between any two observations participants can change in
a variety of ways.
Statistical regression – “subjects scoring highest on a pretest are likely to
score relatively lower on a post-test; conversely, those scoring lowest
on pretest are likely to score relatively on a post-test” [Cohen, Manion
et al. 2000] p.126.
Testing – Pretests at the beginning of experiments can produce effects
other than those due to the experimental treatments.
Instrumentation – Unreliable tests or instruments can introduce serious
errors into experiments.
Experimental mortality – The loss of participants through dropout, not
completing all the tests
Instrument reactivity – The effects that the instruments of the study exert
on the participants in the study.
Hawthorne effect – Improved results among the subjects because they
know they are getting a treatment. Psychological effect.
Since the overall time for the test occasion – a general questionnaire (age,
gender, previous interests etc.), pretest (knowledge questionnaire, 12+10
questions), teaching method (book or game), post-test (knowledge
questionnaire, 12+10 questions) and an evaluation form – was set to 90
minutes and carried out on a computer at an hour of their choice in an
environment they already felt familiar with, it was judged to include low
78
Research design and methodology
maturation, low testing and instrument reactivity (because the familiarity) and low in
some of the aspects of instrumentation44 (because everything was computerised).
Statistical regression – Instead of measuring the difference between post-test and
pretest in points we are comparing the participant’s percentage score of the
max score in the post-test with the percentage score of the max score in the
pretest. The differences between the tests, we measured in percentage units.
In this way, we are only measuring the improvements.
Selection of participants, we wanted a wide range of previous experiences
among the students, so at the time for the test period, we selected a typical
one of the ongoing classes among the first-year undergraduates and a second
among the third-year undergraduates. We asked them to participate, and they
did. However, there was a high degree of experimental mortality.
Hawthorn effect – This is usually dealt with by using ‘double-blind’ design. In
medical research the control groups usually get placebos. In this experiment,
the subject did not know that we used two different types of instruction. The
effect of this is particularly noticeable in the evaluation forms where the
subjects evaluated the linear instruction in terms of comparing it with reading
a ‘real’ (physical) book.
5.10 Ethical considerations
All subjects involved in the study participated voluntarily.
Before the experiment all participants were informed that the data collected
would be used for research about experimental forms of instruction within
IT security and that data on all the participants would be pseudonymised, and
no identifiable variables would be stored.
Afterwards, the involved participants were informed about the ongoing
research and were given a possibility to share their own results. The
44
This threat was intended to be minimised through two pilot studies, the evaluation of which resulted
in improvement of the quality of the instrumentations used.
79
Chapter 5
participants were also given the possibility to give feedback during and after
the experiment occasion.
5.11 Limitations in the study
Though experiments are accepted in educational research it is always going to
be difficult to find suitable control groups. In this experiment the linear
instruction was regarded as ‘normal’, therefore the groups B and D, in Figure
10, receiving this type of instructional treatment functioned as control groups.
Selecting participants on a voluntary basis may also be regarded as a
limitation.
5.12 Chapter conclusion
In this chapter we have presented the methodology and the design of the two
experiments used in this study.
80
Chapter 6
FINDINGS AND LIMITATIONS
This chapter presents and discusses the findings of the previous chapter. It
also discusses the limitations of this study.
6.1
Chapter introduction
The evaluations were conducted in Sweden, so the collected data are all in the
Swedish language. The cited remarks from the participants to support the
qualitative results are all translations.
6.2
Evaluation of phase 3 – professionals
This section presents the outcome of phase 3, which was conducted by
representatives of our industry partners SEIS and Nexus AB. The ‘acquired
knowledge’ was not evaluated for reasons explained in previous chapter,
instead we evaluated user-acceptance of the game as an instruction.
The outcome was of qualitative nature and the findings were almost
unambiguous; the higher pre-knowledge and age the fewer acceptances.
Almost everyone believed in the teaching methodology but were irritated on
the simple design and graphics of this particular game prototype, “Who is the
target group, children?”
They also requested more action in the game, and some even gave
suggestions like “Why not more action? For example, agent 007 that needs to
send a secret message to M”(male, 39).
At the same time they also requested more simplicity and clarity, and stressed
the importance to receive the required information as fast as possible “...I
usually don’t play computer games and have very high demands on simplicity
and clarity” (male, 40). “One should not need to search for information”.
81
Chapter 6
Time-efficiency was quite important, a common remark was, “Who is going
to pay for the time I spent playing?”
As one participant expressed himself, and this also concludes the results from
the professionals, “I believe very much in this type of instruction that
activates all senses, but it really need to have a very high standard”(male, 39).
6.3
Evaluation of phase 4 – students
This section presents the outcome of phase 4, which was conducted by the
research leader in a controlled environment with students as participants. The
analyses are based on quantitative as well as qualitative data. The first section
presents the quantitative differences between the participating groups of
subjects in acquired knowledge. The second section presents the quality of the
individuals that made significant improvements in the tests and those who did
not. The differences will be presented qualitatively.
6.3.1 Quantitative results
The quantitative results from the two experiments (holistic and serialistic
group) are presented together in appendix C and concluded in Table 3. The
numbers represents the difference as the mean value for each of the
participating group between post-test and pretest as percentage units. In the
table the time spend on learning is also presented for each group.
In Table 3, which only shows the improvements of the participating groups,
we can see that:
1) Holist-playing group improved45 37,78 percentage units in part 2 –
‘comprehension’ questions – and 29,17 percentage units in part 1 –
‘knowledge’ questions.
2) Holist-reading group improved46 15,88 percentage units in part 2 –
‘comprehension’ questions – and 30,88 percentage units in part 1 –
‘knowledge’ questions.
45
Holist-playing group’s improvement test E-A in Figure 10
82
Findings and limitations
This implies that the holist-playing group made 37,78/15,88≈137% better
result in ‘comprehension’ than the holist-reading group.
We can also see that even the serialist-playing group made a better result in
‘comprehension’ than the serialist-reading group, 24,62/10,67≈131%,
although they did not receive as high improvement as the holists.
Improvements
∆ t1 − t 0 (% units), Standard deviation; used time (min:sec)
‘Knowledge’
Part 1
‘Comprehension’
SD
Part 2
SD
Time
22,70
42:54
32,26
13,44
27,14
17,78
0,2266
0,2320
0,1807
0,2257
0,2207
All
Game
Text
Holist
Serialist
31,88
30,65
33,07
30,00
35,19
0,2066
0,2261
0,1887
0,1909
0,2245
Holist game
Holist text
29,17
30,88
0,1836
0,2036
37,78
15,88
0,2264
0,1661
60:31
Serialist game
Serialist text
32,69
35,56
0,2815
0,1739
24,62
10,67
0,2259
0,1981
43:33
59:52
23:42
50:13
33:51
37:40
24:51
Table 3. Results – Relative difference as
percentage units
In the table, the improvements of five more groups are presented for
comparison purposes all subjects, subjects reading, subjects playing, all holists
and all serialists.
The distribution of the subjects’ improvements is shown in Figure 14. Each
subject is put in a coordinate system; the vertical line represents the acquired
‘comprehension’ (improvements in part 2) and the horizontal line the
acquired ‘knowledge’ (improvements in part 1). The axes are divided by the
median of the improvements in part 1 and part 2. The ‘knowledge quadrants’,
divided by the two medians, shows the four categories of the subjects:
A - Those who acquired low improvements in ‘knowledge’ and high
improvements in ‘comprehension’.
46
Holist-reading group’s improvement i.e. test F-B in Figure 10
83
Chapter 6
B - Those who acquired high improvements in ‘knowledge’ and
‘comprehension’.
C - Those who acquired low improvements in ‘knowledge’ and
‘comprehension’.
D - Those who acquired high improvements in ‘knowledge’ and low
improvements in ‘comprehension’.
These will be further discussed in the presentation of the qualitative results.
Table 4 shows how the improvements for each subject group (100%) were
distributed within the knowledge quadrants e.g. 38% of the holist-players are
represented in quadrant A – high improvements in ‘comprehension’ and low
in ‘knowledge’ – while only 5% are represented in quadrant D – high
improvements in ‘knowledge’ and low in ‘comprehension’.
Figure 14. Distributions of Individuals in
knowledge quadrants
In the table we can see that the subject groups, which received the computer
game as instruction are well represented in knowledge quadrant B i.e. high
improvements in ‘knowledge’ and high improvements in ‘comprehension’.
We can also see that no serialist-reader is represented in this quadrant.
84
Findings and limitations
Group distributions in
knowledge quadrants
Serialist/Game
Serialist/Text
Holist/Game
Holist/Text
0,00%
20,00%
38,89%
17,65%
Serialist/Game
Serialist/Text
Holist/Game
Holist/Text
30,77%
0,00%
33,33%
11,76%
Serialist/Game
Serialist/Text
Holist/Game
Holist/Text
61,54%
40,00%
22,22%
41,18%
Serialist/Game
Serialist/Text
Holist/Game
Holist/Text
7,69%
40,00%
5,55%
29,41%
A B
C D
Knowledge
Mean = 31,88
Median = 33,33
Comprehension
Mean = 22,70
Median = 20,00
Table 4. Distribution of Groups’ improvements
When measuring improvements, subjects that scored a high percentage
correct answer in the pretest is not shown correctly because they are likely to
score lower in improvements, which could be one of the reasons that so
many subjects are represented in knowledge quadrant C – low improvements
in ‘comprehension’ and low improvements in ‘knowledge’. In Figure 15 and
Figure 16 the mean of each groups’ acquired result in the pretest and post-test
are therefore presented. The results are shown in percentage correct answers
of max score. In these figures we can see that the groups’ initial
pre-knowledge was approximately the same but depending on the instruction
they showed different amount of acquired ‘knowledge’ and ‘comprehension’
in the post-test.
85
Chapter 6
Figure 15. Results in pretest and post-test –
Learning preference and Instruction
To our initial question – the tentative hypothesis from phase 1 – whether
computer games are a suitable teaching method for people with holistic
learning preferences to understand IT security related issues the answer is yes.
We have already showed that the holists that played the computer game
acquired 137% better ‘comprehension’ than the holists that read the text. The
method for analysis of variation, ANOVA also shows that this difference is
significant at confidence level 99.5% (F(3,59) 5,646, p<.005)47.
47
The statistical analyses are presented in appendix B
86
Findings and limitations
Figure 16. Results in pretest and post-test –
Learning preference or Instruction
Additionally findings, all referring to Table 3 -Table 4 and illustrated in Figure
14 - Figure 16, indicates:
1. There is a noticeable connection between learning preference and the
time used for learning.
a. Holists spent 37.4/24.5≈52% more time on reading, than the
serialists.
b. Holists spent 60.3/43.3≈39% more time on playing, than the
serialists.
c. Holists spent 60.3/37.4≈61% more time on playing than
reading.
d. Serialists spent 43.3/24.5≈77% more time on playing than
reading
Comment:
Apparently the serialists acquired information more easily when
reading. Holists spent in average 50.1/33.5≈50% more time on
learning than the serialists.
It seems that holists need more time to learn, no matter kind of
instruction.
87
Chapter 6
Even when holists spent more time than the serialists on reading they
nevertheless acquired less knowledge.
2. There is a noticeable connection between teaching method and type
of knowledge acquired.
a. When the text was used, increase in ‘knowledge’ was mainly
acquired (part 1 in Table 3), and less of ‘comprehension’,
(part 2, in Table 3).
b. When the game was used, increase in ‘comprehension’ was
mainly acquired, and less of ‘knowledge’.
Comment: The total amount of increased acquired knowledge
(‘knowledge’ ∧ ‘comprehension’) was higher when the computer
game was used, regardless of learning preference.
3. There is also a noticeable connection between learning preference,
teaching method and type of knowledge acquired.
a. Serialists that used the text as teaching method increased
more in acquired ‘knowledge’ than the corresponding holistic
group.
b. Holists that used the game as teaching method increased
more in acquired ‘comprehension’ than the corresponding
serialistic group.
The researcher did not notice any connections between gender, age and the
results.
6.3.2 Qualitative results
This section presents the findings from the qualitative analyses described in
the previous chapter. The quantitative results, in the previous section,
described the connections between the participating groups and the
improvements, in this section we will look at the connections between the
individuals and improvements. These analyses were based on the recorded
improvements presented in Figure 14 in which all individuals have been
categorised in either of four ‘knowledge quadrants’. The qualitative results are
collected in appendix D.
88
Findings and limitations
6.3.2.1 Low knowledge and high comprehension
The individuals in ‘knowledge quadrant A’ are characterised by, during the
experiments they achieved low improvements in ‘knowledge’ and high
improvements in ‘comprehension’.
The recordings from the pretest showed that most of these individuals had
initially low pre-knowledge in PKI.
Average time used for learning within this category
All: 54 min
Game: 56 min
Text: 51 min
The individuals liked the game as instruction and even thought it was much
more fun than conventional learning methods e.g. “Superb way of
understanding a bit more about incomprehensible encryption” (#201hs).
They also gave a lot of feedback on how to improve the game.
Apparently these individuals liked to express themselves, they used long
sentences and many words. Several of the individuals also expressed that the
symbols were very pedagogic and understandable e.g. “The idea is funny. A
good pedagogic trick with using funny messages...this form of pedagogy really
encourages experiential learning” (#188hs).
All individuals but one perceived that they raised their understanding of PKI.
6.3.2.2 High knowledge and high comprehension
The individuals in ‘knowledge quadrant B’ are characterised by, during the
experiments they achieved high improvements in ‘knowledge’ and high
improvements in ‘comprehension’.
The recordings from the pretest showed that most of the individuals had
initially very little pre-knowledge in PKI.
89
Chapter 6
Average time used for learning within this category
All: 62 min
Game: 67 min
Text: 42 min
Several of the individuals mentioned that they thought this was much more
fun and better than conventional instructions. They also gave a lot of
feedback on how to improve the game.
The individuals expressed themselves with long sentences and many words.
They also thought the symbols were very pedagogic and understandable e.g.
“Funny, using Paradise with Adam and Eve. This gives a feeling of familiarity.
One already knows the background” (#103ss). Though several of the
individuals thought the graphics were too simple some comment the
advantage of this e.g. “Funny! Especially the Snake. It’s easier and funnier to
experiment and to test what’s happening, when the graphics are childish and
simple” (#186hs).
All individuals perceived that they raised their understanding of PKI. Several
also expressed they have become interested in the subject and want to learn
more e.g. “I would gladly learn even more about PKI, more details and the
theories behind it. Now when I understand what its all about” (#186hs).
6.3.2.3 Low knowledge and low comprehension
The individuals in ‘knowledge quadrant C’ are characterised by, during the
experiments they achieved low improvements in ‘knowledge’ and low
improvements in ‘comprehension’.
The recordings from the pretest showed that most of the individuals had
initially good pre-knowledge in PKI. This is also one of the reasons they did
not improve their prior knowledge.
90
Findings and limitations
Average time used for learning within this category
All: 31min
Game: 42 min
Text: 24 min
Though the individuals expressed themselves with short sentences and very
few words some really liked the game as instruction e.g. “It was funny. One
learnt a lot, in such a short time” (#111ss). The language was ‘objective’ using
expressions like “probably ok”, “entertaining”, “good in general”, and “there
is a need for this type of instruction”. They often talked about what others
might find useful, not about their own experiences or feelings. Several
individuals would like to see games as a complement to conventional
education e.g. “It’s always good with this type of instruction when the subject
feels so heavy/difficult” (#200ss).
6.3.2.4 High knowledge and low comprehension
The individuals in ‘knowledge quadrant D’ are characterised by, during the
experiment they achieved high improvements in ‘knowledge’ and low
improvements in ‘comprehension’.
The recordings from the pretest showed that most of the individuals had
initially little pre-knowledge in PKI.
Average time used for learning within this category
All: 41 min
Game: 75 min
Text: 38 min
These individuals gave almost no comments at all. They only answered on the
necessary questions and limited their number of words at a minimum.
The individuals perceived they acquired poor or moderately new knowledge.
91
Chapter 6
6.3.2.5 Summary of qualitative results
Most of the individuals believed in using computer games in the learning
process, by itself or as a complement to conventional instructions.
Individuals with low prior knowledge and at the same time explorative in
nature (in this case playing with words in the questionnaires and investigating
symbols on the screen) tend to increase their ‘comprehension’ more when
using computer games as instruction than reading a text.
Though there are exceptions, individuals that felt satisfaction with the
instruction seem to perform better. Although, there are examples of
individuals that did not felt satisfaction with the game at all but still achieved
high improvements.
Unexpectedly, there were several examples showing no relation between
perceived acquired knowledge and the actual measured/recorded
improvement.
6.4
Limitations in the results
A general problem with experimental approaches is that the experiment
situation differs from real life, which makes generalisation outside the
experiment environment difficult [Halvorsen 1992]. Although, the
experimental environment we have used, is already familiar to the students.
The environment is also the same as the students use during their studies.
6.5
Chapter conclusion
The results support the research hypothesis. In the conducted experiment the
holists that used the computer game as instruction made significant
improvements in ‘comprehension’ compared to the holists that read a text.
The results also show that the holists needed more time to learn than the
serialists. In spite of the fact that holists spent more time on reading than the
serialists they still acquired less ‘knowledge’.
The teaching method determined the type of knowledge acquired.
92
Findings and limitations
It seems that an individual with low prior knowledge and at the same time
explorative in nature is the ideal learner for using computer games as
instruction.
The majority of the participants expressed their support for using computer
games in education, although, the game-design elements (graphics, sound,
interface, gameplay and story) needs to be better designed than in our
prototype; this was especially noticeable for the participating professional
group.
93
Chapter 6
94
Chapter 7
CONCLUDING DISCUSSION AND
SUGGESTIONS FOR FUTURE RESEARCH
The conclusion from this thesis’ research is that computer games can be a
suitable holistic teaching method when learning to understand IT security and
therefore also a suitable alternative/complement to conventional linear
instruction.
The findings when answering the working hypothesis that have guided us in
the design of the research indicates that:
1. Subjects who receive treatments matched with learning preference
will have higher amount of acquired understanding about IT security
than those who are mismatched.
2. Subjects who receive treatments mismatched with learning preference
will have lower amount of acquired understanding about IT security
than those who are matched.
The idea was based on that holists are matched in instruction when playing
computer games and serialists are matched when reading.
We have seen that the tentative hypothesis from phase 1 still holds; in our
experiments the learners with holistic learning preferences acquired a
significant higher amount of understanding and did so more efficiently when
they learnt through playing instead of reading.
This was also the only significant result we received regarding matching/
mismatching learning preference with teaching strategy and then only in terms
of the amount of acquired ‘comprehension’.
3. Subjects who receive treatments matched with learning preference
will have higher amount of satisfaction than those who are
mismatched.
95
Chapter 7
4. Subjects who receive treatments mismatched with learning preference
will have lower amount of satisfaction than those who are matched.
The results from the qualitative analyses showed no connection between the
matching/mismatching and satisfaction with the instruction.
5. Subjects who receive treatments with a non-linear teaching strategy
will have higher amount of ‘comprehension’ than those who received
treatments with a linear teaching strategy.
6. Subjects who receive treatments with a linear teaching strategy will
have higher amount of ‘knowledge’ than those who received
treatments with a non-linear teaching strategy.
The idea was based on that linear teaching methods have a predetermined
structure set by the author/teacher and therefore the learners are more
focused to remember than to actually understand i.e. ‘knowledge’ according
to Bloom’s taxonomy.
The answer is yes, as can be seen in Figure 16 all players i.e. holists and
serialists, acquired a better result in ‘comprehension’ than in ‘knowledge’ in
contrast to all readers. All readers acquired better result in ‘knowledge’ than in
‘comprehension’ in contrast to all players.
7. To what extent does learning preference (serialist/holist), teaching
strategy (linear/non-linear), matching/mismatching learning
preference and teaching strategy, age, gender and satisfaction with the
instruction singly or in combination, affect achievement in acquiring
understanding of IT security.
This was more of a working question than a working hypothesis, suitable for
discussions. However, the findings show that there is a difference in how
people acquire knowledge in the most efficient way, depending on the
learning preference i.e. holists seem to be more focused to understand, and
serialists seem to be more focused to remember. Readers with serialistic
learning preferences acquired more of ‘knowledge’ than holistic readers in
spite of the fact that the holists spent in average 52% more time on reading
than the serialists.
96
Concluding discussion and suggestions for future research
Furthermore, there is apparently a connection between teaching method and
type of acquired knowledge. In general, the readers acquired more of
‘knowledge’ and the players acquired more of ‘comprehension’.
Unexpectedly was that even the serialists received a better understanding of
IT security by using computer games.
Quantitatively, there were week relations between type of knowledge
acquired, learning preference, type of instructions and matching/
mismatching instruction with learning preference, see Figure 15 and Figure 16.
Qualitatively, individuals with low prior knowledge and explorative in nature
tend to increase their ‘comprehension’ more when using computer games as
instruction than reading a text.
The evaluation forms from students are throughout positive about the game
prototype and they enjoyed the learning process more than reading a book or
even classroom teaching. Although, some students pointed out that
professional made design and graphics would really improve their learning
capabilities.
The evaluation forms from the industry people showed a different picture.
The game has to astonish the players, with graphics and sound. The more
expensive it looks, the more people will consider spending time with it.
An interesting finding was that even if the subject disliked the game – but
continued anyway – s/he still acquired a higher amount of understanding than
the subjects in the control group.
To be able to generalise the findings additional experiments may be
conducted, with a stable selection of subjects including a larger number of
subjects.
In the experiments we have compared one computer game and one text, one
could evaluate other kinds of computer games and other kinds of linear
teaching methods. In the presented computer game there is a low degree of
97
Chapter 7
excitement and story. If we could improve the ‘gameplay’ (fun to play) how
would this reflect the amount of acquired understanding in IT security?
Additional experiments also require improving the evaluation process of
acquired understanding and the methods to decide learning preference.
Interesting would also be to carry out the experiment as an international
effort in different cultures to map the different needs. The qualitative analyses
indicate that other aspects of learning than serialist/holist dimension may
benefit more from GBI such as an explorative nature of the individual. [Ward
1994] mention for instance that visual teaching strategies can be important for
people with learning disabilities.
We introduced the term knowledge quadrants in Figure 14. It is important in
the development of a curriculum to decide what kind of knowledge according
Bloom’s taxonomy (in which quadrant) we want the learner to acquire.
How does these findings correspond with the teaching methods for
IT security used today?
Are GBI motivated to develop? Because of the complex dynamic
multidimensional nature of modern IT security, it is this researchers opinion
that we need to be open-minded in the search for new educational means;
non-linear teaching methods like GBI are definitely an alternative to
conventional teaching methods when learning IT security. GBI are usually
more expensive to develop but by using the suggested framework for the
design of GBI, which combines design of computer games and design of
instructional teaching methods, the development process can be less complex
and less expensive with a higher amount of quality as a result that can be
evaluated and improved.
People talk about individualised teaching methods but this is not realistic
today, perhaps this can be a step towards that goal.
98
BIBLIOGRAPHY
Araï, D. (1999). (in Swedish) Statistiska metoder för
beteendevetenskap och medicin. Lund, Sweden,
Studentlitteratur.
Bell, T., H. Thimberly, M. Fellows, I. Witten
and N. Koblitz (1999). Explaining cryptossystems
to the general public. WISE1, IFIP TC11.8,
Proceedings of the First World Conference on
Information Security Education, Kista,
Stockholm, Sweden. pp.221-233.
Bjärbo, G., A. Boström and C. Åhnberg
(1996). (In Swedish) Användning av Elektroniska
IDentitetskort, Master Thesis, Stockholm,
Sweden, Stockholm University (SU)/ Royal
Institute of Technology(KTH), Department
of Computer & Systems Sciences(DSV).
Björck, F.(1998a). Information Security Survey Sweden 1998 Ernst & Young, Stockholm
Company Report.
Björck, F.(1998b). Measuring Information Security
Awareness Department of Computer &
Systems Sciences(DSV) / Stockholm
University (SU)/ Royal Institute of
Technology(KTH), Kista, Stockholm.
Björck, F.(1998c). Trends in IT security
Department of Computer & Systems
Sciences(DSV) at Stockholm University (SU)/
Royal Institute of Technology(KTH), Kista,
Stockholm.
Björck, F. and L. Yngström (2001). IFIP World
Computer Congress / SEC 2000 Revisited. Wise2:
proceedings for IFIP TC11 WG11.8 Second
World Conference on Information Security
Education, Pearth, Australia, School of
Computer and Information Science, Edith
Cowan University, Perth, Western Australia.
pp.209-223.
Bloom, B. S., M. Engelhart, E. Frost and D.
W. & Krathwohl (1956). Taxonomy of educational
objectives : the classification of educational goals,
Handbook 1, Cognitive Domain. New York,
Longmans.
Brown, J. S., A. Collins and P. Duguid (1989).
Situated Cognition and the Culture of Learning
Educational Researcher 18(1): pp.32-42.
Bruner, J. S. (1960). The process of education.
Cambridge, Mass., Harvard U.P.
BRÅ(2000). (in Swedish) BRÅ-rapport 2000:2
Brottsförebyggande rådet, Report no.91-3831608-0, Sweden.
Campbell, D. T. and J. Stanley (1963).
Experimental and quasi-experimental designs
for research on teaching. Handbook of Research
on Teaching. N. Gage(ed.). Chicago, Rand
McNally: pp.171-246.
Caroll, J. M. (1990). The Nurnberg Funnel:
Designing Minimalistic Instruction for Practical
Computer Skill. Cambridge, Massachusetts, The
MIT Press.
Cohen, A. M. (1969). The relationships among
student characteristics, changed instructional practices
and student attrition in junior college: final report.
Washington, D.C., Educational Resources
Information Center.
Cohen, F. (1998). Fred Cohen & Associates,
Electronic source, last accessed: 16 Dec 1998,
http://all.net/
Cohen, L., L. Manion and K. Morrison (2000).
Research Methods in Education. London,
RoutledgeFalmer.
99
Bibliography
Conner, M. (2000). How adults learn,
Electronic source, last accessed: 5 May 2001,
http://www.learnactivity.com/ ,search for
adult learning.
Felder, R. (1993). Reaching the Second Tier:
Learning and Teaching Styles in College Science
Education J. College Science Teaching 23(5):
pp.286-290.
Cornett, C. E. (1983). What you should know
about teaching and learning styles. Bloomington,
Ind., Phi Delta Kappa Educational
Foundation.
Felder, R. (1995). Learning and Teaching Styles In
Foreign and Second Language Education Foreign
Language Annals 28(No.1): pp.21-31.
Crawford, C. (1984). The art of computer game
design. Berkeley, Calif., Osborne/McGraw-Hill.
Dalgarno, B. (1998). Tools for Authoring
Constructivist Computer Assisted Learning Resources:
a Review. ASCILITE '98, [online]. Available:
http://cedir.uow.edu.au/ASCILITE98/asc98pdf/dalgarno0153.pdf.
Davison, L., T. Bryan and R. Griffiths (1999).
Reflecting students learning styles Active Learning
10(July 1999): pp.10-13.
Dewey, J. (1938). Experience and Education. New
York, MacMillan.
Felder, R. and L. Silverman (1988). Learning and
Teaching Styles in Engeneering Education
Engineering Education 78: pp.674-681.
Fillery-James, H. (1999a). A Soft Approach To
Management of Information Security, Ph.D. thesis,
Perth, Australia, Curtin University of
Technology, School of Public Health.
Fillery-James, H. (1999b). Teaching Computer
Security. WISE1: Proceedings for the IFIP
TC11 WG11.8 First World Conference on
Information Security Education, Kista,
Sweden. pp.197-208.
Dryden, G. and J. Vos (1994). The Learning
Revolution. California, Jalmar Press.
Fjellman, E. and J. Sjögren(2000). Interaktiv
underhållning inför framtiden
Kommunikationsforskningsberedningen,
Report no. KFB-rapport 2000:10, Stockholm,
Sweden
Dunn, R. S. and K. J. Dunn (1978). Teaching
students through their individual learning styles: a
practical approach. Reston, Va., Reston Pub. Co.
Ford, N. (1985). Learning Styles and Strategies of
Postgraduate Students British Journal of
Educatonal Technology 1(16): pp.65-77.
Ehrman, M. (1989). Ants and Grasshoppers,
badgers and butterflies: Qualitative and
quantitative investigation of adult learning
styles and strategies. Bibliography of Language
Learning Resources. I. L. L. Department, SIL
International.
Frank, A. (2001). (In Swedish) Myter och sanningar
om spelindustrin Rockad no.1 | February 2001:
pp. 3-12.
Ehrman, M. and R. Oxford (1990). Adult
Language Learning Styles and Strategies in an
Intensive Training Setting Modern Language
Journal; v74 n3 (Fall 1990): pp.311-327.
100
Garefelt, J. and A. Westerlund (1997). (in
Swedish) Kort om krypto-Nytt behov för näringsliv och
samhälle. Stockholm, Sweden, Svenska
Arbetsgivareföreningen.
Goforth, D. (1994). Learner Control = Decision
Making + Information: a Model and Meta-analysis
Journal of Educational Computing Research
11(1): pp.1-26.
Bibliography
Halvorsen, K. (1992). Samhällsvetenskaplig metod
(org. "Å forske på samfunnet"). Lund, Sweden, ed.
Studentlitteratur.
Harapnuik, D. (1998). Inquisitivism or "The
HHHMMM??? What Does This Button Do?"
Approach to Learning: The Synthesis of Cognitive
Theories into a Novel Approach to Adult Education.
Proceedings for WebNet-World Conference
'98 of the WWW, Internet, & Intranet.
Heimbach, A. (2001). Herr der Emotionen.
Deutshe Allgemeine Sonntagsblatt, no.10. Die
Zeit. week.42, 2001.
Highland, J. (1992). Perspectives in
Information Technology Security in Aiken.
Information Processing 92. e. E. a. Society, Elsevier
Science Publishers B.V. (North-Holland), IFIP
transactions -13. Volume LL: pp.440-447.
Howland, G. (1998). Game Design: The
Essence of Computer Games, Electronic
source, last accessed: 20 Sep 1999,
http://www.lupinegames.com/articles/essgam
es.htm
Irvine, C. E. (1999). Amplifying Security Education
in the Laboratory. WISE1: Proceedings for the
IFIP TC11 WG11.8 First World Conference
on Information Security Education, Kista,
Sweden. pp 139-146.
Johansson, H. and M. Kager (1995). (In
Swedish) Upplevda hot mot ADB-lagrad informationen studie av svenska ADB-chefers uppfattning,
Master Thesis, Stockholm, Stockholm School
of Econoimics, Information Management
Jonassen, D. H. (1991). Objectivism vs.
Constructivism: Do we need a new philosophical
paradigm? Educatiolnal technology: Research
and Development 39.
Jonassen, D. H. (1997). A Model for Designing
Constructivist Learning Environments. Proceedings
for the International Conference on
Computers in Education, Kuching, Sarawak,
Malaysia: Univesiti Malaysia Sarawak.
Jung, C. G. (1923). Psychological types, Pantheon
Books.
Kearsley, G. (1997). Explorations in Learning
& Instruction: The Theory Into Practise,
Electronic source, last accessed: 4 Aug 2001,
http://tip.psychology.org/
Keirsey, D. and M. Bates (1984). Please
understand me : character & temperament types. Del
Mar, CA, Gnosology Books.
Kindell, G. and P. Hollman (1989). A
comparison of linear and global approaches SIL
handout, Dallas, Texas.
Knowles, M. (1975). Self-Directed Learning.
Chicago, Follet.
Knowles, M. (1984a). The Adult Learner: A
neglected Species. Houston, Texas, Gulf
Publishing.
Knowles, M. (1984b). Andragogy in Action. San
Fransisco, Jossey-Bass.
Kolb, D. (1984). Experimental Learning:
Experience as the source of learning and development.
New Jersey, Prentice-Hall.
Kolb, D., I. M. Rubin and J. M. McIntytre
(1974). Organizational psychology: An experiential
approach. Englewood Cliffs, New Jersey,
Prentice-Hall.
101
Bibliography
Kowalski, S. (1994). IT insecurity : a multidisciplinary inquiry, Ph.D. thesis, Stockholm,
Sweden, Stockholm University & Royal
Institute of Technology, Report series Department of Computer & Systems Sciences
94:004.
Lewin, K. (1951). Field theory in social science.
New York, Harper Collins.
Lidholm, M. (1999). (In Swedish)
Informationssäkerhet i Sverige - en studie av SIG
Secuitys företagsmedlemmar i Stockholm., Master
thesis, Kista, Sweden, Stockholm University
(SU)/ Royal Institute of Technology(KTH),
Department of Computer & Systems Sciences,
Master's series-No-99-07-DSV-SU.
Lönnqvist, P. (2000). (in Swedish) Navigation och
lärande- en studie av kognitiva förmågor,
inlärningsstilar och navigation i hypermedia, Master
thesis, Lund, Sweden, Lunds Universitet,
Psykologiska Institutionen
Malekani, X. (1998). (In Swedish)
Säkerhetsaspekter, Mänsklig Interaktion &
Erfarenheter av Dataspel, Master thesis, Kista,
Sweden, Stockholm University (SU)/ Royal
Institute of Technology(KTH), Department of
Computer & Systems Sciences, Master's series
No-98-31-DSV-SU.
Marton, F. (1970). Structural dynamics of learning,
Doctoral thesis, Göteborg, Sweden, Göteborgs
universitet, Institutionen för Pedagogik.
McCarthy, B. (1981). The 4MAT system: teaching
to learning styles with right left mode techniques. Oak
Brook, Ill., Excel.
McCaulley, M. H. (1979). Psychological types in
engineering: implications for teaching. Engineering
Education 66(7): pp.729-736.
102
McCaulley, M. H., C. F. Godleski, C. F.
Yokomoto, L. Harrisberger and E. D. Sloan
(1983). Application of psychological styles in
engineering Engineering Education 78(5):
pp.394-400.
Myers, I. B. and M. H. McCaulley (1985). A
guide to the Development and Use of the Myers-Briggs
Type Indicator, California: Consulting
Psychologists Press.
Nelson, W. and D. B. Palumbo (1992).
Learning Instruction, and Hypermedia Journal of
Educational Multimedia and Hypermedia 1(3):
pp.281-299.
Näckros, K. (1999a). Approaching the concept of
IT-security for young users. WISE1: Proceedings
for the IFIP TC11 WG11.8 First World
Conference on Information Security
Education, Kista, Sweden. pp.237-246.
Näckros, K.(1999b). (In Swedish) Design
dokumentation över Spelprototypen Seclab vid
institutionen för data- och
systemvetenskap(DSV) SU/ kungliga tekniska
högskolan(KTH), No. P10535-3, Kista,
Sweden.
Näckros, K.(1999c). (in Swedish) Modell och
specifikation av demoversion 1 Seclab vid
institutionen för data- och
systemvetenskap(DSV) / kungliga tekniska
högskolan(KTH), No. P10535-3, Kista,
Sweden.
Näckros, K. (2000). Using Computer Games in IT
Security Education - preliminary results of a study.
Nordsec2000: Proceedings of the fifth Nordic
Workshop on Secure IT systems - encouraging
co-operation, Reykjavik, Iceland. pp.251-258.
Bibliography
Näckros, K. (2001). Game-Based Learning within
IT Security Education. Wise2: Proceedings for
the IFIP TC11 WG11.8 Second World
Conference on Information Security
Education, Pearth, Australia, School of
Computer and Information Science, Edith
Cowan University, Perth, Western Australia.
pp.243-260.
Pask, G.(1973). Educational Methods Using
Information About Individual Styles and Strategies of
Learning , HR 1424/1 (2 vols).
Pask, G. (1976a). Conversation Theory: applications
in education and epistemology. Amsterdam,
Elsevier.
Pask, G. (1976b). Conversational techinques in the
study and practice of education British Journal of
Educational Psychology 46: pp.12-25.
Pask, G. (1976c). Styles and strategies of learning
British Journal of Educational Psychology 46:
pp.128-148.
Pask, G. (1988). Learning Strategies, Teaching
Strategies, and Conceptual or Learning Style.
Learning Strategies and Learning Styles. R. In
Schmeck, Ed, Plenum Press, New York. 46:
pp.83-100.
REPLAY (2001). Seminar about current and future
trends of Computer Games, 27 Apr 2001, arranged
by Riksutställningar, funded by KK-stiftelsen
(the Association for Knowledge and
Competence Development in Sweden).
Smith, G. B., Ed. (1995). More about learning
styles of the gifted. Our Gifted Children.
Melbourne, Hawker Brownlow Education.
Sommerville, I. (1996). Software engineering.
Wokingham, Addison-Wesley.
Sticht, T. G. (1975). Applications of the audread
model to reading evaluation and instruction.
Hillsdayle,NJ, Erlbaum.
Summerville, J. (1999). Role of awareness of
cognitive style in hypermedia International Journal
of Educational Technology 1999 v1(July, n1).
Tholander, J. (2001). Designing for Cognitive
Apprenticeship: A learning Environment for Object Oriented Modeling, Licentiate thesis, Kista,
Sweden, Stockholm University (SU)/ Royal
Institute of Technology(KTH), Department of
Computer & Systems Sciences(DSV), Report
Series: No-01-014-DSV-SU.
Travers, M. W. (1977). Essentials of Learning
(4th ed.). New York, NY, MacMillan.
Pask, G. and B. C. E. Scott (1972). Learning
strategies and individual competence International
Journal of Man-Machine Studies 4: pp.217253.
Piaget, J. (1952). The Origins of Intelligence in
Children, New York: International University
Press.
Piaget, J. and B. Inhelder (1971). The Psychology
of the Child, New York: Basic Books.
Turkle, S. (1990). Style as Substance in
Educational Computing. The information society:
evolving landscapes. J. (ed.) Berleur. New York
Heidelberg, North York, Ontario, Springer:
pp.145-160.
Ward, L. (1994). Specific Developmental Dyslexia
Canadian Dyslexia Association(September,
1994).
Prensky, M. (2001). Digital game-based learning.
New York ; London, McGraw-Hill.
103
Bibliography
White, G. B. and R. E. Sward (1999). Developing
an Undergraduate Lab for Information Warfare and
Computer Security. WISE1: Proceedings for the
IFIP TC11 WG11.8 First World Conference
on Information Security Education, Kista,
Sweden. pp.163-170.
Whitten, A. (1999). Why Johnny Can't Encrypt: A
Usability Evaluation of PGP 5.0. Proceedings of
the 9th USENIX Security Symposium.
Witkin, H. A. and D. R. Goodenough (1981).
Cognitive styles, essence and origins : field dependence
and field independence. New York, International
Universities Press.
Yngström, L. (1996). A systemic-holistic approach
to academic programmes in IT security, Ph.D. thesis,
Stockholm, Sweden, Stockholm University &
Royal Institute of Technology, Report series Department of Computer & Systems Sciences
96:021.
104
Yngström, L.(1999). (In Swedish) Reflektioner
kring utvecklingsmetodik för spel Stockholm
University (SU)/ Royal Institute of
Technology(KTH), Kista, Stockholm.
Yngström, L. and F. Björck (1999). The Value
and Assesment of Information Security Education and
Training. WISE1: Proceedings for the IFIP
TC11 WG11.8 First World Conference on
Information Security Education, Kista,
Sweden. pp.271-292.
Östfeldt, J. (1999). (In Swedish) Barn och
ungdomar i datorspelens värld - en studie om
datorspelserfarenheter och datorspel som stöd för
inlärning, Master thesis, Stockholm, Sverige,
Stockholm University (SU)/ Royal Institute of
Technology(KTH), Department of
Computer & Systems Sciences(DSV), Master's
series: No-99-17-DSV-SU.
APPENDICES
Appendices
Appendix A – Material to participants
Appendix B – Data Analyses with Anova
Appendix C – Quantitative Results, Pretest and Post-test
Appendix D – Qualitative Results
105
APPENDIX A – MATERIAL TO PARTICIPANTS
Instructions to the participants (in Swedish)
General question form (in Swedish)
Knowledge form (in Swedish)
Evaluation form (in Swedish)
7 augusti 2000
DSV, Säkerhetsinformatik
Kjell Näckros, [email protected], tel 08-16 17 65
Projektbeskrivning
“Man kan leda hästen till vattnet men inte tvinga den till att dricka”- okänt ursprung, dock funnen på toaletten
på kafé "haggan i backen" i Sigtuna stad.
Det går inte att tvinga människor att lära sig ny kunskap eller att få dem att förändra sitt beteende om de inte
är mottagliga.
Syftet med projektet är att finna effektiva metoder att få de människor som har svårt att tillgodogöra sig
information genom existerande metoder, att bli medvetna om de säkerhetsrelaterade frågeställningar som
uppkommer då elektroniska kommunikationteknologier används.
Trots att det finns ett stort utbud av böcker och kurser med syfte att lära ut säkerhet används inte dessa
kunskaper när de behövs, vilket har lett till en ökad osäkerhet. Det kan naturligtvis bero på en mängd olika
orsaker men vi har valt att utgå från brist på medvetenhet (till skillnad från kunskaper) inom
säkerhetsrelaterade frågeställningar hos de som använder elektroniska kommunikationsteknologier.
Utgående från vår tes om brist på IT-säkerhetsmedvetenhet bland datoranvändarna kompletterar vi denna med
att människor har olika inlärningspreferenser, de som lär sig från del till helhet (serialister) och de som lär sig
från helhet till del (holister). Dagens utlärningsformer stimulerar serialisterna och därmed förbiser holisterna,
fortsättningsvis har vi inte råd att utlämna denna grupp från det moderna samhället. Syftet med detta projekt
är följdaktligen att finna metoder som kan bidra till ökat säkerhetsmedvetande bland de som har holistiska
inlärningspreferenser.
Hypotesen som vi vill testa är; att användandet av datorspel som inlärningsmetod bidrar effektivare till ökat
säkerhetsmedvetande bland de som har holistiska inlärningspreferenser än traditionella didaktiska metoder.
Test beskrivning
Ni skall ha fått ett kuvert, på detta finns det nummer som ert material kommer att identifieras genom. Testets
utforming kommer att utföras som följer:
1. Allmänna frågor. Fyll i så gott ni kan.
2. Test 1. Frågor om era nuvarande kunskaper inom PKI.
3. Speltest. Sitta vid datorn och testa "PKI-applikation"
4. Test 2. Samma som Test 1 för att observera tillägnad kunskap.
5. Evaluering. Hjälp oss genom tycka till om prototypen och idén.
6. Avslut. Lägg in alla formulär i kuveret och lämna in det till ansvarig.
7. Personlighetstest. Frivillig, men viktig för oss. Lämnas in i efterhand, se nedan.
Vi är intresserade av att evaluera vår hypotes om inlärande av medvetenhet.
För att kunna göra detta vädjar vi till er att i efterhand göra en personlighetstest, Keirsey temperament sorter
II, som finns on-line på Internet med adressen: http://keirsey.com/
Om ni vill medverka ber vi er att utföra denna test på ert modersmål. Försök att svara uppriktigt och gör en
utskrift på det svar ni får från detta frågetest.
Problem med Utskrift - Gör så här!
1. När ni har fått resultatet i webbläsaren, välj ”Markera Allt” i ,”Redigera”-menyn (alt. ”Select All” i
”Edit”-menyn, beroende på version) och Kopiera (Copy)
2. Öppna Microsoft Word (eller liknande) och ”Klistra in” (”Paste”) det kopierade materialet.
3. Tabellen får inte plats så, Välj Arkiv->Utskriftformat->Pappersstorlek->Liggande (Page Setup->
Paper Size->Orientation, välj Landscape). Tryck på OK
4. Gör sedan en utskrift av detta.
Skriv det identifierande nummer som står på ert kuvert och skicka detta till:
papper
elektroniskt, som bifogad fil
DSV SU/KTH
[email protected]
att: Kjell Näckros
Electrum 230
164 40 Kista
Tack för er medverkan.
7 augusti 2000
DSV, Säkerhetsinformatik
Kjell Näckros, [email protected], tel 08-16 17 65
Allmänna frågor
Kön:
Ålder:
Befattning:
Tidigare studerade kurser, behöver ej vara högskolerelaterat (ex ekonomi, design, litteratur):
Intressen (ex fotografi, politik, grottletning):
Brukar du spela dataspel, i så fall hur mycket, hur ofta och vilken typ:
Hur uppfattar du att du behärskar PKI (Public Key Infrastructure):
a) Inte alls.
b) Lite
c) Bra
d) Mycket Bra
7 augusti 2000
DSV, Säkerhetsinformatik
Kjell Näckros, [email protected], tel 08-16 17 65
Test
Kryssdel
Ringa in ett eller flera av de val som förekommer. Lägg inte ner alltför mycket tid på detta.
1. En symmetrisk nyckel kan användas för (ett svar);
a) både kryptering och dekryptering
b) antingen kryptering eller dekryptering
c) bara kryptering
d) bara dekryptering
e) vet inte
2. En asymmetrisk nyckel kan användas för (ett svar);
a) både kryptering och dekryptering
b) antingen kryptering eller dekryptering
c) bara kryptering
d) bara dekryptering
e) vet inte
3. Ett symmetriskt nyckelpar är (ett svar);
a) exakt lika
b) helt olika
c) vet inte
4. Ett asymmetriskt nyckelpar är (ett svar);
a) exakt lika
b) helt olika
c) vet inte
5. Konfidentialitet kan erhållas genom (ett eller flera svar);
a) kryptering med symmetrisk nyckel
b) kryptering med privat nyckel
c) kryptering med publik nyckel
d) certifikat
e) digital signatur
f) PKI (Public Key Infrastructure)
g) certifikatutfärdare (CA-Certification Authority), CP
h) betrodd tredjepart (TTP-Trusted Third Party)
i) vet inte
6. Integritet kan erhållas genom (ett eller flera svar);
a) kryptering med symmetrisk nyckel
b) kryptering med privat nyckel
c) kryptering med publik nyckel
d) certifikat
e) digital signatur
f) PKI (Public Key Infrastructure)
g) certifikatutfärdare (CA-Certification Authority), CP
h) betrodd tredjepart (TTP-Trusted Third Party)
i) vet inte
1 (4)
7 augusti 2000
DSV, Säkerhetsinformatik
Kjell Näckros, [email protected], tel 08-16 17 65
7. Autenticiering kan erhållas genom (ett eller flera svar);
a) kryptering med symmetrisk nyckel
b) kryptering med privat nyckel
c) kryptering med publik nyckel
d) certifikat
e) digital signatur
f) PKI (Public Key Infrastructure)
g) certifikatutfärdare (CA-Certification Authority), CP
h) betrodd tredjepart (TTP-Trusted Third Party)
i) vet inte
8. Oavvislighet kan erhållas genom (ett eller flera svar);
a) kryptering med symmetrisk nyckel
b) kryptering med privat nyckel
c) kryptering med publik nyckel
d) certifikat
e) digital signatur
f) PKI (Public Key Infrastructure)
g) certifikatutfärdare (CA-Certification Authority), CP
h) betrodd tredjepart (TTP-Trusted Third Party)
i) vet inte
9. I ett symmetriskt nyckelsystem (ett svar);
a) får det finnas en öppen nyckelkatalog
b) får det inte finnas en öppen nyckelkatalog
c) vet ej
10. I ett asymmetriskt nyckelsystem (ett svar);
a) får det finnas en öppen nyckelkatalog
b) får det inte finnas en öppen nyckelkatalog
c) vet ej
11. Ett certifikat innehåller bl a (ett svar);
a) nyckelägarens namn, dess publika nyckel, giltighetsdatum
b) nyckelägarens namn, dess privata nyckel, tidsstämpel
c) nyckelägarens namn, dess symmetriska nyckel, giltighetsdatum
d) nyckelutgivarens namn, mottagarens publika nyckel, giltighetsdatum
e) nyckelutgivarens namn, dess privata nyckel, tidsstämpel
f) nyckelutgivarens namn, dess symmetriska nyckel, giltighetsdatum
g) vet ej
12. Ett trovärdigt certifikat skall (ett svar);
a) signeras av användaren
b) signeras av CA/CP/TTP
c) signeras av mottagaren
d) vet ej
2 (4)
7 augusti 2000
DSV, Säkerhetsinformatik
Kjell Näckros, [email protected], tel 08-16 17 65
Fritextdel
Denna del är tänkt för fria tolkningar. Förklara med egna ord (gärna läsligt), inte bilder, vad det är som beskrivs.
13.
Förklara och exemplifiera följande:
Vad är IT-säkerhet?
Varför behövs IT-säkerhet?
14.
Förklara och exemplifiera följande;
TTP (Trusted Third Part) - pålitlig tredjepart
CA (Certification Authority) - Certifikatutfärdare
15.
Förklara och exemplifiera följande;
PKI (Public Key Infrastructure) - publik nyckel infrastruktur
16.
Förklara och exemplifiera följande;
Hur garanteras konfidentialitet (confidentiality), omöjligt att läsa, i ett elektroniskt meddelande?
17.
Förklara och exemplifiera följande;
Hur garanteras riktighet (integrity), omöjligt att förfalska, i ett elektroniskt meddelande?
18.
3 (4)
7 augusti 2000
DSV, Säkerhetsinformatik
Kjell Näckros, [email protected], tel 08-16 17 65
Förklara och exemplifiera följande;
Hur garanteras autenticitet (authenticity), veta att avsändaren är den han/hon utger sig för att vara, i ett
elektroniskt meddelande?
19.
Förklara och exemplifiera följande;
Hur garanteras oavvislighet (non-repudiation), avsändaren skall inte kunna förneka att han/hon skickat, i ett
elektroniskt meddelande?
20.
Adam och Eva har var sitt asymmetriskt nyckelpar, en privat och en publik nyckel.
Hur kan Adam sända information som bara Eva kan läsa och hur kan Eva läsa informationen?
21.
Adam och Eva har var sitt asymmetriskt nyckelpar, en privat och en publik nyckel.
Eva har tagit emot ett meddelande från Adam.
Hur kan Eva vara säker på att Adams meddelande inte har blivit förändrat sedan det skrevs?
22.
Adam och Eva har var sitt asymmetriskt nyckelpar, en privat och en publik nyckel.
Eva har tagit emot ett meddelande från Adam.
Hur kan Eva vara säker på att meddelandet kommer från Adam?
4 (4)
7 augusti 2000
DSV, Säkerhetsinformatik
Kjell Näckros, [email protected], tel 08-16 17 65
Utvärderingsformulär
Här vill vi gärna ta emot kritik och önskemål
rörande innehållet i det som ni varit med om.
Dåligt
O
O
Bra
O
O
Mycket bra
O
Dåligt
O
O
Bra
O
O
Mycket bra
O
Dåligt
O
O
Bra
O
O
Mycket bra
O
Dåligt
O
O
Bra
O
O
Mycket bra
O
Dåligt
O
O
Bra
O
O
Mycket bra
O
Dåligt
O
O
Bra
O
O
Mycket bra
O
Dåligt
O
O
Bra
O
O
Mycket bra
O
Förståelse - generellt
Tycker du att det här är ett bra sätt att lära sig hur
PKI fungerar?
Förståelse - personligt
Tycker du att du har lärt dig någonting hur PKI
fungerar, utöver vad du eventuellt visste tidigare?
Design - Grafik
Vad tycker du om grafiken?
- Grafik är alla de bilder som visas och alla de
effekter som används på dessa. Det vill säga allting
som du kommer att se.
Design - Ljud
Vad tycker du om ljudet?
- Musik, ljudeffekter mm. Det vill säga allting som
du kommer att höra.
Design - Gränssnitt
Vad tycker du om gränssnittet?
- Navigering i applikationen. Menyer, dialogrutor
och allting som du använder dig av för att
interagera och hitta i applikationen.
Design - Spelbarhet
Vad tycker du om spelbarheten?
- 'Rolighetsfaktor', hur engagerande applikationen
är. Tiden som du kan upprätthålla ditt intresse.
Design - Handling
Vad tycker du om handlingen?
- Bakgrundsinformation, nödvändiga kunskaper för
att du skall kunna agera. Innehållet i applikationen
och all den information som du tillägnar dig genom
applikationen.
7 augusti 2000
DSV, Säkerhetsinformatik
Kjell Näckros, [email protected], tel 08-16 17 65
Teknisk Miljö
Hur tycker du att applikationen och den här testen
rent tekniskt har fungerat?
Tala gärna om vilken datormiljö du sitter vid.
Dåligt
O
O
Bra
O
O
Övriga kommentarer
- Hur kan prototypen förbättras?
- Har det varit kul, lärande?
- Alla kommentarer är viktiga.
Mycket bra
O
APPENDIX B – DATA ANALYSES WITH ANOVA
ANOVA Table for Part1*100
Group
Residual
DF
3
59
Sum of Squares
360,663
26103,182
Mean Square
120,221
442,427
Means Table for Part1*100
Effect: Group
hol/spel
hol/text
ser/spel
ser/text
Count
18
17
13
15
Mean
29,167
30,882
32,692
35,556
Std. Dev.
18,358
20,362
28,149
17,385
Std. Err.
4,327
4,939
7,807
4,489
Interaction Bar Plot for Part1*100
Effect: Group
40
35
Cell Mean
30
25
20
15
10
5
0
hol/spel
hol/text
ser/spel
ser/text
Cell
Fisher's PLSD for Part1*100
Effect: Group
Significance Level: 1 %
hol/spel, hol/text
hol/spel, ser/spel
hol/spel, ser/text
hol/text, ser/spel
hol/text, ser/text
ser/spel, ser/text
Mean Diff.
-1,716
-3,526
-6,389
-1,810
-4,673
-2,863
Crit. Diff
18,935
20,378
19,573
20,628
19,833
21,215
P-Value
,8103
,6468
,3885
,8161
,5330
,7207
F-Value
,272
P-Value
,8455
Lambda
,815
Power
,026
ANOVA Table for Part2*100
Group
Residual
DF
3
59
Sum of Squares
7101,984
24739,286
Mean Square
2367,328
419,310
F-Value
5,646
Means Table for Part2*100
Effect: Group
hol/spel
hol/text
ser/spel
ser/text
Count
18
17
13
15
Mean
37,778
15,882
24,615
10,667
Std. Dev.
22,637
16,605
22,589
19,809
Std. Err.
5,336
4,027
6,265
5,115
Interaction Bar Plot for Part2*100
Effect: Group
40
35
Cell Mean
30
25
20
15
10
5
0
hol/spel
hol/text
ser/spel
Cell
ser/text
Fisher's PLSD for Part2*100
Effect: Group
Significance Level: 1 %
hol/spel, hol/text
hol/spel, ser/spel
hol/spel, ser/text
hol/text, ser/spel
hol/text, ser/text
ser/spel, ser/text
Mean Diff.
21,895
13,162
27,111
-8,733
5,216
13,949
Crit. Diff
18,434
19,839
19,055
20,082
19,308
20,654
P-Value
,0025 S
,0826
,0004 S
,2517
,4750
,0774
Fisher's PLSD for Part2*100
Effect: Group
Significance Level: 0,5 %
Mean Diff.
21,895
Crit. Diff
20,197
P-Value
,0025 S
hol/spel, ser/spel
13,162
21,737
,0826
hol/spel, ser/text
hol/text, ser/spel
27,111
-8,733
20,878
22,003
,0004 S
,2517
hol/text, ser/text
ser/spel, ser/text
5,216
13,949
21,156
22,630
,4750
,0774
hol/spel, hol/text
P-Value
,0018
Lambda
16,937
Power
,800
APPENDIX C – QUANTITATIVE RESULTS,
PRETEST AND POST-TEST
Pretest
part 1 (score)
s/g
s/t
0
10
0
0
6
0
3
1
2
0
6
7
9
part 1 - covariance (score)
s/g
s/t
h/g
h/t
s/g
12,85207
s/t
5,201183 6,488889
h/g
-0,94083 -0,95556 9,311728
h/t
-3,21893 2,444444 0,757785 8,110727
part 1 - covariance (percentage of max score)
s/g
s/t
h/g
h/t
s/g
892,5049
s/t
311,6427 450,6173
h/g
-195,902 -258,916 646,6478
h/t
-228,909 122,1708 -31,5072 563,2449
sum=
sd=
mean=
part 2 - covariance (score)
s/g
s/t
h/g
h/t
s/g
5,147929
s/t
-0,05325 4,506667
h/g
-1,62722 -2,09333 8,608025
h/t
-1,73964 1,053333 0,788927 3,148789
part 2 - covariance (percentage of max score)
s/g
s/t
h/g
h/t
s/g
514,7929
s/t
-5,32544 450,6667
h/g
-162,722 -209,333 860,8025
h/t
-173,964 105,3333 78,89273 314,8789
44
50
3,731364 2,636737
3,38
3,33
part 2 (score)
s/g
s/t
0
4
2
0
1
1
2
0
0
0
3
4
8
sum=
sd=
mean=
part 1+ part 2 - covariance(score)
s/g
s/t
h/g
h/t
s/g
9,534024
s/t
3,220414 6,248889
h/g
-0,91864 -1,06222 9,654321
h/t
-1,97337 2,442222 1,378893 6,307958
part 1+part2 - covariance(percentage of max score)
s/g
s/t
h/g
h/t
s/g
723,7837
s/t
206,1226 485,321
h/g
-102,235 -119,525 781,0935
h/t
-178,041 169,6049 90,56373 468,6683
h/t
8
1
0
1
2
1
9
1
1
2
0
4
0
0
1
4
0
3
0
1
0
0
1
0
2
0
0
4
0
0
7
6
1
1
5
25
24
37
29
2,361551 2,197401 3,019003 1,829095
1,92
1,60
2,06
1,71
0
4
2
0
1
1
2
0
0
0
3
4
8
sd
0,4624
0,1838
sd
4,0429
0,1769
sd
0,4974
0,2062
part 2 (percentage of max score)
s/g
s/t
h/g
h/t
0,00
0,00
80,00
10,00
40,00
10,00
0,00
10,00
20,00
0,00
20,00
10,00
0,00
0,00
90,00
10,00
10,00
50,00
10,00
20,00
10,00
50,00
0,00
40,00
20,00
50,00
0,00
0,00
0,00
0,00
10,00
40,00
0,00
0,00
0,00
30,00
0,00
0,00
0,00
10,00
30,00
0,00
0,00
0,00
40,00
0,00
10,00
0,00
80,00
10,00
20,00
0,00
50,00
0,00
40,00
20,00
0,00
0,00
70,00
60,00
10,00
10,00
50,00
250
240
320
290
23,61551 21,97401 30,18326 18,29095
19,23
16,00
18,82
17,06
sd
4,9713
1,5146
sd
0,1865
0,3488
part 1+part2 (percentage of max score)
s/g
s/t
h/g
h/t
0,00
33,33
58,33
58,33
83,33
50,00
0,00
25,00
0,00
0,00
8,33
0,00
0,00
0,00
41,67
0,00
50,00
50,00
16,67
33,33
0,00
16,67
16,67
16,67
25,00
58,33
0,00
25,00
8,33
25,00
16,67
75,00
16,67
25,00
25,00
33,33
0,00
0,00
25,00
33,33
50,00
0,00
25,00
0,00
58,33
16,67
25,00
0,00
75,00
58,33
50,00
0,00
50,00
0,00
66,67
33,33
25,00
16,67
83,33
50,00
58,33
41,67
83,33
0,00
0,00
80,00
10,00
40,00
10,00
0,00
10,00
20,00
0,00
20,00
10,00
0,00
0,00
90,00
10,00
10,00
50,00
10,00
20,00
10,00
50,00
0,00
40,00
20,00
50,00
0,00
0,00
0,00
0,00
10,00
40,00
0,00
0,00
0,00
30,00
0,00
0,00
0,00
10,00
30,00
0,00
0,00
0,00
40,00
0,00
10,00
0,00
80,00
10,00
20,00
0,00
50,00
0,00
40,00
20,00
0,00
0,00
70,00
60,00
10,00
10,00
50,00
617
657
928
765
23,72
21,89
25,79
22,50
27,43602 22,40661 28,3445 21,97431
sd
1,7196
3,3163
h/g
0
1
0
0
5
5
5
0
0
0
0
0
1
5
2
part 1+part2 (score)
s/g
s/t
0
4
10
6
0
0
0
0
6
6
0
2
3
7
1
3
2
3
0
0
6
0
7
2
9
7
6
4
sum=
mean=
sd=
h/t
7
7
0
3
1
0
5
0
2
4
2
2
0
3
2
9
3
4
3
4
3
0
3
0
6
0
0
8
3
2
10
6
7
5
10
67
57
3,13998 2,935583
3,72
3,35
part 1 (percentage of max score)
s/g
s/t
h/g
h/t
0,00
33,33
58,33
58,33
83,33
50,00
0,00
25,00
0,00
0,00
8,33
0,00
0,00
0,00
41,67
0,00
50,00
50,00
16,67
33,33
0,00
16,67
16,67
16,67
25,00
58,33
0,00
25,00
8,33
25,00
16,67
75,00
16,67
25,00
25,00
33,33
0,00
0,00
25,00
33,33
50,00
0,00
25,00
0,00
58,33
16,67
25,00
0,00
75,00
58,33
50,00
0,00
50,00
0,00
66,67
33,33
25,00
16,67
83,33
50,00
58,33
41,67
83,33
367
417
475
475
31,0947 21,97281 23,37444 24,46319
28,21
27,78
27,94
27,94
h/g
4
6
0
0
6
2
7
3
3
0
0
2
7
6
4
0
1
0
0
5
5
5
0
0
0
0
0
1
5
2
69
74
2,65
2,47
3,14887 2,542512
h/g
h/t
7
7
0
3
1
0
5
0
2
4
2
2
0
3
2
9
3
4
3
4
3
0
3
0
6
0
0
8
3
2
10
6
7
5
10
8
1
0
1
2
1
9
1
1
2
0
4
0
0
1
4
0
3
0
1
0
0
1
0
2
0
0
4
0
0
7
6
1
1
5
104
86
2,89
2,53
3,151215 2,549335
Post-test
part 1 - covariance(score)
s/g
s/t
h/g
h/t
s/g
3,905325
s/t
1,118343 2,106667
h/g
-0,21302 -0,45333 4,617284
h/t
-1,95858 -0,94667 1,055363 4,878893
part 1 - covariance(percentage of max score)
s/g
s/t
h/g
h/t
s/g
271,2032
s/t
0,007766 146,2963
h/g
-0,00148 -0,00315 320,6447
h/t
-0,0136 -0,00657 0,007329 338,812
part 1 (score)
s/g
s/t
7
8
8
9
4
4
8
6
7
9
8
7
6
8
9
8
4
8
7
9
6
7
10
6
11
10
7
8
sum=
sd=
mean=
part 2 - covariance(score)
s/g
s/t
h/g
h/t
s/g
5,775148
s/t
-2,0355 4,755556
h/g
-0,17751 0,888889 5,027778
h/t
-1,94675
0,8 -0,08997 4,795848
part 2 - covariance(percentage of max score)
s/g
s/t
h/g
h/t
s/g
577,5148
s/t
-203,55 475,5556
h/g
-17,7515 88,88889 502,7778
h/t
-194,675
80 -8,99654 479,5848
h/t
8
7
7
3
8
6
8
7
4
7
5
7
5
7
6
9
9
9
8
9
4
7
8
7
7
1
6
10
5
6
12
9
10
9
10
95
114
130
120
2,056883 1,502379 2,211083 2,276801
7,31
7,60
7,22
7,06
part 2 (score)
s/g
s/t
3
6
3
6
3
0
3
5
1
7
5
6
9
sum=
sd=
mean=
part 1+part 2 - covariance(score)
s/g
s/t
h/g
h/t
s/g
6,976331
s/t
3,139053 9,515556
h/g
0,704142 1,533333 5,304784
h/t
0,352071 4,366667 1,756055 8,380623
part 1+part 2 - covariance(percentage of max score)
s/g
s/t
h/g
h/t
s/g
497,0455
s/t
357,9861 647,037
h/g
-162,028 -154,167 412,5686
h/t
1,929012 231,8673 -15,9722 576,6724
h/t
8
3
2
1
9
1
9
5
6
6
6
6
4
4
2
7
4
4
4
5
3
1
7
1
7
1
7
3
4
0
8
6
7
2
8
57
40
105
56
2,501282 2,257263 2,307277 2,257341
4,38
2,67
5,83
3,29
sum=
sd=
mean=
sd
2,8644
2,1450
sd
0,1161
1,3877
part 2 (percentage of max score)
s/g
s/t
h/g
h/t
30,00
60,00
80,00
30,00
60,00
40,00
20,00
10,00
30,00
10,00
90,00
10,00
60,00
0,00
90,00
50,00
30,00
40,00
60,00
60,00
0,00
60,00
60,00
60,00
30,00
50,00
40,00
40,00
50,00
10,00
20,00
70,00
10,00
10,00
40,00
40,00
70,00
10,00
40,00
50,00
50,00
0,00
30,00
10,00
60,00
40,00
70,00
10,00
90,00
20,00
70,00
10,00
50,00
70,00
30,00
0,00
40,00
0,00
80,00
60,00
70,00
20,00
80,00
570
400
970
560
25,01282 22,57263 23,12053 22,57341
43,85
26,67
57,06
32,94
sd
1,1578
13,3334
sd
0,1460
1,1950
part 1+part 2 (percentage of max score)
s/g
s/t
h/g
h/t
58,33
66,67
66,67
58,33
66,67
75,00
58,33
25,00
33,33
33,33
66,67
50,00
66,67
50,00
66,67
58,33
58,33
75,00
33,33
58,33
66,67
58,33
41,67
58,33
50,00
66,67
41,67
58,33
75,00
66,67
50,00
75,00
33,33
66,67
75,00
75,00
58,33
75,00
66,67
75,00
50,00
58,33
33,33
58,33
83,33
50,00
66,67
58,33
91,67
83,33
58,33
8,33
58,33
50,00
83,33
66,67
41,67
50,00
100,00
75,00
83,33
75,00
83,33
30,00
60,00
80,00
30,00
60,00
40,00
20,00
10,00
30,00
10,00
90,00
10,00
60,00
0,00
90,00
50,00
30,00
40,00
60,00
60,00
0,00
60,00
60,00
60,00
30,00
50,00
40,00
40,00
50,00
10,00
20,00
70,00
10,00
10,00
40,00
40,00
70,00
10,00
40,00
50,00
50,00
0,00
30,00
10,00
60,00
40,00
70,00
10,00
90,00
20,00
70,00
10,00
50,00
70,00
30,00
0,00
40,00
0,00
80,00
60,00
70,00
20,00
80,00
570
350
710
450
25,01282 22,13015 24,70337 22,58886
43,85
26,92
54,62
34,62
sd
1,4601
11,9499
h/g
6
4
1
0
4
6
5
1
1
1
0
4
2
5
0
part 1+part 2(score)
s/g
s/t
7
8
8
9
4
4
8
6
7
9
8
7
6
8
9
8
4
8
7
9
6
7
10
6
11
10
7
8
3
6
3
6
3
0
3
5
1
7
5
6
9
sd
0,3519
0,2267
part 1 (percentage of max score)
s/g
s/t
h/g
h/t
58,33
66,67
66,67
58,33
66,67
75,00
58,33
25,00
33,33
33,33
66,67
50,00
66,67
50,00
66,67
58,33
58,33
75,00
33,33
58,33
66,67
58,33
41,67
58,33
50,00
66,67
41,67
58,33
75,00
66,67
50,00
75,00
33,33
66,67
75,00
75,00
58,33
75,00
66,67
75,00
50,00
58,33
33,33
58,33
83,33
50,00
66,67
58,33
91,67
83,33
58,33
8,33
58,33
50,00
83,33
66,67
41,67
50,00
100,00
75,00
83,33
75,00
83,33
792
950
1000
1000
17,14069 12,51983 18,03512 18,97334
60,90
63,33
58,82
58,82
h/g
6
4
1
0
4
6
5
1
1
1
0
4
2
5
0
57
35
2,501282 2,213015
4,38
2,69
h/g
h/t
8
7
7
3
8
6
8
7
4
7
5
7
5
7
6
9
9
9
8
9
4
7
8
7
7
1
6
10
5
6
12
9
10
9
10
8
3
2
1
9
1
9
5
6
6
6
6
4
4
2
7
4
4
4
5
3
1
7
1
7
1
7
3
4
0
8
6
7
2
8
71
45
2,470337 2,258886
5,46
3,46
APPENDIX D – QUALITATIVE RESULTS
Subjects, category A
low improvements in 'knowledge'
high improvements in 'comprehension'
Key:
VP
Very Poor
P
Poor
Lerning Preference
Ser Hol Hol Hol Ser Hol Hol
Type of Instruction
T
G
T
T
T
G
T
Age
20 21 42 23 25 31 34
Gender
F
M M
F
F
F
M
Perceived Knowledge
VP VP P VP P VP VP
Measured Knowledge
Part 1, %
33 16 33 25 50 16 75
Part 2, %
0 10 20 0 10 0 40
Experience of Games
VG VG P VP VP VP VP
Is this a good way of teaching PKI?
G VG G G
~
~
~
Have you understood PKI?
E G VG VG ~
~
~
What do you think about the design?
~
G
P VP ~
~
~
What do you think about the inteface/navigation?
~
P G
P
~
~
~
What do you think about the 'gameplay'/fun?
~
P VP P
~
~
~
What do you think about the 'story'/content?
~ VG VP P
~
~
~
Average time (min): 54
45 29 71 38 ~
~
~
Average time, Game (min): 56
Average time, Text (min): 51
Categories of Utterances:
Better than conventional teaching.
1
A good complement to traditional teaching.
More fun than conventional teaching.
I felt stressed. Need more time.
1
Easier navigation.
1
Good/illustrative symbols.
1
Want to learn more.
1
Difficult to get started.
Need better feedback.
Need a clearer goal.
Better graphics.
Difficult to understand 'big picture' of PKI
Easy to understand 'big picture' of PKI
Stimulates experiental learning.
Booring. Difficult to maintain interest
I really learned something.
1
More teaching like this.
Practice makes understanding
Demands interest in forehand
1
Learnt a lot in short time
Learnt nothing
Better for children
Demands knowledge in forehand
Ser=Serialist
T=Text
hs=Holist/Game
ss=Serialist/Game
Hol=Holist
G=Game
ht=Holist/Text
st=Serialist/Text
G
VG
E
Good
Very Good
Excellent
~ no answer
Hol Hol Ser Hol Hol Hol
G G
T
G G G
22 32 27 48 25 31
F
F
M M
F
F
P VP P VP P
P
25
0
VG
VG
G
P
P
P
G
53
50 16 25 58
20 0
0 10
VP VG VP VP
VG G
P
~
G VG P
~
VG G G
~
P
P
P
~
VG G
P
~
G G
P
~
52 53 76 72
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Typical Utterances (Game):
1.1) "The idea is funny. A good pedagogic trick with using funny messages." (#188hs)
1.2) "This form of pedagody encourages experiental learning" (#188hs)
1.3) "Superb way of understanding a bit more about incomprehensible encryption." (#201hs)
Typical Utterances (Text):
2.1) "The text is very good, many facts with a focus on what is interesting." (#120ht)
1
83
50
P
~
~
~
~
~
~
54
Subjects, category B
high improvements in 'knowledge'
high improvements in 'comprehension'
Key:
VP
Very Poor
Lerning Preference
Ser Hol Hol
Type of Instruction
G
T
G
Age
20 21 31
Gender
M
F
M
Perceived Knowledge
VP VP G
Measured Knowledge
Part 1, %
0
0
8
Part 2, %
0 10 20
Experience of Games
VG VP VG
Is this a good way of teaching PKI?
P VG G
Have you understood PKI?
G VG G
What do you think about the design?
P G G
What do you think about the inteface/navigation?
G
~
G
What do you think about the 'gameplay'/fun?
G
~
E
What do you think about the 'story'/content?
G
~
E
Average time (min): 62
44 59 93
Average time, Game (min): 67
Average time, Text (min): 42
Categories of Utterances:
Better than conventional teaching.
A good complement to traditional teaching.
1
More fun than conventional teaching.
1
1
I felt stressed. Need more time.
Easier navigation.
Good/illustrative symbols.
1
Want to learn more.
1
Difficult to get started.
Need better feedback.
1
Need a clearer goal.
Better graphics.
1
Difficult to understand 'big picture' of PKI
Easy to understand 'big picture' of PKI
Stimulates experiental learning.
Booring. Difficult to maintain interest
I really learned something.
1
More teaching like this.
Practice makes understanding
Demands interest in forehand
Learnt a lot in short time
Learnt nothing
Better for children
Demands knowledge in forehand
P
Poor
Ser=Serialist
T=Text
hs=Holist/Game
ss=Serialist/Game
Hol=Holist
G=Game
ht=Holist/Text
st=Serialist/Text
G
VG
E
Good
Very Good
Excellent
~ no answer
Ser Hol Hol Ser Hol Hol Ser Hol Hol
G G G G G
T
G G G
40 42 25 25 23 23 38 26 26
M
F
M
F
F
F
F
F
F
VP VP VP VP VP VP VP VP G
0
0
VP
G
G
G
G
E
G
87
0 25 8 25 33 0 25 0
0
0
0
0 10 0 10 0
G
P VP P G VG VP G
E VP E
~
G VG G
E
E G VG ~ VG VG VG E
E G G
~
G
E VG VG
E
P
P
~ VG VG G G
E VP ~
~
P G
P VG
E VP ~
~ VG VG G VG
70 46 32 ~
25 82 82 69
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Typical Utterances (Game):
1.1) "Funny, using Paradise with Adam and Eve. This gives a feeling of familiarity. One already knows the background" (#103ss)
1.2) "A funny way to teach" (#156hs)
1.3) "Funny! Especially the Snake. It's easier and funnier to experiment and to test what's happening, when the graphics are childish and simple" (#186hs)
1.4) "Very pedagogically! Almost too obvious, but rather funny and that is not too common." (#186hs)
1.5) "I would gladly learn even more about PKI, more details and the theories behind it. Now when I understand what its all about." (#186hs)
Typical Utterances (Text):
Subjects, category C
low improvements in 'knowledge'
low improvements in 'comprehension'
Lerning Preference
Type of Instruction
Age
Gender
Perceived Knowledge
Measured Knowledge
Part 1, %
Part 2, %
Experience of Games
Is this a good way of teaching PKI?
Have you understood PKI?
What do you think about the design?
What do you think about the inteface/navigation?
What do you think about the 'gameplay'/fun?
What do you think about the 'story'/content?
Average time (min):
31
Average time, Game (min):
42
Average time, Text (min):
24
Categories of Utterances:
Better than conventional teaching.
A good complement to traditional teaching.
More fun than conventional teaching.
I felt stressed. Need more time.
Easier navigation.
Good/illustrative symbols.
Want to learn more.
Difficult to get started.
Need better feedback.
Need a clearer goal.
Better graphics.
Difficult to understand 'big picture' of PKI
Easy to understand 'big picture' of PKI
Stimulates experiental learning.
Booring. Difficult to maintain interest
I really learned something.
More teaching like this.
Practice makes understanding
Demands interest in forehand
Learnt a lot in short time
Learnt nothing
Better for children
Demands knowledge in forehand
Key:
VP
Very Poor
Ser=Serialist
T=Text
hs=Holist/Game
ss=Serialist/Game
Hol=Holist
G=Game
ht=Holist/Text
st=Serialist/Text
G
VG
E
Good
Very Good
Excellent
~ no answer
P
Poor
Hol Hol Hol Ser Ser Hol Ser Ser Ser Ser Hol Ser Ser Hol Ser Hol Hol Ser Ser Hol Ser Hol Hol Ser Ser
G
T
T
G G G G
T
T
G G
T
G
T
G
T
T
T
T
G G
T
T
T
G
26 41 32 32 21 36 40 21 ~ 44 25 32 21 21 35 27 37 22 25 31 35 22 27 25 26
F
F
M M M M
F
F
M
F
F
F
F
F
F
M
F
F
M M
F
M M M
F
P VP VP P VP P VP VP ~
G VP G VP VP P
P VP P
P G
P
P VP VP P
58 58 25 83 0 41 50 0 50 25 16 58 16 0 50 66 16 58 50 83 58 50 41 33 75
80 10 10 40 20 90 10 0 50 20 10 50 0
0 30 40 0 10 50 70 40 60 10 20 80
G VP G G VP VG VG VG ~ VP G VP VP VP VP VG P G VP G VG VG G
P VP
VG G G G
E G
~
~
P
~
G
P
~
G
~
~
~
~
~
~
~
~
~
~
G
VG G G G
E VG ~
~
P
~
P
P
~
G
~
~
~
~
~
~
~
~
~
~
~
G
P G G
E VG ~
~ VP ~ VP P
~
P
~
~
~
~
~
~
~
~
~
~
~
VG ~
G G VG G
~
~
P
~
G
~
~ VG ~
~
~
~
~
~
~
~
~
~
~
P
~
G G G G
~
~
P
~ VP ~
~
P
~
~
~
~
~
~
~
~
~
~
~
VG ~ VG G G G
~
~
P
~
P
~
~ VG ~
~
~
~
~
~
~
~
~
~
~
25 40 22 60 34 71 ~
~
44 ~
17 31 ~
36 38 26 18 14 13 43 35 17 18 13 56
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Typical Utterances (Game):
1.1) "It was funny. One learnt a lot, in such a short time." (#111ss)
1.2) "The language and the examples with Adam and Eve, were very good." (#111ss)
1.3) "...rather entertaining..." (#115hs)
1.4) "It's always good with this type of instruction when the subject feels so heavy/difficult." (#200ss)
Typical Utterances (Text):
2.1) "Not the most funny thing I've ever done...my lack of intrest in security, needs a funnier instruction to learn." (#185ht)
2.2) "The text is well written and easy to understand." (#196ht, #197st)
1
Subjects, category D
high improvements in 'knowledge'
low improvements in 'comprehension'
Key:
VP
Very Poor
Lerning Preference
Hol Hol
Type of Instruction
G
T
Age
35 41
Gender
F
M
Perceived Knowledge
VP VP
Measured Knowledge
Part 1, %
0
0
Part 2, %
0 10
Experience of Games
G VP
Is this a good way of teaching PKI?
P VG
Have you understood PKI?
P
P
What do you think about the design?
P G
What do you think about the inteface/navigation?
P
~
What do you think about the 'gameplay'/fun?
P
~
What do you think about the 'story'/content?
P
~
Average time (min):
41
75 28
Average time, Game (min):
75
Average time, Text (min):
38
Categories of Utterances:
Better than conventional teaching.
A good complement to traditional teaching.
More fun than conventional teaching.
I felt stressed. Need more time.
1
Easier navigation.
Good/illustrative symbols.
Want to learn more.
Difficult to get started.
Need better feedback.
Need a clearer goal.
Better graphics.
Difficult to understand 'big picture' of PKI
Easy to understand 'big picture' of PKI
Stimulates experiental learning.
Booring. Difficult to maintain interest
I really learned something.
More teaching like this.
Practice makes understanding
Demands interest in forehand
Learnt a lot in short time
Learnt nothing
Better for children
Demands knowledge in forehand
P
Poor
Ser=Serialist
T=Text
hs=Holist/Game
ss=Serialist/Game
Hol=Holist
G=Game
ht=Holist/Text
st=Serialist/Text
G
VG
E
Good
Very Good
Excellent
~ no answer
Hol Ser Ser Hol Ser Ser Hol Ser Ser Hol Ser
T
G
T
T
T
T
T
T
T
T
T
42 38 26 34 33 ~ 25 20 28 20 34
F
F
M M M
F
M
F
F
F
F
VP VP VP VP VP VP VP VP VP VP VP
16 0
0 33 16 25 0 25 0
0
0
40 10 0 30 50 0
0
0
0
0
0
G VP VG VG VP VP VP P VP VP VG
P
~
G G G G
~
P G
P
~
P
~ VG G G
P
~ VG VG P
~
~
~ VP P
~
G
~
G VG P
~
~
~ VG P G G
~ VG G
E
~
~
~
P
P
P G
~
G G
P
~
~
~
~
G
~
P
~
G G
P
~
41 ~
13 152 31 ~
29 37 31 13 11
1
1
1
1
1
1
Typical Utterances (Game):
Typical Utterances (Text):
2.1) "The presentation is similar to a physical book" (#110ht)
2.2) "I would not like to read this once again, the subject is too booring..." (#177st)
1
1
1