CITY UNIVERSITY OF HONG KONG
香港城市大學
Multi-task Performance in Processing Fourchoice Spatial Stimulus-Response (S-R)
Mappings: Implications for Multimodal Humanmachine Interface Design
刺激與反應物的四種不同空間相容關係
對多重任務處理的影響：
多模式人機界面設計啟示
Submitted to
Department of Systems Engineering
and Engineering Management
系統工程及工程管理學系
in Partial Fulfillment of the Requirements
for the Degree of Doctor of Philosophy
哲學博士學位
by
Tsang Ngai Hung
曾毅雄
July 2014
二零一四年 七月
Abstract
The spatial stimulus-response compatibility (SRC) effect refers to the robust finding
that human performance is better for some spatial arrangements of controls and displays
than for others. Usually the effect is most marked when components of the response
panel physically correspond in some obvious way with those of the stimulus panel.
Spatial stimulus-response compatibility (SRC) effects have far-reaching implications for
optimizing human-machine interface design. Yet, previous studies on display and control
compatibility and response performance have been mostly limited to a single-task
paradigm. With the increased complexity of human-machine interfaces, there is an
increase in the number and variety of stimulus modalities and control devices to be
handled concurrently by operators in a control room. The study for resources allocation
and capacity limitations in multimodal information processing in the context of spatial
compatibility has therefore become important for enhanced human performance and
overall system performance.
The first experiment examined the effect of spatial compatibility on dual-task
performance for various display-control configurations using a tracking task and a
discrete four-choice response task. Different levels of compatibility between the stimuli
and responses of the discrete response task were found to lead to different degrees of
influence on the tracking task, such that the more incompatible the stimulus-response
mapping, the more severe the interference with the tracking task. However, degradation
of performance was observed for both tasks. This was probably due to resource
competition for the visual and spatial resources required for simultaneous task operation
within the visual modality and as required for bimanual responses. The dual tasks were
not closely spaced and both required focal vision for processing, involving scanning
i
back and forth between the two visual tasks. This may explain part of the delay found
for dual-task processing. No right-left prevalence effect for the spatial compatibility task
was observed in this study, implying that the use of unimanual two finger responses may
not provide the right conditions for a significant effect in the horizontal right-left
dimension, as may be found when both hands are used for responses.
The second experiment tried to explore the feasibility of superimposing the two
independent visual tasks in Experiment 1 on a single display. In this, the task stimuli
were placed in close proximity so that focal and ambient vision could be utilized
concurrently to minimize resource competition due to excessive demand on the same
visual channel (focal) for visual processing. The results showed that although
performance with respect to both the tracking and spatial response tasks was impaired,
the magnitudes of impairment were not as great as expected (compared with Experiment
1). This implies that focal and ambient vision required for the tracking task and spatial
task, respectively, might be deployed, at least partly, from separate resources.
Participants here seemed to successfully use focal vision for tracking and ambient vision
for identifying signal lights concurrently, reducing the expected keen competition for
visual resources.
The third experiment was to investigate the interaction between and the
performance of a tracking task and an auditory spatial compatibility task with concurrent
processing of visual and auditory inputs. As there are not always clear cut predictions
concerning the performance of cross-modality (auditory-visual) versus intra-modality
(visual-visual) configurations, it was found here that the cross-modality configuration
was superior to the intra-modality configuration only when visual scanning was
ii
necessary between the intra-modal dual tasks (Experiment 1). When the dual tasks were
spaced close enough such that focal and ambient vision were utilized simultaneously for
processing (Experiment 2), the intra-modality configuration resulted in slightly better
dual-task performance than the cross-modality configuration.
The last experiment used a multi-task paradigm involving dual hand and foot
tracking and a discrete choice response task to study the effect of spatial compatibility
for various display-control configurations on human performance. Delay in multi-task
processing was observed when more than one task demanded for the same pool of
resource for processing. It was found that cross-modal time-sharing is superior to intramodal time-sharing under most dual-task/multi-task circumstances because of the use of
different perceptual channels for task processing. However, such cross-modal benefit has
seldom been studied within a task, that is, more than one modality is used for stimulus
presentation within a task. Here, it was found that compared with the visual-visual signal
presentation (intra-modality), the auditory-visual signal presentation (cross-modality)
resulted in a significantly higher hand tracking error, response time, and response error.
This implies that it is very likely that a mixed-modality for stimulus presentation within
a task may impair multi-task time-sharing. This is probably due to response conflict and
the modality shifting between visual and auditory modalities across trials.
The deliverables of this work will help industrial designers and ergonomists in
developing effective and intuitive multimodal interfaces so as to improve multi-task
performance in control rooms. They are helpful for improving efficiency and overall
system performance in human-machine systems – in particular in emergency situations.
Keywords: Human Factors, Ergonomics, Spatial Compatibility, Control/Display
Relationship, Intra- and Cross-modalities, Multiple Resources
iii
TABLE OF CONTENTS
Abstract
i
Qualifying Panel and Examination Panel
iv
Acknowledgements
v
Table of Contents
vi
List of Tables
xii
List of Figures
xiv
Chapter 1 Introduction ..................................................................................................... 1
1.1 Background .............................................................................................................. 1
1.2 Aim ........................................................................................................................... 7
1.3 Objectives ................................................................................................................. 7
Chapter 2 Literature Review ........................................................................................... 8
2.1 Multimodal Interface Design ................................................................................... 8
2.1.2 Visual Modality ................................................................................................. 9
2.1.3 Auditory Modality ........................................................................................... 10
2.1.4 Tactile Modality .............................................................................................. 10
2.1.5 Auditory versus Tactile Signals ....................................................................... 11
2.2 Spatial Stimulus-Response (S-R) Compatibility.................................................... 12
2.3 Multitasking Theories ............................................................................................ 13
2.3.1 Resource Capacity........................................................................................... 13
2.3.2 Bottleneck Theories ........................................................................................ 14
2.3.2.1 Psychological refractory period (PRP)..................................................... 14
2.3.3 Multiple Resource Theory............................................................................... 16
2.3.3.1 Stages ....................................................................................................... 17
2.3.3.2 Perceptual Modalities ............................................................................... 18
2.3.3.3 Visual Channels ........................................................................................ 18
2.3.3.4 Processing Codes ..................................................................................... 19
2.3.4 Threaded Cognition ......................................................................................... 20
2.4 Tracking ................................................................................................................. 23
2.4.1 Pursuit Displays versus Compensatory Displays ............................................ 25
2.4.2 Control Order of Tracking Systems ................................................................ 28
2.4.2.1 Position Control ....................................................................................... 28
2.4.2.2 Rate Control ............................................................................................. 28
vi
2.4.2.3 Acceleration Control ................................................................................ 28
2.4.2.4 Higher-order Control................................................................................ 28
2.4.3 Time Lags in Tracking .................................................................................... 29
2.4.4 Measurement of Tracking Accuracy ............................................................... 29
Chapter 3 Experiment 1 – Tracking and Discrete Dual-Task Performance
Assessment of Different Spatial Stimulus-Response Mappings .................................. 31
3.1 Introduction ............................................................................................................ 31
3.1.1 Aim .................................................................................................................. 33
3.1.2 Objectives ........................................................................................................ 33
3.1.3 Hypotheses ...................................................................................................... 34
3.2 Methods .................................................................................................................. 34
3.2.1 Participants ...................................................................................................... 34
3.2.2 Apparatus and Stimuli ..................................................................................... 35
3.2.3 Design ............................................................................................................. 36
3.2.4 Procedure......................................................................................................... 38
3.3 Results .................................................................................................................... 39
3.3.1 Primary Task – Root Mean Square Tracking Error (RMSTE) with Interruption
.................................................................................................................................. 39
3.3.2 Comparison of RMSTEs with and without Interruption from the Secondary
Task .......................................................................................................................... 41
3.3.3 Secondary Task – Reaction Time (RT) ........................................................... 42
3.3.4 Secondary Task – Response Error ................................................................... 47
3.4 Discussion .............................................................................................................. 48
3.5 Conclusion ............................................................................................................. 55
Chapter 4 Experiment 2 - Tracking and Discrete Dual-Task Performance
Assessment of Different Visual Spatial Stimulus-Response Mappings with Focal
and Ambient Vision ......................................................................................................... 58
4.1 Introduction ............................................................................................................ 58
4.1.1 Aims ................................................................................................................ 59
4.1.2 Objectives ........................................................................................................ 60
4.1.3 Hypotheses ...................................................................................................... 60
4.2 Method ................................................................................................................... 61
4.2.1 Participants ...................................................................................................... 61
4.2.2 Apparatus and Stimuli ..................................................................................... 61
vii
4.2.3 Design ............................................................................................................. 62
4.2.4 Procedure......................................................................................................... 64
4.3 Results .................................................................................................................... 65
4.3.1 Primary Task – Root Mean Square Tracking Error (RMSTE) ........................ 65
4.3.2 Comparison of RMSTE with and without Visual Scanning in a Dual-task
Paradigm .................................................................................................................. 67
4.3.3 Secondary Task- Reaction Time (RT) ............................................................. 68
4.3.4 Comparison in RT with and without Visual Scanning in Dual-task Condition
.................................................................................................................................. 73
4.3.5 Secondary Task – Response Error ................................................................... 74
4.3.6 Correlation between Root Mean Square Tracking Errors (RMSTEs) and
Reaction Times (RTs) ............................................................................................... 76
4.4 Discussion .............................................................................................................. 76
4.5 Conclusion ............................................................................................................. 83
Chapter 5 Experiment 3 - Tracking and Discrete Dual Task Performance
Assessment of Different Auditory Spatial Stimulus-Response Mappings:
Implications for Human Machine Interface Design .................................................... 85
5.1 Introduction ........................................................................................................... 85
5.1.1 Aim .................................................................................................................. 86
5.1.2 Objectives ........................................................................................................ 86
5.1.3 Hypotheses ...................................................................................................... 87
5.2 Method ................................................................................................................... 88
5.2.1 Participants ...................................................................................................... 88
5.2.2 Apparatus and Stimuli ..................................................................................... 88
5.2.3 Design ............................................................................................................. 89
5.2.4 Procedure......................................................................................................... 91
5.3 Results .................................................................................................................... 92
5.3.1 Primary Task - Root Mean Square Tracking Error (RMSTE) during
Interruption ............................................................................................................... 92
5.3.2 Comparison of RMSTEs with and without Interruption from the Secondary
Task .......................................................................................................................... 93
5.3.3 Secondary Task - Reaction Time (RT) ............................................................ 94
5.3.4 Secondary Task – Error Percentage (EP) ........................................................ 98
5.3.5 Comparison in RMSTE between Auditory and Visual Presentations for the
Discrete Response Task .......................................................................................... 100
5.3.6 Comparison in RT between Auditory and Visual Presentations for the Discrete
viii
Response Task ........................................................................................................ 102
5.3.7 Comparison in EP between Auditory and Visual Presentations for the Discrete
Response Task ........................................................................................................ 104
5.3.8 Correlation between root mean square tracking errors (RMSTEs) during
interruption and reaction times (RTs) ..................................................................... 106
5.4 Discussion ............................................................................................................ 106
5.4.1 Root Mean Square Tracking Error (RMSTE) ............................................... 107
5.4.2 Reaction Time (RT) ....................................................................................... 112
5.4.3 Error Percentage (EP) ................................................................................... 116
5.5 Conclusion ........................................................................................................... 116
Chapter 6 Experiment 4 – Hand and Foot Controlled Dual Tracking Task
Performance Together with a Discrete Spatial Stimulus-Response Compatibility
Task................................................................................................................................. 119
6.1 Introduction .......................................................................................................... 119
6.1.1 Aims .............................................................................................................. 120
6.1.2 Objectives ...................................................................................................... 121
6.1.3 Hypotheses .................................................................................................... 121
6.2 Method ................................................................................................................. 122
6.2.1 Participants .................................................................................................... 122
6.2.2 Design ........................................................................................................... 122
6.2.3 Apparatus and Stimuli ................................................................................... 126
6.2.4 Procedure....................................................................................................... 128
6.3 Results .................................................................................................................. 129
6.3.1 Root Mean Square Hand Tracking Error (RMSHTE) during Interruption ... 129
6.3.2 Comparison of RMSHTEs with and without Interruption from the Concurrent
Task(s) .................................................................................................................... 131
6.3.3 Root Mean Square Foot Tracking Error (RMSFTE) during Interruption ..... 132
6.3.4 Comparison of RMSFTEs with Interruption from the Concurrent Task(s) .. 133
6.3.5 Reaction Time (RT) for the Spatial Compatibility Task ............................... 134
6.3.6 Error Percentage (EP) for the Spatial Compatibility Task ............................ 140
6.3.7 Correlation between Root Mean Square Hand Tracking Errors (RMSHTEs),
Root Mean Square Foot Tracking Errors (RMSFTEs), and Reaction Times (RTs)
................................................................................................................................ 142
6.4 Discussion ............................................................................................................ 142
6.4.1 Root Mean Square Hand Tracking Error (RMSHTE)................................... 143
ix
6.4.2 Root Mean Square Foot Tracking Error (RMSFTE) .................................... 149
6.4.3 Reaction Time (RT) ....................................................................................... 149
6.4.4 Error Percentage ............................................................................................ 152
6.5 Conclusion ........................................................................................................... 153
Chapter 7 Summary ...................................................................................................... 155
7.1 Overall Discussion ............................................................................................... 155
7.1.1 Experiment 1 ................................................................................................. 155
7.1.2 Experiment 2 ................................................................................................. 156
7.1.3 Experiment 3 ................................................................................................. 156
7.1.4 Experiment 4 ................................................................................................. 157
7.2 Spatial S-R Compatibility Effect ......................................................................... 158
7.3 Modality Effect .................................................................................................... 160
7.4 Dual-task/Multi-task Processing .......................................................................... 161
7.5 Overall Conclusion............................................................................................... 162
References ...................................................................................................................... 166
Appendix of Experiment 1............................................................................................ 178
1.1 Results of repeated-measures ANOVA performed on root mean square tracking
error (RMSTE) and reaction time (RT) for S-R mapping condition. ......................... 178
1.2 Results of repeated-measures ANOVA performed on reaction time (RT) for spatial
compatibility. .............................................................................................................. 178
1.3 Results of repeated-measures ANOVA performed on reaction time (RT) for
stimulus position, response position and their interaction. ........................................ 178
Appendix of Experiment 2............................................................................................ 179
2.1 Results of repeated-measures ANOVA performed on root mean square tracking
error (RMSTE) and reaction time (RT) for S-R mapping conditions. ....................... 179
2.2 Results of repeated-measures ANOVA performed on reaction time (RT) for spatial
compatibility. .............................................................................................................. 179
2.3 Results of repeated-measures ANOVA performed on reaction time (RT) for
stimulus position, response position and their interaction. ........................................ 179
Appendix of Experiment 3............................................................................................ 180
3.1 Results of repeated-measures ANOVA performed on root mean square tracking
error (RMSTE) and reaction time (RT) for S-R mapping conditions. ....................... 180
3.2 Results of repeated-measures ANOVA performed on reaction time (RT) for spatial
compatibility. .............................................................................................................. 180
3.3 Results of repeated-measures ANOVA performed on reaction time (RT) for
stimulus position, response position and their interaction. ........................................ 180
x
3.4 Results of ANOVA performed on root mean square tracking error (RMSTE) in
different modalities and visual conditions under the different spatial mapping
conditions. .................................................................................................................. 181
3.5 Results of ANOVA performed on reaction time (RT) in different modalities and
visual conditions under the different spatial mapping conditions. ............................. 181
3.6 Results of ANOVA performed on error percentage (EP) in different modalities and
visual conditions under the different spatial mapping conditions. ............................. 181
Appendix of Experiment 4............................................................................................ 182
4.1 Results of repeated-measures ANOVA performed on root mean square hand
tracking error (RMSHTE), root mean square foot tracking error (RMSFTE) and
reaction time (RT) for S-R mapping conditions, signal modality and their interaction.
.................................................................................................................................... 182
4.2 Results of repeated-measures ANOVA performed on reaction time (RT) for spatial
compatibility under different signal modalities. ........................................................ 183
4.3 Results of repeated-measures ANOVA performed on reaction time (RT) for
stimulus position, response position, signal modality, and their interactions. ........... 183
xi