-CALIFORNIA STATE UNIVERSITY, NORTHRIDGE
THE EFFECT OF TRAINING AND METHOD OF STIMULUS
PRESENTATION ON ESTIMATES OF TASK-PERFORMANCE TIMES
A thesis submitted in partial satisfaction of
the requirements for the degree of
Master of Arts in
Psychology
by
Steve Gabany
June, 1977
L_
-----------------------------
The Thesis;{f
Mar~~.
S~ft)~any
is approved:
Sanders
~9 .Y,7fabac/m/k
California State University, Northridge
ii
TABLE OF CONTENTS
......... •...................................... . v
List of Figures ............................................... vii
Abstract ..................... -· .............................. . viii
Introduction .................................................. 1
General Objectives ....................................... 1
List of Tables
Background •••••••••••••••••••••••••••••••••••••••••••••••
1
........................
1
Distributions of Performance Time
.......................
Training .................................................
Stimulus Presentation ....................................
8
Statement of the Problem •••••••••••••••••••••••••••••••••
9
Variables Affecting Time Estimates
........................................................
Independent Variables ....................................
Dependent Variables ......................................
Procedure ................................................
Design ...................................................
Subjects .................................................
Results .......................................................
Effect of Training and Stimulus Presentation Method
for the 50th Percentile Values ........................
I
Effect of Training and Stimulus Presentation Method
for the 95th Percentile Values ........................
!
~1ethod
I
I
I
4
5
11
11
11
12
15
16
19
18
21
Effect of Same versus Different Pretrainin9 and Posttraining Method of Stimulus Presentation ••••••••••••••
29
...........................
33
Performance Time Distributions
L
_____ _ j
iii
Page
....................................................
Discussion of the Hypotheses .............................
Application of Findings ..................................
Theoretical Implications .................................
Further Research .........................................
Conclusions ...................................................
References ....................................................
·Appendices ....................................................
Appendix A. Literature Review ...........................
:Discussion
Appendix B.
Appendix
c.
............................... .
Posttraining Deviation Scores .............. .
Instructions
...................... .
....................................
38
38
41
42
44
46
47
50
51
54
58
Appendix D.
Mean Deviation Scores
60
Appendix E.
Raw Data
62
iv
TABLES
Table 1.
Estimated 5th and 95th Percentile Performance
Times for Twelve Unvalidated Tasks, and
Observed 50th and 95th Percentile Performance
Times for Five Validated Tasks ••••••••••••••••••••••
13
Analysis of Variance of Time Estimates for 50th
Percentile Values for Pretraining Stimulus
Presentation Method and Task ••••••••••••••••••••••••
19
Mean Deviation and Value of 't' of Statistically
Significant Effects of Pretraining Estimates of
50th Percentile Performance Times •••••••••••••••••••
20
Analysis of Variance of Time Estimates for 50th
Percentile Values for Pretraining Stimulus Presentation Method, Traininq Method, Posttraining
Stimulus Presentation Method, and Task ••••••••••••••
22
Mean Deviation of Statistically Significant
Effects of Posttraining Estimates of 50th
Percentile ••••••••••••••••••••••••••••••••••••••••••
23
Analysis of Variance of Time Estimates for 95th
Percentile Values for Pretraining Stimulus
Presentation Method, and Task •••••••••••••••••••••••
24
Mean Deviation of Statistically Significant
Effects of Pretraining Estimates of 95th
Percentile ••••••••••••••••••••••••••••••••••••••••••
25
Analysis of Variance of Time Estimates for 95th
Percentile Values for Pretraining Stimulus Presentation Method, Training Method, Posttraining
Stimulus Presentation Method, and Task ••••••••••••••
26
Mean Deviation of Statistically Significant
Effects of Posttraining Estimates of 95th
Percentile •••••••.•••••.••••••••••••••••••••••••••••
28
Table 10. Analysis of Variance of Posttraining Estimates of
50th Percentile Performance Times for Same versus
Different Methods of Pretraining and Posttraining
Stimulus Presentation •••••••••••••••••••••••••••••••
31
Table 2.
'Table 3.
'Table 4.
:Table 5.
iTable 6.
!
iTable 7.
Table 8.
Table 9.
!Table 11. Mean Deviation and Value of 't' of Statistically
i
Significant Effects of Posttraining Estimates
l
of 50th Percentile Performance Times for Same
. ____ve_r_s_u_s_D_,_·f_f_e_r_e_n_t_M_et_h_o_d_s_of_P__r_e_tr_a_i_n_ing and
L
\1
i
_______!
Posttraining Stimulus Presentation
..................
Page
32
'Table 12. Analysis of Variance of Posttraining Estimates of
95th Percentile Performance Times for Same versus
Different Methods of Pretraining and Posttraining
Stimulus Presentation •••••••••••••••••••••••••••••••
34
Table 13. Mean Deviation and Value of •t• of Statistically
Significant Effects of Posttraining Estimates
of 95th Percentile Performance Times for Same
versus Different Methods of Pretraining and
Posttraining Stimulus Presentation ••••••••••••••••••
35
Table 14. Percentile Corresponding to Mean Estimates of
50th and 95th Percentile Performance Times,
and Standard Deviation across Percentiles •••••••••••
36
'
iL___ _
FIGURES
Figure 1.
Estimated Times Compared with Empirical
Cumulative Performance Distributions
(Burger et al., 1970) •••••••••••••••••••••••••••••••
3
:Figure 2.
Flow of Subjects through Experimental
Conditions ••••••••••••••••••••••••••••••••••••••••••
17
Figure 3.
Mean Deviation of Posttraining Estimates
of 95th Percentile Performance Time •••••••••••••••••
30
i
L
:'
i
'
!
_ _ _ _ _ _ _ _ _ _ _ _ _j
vii
ABSTRACT
THE EFFECT OF TRAINING AND METHOD OF STIMULUS
PRESENTATION ON ESTIMATES OF TASK-PERFORMANCE TIMES
by
Steve Gabany
Master of Arts in Psychology
The present study was designed to determine: 1) if subjects
'can be trained to estimate task performance times; 2) if training
by a paired-comparison paradigm would produce more accurate estimates
·than training through performing the actual task; and 3) if viewing
·a video tape of the tasks being performed would provide more accurate
;estimates than reading the names of the tasks.
Subjects were presented with instructions and with seventeen
tasks.
The seventeen tasks were presented either by video or
1written stimuli, and the subjects were then required to make estimates
iof the 50th and 95th percentile performance times.
That is, respec-
!tively, the time above which 50% and 5% of trained operators would
i
lneed more time to complete the task correctly. These tasks were
!selected
from a list of one-hundred on which estimates of performance
I
/times had been published.
They were selected on the basis of
l
!familiarity to the general public.
!
l----~-----~---
Estimates of performance times
of five of the tasks had been validated by obtaining actual performance
times.
These five tasks were embedded within the remaining twelve
.tasks.
Following an introduction to the concept of estimating task per: formance times, twelve subjects
~vere
trained to make estimates by
'Viewing as many pairs of the tasks as time permitted, and indicating
which took more time to complete.
trained by
perf6r~ing
The other twelve subjects were
twelve of the tasks, including the five validated
tasks on a device which simulated equipment designed to test electronic
components.
The
remainin~
five tasks could not be performed on the
equipment.
Finally, all subjects were again presented the seventeen tasks
:by one of the two methods of stimulus presentation, and were asked
I
:to estimate their performance times of the tasks.
Estimates of performance times for the five validated tasks comprised the dependent variable.
The study utilized eight experimental groups to include all com: binations of the two pretraining stimulus presentation methods, two
!
J
I
training methods, and two posttraining stimulus presentation methods.
It was hypothesized that the subjects who were trained by making
I
i
paired comparisons would provide more accurate estimates than subjects
!
! who were trained by performing the tasks; that subjects who observed
I
I video tape recordings of the tasks would provide more accurate esti1
i mates than subjects who read the names of the tasks; and that there
I
I would be no difference in the accuracy of the estimates between subljects who were presented with the same method of stimulus presentatio~
before and after training, and subjects who \'Jere presented with a
'different method of stimulus presentation before and after training.
Estimates of the 50th percentile values of performance times
:were too variable and inaccurate and task specific to have general
fapplicability.
Those judges who received the compatible combination
of paired-comparison training and written stimulus presentation or
of operational training and video stimulus presentation provided more
accurate estimates of the 95th percentile than subjects who received
·an incompatible combination of paired-comparison training and video
stimulus presentation, or of operational training and written stimulus
presentation.
i
I
1_ _ _ _ _
X
INTRODUCTION
:General Objectives
The main objective of this study was to determine if the accuracy
!of estimates of task performance times is a function of training and
I
i
tof the method used to present information.
Accuracy was determined
'by comparing estimated performance times with observed performance
:times •
. Background
The use of human judgements is an inherent part of system design,
and appears to be one cost-effective method of obtaining reliable
and valid evaluations of some task and equipment characteristics
. (Knowles, Burger, Mitchell, Hanifan and Wulfeck, 1969).
Task performance time is a major variable in operability and
!maintainability models.
r~eister (1971) discussed the applicability
lof task performance time to three classes of human reliability models
1
1) analytical methods of predicting operability 2) simulation methods
of predicting operability, and 3) various methods of predicting maintainability.
Distributions of Performance Time
For each task or piece of equipment, it is assumed that a distribution of performance or repair times exists.
However, in most cases
of system design or model application, these distributions are unknm'ln, ·
and estimates must be used to determine various factors such as the
acceptable level of performance of system effectiveness (Smith, Blanchard and Westland, 1971).
Since it is impractical to estimate each
lpoint on the continuum, researchers have constructed the distributions
__ j
1
:bY assuming the shape of the distribution and asking judges to estimate
'
:parameters which define the distribution.
Blanchard, Mitchell and
;smith (1966) asked judges to estimate 11 minimum 11 and 11 maximum 11 task
performance times which the authors defined as the 5th and 95th percentiles. Blanchard, Westland, Mitchell, Smith and Sklar (1966) asked
judges to 11 mentally reconstruct .. the distributions of total repair
and trouble-shootin9 times and to estimate directly the 1st, 25th,
50th (median), 75th, and 99th percentiles.
While the choice of
percentiles appears to be arbitrary, the desired result is usually
an estimation of central tendency and variability.
The present inves-
tigator found only one study that validated estimates of task perforimance time.
In that study, Burger et al. (1970) found that expert
[judges were unable to estimate accurately minimum or 5th percentile
I
iva 1ues of task performance times, but were able to 'estimate maximum
itimes accurately.
More accurate distributions might be constructed
'by training judges to estimate the 50th and 95th percentiles directly.
Cumulative performance distributions.
Hanifan and Sklar (1967)
,developed a technique to construct cumulative empirical time distribu-
!
~
tions. Burger et al. (1970) constructed observed time distributions
ifor five tasks and related estimated minimum and maximum times to
i
the distributions by superimposing points along each empirical time
distribution which corresponded to minimum and maximum time estimates.
This procedure allowed Burger et al. (1970) to determine the percen-
itile at which judges were estimating. Figure 1 shows those empirical
I
ltime distributions and the minimum and maximim estimates (Burger
I
i
!et al., 1970).
I
L_____
For example, Figure 1 shows that the mean minimum
__
_j
100
90
80
70
Cl.)
-
60
~ 50
u
....
:. 40
2
8
10
12
Task Performance (sec.)
Figure 1. Cumulative performance time distributions for
five tasks fitted from empirical data. • = estimated 5th
percentile; 0 = estimated 95th percentile. Task numbers
on curves correspond to task identification numbers in
Blanchard et al. (1966).
L
1
time estimate of 1.4 seconds for Task 29 was over the 60th percentile.
!If the mean minimum time estimate had been accurate, that is at the
'5th percentile, the judges, on the average, would have estimated
iminimum performance time to be about .7 seconds.
'
!Variables Affecting Time Estimates
The present investigator found only two papers which reviewed
'studies of time estimates; Leuba (1963), and Bongers (1969).
There
is little agreement between the authors as to which variables affect
time estimates, or as to the type and amount of the effect of each
variable.
The following list appears to represent variables which
affect the consistency or accuracy of time estimates:
1. Noise and pain (Stuart, 1923);
2. Personality characteristics (Roberts and Herrman,
1961; Siegman, 1961; Falk and Bindra, 1954);
3. Training (White, 1963; Friedman, 1965; Bakan,
Nagle and Denny, 1959);
4. Age (Le Comte du Nouy, 1937);
5. Temperature (Hoagland, in Abramson, 1951);
6. Stimulus complexity (Gulliksen, 1927);
7. Task experience (Bongers, 1969).
These variables have been investigated using the three classic
:psychophysical methods of time estimation: production of a standard,
reproduction of a standard, and estimation of the duration of a standard.
to
11
Leuba (1963) has suggested that such methods require the judge
live through 11 the interval.
However, in a system design setting,
I the judge is often required to conceptualize an entirely new task,
I
i
L---~------------
and to estimate the time required to perform it.
Since the task
.does not exist, the judge cannot live through its performance.
Fur-
ther, the judge may make an estimate in more or less time than the
I
1actual time required to perform the task.
Finally, Smith et al.
(1971) point out that subjects who are estimating repair times are
not dealing with time-duration stimuli.
The stimuli in system design
or model application are tasks or equipment types, rather than the
auditory or visual signals which indicate the start or finish of
a time interval.
The present investigator did not find any studies which examined
the effects of environmental, task or subject variables on those
time estimates typically used in system design.
The levels of pain,
!temperature and noise are probably not high enough in most work envi:ronments to have an adverse affect on
1
performanc~
time estimates.
However, variables such as personality characteristics, training,
'and stimulus and task complexity seem, intuitively, to have the poten•tial of affecting such estimates.
!
Training
i
'
Burger et al. (1970) suggest that training may improve the
i accuracy of estimates of minimal task-performance times. White (1963)
i
I'
I concluded that persons can be trained to produce an interval accurately;
I
I
!
by counting to themselves and that the transfer of training is positive
I and equal to producing intervals as far as sixty-seconds removed
!
I from the training interval. Clearly, production of intervals is
I
I not applicable to estimating unknown task performance time.
i et al. (1959) found that training subjects to estimate
1
L ______ _
Bakan
0
:the duration of a standard by providing knowledge of results transferred from twenty-five-second judgements to two-minute judgements,
1
~nd
that the effect carried over to retention trials the following
:day.
However, training judges to produce or estimate an interval
jby psychophysical methods may be different from estimating task
~erformance
time.
Blanchard, Westland, Mitchell, Smith and Slkar (1966) reviewed
subjective techniques for obtaining human reliability data.
They
concluded that 1) an absolute judgement method is probably quite
imprecise since elemental man-machine reliabilities are usually
~estimated
to the nearest ten-thousandth, while humans have difficulty
1estimating to the nearest hundredth; 2) rank-ordered techniques
;result in transformation to probability scales without 11 anchor 11
ireliabilities; and 3) a paired-comparison methop, while unwieldy,
'is generally accepted to be the most versatile and reliable of
available methods.
Blanchard et al. (1966) subsequently used
1a paired-comparison technique to obtain estimates of one-hundred
jpsychomotor tasks.
Wallace and Rabin (1960) and Leuba (1963),
physical time-estimation studies.
f
A paired-comparison paradigm has also been used in verbal
!learning settings in which the aim was to promote aspects of cognitive
i
)learning, such as insight, problem-solving strategies, organization
i
iof information and learning of principles (Jung, 1968).
Guilford
j(1954) states that although usage has been limited primarily to
L____ _ _ _ _ _ _ __
I
----- --···-- ---- -· ----·-- --···-·
··-·---
---
---.- -----------.
-
!a determination of affective and aesthetic values, the procedure
''
ican be applied whenever stimuli can be presented in pairs, either
~imultaneously
/
or in succession.
Work done by Newell and Simon (1967) in the area of information
i
!processing, suggests that judges trained to estimate psychomotor
'task performance time by a paired-comparison method might first
.learn to form groups of tasks or subtasks according to their complexity ;
:and, with feedback, might order the clusters according to some
idimension of a completion-time continuum.
When called upon to
:estimate the performance time of a new task, he might compare
.the psychomotor attributes of the new task to those in the memorized
;clusters to determine the closest similarity.
His response would
ibe the performance time associated with the cluster.
If wrong,
[the judge might either place the task in a different cluster,
!place the cluster at a different point on the performance-time
icontinuum, or decide that the task represents a new cluster.
!Subsequent responses and questioning might reveal his choice
of strategy.
If learning'to estimate task performance times is a type
of concept formation, it appears that a paired-comparison technique
might be applied to train judges to form a base of task-performance
times by dividing the relative differences into time categories.
However, if learning to make such estimates depends upon mastering
the psychomotor skills involved, another technique might be to
train judges to perform tasks by operating a piece of equipment
lor a simulator.
'
Most of the studies that actually collected subjective:
'
_ _ _ _ _ _ _ _1
!performance-time data utilized judges who were experienced with
!the equipment (Bongers, 1969; Smith et al, 1971), or who were
trained to perform the tasks prior to estimating (Burger et al.,
1970).
Under a concept formation theory, if positive transfer
of training occurs by either method, the judge would compare the
psychomotor requirements of the experimental task to his newly
acquired store of experiences and generate a subjective estimate
of the performance time that will be reliable, although not necessarily
accurate.
If the actual performance times are known, and the
judge is provided feedback that indicates the relative accuracy
of his estimate, it is possible that accuracy will also increase •
. However, even if his estimates are inaccurate but consistent,
!
a correction factor could be applied.
Without such consistency,
;correcting the estimates would be impossible.
;Stimulus Presentation
Trattner, Fine and Kubis (1955) suggest that the method of
!Stimulus presentation may affect the accuracy of human judgement.
:In an investigation of the effect of method of stimulus presentation
!
'
/on
expert judgements of abilities essential for successful task
!performance, the authors concluded that although reliabilities
of the judges who observed the tasks directly were significantly
higher than those who read a task description, no significant
differences were found for the accuracy of the two group's
J~atings.
However, the authors stated that the instructions and training
may have served as mediating conditions and may have obscured
iact~al differences in the validity of the concepts.
In effect,
J
..... ----·---···----- ... --- - - -
-
-
:concepts rather than information about job descriptions may have
t
been rated.
The present investigator did not find any studies which inves-
I
tigated the effect of the method of stimulus presentation on estimates
I
lof task performance time.
Bongers (1969) studied estimates by
experienced operators of performance times for their own tasks.
:Blanchard et al. (1966) presented written stimuli to expert judges,
while Leuba (1963) presented verbal and written material to naive
subjects.
Miller (1954) points out the need to consider the physical
and psychological fidelity of simulation in training, since training
;programs must ultimately be concerned with the transfer of the
i
;critical components of the actual task.
At issue is the trade-off
I
i
! between transfer of training, the physical fidelity of the simulator,
'
i
! and the cost of providing the simulator. These considerations
·seem germane to time-estimation training and model application,
since judges are likely to be estimating performance times for
non-existent tasks.
A direct relatinnship probably exists between
physical fidelity and construction cost with an aim toward obtaining
accurate estimates with the least amount of physical fidelity.
Statement of the Problem
The present study was designed to determine: 1) if subjects
can be trained to estimate task-performance time; 2) if training
by a paired-comparison paradigm would produce more accurate estimates
/ than training through performance of the actual task; and, 3)
I if
viewing a video tape of the task being perfonned
wou~rovid~---~
10
1more accurate estimates than reading the name of the task.
The study utilized eight experimental groups to include all
!combinations of two pretraining stimulus presentation methods,
ltwo training methods, and two posttraining stimulus presentation
!methods.
It was hypothesized that the subjects who were trained by
:making paired comparisons would provide more accurate estimates
than subjects who were trained by performing the tasks; that subjects
'Who observed video tape recordings of the tasks would provide
.more accurate time estimates than subjects who read the names
·of the tasks; and that there would be no difference in the accuracy
:of estimates between subjects who were presented with the same
'
!method of stimulus presentation before and after training, and
;subjects who received a different method of stimulus presentation
ibefore and after training.
I I
METHOD
'Independent Variables
The four independent variables were: the method used to describe
I
1the tasks when the subjects made their pretraining estimates; the
l
l
imethod used to train subjects to make performance time estimates;
the method used to de-cribe the tasks when the subjects made their
:posttraining estimates; and the task.
Task was not considered to
be a major variable.
Training methods.
Subjects were trained either by making
paired comparisons of performance times of seventeen tasks of
varying psychomotor complexity, or by learning to perform twelve
iof the same tasks on a piece of simulated test equipment.
Stimulus presentation.
In obtaining estimates of task performance
!times, two methods of presenting stimuli to the subjects were used;
;the written names of the tasks or a video tape display of the task
:being performed by a trained operator.
Task.
i
The seventeen tasks are listed in Table 1. They were
selected from a list of one hundred for which Blanchard et al. (1966)
had obtained estimates of performance times, and included five on
which Burger et al. (1970) had validated estimates by obtaining actual
I
!performance times.
Table 1 shows estimates of the 5th and 95th per-
!centile times for the twelve unvalidated tasks (Blanchard et al.,
I 1966). Performance times for the 50th and 95th percentiles for the
i
!five validated tasks are also shown (Burger et al., 1970).
I Dependent Variables
i
First, all subjects made pretraining estimates of performance
'------------~--
~~~--~-
-------
_j
IL
·TABLE 1
'Estimated 5th and 95th Percentile Performance Times for Twelve Unvali'dated Tasks (Blanchard et al., 1966), and Observed 50th and 95th Per;centile Performance Times for Five Validated Tasks (Burger et al.,
1970).
1
95th
5th
·unvalidated Tasks
Percentile Percentile
1. Observe Several Dials and Record Data
Quantity.
12.0
30.0
3. Change Typewriter Ribbon.
90.0
30.0
4. Install Tape in Tape Recorder.
45.0
13.5
5.5
6. Observe Dial Quantity for Correct Readout.
2.5
9.0
7. Adjust Dial Quantity with Crank.
4.5
8. Throw Toggle Switch and Observe Dial
6.0
Quantity.
3.0
4.0
10. Observe Dial Quality for Correct Readout.
1.5
:12. Adjust Dial Quality with Knob.
2.5
7.5
14. Throw Toggle Switch and Observe Dial
5.5
Quality.
2.0
15. Depress Button, Turn Knob with Other Hand,
10.0
and Observe Dial Quantity.
5.0
.16. Observe Several Quantity and Quality
Indicators and Determine if Equipment is
25.0
10.0
Operating Correctly.
17. Throw Toggle Switch, Hold, and with Other
Hand Adjust Dial Quantity with Knob.
10.0
5.0
Validated Tasks
50th
95th
Percentile Percentile
2. Throw Toggle Switch and Observe Switch
Position.
1.3
3.2
5. Turn Rotary Selection Switch to Specific
Position.
1.7
5.1
9. Observe Dial Quantity and Record Data.
5.2
9.4
11. Push Toggle and Note if On-Off Lamp is Lit.
1.9
4.2
13. Adjust Dial Quantity \'lith Knob.
5.5
9.1
________________________________________________________
:
L
I
I
_ _ _ _ _ _ _ _ _ _j
13·
times for the seventeen tasks shown in Table 1.
'subjects.
Then, all subjects
Again, these five tasks were embedded within the remaining
twelve.
Procedure
The instructions given to the subjects are presented in Appendix
B.
No clocks were in the room, and subjects were asked to put their
:watches out of sight.
Subjects were tested and trained individually.
i
·To familiarize them with the concept and use of time estimation,
!subjects were shown a ten-minute video tape recording of a segment
lof a commercial 16mm film on time-and-motion methods.
To introduce the subjects to the specific type of time estimation
~under
consideration, they were also shown a ten-minute video tape
'
1recording of trained operators performing fifteen tasks of differing
I
fpsychomotor complexity using a variety of electronic, electromechanical,
i
[
land mechanical devices.
These tasks were chosen to represent a variety:
!
iof performance times and complexity, and were performed on testing
i
land repair equipment located within the Psychology Department at
!california State University, Northridge.
Although none of these
'
I
!tasks were duplicated in those presented to the subjects for estimation
I
!performance times were omitted in order to reduce contaminating the
i
jtraining an~ stimulus presentation variables.
-·
---
~--
Video stimulus presentation.
- _.__
Twelve subjects were shown a video
:recording
of one of the seventeen tasks being performed once by a
i
:trained operator. Subjects were told that the operator was, 11 Wellltrained.11
Subjects were required to make 50th and 95th percentile
!
1estimates of performance time for each task before the video clip
I
of the next task was presented.
The order of presentation was the
same for all subjects.
Written stimulus presentation.
Twelve subjects were presented
with a list of the names of seventeen tasks of varying psychomotor
complexity.
Subjects were required to read down the list and make
• 50th and 95th percentile estimates of performance ~ime for each task
in order.
The order of presentation was the same for all subjects.
Paired-comparison training.
i
A paired-comparison psychophysical
paradigm was used to present seventeen tasks of varying psychomotor
·complexity to b1el ve subjects.
Descriptions of all possible pairs
· of the tasks were typed on 4 X 6 11 index cards.
The order of the
'cards was randomized by shuffling them before presenting them to
the next subject.
The cards were displayed one at a time to the
subject by the experimenter.
After each pair was presented, the
subject was asked to say which member of the pair, on the average,
was likely to take more time to complete.
i
The experimenter provided
immediate feedback whether or not the choice was correct, since the
) experimenter wanted to approach a level of psychological fidelity
I
i of simulation found in an actual training situation. If the choice
i
j
was incorrect, the experimenter told the subject the average perfor-
i mance time of the longer task, (Blanchard et al.,
!
i
$-------
1966) except in
1;.)
:those comparisons involving the five tasks validated by Burger et
i
;al. (1970).
Training continued for twenty minutes.
Operational training. ·Twelve subjects were required to learn
!to operate a piece of electronic test equipment consisting of the
!five validated tasks embedded within seven of the remaining twelve
tasks used in the paired-comparison training method.
Each subject
·was given instruction in operating an Operational Amplifier Test
'Console, an apparatus designed by Burger et al. (1970) to simulate
psychomotor tasks involved in operating electronic testing equipment.
The subject was asked to perform the tasks as quickly as possible,
and was allowed to refer to a set of operational procedures whenever
•necessary.
~
After the subject completed the twelve tasks, the experi-
imenter returned the apparatus to the ready state, provided a different
!operational amplifier, and asked the subject to perform the sequence
:again in the same order.
N~
feedback was provided since the experimen-
'
:ter wanted to approach a level of psychological fidelity of simulation
similar to that found in on-the-job training.
iPostexperimental Interviews
Subjects were asked to describe how they made their estimates.
iThe experimenter recorded a summary of their answers.
Subjects were
I
!then told the purpose of the experiment.
loesign
I
A 2 X 2 X 2 X 5 mixed analysis of variance design with repeated
/measures on the task variable, was used to determine any significance
I of the main effects of method of pretraining stimulus presentation,
\training, method of posttraining stimulus presentation, task, and
L__ _ _ ~
i
~_j
:of the interactions among the four factors.
:Subjects
Twenty-four students, fourteen male and ten female, \'/hose participation in experiments is one method of partially satisfying the course
requirements in lower-division Psychology 150 and 250 classes at
California State University, Northridge, volunteered for the study.
No subjects reported hobbies or professions involving time estimation;
therefore, all were considered naive about estimating time.
Subjects
were randomly assigned to the eight experimental conditions summarized
in Figure 2.
Posttraining
Stimulus
Presentation
Training
Pretraining
Stimulus
Presentation
"
Written
N=3
--..."
Video
N=3
--...'
Written
N=3
'
Video
N=3
....
7
Written
N=3
'
Video
N=3
..."-
Written
N=3
.........
Video
N=3
--;7
....
...
Operate
N=6
Written
N=12
...
......
Paired
Comparison
N=6
""7
.........
Operate
N=6
7
Video
N=12
,q
......
""7
j
Figure _2.
IL _
Paired
Comparison
N=6
Flow of subjects through experimental conditions.
i
----~-------~-------_______)
RESULTS
Estimates of 50th and 95th percentile values of performance
'times for the five tasks for which actual times for practiced subjects
were available were obtained prior to and after training.
The dif-
·ferences between each subject's estimated values and actual values
observed by Burger et al. (1970) were calculated and used for all
subsequent calculations.
Analysis of Deviations of the 50th Percentile Performance Times
Pretraining.
A two-factor mixed-design analysis of variance,
using BMD-08V, was conducted on the pretraining difference values
of the 50th percentile to determine any effect of stimulus presentation
and task on estimates made prior to training.
Table 2 shows that
the following effects were statistically significant: 1) task, [
· (4,88) = 47.15, £
. (4,88) = 3.04, £
<
<
.001, and 2) stimulus presentation x task, [
.05.
Table 3 shows the mean deviation of the
. statistically significant effects and the values oft-tests performed
· on the mean estimates of the main effects to determine if they differed
i significantly from zero.
1
Subjects who were presented with written
stimulus material estimated Task 5 least accurately:
and Task 9 most accurately:
X= -.84.
X = -2.05,
Only the mean deviation of
Task 9 and Task 11 was significantly different from zero, and could
i
! be considered as accurate estimates of performance time.
All other
/deviations were significantly different from zero, and were considered
i
I to
be inaccurate estimates. T-tests were not performed on task inter-
/ actions since they were not central to the study.
i
L_
-----~-~J
1::1
TABLE 2
Analysis of Variance of Pretraining Estimates of
50th Percentile Performance Times.
Source of Variance
Pretraining Stimulus
Presentation (Pre)
Task (T)
Subjects (S) x Pre
"
Pre x T
T x S x Pre
Total
DF
MS
1
4
22
4
88
119
60.49
55.49
20.39
3.58
1.18
*£.< .05
** £. < .001
I
!
I
Ii
I
i
L_
F
2.96
47.14**
3.04*
c..u
TABLE 3
Mean Deviation and Value of •t• of Statistically Significant Effects of Pretraining Estimates of 50th Percentile
Performance Times.
Ex~eriMental
Effect
Task
Task 2
Task 5
Task 9
Task 11
Task 13
Stimulus Presentation
x Task
Video
Task 2
'
..Task 5
Task 9
Task 11
Task 13
Written
Task 2
Task 5
Task 9
Task 11
Task 13
Nean
DF
tj
-1.62
-1.35
.43
-.43
2.14
23
23
23
23
23
3.95**
2.81*
.75
.98
4.55**
-1.44
-.65
1. 70
.25
2.86
-1.80
-2.05
-.84
-1.11
1.42
*.E.< .005
**E.< .QQQ?
1 Tested against mean value of 11 011
i
lL__ _ _ _
__________i
Ll
TABLE 4
Analysis of Variance of Posttraining Estimates of
50th Percentile Performance Times.
c.. c..
Posttraining.
A four-factor mixed-design analysis of variance,
also using BMD-08V, was calculated to determine the effect of pretraining method of stimulus presentation, training, posttraining
method of stimulus presentation, and task on 50th percentile estimates.
Table 4 indicates that the following experimental effects were significant: 1) task, [ (4,64) = 9.26, £
= 3.01, £
<
<
.001; 2) training x task, [ (4,64)
.05; and 3) training x stimulus presentation x task,
[ (4,64) = 4.40, £
<
.01.
Table 5 shows the mean deviations of the
statistically significant effects and the values of t-tests performed
'on the mean estimates of the main effect to determine if they differed
significantly from zero.
Analysis of Deviations of the 95th Percentile Performance Times
Pretraining.
Using BMD-08V, a two-factor mixed-design analysis
iof variance was conducted on the pretraining difference values of
:the 95th percentile.
Table 6 shows that the only statistically signi-
: ficant effect was task, [ (4,88) = 53.78, £
<
.001.
Table 7 shows
, the mean deviation and the t-value of the statistically significant
effect.
It shows that subjects estimated the 95th percentile time
of Task 13 least accurately:
X= 4.10, and Task 11 most accurately:
X= .24. All mean deviations except for Task 11 were significantly
different from zero.
Only the estimate of the performance time of
! Task 11 was considered to be accurate.
I,
Posttraining.
A four-factor mixed design analysis of variance,
) using BMD-08V, was conducted to determine the effect of pretraining
I method of stimulus presentation, training, posttraining method of
I stimulus presentation, and task. Table 8 indicates that the
I
l________ -----
------ ---~-_j
'-'-'
TABLE 5
Mean Deviation of Statistically Significant Effects of
·Posttraining Estimates of 50th Percentile Performance Times.
Exeerimental Effect Task 2
Task
-.79
1
Value of •t• oF=23 3.-22*
·Training x Task
P.C.
-1.01
Op.
-.58
!Training x Stimulus
!Presentation x Task
P.C.-Video
-1.02
P.C.-Written
-1.00
Op.-Video
-.75
Op.-Written
-.42
Task 5
-1.69
3.70*
Task 9
-.02
.03
-1.63
1. 76
-1.48
1.44
• 91
.06
2.26
1.26
-.57
-2.70
-3.05
-.47
.02
-2.98
1.20
1.68
-.63
-1.18
-.03
.15
1.67
2.85
2.90
-.33
*p_ < • 005
1 Tested against mean value of non.
I
i
L __
Task 11
-.42
1.59
Task 13
1. 77
3.40*
TABLE 6
Analysis of Variance of Pretraining Estimates of
95th Percentile Performance Times.
Source of Variance
Pretraining Stimulus
Presentation (Pre)
Task (T)
Subjects (S) x Pre
Pre x T
T X s X Pre
Total
OF
~1S
F
1
4
22
4
88
119
106.03
144.73
40.97
.82
2.69
2.59
53.78*
• 30
* E.< .001
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _!i
C..;)
- TABLE 7
Mean Deviation and Value of •t• of Statistically Significant Effects of Pretraining Estimates of 95th Percentile
Performance Times.
Ex~erimenta 1
Task
Task
Task
Task
Task
Task
2
5
9
11
13
Mean
OF
tl
-2.20
.67
2.91
.24
4.10
23
23
23
23
23
2.23*
1.31
4.85**
.36
6.72**
Effect
* £. < .025
** £. < .0005
1 Tested against mean value of
11
0
11
•
'
I
i
I
I
I
L____________ - - - - - - - - - - - - - - - - - - - - -
;
·---·---------
<-V
TABLE 8
Analysis of Variance of Posttraining Estimates of
95th Percentile Performance Times.
Source of Variance
Pretraining Stimulus
Presentation (Pre)
Training (Tr)
Posttraining Stimulus
Presentation (Post)
Task (T)
Pre x Tr
Pre x Post
Tr x Post
Pre x T
Tr x T
Post x T
Pre x Tr x Post
Pre x Tr x T
Pre x Post x T
Tr x Post x T
Subjects (S) x
Pre x Tr x Post
Pre x Tr x Post x T
T X s X Pre X Tr X
Post
Total
DF
MS
F
1
1
3.78
8.27
.12
.26
1
4
1
1
1
4
4
4
1
4
4
4
13.13
91.27
56.17
24.57
151.65
6.07
10.82
6.40
69.77
7.70
7.25
9.96
.41
16.19**
1. 74
.76
4.69*
1.08
1. 92
1.14
2.16
1.37
1.29
1. 77
16
4
32.30
7.78
1.38
64
119
5.63
* E. < • 05
** E.< .001
i_________________...;_______
[
~-------.
·-----------------~--
C... I
following experimental effects were statistically significant: 1)
task, I (4,64) = 16.69,
~ <
.001; and 2) training x posttraining
method of stimulus presentation, I (1,16) = 4.69,
~ <
.05. Table 9
shows the mean deviation and the value of t-tests performed to deterImine if the mean estimates differed significantly from zero.
Overall,
subjects estimated Task 13 least accurately: X = 3.43, and Task 5
most accurately: X = -.59.
The two-way interaction shows that subjects
'who received a combination of paired-comparison training and written
, stimulus presentation provided a mean deviation of -.72, and that
• subjects who received a combination of operational training and video
, stimulus presentation provided a mean deviation of .46.
Since this
, interaction of training and stimulus presentation was the only statis!tically significant effect that did not involve task, at-test was
I
1
Calculated to determine if any of the group means failed to differ
!significantly from a mean deviation of zero.
That is, did any combi-
1nation of training and stimulus presentation result in estimates
'that could be considered to be statistically accurate?
Neither
of the compatible combinations of paired-comparison training and
written stimulus presentation or operational training and video
I
i stimulus
I
presentation were significantly different from zero, and
I
I both were considered to be accurate estimates of the 95th percentile
i performance times. Conversely, both of the incompatible combinations
i
jof paired-comparison training and video stimulus presentation, and
!
!operational training and written stimulus presentation were signifilcantly different from zero and both were considered to be inaccurate
\estimates.
Also, it indicates that neither of the compatible
<.U
- TABLE 9
Mean Deviation and Value of 't' of Statistically Significant Effects of Posttraining Estimates of 95th Percentile
Performance Times.
Ex~erimental Effect
Task
Task 2
Task 5
Task 9
Task 11
Task 13
Training x Stimulus
Presentation
P.C.-Video
P.C.-Written
Op.-Video
Op.-Written
Mean
OF
t1
-1.15
-.59
2.44
.84
3.43
23
23
23
23
23
1.92*
.60
3.40**
1.66
4.57***
2.19
29
29
29
29
4.56**
1.06
.49
3.97**
-. 72
.46
2.05
* £. < • 05
** £. < .005
*** £. < .0005
1 Tested against mean value of 0".
11
l
IL~------~-·----------~-------
combinations of training and stimulus presentation were more accurate
than the other.
Figure 3 graphs the mean deviations.
The smaller
!deviations represent compatible combinations of training and stimulus
i
!presentation; larger deviations represent incompatible combinations.
;
!Analysis of Same versus Different Pretraining and Posttraining Method
.of Stimulus Presentation
An analysis was performed to compare the effect of presenting
the subject with the same method of stimulus presentation before
and after training (i.e., written-written or video-video) with a
different method of stimulus presentation (i.e., written-video or
.video-written) before and after training.
Analysis of estimates of the 50th percentile performance times.
:A two-factor analysis of variance, using UCLA Biomedical (statistical)
program BMD-08V (Dixon, 1973) was conducted on the posttraining difference values of the 50th percentile to determine any effect of
combination of stimulus presentation and task.
Table 10 shows that
task was the only statistically significant effect: [ (1,88) = 8.97,
£
<
.001.
Table 11 shows the mean deviation and the t-value of the
significant effect, and indicates that Task 13 was estimated least
accurately:
X= 2.11,
and that Task 9 was estimated most accurately:
X = -.28. Only the mean deviation of Task 9 and Task 11 were not
significantly different from zero.
The estimates of the other three
tasks were considered to be inaccurate.
Analysis of estimates of the 95th percentile performance
time~.
A two-factor analysis of variance, using BMD-08V, was conducted on
/the posttraining difference values of the 95th percentile to determine
'
_j'
JU
Stimulus Presentation
=
Written
= Video
3
2
.
u
CI.J
Ill
s::
0
1
•r.jJ
n::l
•r-
>
CI.J
Cl
0
-1
PairedOperate
Comparison
Training Methods
Figure 3. Mean deviations of
posttraining estimates of 95th
percentile performance time.
_j
Jl
TABLE 10
Analysis of Variance of Posttraining Estimates of
50th Percentile Performance Times for Same versus
Different Methods of Pretraining and Posttraining
Stimulus Presentation.
Source of Variance
Stimulus Presentation
Method (Stirn)
Task (T)
Subjects (S) x Stirn
Stirn x T
T X s X Stirn
Total
OF
MS
F
1
4
22
4
88
119
23.23
48.31
6.90
3.94
5.38
3.36
8.98*
.73
*.e.< .001
'
__ j
32
TABLE 11
Mean Deviation and Value of 't' of Statistically Significant Effects of Posttraining Estimates of 50th Percentile
Performance Times for Same versus Different Methods of Pretraining and Posttraining Stimulus Presentation.
Ex12erimental Effect
Task
Task 2
Task 5
Task 9
Task 11
Task 13
Mean
DF
t1
-.79
23
23
23
23
23
3.59*
3.74*
.08
.25
3.77**
-1.72
-.28
-.42
2.11
* E. < • 005
** E. < • 0005
1 Tested against a mean value of "0".
iany effect of combination of stimulus presentation, and task.
Table
;12 shows that task was again the only statistically significant effect:
;I (1,88) = 11.35, R < .001.
Table 13 shows the mean deviation and the t-value of the signifiicant effect, and indicates that subjects estimated Task 13 least
:accurately:
X= -3.21, and that they estimated Task 5 most accurately:
X= -.37, and that the mean deviations of Task 2 and Task 5 were
not significantly different from zero.
Performance Time Distributions
The construction and application of cumulative performance times
was discussed earlier.
Briefly, Burger ct al. (1970) states that
. these distributions are often more useful than a measure of central
tendency, since the percentile at which the subjects are estimating
i
:can be determined by superimposing the mean estimates along the distri-•
; bution.
Table 14 shows the percentile at which the four posttraining
groups estimated the 50th and 95th percentile performance times of
. the five validated tasks.
The standard deviation across tasks of
the percentiles of each group is also shown.
It indicates that esti-
mates of the 50th percentile varied between the 5th and 95th percentile, and that the variability of scores was higher for estimates
of the 50th percentile than for the 95th percentile.
Additionally,
it shows that those subjects who were presented with compatible com1
binations of:training·and stimulus presentation provided the least
i variable estimates of the 95th percentile.
I Postexperimental
I
Interviews
Subjects were asked to respond to the open-ended question,
L__------------,------------------
TABLE 12
Analysis· of Variance of Posttraining Estimates of
95th Percentile Performance Times for Same versus
Different Methods of Pretraining and Posttraining
Stimulus Presentation.
Source of Variance
Stimulus Presentation
Method (Stirn)
Task (T)
Subjects (S) x Stirn
Stirn x T
T x S x Stirn
Total
OF
MS
F
1
4
22
4
88
119
21.17
74.15
38.25
4.60
6.53
.55
11.35*
.70
* E.< .001
I
L----~-~---
--------~----------~----
--~~ ----~-
TABLE 13
Mean Deviation and Value of 't' of Statistically Significant Effects of Posttraining Estimates of 95th Percentile
Performance Times for Same versus Different Methods of Pretraining and Posttraining Stimulus Presentation.
ExEerimental Effect
Task
Task 2
Task 5
Task 9
Task 11
Task 13
Mean
DF
-.87
-.37
2.44
.84
3.21
23
23
23
23
23
tl
1.45
.37
3.39**
1. 79*
4.22**
* £. < • 005
** £. < .0005
1 Tested against a mean value of 0
11
'-------
11
•
_________ _j
.:Sb
iTABLE 14
Percentile at which the Four Posttraining Groups Estimated
the 50th and 95th Percentile Performance Times of the Five
'Tasks, and the Standard Deviation Across Tasks.
Grou~
50%
P.C.-Video
P.C.-Written
Op.-Video
Op.-Written
:95%
, P.C.-Video
P.C.-Written
Op.-Video
Op.-Hritten
· Task 2 Task 5 Task 9 Task 11 Task 13 Std. Dev.
84
85
82
72
70
94
95
58
52
91
10
20
63
85
66
43
5
61
36
5
30.16
13.01
34.63
27.30
95
100
97
93
84
100
100
88
71
98
82
100
90
82
27
90
84
35
26.45
4.34
9.40
23.52
77
65
.J/
i "How
did you make your estimates?"
Results of the summarized responses
i
i
1indicated that subjects attempted to visualize the task being performed
as one step, in a wholistic manner.
l_______ ----------------------------------------
DISCUSSION
The discussion is divided into four sections; discussion of
!the hypotheses; practical application of the results; theoretical
!implications; and suggestions for additional research.
!Discussion of the Hypotheses
This study tested two main hypotheses; first, that subjects
.who were trained by'a paired-comparison paradigm would provide more
accurate posttraining estimates than subjects who were trained by
performing the tasks; and second, that subjects who observed video
tape recordings of the tasks being performed would provide more accur. ate posttraining estimates than subjects who read the name of the
task.
Since the analysis of variance of posttraining deviations
'of the 50th percentile performance times indicated a statistically
I significant
training x posttraining method of stimulus presentation
x task interaction, and a statistically significant training x post, training method of stimulus presentation interaction for estimates
• of the 95th percentiles, it was concluded that neither hypothesis
was supported independently.
Interpretation of both the two-way
and three-way interaction of posttraining deviations of the 50th
i percentile adds little meaning, since the effect of training or method
i
i
I of stimulus presentation is dependent upon the task being estimated.
i
i This task specificity is also apparent in studying the percentiles
/at which the subjects were estimating the actual 50th percentile
!
/performance times, and the standard deviations of the percentiles
/ of each group shown in Table 14.
t
!
The usefulness of the 50th percen-
tile estimates are reduced by inaccuracy and inconsistency; for
~----------
39
1i example, the group with the most consistency overestimated every
:task with a range from 61% to 94%.
I
The two-way interaction of deviations of the 95th percentile
!is more interpretable and seems to aid in understanding the effect
iof the two main variables.
The finding that subjects who received
a compatible combination of training and stimulus presentation provided
imore accurate estim~tes than those who received an incompatible combi- :
'
nation was not expected. This finding may be encouraging since subjects
:who were trained by reading pairs of tasks and indicating which took
longer and who were presented with written stimuli estimated 95th
percentile performance times as accurately as those who learned to
estimate by performing the tasks and who were presented with video
; stimuli.
This result may be confounded by differential feedback,
; since feedback was provided to the paired-comparison training group,
~but
not to the operational training group.
Finally, Table 14 shows
:that subjects who received a combination of paired-comparison training
:and written stimulus presentation provided the most accurate and
least deviant equivalent percentiles when superimposed along the
cumulative performance distirbutions.
Subjects who received the
other compatible combination of operate training and video stimuli
provided reasonably accurate, although slightly less consistent estimates.
It should be pointed out that a "no-training" group was not
included, although the effect of the method of pretraining stimulus
presentation did not result in any significant effect when the posttraining deviations were analyzed.
This finding lends some support
'tU
!to resulting effects of training, since stimulus presentation should
:not train subjects.
Support was found for the third hypothesis, that providing the
!same
or different methods of stimulus presentation prior to making
i
'
ipretraining and posttraining estimates would have no effect on the
estimates.
This findin9 lends support to the effect of training
and stimulus presentation by reducing one contaminating effect.
The effect of task was also not expected, but is not suprising.
Although Task 13, the most complex of the five validated tasks, appeared as the task least accurately estimated in all but three of
the significant effects, the t-tests indicate that it was not the
only task estimated inaccurately.
Further, no consistent pattern
:emerged for tasks that were estimated accurately.
Results may be
attributible to psychomotor complexity, differential familiarity,
'Or a combination of both.
Postexperimental Interviews
Since the five validated tasks were relatively simple, it is
:not suprising that naive subjects attempted to visualize them being
!performed wholistically, rather than subtask by subtask.
However,
tit was thought that those subjects who were trained by performing
lthe tasks might break down the more complex tasks into their psycho!
!motor components when making posttraining estimates. That this group
I
!did not report a detailed procedure of estimating may mean the proce'
[dure
did not occur, or that the naive subjects did not know how to
jreport the procedure.
I
I
;Application of Findings
Burger et al. (1970) found that 11 minimum 11 time estimates were
:unreliable.
The present study indicates that estimates of the 50th
percentile values are also unreliable.
However, estimates of the
95th percentile, or 11 maximum 11 values may have more practical application at any rate, since, at least for two groups, the estimates were
fairly accurate.
The importance of this finding is that reliable
estimates of the maximum performance time of a task may be the crucial
dimension, especially for hazardous or otherwise critical tasks.
While it might be helpful to know a minimum time or average time
of completion, the model builder or system designer might well be
more concerned with obtaining a reliable estimate of maximum time.
:Further, if one knows or can assume the shape of the distribution,
:and can estimate one point of that distribution, the rest of the
distribution can be estimated.
Although valid estimates are certainly
desirable, the reliability of the estimate is probably as or more
important, since a correction factor can be applied to inaccurate
·but reliable estimates.
I of
11
The current study supports the usability
maximum 11 estimates found by Burger et al. (1970).
The finding that subjects who received a combination of paired-
comparison training and written stimulus presentation provided posttraining estimates of the 95th percentile as, or more accurate than
any other combination indicates that subjects can be trained, with
feedback, to make accurate and consistent estimates of task performance
times without either seeing the task performed or performing it themselves.
If this finding is confirmed for other tasks and different
.,. ...
groups of subjects, it is possible that subjects can be trained to
·mak~
accurate estimates of task performance time without simulators
of high physical fidelity or long-term training procedures.
!Theoretical Implications
The study provides information related to theories of concept
formation fidelity of simulation, and method of stimulus presentation.
Concept formation.
Even estimating the task performance time
of simple tasks seems to the present investigator to be a process
of identifying complex attributes, rather than a single attribute.
Posttraining interviews with the subjects indicated that at least
posttraining estimates were made by trying to visualize the task
being performed, and by comparing the operation to other tasks ex'perienced in or prior to this study.
Bourne and Dominowski (1972)
;indicate that such reports appear consistent with the descriptions
of processes obtained from an information-processing theory.
Neimark and Santa (1975) point out that some other theories of
1concept formation seem to limit the subject's classification method
:to a single attribute, such as sex or color.
Even the Bower-Trabasso
lmodel treats the identification of complex attributes as two or more
independent subproblems (Bourne and Dominowski, 1975), rather than
the wholistic approach reported by subjects in the present study
with simple tasks.
Garner and Felfoldy (1970), Lockhead (1972), and Shepard and
!Cermak (1973) suggest that subjects often react to complex stimuli
las integrated entities prior to the full dimensional analysis pro1
!posed by conventional models of concept identification and formation.
~
:~----~~------··-------------
------~-
------------·· --
--------~-----·--
-----------
-~
Certainly the subjects in the present study reported such an overall
examination of the task, rather than a building process, subtask
by subtask.
Again, the tasks may have been too elementary for a
building process.
Fidelity of simulation.
It was assumed that both training methods
,provided a reasonable amount of psychological fidelity of simulation
!found in real-life training situations.
If so, the findings provide
support for the importance of psychological fidelity of simulation,
·since the combination with the least amount of physical fidelity
·was probably paired-comparison training and written stimulus presentation.
Stimulus presentation.
i determine
Little research has been conducted to
the effect of methods of stimulus presentation on human
:judgements in general.
The present study did not support the superi-
• ority of the observation method found by Trattner et al. (1955).
:It is likely that judging "essential task performance abilities"
:is not comparable to "estimating task performance time," although
same type of concept formation learning may take place.
The
few studies that obtained performance time estimates (Burger et al.,
1970; Bongers, 1967; and Blanchard et al., 1966) employed expert
judges who were asked to provide information concerning tasks with
which they were very familiar.
In such cases, it seems unlikely
that any differences in consistency of estimates would have occured,
since judges would be observing operators performing tasks well known
i
to the judges.
I with
While it may seem that all subjects would be familiar
the simple tasks used in the present study, differences in
accuracy between groups and tasks indicate that differential familiarity
occured.
Also, previous investigators were not concerned with training
the judges to make estimates and to transfer that learning to new
tasks.
When training is conducted, the present study indicates that
!subjects who are presented with written task descriptions coupled
with a compatible method of training will provide estimates of perfor:mance time that are as accurate as those provided by subjects who
are presented video stimuli coupled with a compatible method of training, and more accurate than subjects who are presented with either
of the two methods of stimulus presentation coupled with incompabtible
training methods.
,Additional Research Suggestions
A number of areas appear to warrant further investigation.
First, the present study was limited to five rather simple, validated
1
.tasks.
Further validation, especially of more complex tasks, may
provide additional generalizability of the results, and may help
researchers understand the thought process used by subjects who are
iasked
to estimate task performance time.
!
Furthermore, the consistent
!appearance of task as a main effect and in interactions with training
!and stimulus presentation indicate that task may be an important
i
!variable itself.
Additional research will, by necessity, have to
iI
!validate additional tasks before much progress can be made in this
area.
I
Second, although subjects who were trained by performing the
[tasks were not 11 formally 11 trained by providing feedback, they cannot
I
!be considered to be a control group in the strict sense of experimental
I
L_ _ _ _
----
/design.
It is conceivable that presenting subjects with written
'or video stimuli constitutes training by itself.
Since subjects
left to their own devices for the twenty-minute training period may
review their estimates in their minds, additional research could
easily fill this time with tasks that are unrelated to estimating
time.
Third, future research should employ a paired-comparison training
group who are not given feedback to determine if the present results
are a function of training, feedback or a combination of the two.
Fourth, additional research might be conducted on estimates
of the 50th percentile to see if the pretraining and posttraining
:estimates are indeed task specific, or a result of method errors.
Finally, although the method of pretraining stimulus presentation
!did not appear to have an effect on the posttraining estimates, addi:tional studies should include groups of subjects who receive only
training and posttraining stimulus presentation to further parcel
lOUt the effect of the pretraining stimulus presentation.
I
I
I
I
I
I
L------------~------------------
"TV
CONCLUSIONS
The present study concluded that:
1. Subjects who were presented with a compatible combination
~f
training and stimulus presentation provided more accurate and
!less deviant posttraining estimates of 95th percentile performance
times than subjects who received an. incompatible combination;
2. Subjects who were presented with a combination of paired<comparison training and written stimulus presentation provided
:posttraining estimates of the 95th percentile that were as accurate
and less deviant than any other group;
3. Pretraining and posttraining estimates of the 50th percentile
;performance times were task specific and did not appear to have
~general
applicability to system design or model application without
further study;
4. Subjects who received the same method of stimulus presentation
.prior to and after training provided estimates of both the 50th
iand 95th percentile that were no different than those of subjects
iWho were presented with a different method before and after training;
5. Although the effect of task was almost universally present,
!its effect depended upon the level of percentile, and the methods
!of stimulus presentation and training.
L____--~--- -·----~-~---~-·---------------------~------------~------~------------------
'+I
REFERENCES
~akan, P., Nagle, L.G., and Denny, M.R.
Learning, transfer, and retention in the judgement of time intervals. Papers of the Michigan
Academy of Sciency, Arts, and Letters, Vol. XLIV, 1959, 219226.
/Blanchard, R.E., Mitchell, M.B., and Smith, R.L. Liklihood-of-Accomplishment Scale for a Sample of Man-Machine Activities. Dunlap
and Associates, Inc., Santa Monica, California, June, 1966.
Blanchard, R.E., Westland, R.A., Mitchell, M.B., Smith, R.L., and Sklar,
L.B. Development of a technique for establishing personnel
performance standards, Phase II, Final Report. Dunlap and
Associates, Inglewood, California, California, 1966.
Bongers, L. Factors affecting retr·eival of task-time data from human
store. Report No. 69-50, Office of Naval Research, Department of
Navy, Washington, D.C., Contract No. 233(89), 1969.
:Bourne, L.B., and Dominowski, R.L., in Mussen, P.H., and Rosenweig,
M.R. (Eds.) Annual Review of Psychology. Vol. 23, Palo Alto,
California: Annual Reviews, Inc., 1972.
Brown, D.R., and Hitchcock. L. Jr. Time estimation: Dependence and
independence of modality-specific effects. Perceptual and Motor
Skills, 1963, 16(2), 597-610.
Bruning, J.L., and Klintz, B.L. Computational Handbook of Statistics.
Glenview, Illinois: Scott, Foresman and Co.
Burger, W.J., Knowles, W.B., and Wulfeck, J.W. Validity of expert
judgement of performance time. Human Factors, 1970, 12(5),
503-510.
!Dixon, W.J. (Ed.) BMD: Biomedical Computer Programming.
·
University of California Press, 1974.
Los Angeles:
iFalk, J.L., and Bindra, D. Judgement of time as a function of serial
position and stress. Journal of Experimental Psychology, 1954,
47, 279-282.
i
/Friedman, K.C. A time comprehension test.
Research, 1965, 39, 62-68.
Journal of Educational
1
I
!Garner, W.R., and Felfoldy, G.L. Integrality of stimulus dimensions in
/
various types of information processing. Cognitive Psychology,
I
197o, 225-242.
I
[Guilford, J.P.
Psychometric Methods.
i
i__ ___________ ___
[
.
.
New York: McGraw-Hill, 1954.
48
!Gulliksen, H. The influence of occupation upon the perception of time.
Journal of Experimental Psychology, 1927, 1Q, 52-59.
;Hanifan, D.T., and Sklar, L.B. A generalized maintainability method: I
Rationale and Procedure. Dunlap and Associates, Inglewood, California, 1967.
H., in Abramson, H.A. (Ed.) Problems of consciousness. Transactions of the first conference, March 20-21, 1950. New York:
Josiah Macy, Jr. Foundation, 1951.
~oagland,
:Hunt, E.B., in Kleinmuntz, B. (Ed.) Concepts and the Structure of Memorx _
New York: ~Ji 1ey, 1967.
Jamison, D., Suppes, P., and Wells, S. The Effectiveness of Alternative
Instructional Media: A Survex. Institute for Communicative Research, Stanford University, March, 1973 •
.Jung, J.
Verbal Learning.
New York: Holt, Rinehart and Winston, 1968.
:Knowles, W.B., Burger, W.J., Mitchell, M.B., Hanifan, D.T., and Wulfeck,
J.W. Models, measures and judgements in system design. Human Factors, 1969, 11(6), 577-590.
'LeComte du Nouy, P.
Biological Time.
New York: MacMillan, 1937.
:Leuba, H.R. A study in time perception: An attention model. Unpublished master's thesis. Washington University, Washington, D.C.,
1963.
Levie, W.H. Pictorial research: An overview.
49(2), 37-37.
Viewpoints, 1973, Vol.
Lockhead, G.R. Processing dimensional stimuli: A note.
Review, ~' 1972, 410-419.
Psychological
iLoehlin, J.C. The influences of different activities on the apparent
length of time. Psxchological Monograph, 1959, ~' No. 474.
i
iMeister, D. Comparative analysis of human reliability models, Final
Report. Bunker Ramo, Electronic Systems Division, Westlake
Village, California, 1971.
[Miller, R.B. Psychological considerations in the design of training
equipment. United States Air Force, WADC, TR 54-563, 1954.
!
i
IRoshal, S.M. Effects of learner representation in film-mediated perceptual-motor learning. Technical Report SOC 269-7-5, Instructional
i
Film Research Program, Pennsylvania State College, State College,
Pa., 1949.
i
L_ _________ _
.
1
1
-----~---~---~___j
iRudnick, MF., Porter, M.C., and Suydam, E.L. Pictorial stimulus variables. Viewpoints, 1973, Vol. 49(2), 21-29.
;Shepard, R.N., and Cermak, G.W. Perceptual-cognitive explanations of a
taroidal set of free-form stimuli. Cognitive Psychology, i' 1973,
351-377.
~iegman,
'
A.W. The relationship between future time perspective, time
estimation and impulse control in a group of young offenders, and
in a control group. Journal of Consulting Psychology, 1961, ~'
470-475.
'Smith, R.L., Blanchard, R.E., and Westland, R.A. Acquisition of restore
time data by subjective techniques. Aerospace Medical Research
Laboratory, Report No. AMRL-TR-70-76, Wright-Patterson Air Force
Base, Ohio, 1971.
Spangenberg, R.W. Procedure learning and display motion. Human Resources Research Organization, Fort Knox, Kentucky, 1971.
Stuart, M. Experiments on the estimation of duration.
of Psychology, 1923, }1, 382-388.
British Journal
:Trattner, M.H., Fine, S.A., and Kubis, J.F. A comparison of worker
requirement ratings made by reading job descriptions and by direct
job observation. Personnel Psychology, 1955, ~' 183-194.
!Wallace, M., and Rabin, A.I. Temporal experience.
letin, 1960, 57(3), 213-236.
Psychological Bul-
·White, H.L. Feedback and Transfer of Training in Time Estimation.
Arbor, Michigan: University Microfilms, Inc., 1963.
L ___________
Ann
50
APPENDIX A
Literature Review
LITERATURE REVIEW
[ask Performance Time
The following are examples of the use of distributions of
/time in system design and analysis as discussed by Meister (1971):
Analytical methods of predicting operability.
Users of the
American Institute of Research (AIR) Data Store method combine
task performance time with performance reliability to estimate
:the probability that an operator will correctly perform a given
task.
Designers of the Technique for Establishing Personnel Perfor-
mance Standards (TEPPS) use the same combination to predict both
completion time and successful performance.
The combination is
also part of the input data used by the Critical Human Performance
and Evaluation Program (CHPAE) to predict human performance.
The Berry-Wulff method defines overall system reliability
!by estimating the minimum acceptable performance time, among other
'
:outputs.
The number of acceptable outputs per unit of time to meet
!design expectations is also used to predict task performance and
i
!system success by the Throughput Ratio.
;
Further, the distribution of error as a function of time
I
lis basic to deriving a prediction in probability terms by the
!
~skren-Regulinski
model.
Finally, the total time required to complete a subtask and
I
fthe time required to transfer information between indicators and
!controls are two measures used to compare two or more design varia!
[~ions of the same equipment by the Display Evaluative Index (D~~~:-----~
Simulation methods of predicting operability.
Task performance
time estimates are used by system designers to evaluate overloads
or underloads for one or more operators by the Digital Simulation
[echnique, the TACDEN Simulation Model, the Boolean Predictive
Technique, and the Personnel Subsystem Effectiveness (PSE) model.
Response time estimates are used by the Operations Research
and Control Link Evaluation (ORACLE) to diagnose system elements
that have resulted in a performance imbalance between inputs and
outputs.
Predicting equipment maintainability.
Task performance time
estimates are used for specific purposes, such as predicting airborne
or shipboard electronic system-downtime, and for predicting general
·system-downtime.
'Stimulus Presentation Methods
Although estimating task performance time may be unrelated
to learning to perform the task, the present investigator surveyed
the literature on instructional media to determine if support
exists for using one method over another in an area related to
:time estimation.
Rudnick, Porter and Suydam (1973) made two general
:statements about the advantages of learning from static versus
imotion pictures:
t
1. Motion should be used when the concept being
taught is defined by motion, such as perceptualmotor tasks.
2. Motion should be used when the perceptual-motor
task involves natural human movement.
·--~---·-·---~--~_:
One common theme that does seem to run through the literature
reviews is that, in learning situations, stimulus presentation
methods which involve motion should be used when the subject matter
!involves natural human movement.
If the judge attempts to concep-
!tualize the psychomotor characteristics associated with each task
to estimate the performance time, some medium involving motion
might produce a more accurate estimation than a medium lacking
motion.
I
l--------·-----~
APPENDIX B
Instructions
I
L__._
i
__I
:Instructions
Introductions (all subjects).
"This is an experiment to
jSee how well you can estimate the time that•s required to perform
!a variety of motor tasks, such as flipping switches and positioning
/knobs.
I 1 11 be more specific in a few minutes, but first, I 1 d
like to show you why we•re interested in time estimation.
!film is 10 minutes long,
~nd
This
illustrates some early uses of time-
and-motion studies."
(Shm'i film.)
"As you can see, there was no reason to estimate the time
required to do the jobs, since the times could be measured directly.
·However, now I 1 ll show you some tasks which might be part of a
1
larger, newly designed job, such as the space program, or repairing
fa new model car.
Inaccurate estimates of the time required to
:perform the tasks might result in very expensive cars, or dangerous
'space vehicles."
(Show video tape.)
Video presentation.
"I m going to show you a film of people
1
actually operating a piece of testing equipment.
After you see
the operator perform each task, I ll ask you to make some estimates
1
about the time required to perform it.
Any questions?"
(Video tape of each of the seventeen tasks.)
"How much time do you think it would take the average operator
to perform that task? Another way of looking at it is that 50%
of the operatore would need more time than your estimate to do
the task. (E records time.)
How much time do you think it would
- - - - - - - - - - ---------------
1take the slowest operator to perform the job? Only 5% of the
Dperators would need more time to do the task.
lone together.
Let's do the first
The 'Average Time• to sharpen a pencil is 6.8 seconds.
!That means that the average pencil sharpener will need 6.8 seconds
Ito sharpen a pencil.
However, the slowest time is 8.5 seconds.
This means that now we've included most of the slower people,
:and that only about 5% of all pencil sharpeners will need more
time to do the job.
Now then, do you have any questions?
Fine,
'take your time, but try not to spend too much time with each one. 11
Written presentation.
Operate training.
about).
11
Same, but list of tasks provided.
This is the machine you saw (or read
I'd like you to act as if you were an actual operator.
Here's how to operate it. 11
(E reads instructions.)
11
You may refer to the instructions as often as you need to.
'Now please try it by yourself.
I'll help you if you get stuck. 11
(Stakes one trial.)
11
Fine.
Now, I'd like you to check out these operational
!amplifiers just as if you were an actual operator of the machine.
,I'll tell you when to stop.
Any questions? Fine, please begin,
land be sure to follow the instructions. 11
(Subject operates for twenty minutes.)
Paired-comparison training.
!tasks, arranged in pairs.
I
11
I'm going to show you seventeen
After you've read the descriptions,
i
II want you to tell me which one probably takes more time to perform.
!
/I 1 11 tell you if you're right or wrong.
L _____________ -
---~-______:
!usually tell you how long the correct task takes to perform.
!try one together.
Let•s
Here•s two tasks, flipping a light switch,
:versus sharpening a pencil. Which one do you think takes more
:
!time?
Right (or Wrong).
Now let•s go on to the actual comparisons.
i
~ny
questions?
Fine.
time with each one.
Take your time, but don•t spend too much
Ready?
Begin ...
(S performs paired comparisons.)
If subject does not choose the correct task in the example,
he will be given another, such as,
11
Let•s try another one.
How
about dialing •o• on the telephone, versus turning on a radio
.if your hand was on the knob. 11 Exampl es wi 11 be given unti 1 the
.subject chooses the correct task.
APPENDIX C
Posttest Deviation Scores
__j
~--------------------
----- ---------
------
I
I
I
Posttest Deviation Scores
i
I
Group
Written-P.C.-Video
I
I
I
Written-P.C.-Written
Video-P.C.-Video
Video-P.C.-Written
Written-Op.-Video
Written-Op.-Written
Video-Op.-Video
Video-Op.-Written
.
s
r
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
Task 2
50%
95%
-2.1
1.3
.2
-.7
0
-.6
-.7 -11.8
-1.7 -1.8
-1.3 -1.8
-.3
1.7
.2
-.7
-1.7 -1.8
-1.7 -1.8
-.7
.7
.5
2.0
.2
.3
-2.7 -2.8
.3 -3.5
.8
2.4
.2
-.2
-1.7 -2.0
.5
2.2
-2.7 -2.8
-.2
.2
-1.7 -1.8
-.7
-.8
1.0 -3.2
Task 5
Task 9
50%
95%
95%
50%
-1.9
1.1 -1.3
1.9
2.6
2.2
5.3
-.3
2.1
3.5
-.3
1.1
-3.3 -2.9
3.4
-.8
-3.3 -2.9 -10.8 -5.6
-3.3 -2.9 -2.8
-.6
2.2
5.4
-.3
2.1
.7
3.1
2.7
5.9
.1 -6.8 -2.6
-1.3
-1.3
1.1 -3.8 -1.6
2.6
-.3
3.9
1.1
-5.3 -2.9
1.4
-.8
.7
2.1
2.7
2.9
-7.3 -5.9
.2
2.4
-1.3
.1
.2
-.6
1.2
4.2
4.2
7.9
.2
2.2
1.1
3.4
-1.3
.8
1.1
3.9
1.1
3.7
7.4
-.3
-6.8 -19.9 -2.8 -1.6
-3.3 -2.1
3.2
6.4
-1.3 -1.1
-1.8 -2.6
2.1
-.3
3.2
5.4
-1.3
.1
1.2
3.4
Task
50%
-.1
.9
.7
-3.1
-1.1
-1.1
-2.1
-9
-4.1
-3.1
.9
.4
.9
-1.2
-.9
1.6
.9
-1.8
1.4
-3.1
.9
-1.1
-.1
1.4
11
95%
1.2
2.7
2.4
-3.8
-.8
-.8
1.2
2.7
-4.3
-1.8
2.7
.2
1.2
1.8
1.2
3.8
2.2
-1.2
3.4
-2.8
2.7
1.8
1.2
3.2
Task 13
50%
95%
5.1
TI
6.1
-1.5
3.5
5.1
4.1 -5.9
2.5 -5.9
2.5
2.1
3.5
3.1
6.6
-2.5
3.5
4.1
2.5
3.1
2.5
6.6
3.0
4.1
2.5
5.1
3.5
.1
3.5
5.6
-4.5
7.5
-3.4
4.1
.5
4.7
6.6
1.4
4.0 -2.9
2.5
6.1
.5
2.1
3.5
4.1
2.5
5.1
- ·--··-·---·--·
U1
"I..C
ou
, APPENDIX D
Mean Deviation Scores
L _____________________________,
61
Mean Deviation Scores for All Pretrafning and Posttralning Combinations of Training and Stimulus Presentation
Methods,
Task ?_
Sot
Pt·etraining estimates
Task
-1,62
Video
-1.44
Written
-1.80
Posttraining estimates
Task
-,79
VidPo pretraining
-.70
~ritten pretraining
-.89
Paired compari~or;
-1.01
Video pre
-.77
Written pre
-1.25
Video post
-1.02
Written post
-1.00
Video-video
-.90
Video-written
-.63
-.70
Written-video
Written-written -1.37
-.58
Operate
VIdeo pre
-.61
Written pre
-.53
Video post
-.75
Written post
-.42
Video-video
-.80
VIdeo-written
-.47
WrltL.•·Video
-.70
-.37
Wrl tten-written
Video-video
-.85
-.55
Video-written
Written-video
-.92
Written-written
-.67
Pretra1n1ng-posttraining
method
lask
-.79
Same method
-.86
Different method
-.73
J!~
I_ask 5
50i
95X
!iO%
2~·
~
95%
50%
Mean
952:
,43 2.91 -.43
1.70. 3.97
.25
-.84 / 1.85 -1.11
.24
1.17
-.70
2.14
2.86
1,42
4.10
5.12
3.07
-.17
.54
-.68
1.14
2.08
.20
,84
.85
.82
1.12
.12
.15
.90
-.72
-.13
.37
2.10
-1.80
1.77
2.24
1.30
2.26
2.08
2.43
1.67
2,85
1.50
2.67
I!~LJ!.
5o~
-2,20 -1.35
-1.15 -.65
-3,26 -2.05
.67
1.29
,05
-1,15 -1.69
~.47 -1.71
-1.83 -1.68
-.62 -1.63
.11 -l.cO
-2.75 -2.07
-.17 -.57
-2.42 -2.70
.03 -.30
.30 -2.10
-.37 -2.63
-5.13 -3.30
-1.68 -1.76
-1.12 -2.22
-.92 -1.30
-1.08 -3.05
-.95 -.47
-.13 -3.47
-2.10 -.97
-2.03 -2.63
.20
.03
-.05 -1.88
-.90 -1.53
-1.20 -1.73
-2.47 -1.63
-.59
-1.14
-.04
-1.12
1.02
-.48
1.85
-1.32
1.77
.27
1.93
-2.90
-.06
-3.30
.• 40
-4,10
1.20
-6.97
.37
-1.23
2.03
-2.60
.32
.35
-.43
-2.93
-.63
-1.17
1.03
-4.80
1.44
1.12
1.77
1.20
1.68
1,37
.87
1.03
2,50
,37
-.15
.85
-1.15
-.87 -1.72 -.37
-1.26 -1.76 -1.07
-.47 -1.68
.33
-.20
-.92
.35
-,02
.11
-.13
-1.4fl
-.so
-2.07
,02
P_5.!
2.44
2.57
2.32
3.03
2.07
1.33
3,25
.15
2,90
1.23
3,60
-.93
1.86
3.07
3.32
2,82
3.57
4.07
2.07
1. 57
5.07
3.48
1.65
2.58
2.07
2.44
?..77
2.12
Task 1_!_
-.42
-.64
-.21
-. 91
-l.1!l
-.63
-.63
-1.18
-1.77
-.60
.20
-1.77
,06
-.10
.22·
-.03
.15
-.27
,07
.20
-1.02
-.27
.35
-.77
.56
1.58
1.5o
1.25
1.83
1.10
2.07
1.40
1.60
.48
1.22
1.75
-.10
-.42
-.89
.04
.84
.21
1.48
.?.3
Task_.l;l_
3.43 -.23
.99
4.06 -.14 1.17
.82
2.01 -,33
4,22 -.33 1.32
4.60 -.39 1.59
1.10 -.72 -.13
5.02 -.12 2.19
,6B -1.00 -.72
4.60 -.42 1.83
4,60 -.37 1.35
:! '.7
5.43
.21 2.54
3.03 -3.23 -1.64 -2.80
1.28 2.64 -.14
,66
.11
.75
2.40 3.52
.06 1.76
.17 4.52
,05
.46
2.90 3.43
.12 2.05
-.33 4.60
.27
2.63 3.27 -.11
.33 1.23
2.17 3.77
.66
.21
3.17 3.60
-2.63 5.43 -.09 2.87
2.07 3.93 -.26 1.05
2.42 4.18 -.02 1.29
.21 1.60
2.50 4.52
.03
.10 1.10 -.86
2.11
1.11
3.11
3.21
2.52
3. 91
-.22
-.66
.22
1.05
.66 \
1.47
i:
_j
APPENDIX E
Raw Data
:______ ______ _
____ j
~---------------------·
I
·---·----··-
- ----·------
----·-···
··-·· . -·
.
50th Percentile Pretest Scores
I
I
I
Task
1
*2
3
4
*5
6
7
8
*9
10
*11
12
*13
14
15
16
17
Written-De-Written
Written-De-Video
52
52
51
53
51
53
2.0 15.0 30.0 20.0
8.0 25.0
1.0
5.0
2.0
1.2
.5
8.0
30.0 15.0 90.0 180.0 10.0 300.0
10.0 20.0 40.0 20.0
30.0 120.0
1.0
5.0
4.0
.9
8.2
.8
3.0
7.0
2.0
3.0
2.0
9.0
2.0
5.0
4.0
4.0
3.0
7.0
2.0
4.0
4.0
2.0
3.0
6.0
2.5
4.0
9.0
4.0
7.0
7.0
9.0
2.0
2.0
2.0
5.0
3.0
1.0
4.0
.8
.5
5.0
1.0
3.0
5.0
2.0
1.0
.9
8.0
2.0
5.0
2.0
1.0
1.0 8.0
2.5
4.0
2.0
2.0
8.0
.6
3.0
9.0
5.0
3.0
9.0
.8
2.5 10.0 30.0 70.0 12.0 10.0
3.5 10.0 15.0
5.0
3.0 10.0
*Tasks validated by Burger et al. (1970).
Video-De-Written
Video-De-Video
52
53
51
52
53
51
9.0 12.0
10.0
9.0 10.0 20.0
1.0
.3
2.0
2.0
6.0
5.0
30.0 25.0 60.0 120.0 60.0 180.0
5.0
30.0 15.0 30.0 30.0 45.0
2.0
1.5
2.0
2.0
3.0
.8
1.0
2.0
2.0
2.0
2.0
1.0
4.0
5.0
4.0
1.5
4.0
1.0
2.0
4.0
3.0
2.0
1.2
3.0
4.0
7.0
2.0
3.0
1.5
3.0
1.5
1.5
2.0
1.5
10.0 2.0
.5
.3
2.0
1.0
3.0
1.0
1.0
2.0 15.0
2.0
1.2
4.0
2.0
2.0
6.0
1.2
4.0
2.0
4.0
5.0
1.0
1.0
1.2
4.0
1.5
6.0
5.0
7.0
3.0
3.0
5.0
8.0
3.0 15.0 10.0 15.0
8.0
3.0
3.0
1.5 20.0
8.0
50th Percentile Pretest Scores
Task
1
*2
3
4
*5
6
7
8
*9
10
*11
12
*13
14
15
16
17
Written-PC-Video
Written-PC-Written
S2
S3
S1
S2
S3
S1
7.8
8.0 180.0 30.0
4.5
7.0
2.0
1.5
5.5
5.0
1.5
4.0
20.0 55.0 40.0 120.0 90.0
3.0
5.0 10.0
4.5 10.0 60.0
8.0
3.0
2.0
2.0 10.5
3.0
4.0
7.0
3.5 30.0
3.0
1.0
4.0
5.0
1.5
2.5 15.0 10.0
6.0
4.5
1.5
2.8 60.0
5.0
7.0
5.5
4.0 12.0 10.0
5.0
2.5
6.5
2.0
4.0 30.0 15.0
3.0
5.0
1.8
9.0
5.0
1.0
2.0
5.5
2.0
2.7 10.0 10.0
3.0
5.5
2.0
2.5 10.0
5.0
4.0
3.0
1.5
2.3 60.0
5.0
4.0
5.5
30.0 10.0
2.5
4.7
7.0
4.5
4.0
5.5 120.0 15.0
9.0
7.5
3.0
5.8 60.0 10.0
4.0
*Tasks validated by Burger et al. (1970).
I
I
Video-PC-Video
S2
S3
S1
6.5
2.0
6.0
3.0
5.0
1.0
60.0 10.0 45.0
45.0 10.0 60.0
' 1.0
4.0
5.0
4.0
3.0
.5
2.0
3.0
5.5
1.0
2.0
7.0
4.0
1.0
5.0
5.0
2.0
5.0
5.0
.5
2.0
1.0
2.0
4.0
1.0
3.0
4.0
.5
2.0
6.0
2.0
3.0 10.0
3.0
5.0 120.0
5.0
3.0
7.0
Video-PC-Written
S1
S2
- S3
8.0
6.0
9.0
2.0
1.6
3.0
20.0 60.0 300.0
20.0 45.0 60.0
2.0
1.0
1.5
.8
2.0
3.0
6.0
4.0
3.0
4.0
3.0
1.0
6.0
2.0
3.5
.8
1.8
4.0
3.0
.5
1.0
2.0
4.0
1.0
.5
2.0
3.0
6.0
3.0
1.0
2.5
7.0
4.1
10.0
5.0 10.1
5.6
7.0
3.0
----··
---·-- -·----··- ---------- -·· --··-·-·--····- ·---- .. ---·
.
. - ··--·
-- --
95th Percentile Pretest Scores
Written-0~-Video
Task
1
*2
3
4
*5
6
7
8
*9
10
*11
12
*13
14
15
16
17
S1
1.0
3.0
60.5
25.0
3.0
2.5
5.5
4.0
5.0
3.5
1.9
3.5
3.0
6.3
5.5
4.0
7.3
Written-0~-Written
S2
S1
S3
20.1 45.0 30.0
7.0
5.0
3.0
20.0 180.0 300.0
25.0 70.0 50.0
7.0
7.5
1.5
9.0
5.0
9.0
6.0
6.0 10.0
6.0
5.0
6.0
12.0
6.0
7.5
12.0 50.0
5.0
5.0
3.0
1.3
7.0
5.0
1.5
7.0
5.0
1.8
6.0
5.0
1.5
11.0 10.0
1.3
15.0 60.0 150.0
15.0 25.0 11.0
S2
S3
12.0 30.0
2.0 12.0
45.0 450.0
60.0 180.0
2.0
7.1
3.0 14.0
6.0 11.0
4.0 10.0
5.0 11.0
3.0
8.0
1.0
8.0
6.0
10.0
3.0 10.0
4.0 10.0
6.0 11.0
30.0 12.0
8.0 12.0
Video-0~-Video
S1
20.0
3.0
60.0
60.0
1.2
2.0
1.5
1.8
2.0
1.8
.8
2.6
2.0
1.8
2.0
5.0
2.2
S2
13.0
4.0
45.0
20.0
4.0
4.0
6.0
5.0
5.0
4.0
4.5
6.0
6.0
6.0
9.0
25.0
25.0
Video-0~-Written
S2
S3
S3
S1
15.0 35.0 12.0 15.0
8.0
9.0
2.0
.7
90.0 210.0 120.0 300.0
45.0 50.0 100.0
8.0
3.0
5.0
7.0
3.0
5.0
5.0
4.0
2.0
6.0
6.0
3.0
8.0
4.0
7.0
7.0
3.0
5.0
7.0 12.0
4.0
3.0
3.0 18.0
3.0
3.0
2.0
6.0
1.0
4.0 25.0
3.0
2.0
4.0 13.0
3.0
4.0
9.0
7.0
2.0
2.0
7.0 12.0
5.0
5.0
13.0 22.0
7.0 12.0
13.0 15.0
5.0
5.0
*Tasks validated by Burger et al. (1970).
-·- ·----- ·--··--·---·-
O'l
01
95th Percentile Pretest Scores
Task
1
*2
3
4
*5
6
7
8
*9
10
*11
12
*13
14
15
16
17
Written-PC-Video
S2
S3
S1
10.0
6.5 10.5
3.5
2.5
2.5
25.0 70.0 95.0
7.0 15.0
7.0
3.0
3.5
3.3
5.2
9.0
2.0
5.9
3.0
4.6
6.0
3.0
4.5
3.5
6.9
7.0
3.0
7.0
8.5
9.5
1.5
2.3
6.5
3.0
4.3
6.5
3.0
4.0
3.0
4.5
4.0
3.5
6.5
5.9
6.5
5.0 10.0
9.5
4.0 10.0
Video-PC-Video
Written-PC-Written
S2
S2
S3
S3
S1
S1
300.0 45.0
10~0
3.0
7.0
8.5
4.0
7.0
8.0
22.0
6.0
2.0
300.0 120.0 12.0 120.0 12.5 60.0
15.0 90.0 15.0 60.0 12.5 75.0
5.0
6.0
2.0
5.0
8.0
11.2
60.0
5.0
6.0
5.0
5.0
1.0
25.0
15.0
8.0
8.0
4.0
3.0
120.0
8.0
9.0
2.0
3.0
8.0
7.5
12.2
7.0
1.5
5.0
7.0
20.0
60.0
5.0
3.0
7.0
1.0
13.8
8.0
4.0
3.0
7.0
1.0
15.0 15.0
5.0
2.0
6.5
3.0
15.0
8.0
6.0
2.0
4.0
6.5
8.0
6.0
3.0
8.0
90.0
1.0
60.0 15.0
9.0
3.0
4.0 12.0
180.0 20.0 12.0
5.0
6.0 15.0
90.0 15.0
6.0
4.0
6.0 10.0
Video-PC-Written
S2
S1
- S3
8.3
11.0 12.0
5.0
3.5
4.0
60.0 90.0 600.0
60.0 25.0 180.0
1.5
3.5
2.5
5.0
1.5
4.0
4.0
8.2
6.0
1.5
5.0
5.0
4.0
4.4
8.0
5.0
1.5
3.2
3.0
4.0
1.0
5.0
6.0
1.5
1.0
5.0
5.0
5.0
1.5
7.0
4.0
6.5
9.0
7.0 25.0
13.0
5.5
9.0
9.7
*Tasks validated by Burger et al. (1970).
0
0
~----~~~:.n-Op-~ideo
I
1
I
I
I
I
Task
1
*2
3
4
*5
6
S1
S2
2.o
5.o
1.0
4.0
3o.o 1o.o
10.0 15.0
1.0
9.0
3.0
4.0
6.0
7
2.0
8
2.0
4.0
*9
5.0
2.5
10
3.0
4.0
*11
1.0
5.0
12
1.0
7.0
*13
2.0
7.0
14
2.5
5.0
15
3.0
9.0
16
2.5 11.0
17
3.5 11.0
*Tasks validated by
50th Percentile Posttraining Scores
--- Wri tten-Op-Wri tten
S3
S1
35.o
.6
1.0
.5
6o.o 3oo.o
35.0 200.0
3.0
.5
3.0
.6
3.0
.4
.5
3.0
5.0
1.0
1.2
4.0
1.0
.3
2.0
1.2
2.0
1.1
1.5
.8
5.0
1.2
45.0 30.0
5.0
3.9
Burger et al.
S2
S3
5.o
5.4
1.5
3.0
1o.o 3oo.o
15.0 240.0
1.5
3.0
2.0
3.2
2.0
3.0
3.2
2.0
3.0
4.0
2.0
4.8
1.0
3.7
2.0
2.9
3.0
3.0
1.0
3.1
4.0
3.0
10.0
4.0
5.0
3.0
(1970).
yideo-Op-Written
Video-Op-Video
S1 - S2
S3
S1 - S2
S3
--2.0
5.0
3.0 10.0
5.0 15.0
2.0
.3
.8
1. 5
3.0
4.0
30.0 30.0 45.0 120.0 60.0 180.0
5.0
15.0 35.0 30.0 45.0 45.0
2.0
2.0
8.5
5.0
3.0
3.0
2.0
2.0
2.0
.8
5.0
1.5
3.0
2.0 15.0
2.0
3.0
3.0
2.0
1.5
2.5
1.0 10.0
4.0
8.0
1.5
2.0
2.0
4.0
7.0
2.0
3.0
5.0
1.0 10.0
.8
2.0
.5
3.0
.5
5.0
1.0
2.0
2.0
2.0
5.0
1.0
7.0
2.0
3.0
8.0
2.5
2.0
3.0
5.0
2.0
3.0
4.0
1.5
1.0
2.0
5.0
2.0
3.5
9.0
4.0
2.0 15.0
4.0 10.0
3.0
8.0
5.0
1.5 10.0
4.0
3.0
6.0
...c
~--
--··-··-·-
· - ------------ ·•···
..
. -·· ···-
--
--·
~-----
...
..
-·
.
--·---- ---·--
---
--
50th Percentile Posttraining Scores
I
!
Task
1
*2
3
4
*5
6
7
8
*9
10
*11
12
*13
14
15
16
17
Written-PC-Video
S2
S3
S1
10.0
3.0
4.9
3.4
2.0
1.9
15.0 15.0 15.0
4.5
13.0 11.5
3.6
2.0
2.0
2.0
11.0
1.5
2.0
3.4
4.0
4.5
2.0
2.1
6.5
3.0
4.1
4.0
2.0
2.3
2.0
1.0
1.2
5.0
2.0
2.1
2.0
2.0
3.0
3.5
2.0
1.5
4.5
2.5
2.7
3.4
4.5
5.0
4.5
3.0
4.8
Written-PC-Written
S2
53
51
10.0 60.0
8.0
2.0
3.0
3.0
30.0 60.0 20.0
5.0 45.0 15.0
5.0
5.0
5.0
5.0 15.0
7.0
5.0 20.0
3.0
10.0
5.0
5.0
6.0 16.0
8.0
6.0
5.0 15.0
5.0
3.0
3.0
8.0
5.0
3.0
10.0 10.0
5.0
10.0
3.0
4.0
20.0 15.0
7.0
30.0 60.0 15.0
15.0 45.0 12.0
*Tasks validated by Burger et al. (1970).
Video-PC-Video
S3
S1
52
2.0
2.5 12.0
3.0
1.0
2.0
60.0 10.0 45.0
60.0
1.5 20.0
2.0
1.0
3.0
2.0
2.0
3.0
2.5
7.0
3.0
2.0
1.5
2.0
2.5 12.0
3.0
1.5
3.0
2.0
6.0
2.0
1.0
4.0
1.0
1.5
1.5
3.0
4.0
3.0
1.5 10.0
6.0
3.0 12.0
2.5 15.0
8.0
2.5 12.0
8.0
Video-PC-Written
S2
S1
S3
3.0
8.5
5.0
.8
2.0
3.0
20.0 60.0 120.0
15.0 30.0 60.0
2.0
7.0
3.0
5.0
3.0
2.0
3.5
5.0
4.0
2.0
5.0
5.0
6.0
9.0
4.0
3.0
2.0
5.0
1.5
5.0
1.0
1.5
4.0
6.0
2.0
3.0
5.0
2.0
8.0
1.0
2.0 10.0
12.0
15.0
3.0
2.0
14.0
4.0 15.0
,------------------------··-··---·--·------··------.---
i
····--
···-
-·-····--·· -·
-··--·-·--·-·-
·-----·-----·· --- --· -
···---·
95th Percentile Posttraining Scores
I
Written-PC-Video
I
i
Task _2!_
1
15.0
Written-PC-Written
Video-PC-Video
Video-PC-Written
S2
4.0
S3
7.0
S1
15.0
S2
90.0
S3
10.0
S1
3.0
S2
3.5
S3
14.5
S1
7.0
_g_
4.0
S3
12.1
*2
3
4.5
16.5
3.0
20.0
3.2
23.7
15.0
35.0
5.0
90.0
5.0
25.0
1.5
120.0
3.0
12.0
5.0
55.0
5.0
60.0
2.5
90.0
1.2
300.0
4
*5
6
5.5
4.0
12.0
17.0
2.5
2.5
22.0
3.0
3.1
8.0
8.0
8.0
60.0
8.0
20.0
20.0
8.0
9.0
120.0
3.0
3.0
2.0
2.0
2.5
25.0
5.0
5.0
50.0
4.0
6.0
45.0
2.5
4.0
120.0
8.0
5.0
7
8
5.6
6.5
3.0
3.0
5.5
3.2
8.0
15.0
30.0
8.0
5.0
7.0
5.0
3.0
3.0
2.0
8.5
10.0
5.0
7.0
5.0
3.0
6.0
7.0
*9
10
*11
12
7.5
5.5
3.0
5.5
4.0
3.0
1.5
3.0
5.9
4.0
1.8
4.0
6.0
8.0
8.0
10.0
15.0
20.0
5.0
8.0
10.0
8.0
5.0
5.0
4.0
3.0
3.0
6.0
3.5
2.0
1.5
2.5
12.0
5.0
8.5
3.0
11.0
6.0
6.0
7.0
5.5
4.0
1.5
2.0
8.0
5.0
4.0
6.0
*13
4.0
3.0
4.0
15.0
15.0
7.0
6.0
2.5
5.0
6.0
2.5
5.0
14
15
4.5
5.5
3.0
3.5
2.5
4.1
15.0
25.0
5.0
20.0
6.0
9.0
4.0
8.0.
2.0
3.5
12.0
14.5
10.0
13.0
1.5
2.5
5.0
12.5
16
17
4.5
5.5
5.5
4.5
10.5
6.3
35.0
20.0
90.0
60.0
20.0
15.0
3.5
4.0
20.0
15.0
18.0
16.0
4.5
6.0
4.8
20.0
Burge~
et al. (1970).
*Tasks validated by
--.-
..
- ·-·--
-
...
- -····-
10.0
10.0
-·· ··- ..
~
u:l
r··--- ---··------------------------------- ---- -. - ·--· --
I
----···-·-~
95th Percentile Posttraining Scores
Written-0~-Video
Task
1
*2
3
4
*5
6
7
8
*9
10
*11
12
*13
14
15
16
17
51
3.5
3.0
60.0
20.5
3.0
6.0
5.5
6.0
6.5
6.7
3.0
3.0
4.0
5.0
8.3
7.9
8.4
Written-0~-Written
53
52
51
7.0 50.0
1.0
6.7
.8
6.0
12.0 120.0 350.0
20.0 60.0 350.0
11.0
5.0
.9
5.0
1.0
6.0
8.0
5.0
1.0
5.0
6.0
.8
1.5
7.0 10.0
6.0
6.0
1.5
6.0
3.0
.4
3.5
1.5
9.0
3.5
1.6
9.0
1.2
7.0
3.0
11.0 10.0
1.8
13.0 60.0 40.0
13.0
8.5
5.0
53
52
10.0
7.8
3.0
5.2
45.0 450.0
60.0 360.0
4.0
4o3
6.0
4.5
6.0
3.2
5.0
4.5
6.0
5.5
4.0
6.2
2.0
5.4
4.5
4.2
5.0
4.4
2.0
4.3
6.0
4.0
40.0
5.2
8.0
4.1
*Tasks validated by Burger et al. (1970).
!
!
I--
Video-0~-Video
51
8.0
1.0
60.0
60.0
4.0
1.0
3.0
2.0
2.0
1.0
.8
2.5
2.5
1.5
3.2
3.0
3.0
52
20.0
6.0
39.0
40.0
25.0
8.0
20.0
15.0
11.0
10.0
7.0
12.0
12.0
10.0
10.0
25.0
15.0
Video-0~-Written
sr
53
52
- 53
5.0 17.0
4.0
7.0
4.0
.8
5.0
3.0
60.0 210.0 120.0 300.0
8.0
45.0 70.0 100.0
3.0
6.0
5.0
7.0
3.0
3.0
2.0
4.0
5.0
6.0
5.0
3.0
5.0
2.0
7.0
3.0
6.0
4.0
3.0 12.0
5.0
2.0 15.0
3.0
6.0
3.0
1.0
1.5
9.0
5.0
2.0
3.0
7.0
5.0
4.0
3.0
7.0
4.0
4.0
3.0
6.0
6.0
4.0 15.0
7.0 12.0
6.0 17.0
5.0 10.0
8.0
6.0