i
CALIFOIUHA STATE UNIVERSITY'· NORTHRIDGE
A ,,Study of Inter and Intra-Judge Reliability
in Assessment of Movement Behavior
of Children During Play
A project submitted in partial satisfaction of
the requirements for the degree of Master of
Science in
Recreation and Leisure Studies
by
Marlene Berger .,//
Morris
June, 1977
The Project of j,Vlarlene Berger Ho:r.rts is approved=
Dr .. Isa.oe1le
C'orr.:mittee Chairman
{Date)
California State University, Northridge
1977
ii
ACKNOWLEDGMENT
I would like to extend appreciation to my advisors,
Isabelle Walker, Vel Moore, and Rose Bromwich, and to my
friends, Scott Needham, Susan Cantor and Lee Morris for their
help in making this project possible.
I would like to
acknowledge and thank the four judges who volunteered their
time and energy so that data could be collected and the
-
Department of Movement Therapy at Immaculate Heart College
for their assistance and sharing of expertise in the field
of movement analysis.
I would also like to acknowledge
the works of Rudolf Laban, Marion North, Martha Davis and
Elissa White,
~Ld
hope that in the application of some of
their ideas to the study of children during play I have done
those ideas some degree of justice.
iii
TABLE OF CONTENTSPage
APPROVAL PAGE .
. ii
ACKNOWLEDGMENT.
iii
ABSTRACT
. . . . . . . . . . . . . . . . . . .
vi
CHAPTER
I
. II.
INTRODUCTION
. 1
Introduction and Justification for Study .
Problem, Purpose, Hypothesis .
. . . .
Conceptual Assumptions, Methodological
Assumptions, Limitations of Study .
Definitions of Unique Terms .
. . .
. 1
. 5
IV.
7
REVIEW OF SELECTED LITERATURE .
Summa'ry .
III.
6
9
. 17
METHOD . . . . . . . . . . . . . . . . . . . . 20
Experimental Design. . . . . . . . .
. ..
Selection of Subjects: Experiments I & II .
Selection of Subject: Experiment III
.
Selection of Judges . . . . . . . .
Instrumentation . . . . . . .
Procedure, Experiments I & II .
. ..
Procedure, Experiment III . .
. ..
Data Collection Techniques . . . . . . .
.
Statistical Procedures . . . . . . . . . . .
31
32
33
35
RESULTS . .
36
Statistical Considerations for Treatment
of Data, Experiments I ~ I I . . . . . .
Results and Discussion, Experiment I . . . .
Results and Discussion, Experiment II .
.
Review and Justification for Experiment III .
36
39
42
45
iv
20
20
22
22
'"'7
t.. ...
CHAPTER
IV.
v.
Page
Results ~nd Discussion, Experimen~ III,
part 1 . . . . . . . . . . . . .
Results and Discussion, Experiment III,
Part 2 . . .
Summary Tables .
48
51
CONCLUSION
54
46
Summary of Entire Study . . . . . . . . . . . . 54
Implications . . . . . . . . . . . . .
56
Recommendations for Future Study .
. . . 56
BIBLIOGRAPHY .
60
APPENDICES .
63
Score Sheets for Experiment I . . . . . . . . 64
Score Sheets for Experiment II . . . . . . . . 66
Score Sheets for Experiment I I I . . . . . . . 6 7
/
v
ABSTRACT
"A Study of Inter and Intra Judge Reliability in Assessment
of Movement Behavior of Children During Play.''
by
Marlene Berger Mortis
The purpose of this study was to explore the inter and
intra judge reliability of four judges in the observation
of movement patterns of behavior of children during play.
It was hypothesized that the observed results would not
differ significantly from results which were obtained if
scores were assigned at random.
Three experiments were designed in order to test the
hypothesis.
Experiment I consisted
of a comparison
of the scores given to ten children through the observations
made by two independent judges.
used.
Two pairs of judges were
Experiment II consisted of a comparison of the
observations mad of four children by all four of the
judges.
Experiment III consisted of a comparison of the
observations of three movement routines with the observations of an exact replication of these one hour later.
The scoring of the three short movement routines was performed by the investigator.
Experiment III was used in
order to test intra as well as inter judge reliability.
vi
The score sheet consisted of 20 items.
The items were
selected from a system of movement analysis developed by
Rudolf Laban and his associates.
The result of Experiment I
showed~
98.5% probability
that the observed results were not random.
The results of
Experiment II showed an 83% probability that the observed
results were not random.
The result of Experiment III,
part 1 showed a 99.9% probability that the results were not
random, and the result of Experiment III, part 2 showed
an 86% probability that the observed results were not random.
In conclusion the Null Hypothesis was rejected.
The study therefore indicates that independent judges
can reliably make repeated observations of movement patterns
of children during play and of structured movement routines.
/
·CHAPTER I
Introduction and Justification for Study:
The child has long been of interest to people who wish
to understand the nature of man.
In this study I wish to
look at just one aspect of the child - his movements.
Movement is action; it is change; it is initiation; it is
feeling; it is life.
Every person who has full use of his
body parts chooses to move in a manner which is unique and
individual.
Heredity, culture, physical trauma, emotional
trauma, training, and motivation are only a few of the factors which are involved ·in determining the development of
each person's individual capacity to move, his style of
movement.
I
''The manner in which the individual prefers to move
will influence the total effort he exerts and the
nature of the observable pattern elicited ... The
available research indicates that there is a remarkable
consistency in this kind of personal performance
quality, both in action patterns in which the total
body is moving through space, in limb movement, and
in tasks involving smaller amounts of space ... These
personal preferences (termed individual preferences,
in movement, personal equations of performance, or
individual biases in movement behavior), taken
together, may be indicative of personality expressed
in movement" (Cratty, 1973, 181-182).
If these statements are valid, then it may be possible to
gain a better understanding of children (and therefore
adults) through a study of movement patterns* (see footnote on next page) and preferences.
1
This may be best
2
be achieved through observation of children who are moving
in an inner-directed, voluntary manner - children, that is,
at play.
I suspect that there are consistencies in the movement
behavior of children both during play and during more
structured activities.
However, because play is self-
motivating, inner -diTected behavior (Caplan, 1974, xii),
it is conceivable that personal preferences for a particular movement pattern becomes more pronounced during play.
This would occur because a child at play is not concerned
with meeting the demands of adults.
He/she is more likely
to exlubit those patterns of movement with which he is
most comfortable, moving not in response
to an external
directive, but in response to internal motivation and
preferences (L.e. no one is telling him to swing his arms;
he swings them just because it feels right).
observations of normal*
Through many
children during play, I anti-
cipate that a child's individual range of movement can be
determined and that a norm for movement behavior can be
established(by norm it is meant a standard or average range
which encompasses the movement patterns of the majority of
children in various age groups).
Such a movement behavior
Movement patterns are observable, consistent qualities of
motor performance which the individual exhibits in using
his body while moving through space, using time, weight,
and energy.
Refers to the conventional dictionary definition of the
word normal.
3
norm would tell us how many movement qualities or elements
need to be present in order that a child, while at play,
may be evaluated as having a normal or abnormal range
of movement.
Analysis of movement in play behavior may therefore
provide a valuable diagnostic tool to assess the degree
to which a child maximizes his/her physiomotor potential.
For example, movement rigidity implies a lessened ability
to change, demanding predictable, recurring responses even
though the responses may not be appropriate for new situations.
If a child's play behavior shows
move~ent
pre-
ferences which are too rigid, to inflexible, or too predictable, indicating that the child has little freedom in
his responses to new situations, then remedial work may
be necessary.
.
/.
That is, if a child's movement patterns are
not broad enough so that he can respond to new situations
appropriately, a program of play or therapeutic recreation
may be helpful in re-exposing him to unused and possibly
enriching patterns of movement.
A therapeutic recreation
program is a sequence of activities which are specifically
designed for the achievement ·of the goal of facilitating
change in behavior through play experiences.
Such a
program would be valuable because it could structure play
activities or self rewarding experiences for the child,
with the purpose of restimulating his natural tendency to
function more fully and more flexibly within his environment.
4
The conceptual framework, then, is threefold.
First, that an individual's movement patterns probably
-
do express his personality (Cratty, 1973), his life experiences and how he has integrated those- experiences into
his self image.
The way he chooses to move reflects the
personal preferences that probably will be involved in all
his cognitive, affective, and psycho-motor decision making
throughout his life.
Second, that most people have the physiological ability
to move fully, but that we prefer to move in more limited
ways due to cultural restrictions, lack of necessity, or
trauma.
By 'moving fully'
it is meant the capacity to
achieve a maximum level of motor performance.
Third, that there is a direct correlation between a
person's ability
to respond physically to stimuli and his
;.
ability to respond emotionally, intellectually and socially.
Richer (meaning more abundant) physical responses indicate
rich~r
emotional, intellectual- and social responses, all
of which are necessary if an individual is to be able to
function optimally in our society.
Therefore, physio-
motor functioning at an optimal level is the goal.
A
program of therapeutic recreation, as was described earlier
might be used in achieving this goal.
In conclusion, the conceptual framework presented in
this chapter will hopefully serve as an introduction to the
need for studies of movement behavior.
5
Problem
The problem, then, under investigation here is the
development of an instrument which will allow for -reliable
observation of movement behavior in children.
A valid
instrument would establish guidelines for focusing attention
on specific aspects of movement behavior.
It also would
establish viable parameters (set by the designer) so that
movement can be seen consistently by independent observers.
Purpose
The purpose of this study is to assess the reliability
between judges 'in their independent observations of movement behavior of children during play, to identify
specific
components of physiomotor movement.
£!ypothesis
1.
/
Movement behavior can be observed and recorded by
independent observers.
2.
The level of agreement between observers will be sig-
nificantly greater than if the scores were assigned at
random.
For support of the Null ·Hypothesis the statistical
analysis of the data would reveal no .significant deviation
from a random mean established for a two-choice test,
filled in by 2 or 4 observers.
Deviation for the experi-
mental sample must be one or more standard deviations away
from the random mean in order to be significant.
6
Conceptual Assumptions
1.
Individuals exhibit movement preferences or patterns
that are observable in-their behavior.
2.
Play is a natural phenomenonwhere unrestricted movement
is manifested (Loizos, 1966, 7).
Methodological Assumptions
1.
The judges understand the unique terms.
2.
The judges understand the test or score sheet.
3.
The judges are honest in the use of the test, both when
they are being compared to themselves and when they are
being compared to another judge.
4.
The judges must be aware of theit own preferences for
certain movement qualities which tend to distort kinesthetic perception and reproduction of another person's
movements. Th1s means, for example, that a person who has
difficulty
touching~th±ngs'lightly ~ight~have
difficulty
in perceiving lightness in others, or just the opposite may see more lightness than what is actually present,
because it is a difficult action for the judge to perform.
Limitations of the Study
The major limitation of the study is that it is an
exploratory one intended as a pilot project, in order to
discover problems in the testing, observing and scoring of
movement behaviors and problems inherent in the selection
and training of judges.
The study is not designed to
make adeauate assessment of a child's total renertoire of
movement.
This studY will onlY explore whether indeuendent
observers can see the same movement behavior occurring during
the same small slice of time.
Other limitations of the study are the small sampling,
the small number of judges used, and the lack of time in
w·hich the judges are allow-ed to practice making specific
movement assessment.
The small sampling, and the small
number of judges are limitations because they limit the
•'"·-
quantity of data which can be collected.
The lack of prac-
tice time limits the amount of knowledge or feedback that
the judges can receive while becoming more proficient in
the use of the instrument in their task of observing physicmotor performance.
The items
O)l
the test demand that the observers make
iudgements based uoon their knowledge and exuerience of
the quality of human movement.
Therefore. the efforts of
the iudges measure qualitative data only.
Definition of Unique Terms
The unique terms employed in this study come from a
system of movement analysis called Effort/Shape, invented
by Rudolf Laban (1947 ) and further refined or expanded upon
by others who worked with him.
In order to fully under-
stand the terms, personal practice, experience, and feedback from a trained instructor are necessary.
The defi-
nitions of the terms will be presented in discussion of the
8
instrument used by the judges (see
page2~
).
In summary, the purpose of this exploratory study is
to discover if movement behavior patterns of children during
play can reliably be observed by indepe-ndent judges.
A
relationship between movement behavior and personality has
been suggested in a conceptual framework with the intention
of providing a context
of the study.
/.
for~the
reader within the limitations
CHAPTER II
Review of Selected Literature
Five studies have been chosen for review in this
chapter.
These particular studies were selected because
they all were concerned with the problem of analyzing movement behavior.
A summary will then follow, stressing the
relevance of these studies to the one explored in this
study.
North (1975), in her investigation of personality
assessment through movement, proceeded to validate a system
of such assessment using movement as the medium.
The
subjects were twelve children (six boys and six girls)
between the ages of eight and nine,
s~lected
at random
I
from a classroom of
twenty~six
children.
The children
were from an underprivileged economic area.
The method used for validating this system was comparison.
The movement assessments were compared to teacher
assessment of child's personality, I.
Q. scores, Apper-
ception test scores, Educatio.nal Psychologists' reports,
and the Child Scale B (test for maladjustment).
It was hypothesized that there would be a correlation
between the assessments made through the observations of
each child's movements and the assessments made by the
other methods of evaluation.
9
10
The observations of each child's movements were focused
on recording efforts, body shape, phrasing, shaping,
adapt~ility,
and characteristic patterns of each child';s
movement behavior.
No statistical analysis of the data collected was done.
Individual reports on each child were compiled so that
the reader could compare for himself the assessment through
movement, the class teacher's report, and comments from
Child Scale B.
However, there seemed to be a high degree
of agreement between all of the assessment methods used.
The highest degree of agreement occurred between the
movement observation assessment and the teacher's assessment,
because the evaluation form the teacher used was designed
so that the experimenter and the teacher would at least be
looking for the same qualities.
/
In conclusion, this study showed that it is feasible
to use the method of personality assessment through movement as a tool for diagnoses and advisement, and that such
a tool can be used to great advantage -- especially when
used as one of a group of diagnostic techniques.
The second study by Kestenberg (1965) was a pilot to
establish guidelines for the classification of movement.
The study consisted of movement behavior observations and
the collection of general behavioral data for three children.
The children were studied for a period of ten years.
Ob-
servations were made regularly between birth and two years,
and then several times during the years of latency.
11
ob~erving
Five observers were used, with each one
a different time period.
were
descriptive~
"Early recordings of
Lat~r r~cordings
during
movem~nts
consisted of
fi~~-
-
hand tracings of the rise and fall of the flow of movements.
Eventually the movement assessment method, as
originated by Laban and developed by Lamb, was used"
(p.
4) •
It was hoped that there would be a correlation between
behavioral data collected and the movement assessments.
The results of the study showed that:
"Rhythms of infantile movement seem to be determined
by a congenital pattern.
Preferences for certain
rhythms may persist into adulthood even though movement becomes more complex as the nervous system
matures. As the child grows up, fantasies provide
content, add purpose, so to speak, to forms of excitation and discharge congenitally determined" (p. 6).
The results also showed that modifications in the orginally
observed rhythms occurred due to maturation and to interactions with persons important to the children, but that
preferences for particular motor patterns were not eradicated.
Kestenberg concluded that there is a "correlation
between the childrens' preferied rhythms of motor discharge and their specific drive endowment" (p. 31).
The
longitudinal study implies that originally preferred
rhythms persist throughout one's life.
An unexpected result of the research revealed that
two of the children functioned better and enjoyed things
12
more when they were free to use their originally preferred
rhythms.
The third study, by James (1931) was concerne-d with
the meaning of body postures.
-
The three-part study
included observation and reporting on photographs of
postures.
The photographs were shown to three judges.
The content of the photographs were determined through
movement of the head, torso, feet, knees and arms of a
mannikin which was dressed in plain white shorts and a
plain white shirt.
face.
A black mask was placed over its
The mannikin was photographed in a total of 347
different postures.
The purpose of the first part was "to determine how
far bodily posture (apart from mimicry and gesture taken
in the technioal sense) may be expressive" (p. 407).
Judges were asked to observe each of the 347 different
postures for thirty seconds and then report on the meaning,
attitude, or significant part of the posture (if there was
one).
The results of this experiment showed that the observers found a great number 'of the various postures
expressive of a mental state or function (thought or emotion).
Occasionally the postures were interpreted as an
expression of a kind of behavior, i.e. the figure was
regarded as a process of movement.
No quantitative study
was done comparing the relative agreement of the observers'
13
descriptions of particular postures.
James' conclusion
was that the postural expression is not specific, but is
more general in nature- (p. 434) .
The purpose of the second part was to determine the
nature of the experience for the observers.
In this expe-
riment, observers were asked to look at photographs of
thirty postures (for which a high level of agreement had
been found in Experiment I) and report on the emotion
or attitude each posture expressed.
The judges were
standing while watching the photographs.
The results of this experiment were that "concommitant sensory, imaginal or affective processes correlate
lowly with the verbal report of the expression'' (p. 420).
The statistics proved that "There can be no question that
the fulfillment of the instruction to find an expression
1-
in the posture requires the observer to place the figure
in a situation before the posture becomes meaningful"
(p.
420).
The purpose of part 3 was to investigate the relation
between the way the posture was seen and the way it felt.
Observers were asked to look .at, and then assume, with
eyes closed, the pose of the photograph of each posture
and then report the meaning of the posture assumed.
The results of part 3 were:
on SO% of the reports
the same generic meaning was given to the posture as was
reported in Experiment I.
In the remaining cases, where
14
differences occurred, the difference was that in Experiment
III the posture was reported as an expression of a movement
rather than of an emotion or mental attitude.
of the reports differed because of the
About 10%
~ifference
between
"the posture as seen and the posture as felt" (p. 423).
In about 25% of the observations "the difference was
conditioned upon the different setting in which the
observer placed the two postures" (p. 423).
James concluded that the predisposition of the
observer to judge a posture as being an attitude or an
expression of movement may be conditioned by "ll) instruction; (2) habituation; (3) the situation or surroundings
in which the posture appears; and (4) by the nature of
the posture itself" (p. 424-S).
The fourth study is entitled "A Study of Autistic
/.
Gestures in Elementary School Age Children," by Maginnis
(1956).
The purpose of this study was to (1) establish
a reliable and objective method of studying autistic
gestures, and (2) to analyze such gestures for organizational form or pattern.
"Autistic gestures are defined as the bodily habit
forms of movement, referred to in some studies as
nervous mannerisms or nervous habits" (p. 1).
"Autistic gestures are not thought of as tsolated.
items specific to eath individual, but rather as a
biological function first, the individuality of which
may be both culturally and psychologically determined"
(p. vi).
The method used for the collection of data was an
open checklist to record an observation of each child for
15
an interval of ten minutes.
Reliability of observers was
established through a training and testing process which
involved repeated observations and scoring of autistic
gestures seen in a film.
301 elementary school children were the subjects
involved in the study.
Three more observations were made
of 40 children who were randomly selected from the 301 in
the total sample.
Analysis of the method used revealed reliability 1n
the 10 minute observation technique, established the randomness of the sample of forty; and showed that the 301
children originally selected were a representative sample
for all children of elementary school age.
Analysis of data revealed "correlations between the
size of the gestural
field and sex, intelligence quotient,
/
.
sociometric status, and age" (p. 98).
Maginnis concluded that autistic gestures can reliably
be observed and that there is a statistical correlation
between the size and content of the child's gestural field
and his age, sex, intelligence, and participation level
in the classroom.
The last study, by Kalish (1976), also involved
autistic as well as normal children.
In this study Kalish
attempted to present a new scale for assessing the movement behaviors of atypical children.
The purpose was to
collect normative data for support of the theoretical
16
assumptions made in a Behavior Rating Instrument for
Autistic and other Atypical Children (BRIAAC), which was
developed by the author.
The subjects for this study were 195 normal children
(Group I) and 75 autistic children (Group II). The subjects
in Group I were selected on the basis of matching sex,
age, and race, so that tour groups could be composed.
The subjects in Group II were selected solely on the basis
of an autistic diagnosis.
THe children in both groups were scored with specific
data collected on (1) movement, (2) vocalization and (3)
psycho-sexual Scales and total BRIAAC Cumulative Scores.
These scores were determined through two hour observations
of each child by a trained and reliable BRIAAC rater.
Inter-judge reliability was achieved through a testing
process with the investigator.
Fou~
raters were used.
The hypothesis of this study was that there would be
significant differences in the scores of the normal chilren and the autistic children, that the scores of the
normal children would be highly correlated 1vith chornological age, while scores of the autistic children would not.
The results showed that there were significant differences (p<.OOl) between these two sets of children.
Further, the scores of the normal group were highly correlated with age (p<.OOl), while those of the autistic
children were not.
The results of this study suggest that
it is feasible to use these scales as a "screening
i7
instrumenta for an objective evaluation of pre-school children in
selected ar·eas.
The studies done by Norths Jamess Kestenberg, Maginnis and Kalish
an
suggest that movement behavior of individuals can be studied.
These studies can be used for
di~gnostic
purposes so that a fuller
understanding of the person can be achieved.
The Nor·th study is re 1 event to the present study for the subjects
were the same age and were observed using the same system for movement
d2.scr·ipt·iorL
The North study, however, is dealing with the correlation
betvteen movement assessments and other types of assessments.
The pre-
sent study is attempting to establish the reliability of the judges
who ar·e making the movement assessments.
jv1any impl·ications can be dravm from the Kestenberg study.
Her
statement that childt'en learn and enjoy more when they are f;"ee to use
th?:ir or··igino.l1y preferTed rhythms is particularly important for the
field of recreation.
ferted
:~hyth:llS
progn1.m,
l'c:t·dh~r
!Wogr·am~ ca.n
be
This statement implies that if a child's pre-
can be identified,, then an individualized recreation
than a standard and perhaps rhythmically inappropri.ate
de~>'fgned
in order to increase that childs ability to
enhance the potential quality of life.
A study of the indivtduality
of rnoveme.rrt can a·td tn the development of more effective recreation
pr0gramming so that recreation professionals will be more capable of
1112~:-ting th:..: sp9.c:idl demands of the clients they serve~ thei~efore aiding
clients in
achi~ving
personal growth towards optimal functioning.
18
James' study seems to suggest that observers cannot
look at a posture without a context.
Individuals must, in
their imaginations' place. the posture in
a
si tua ti_on. in
order for the posture to have meaning. - James found that
errors in reporting on the expression of a posture occurred
because of the difference between the way a posture was
perceived visually and the way it "\'las perceived kines.:.: ,_;
thetically.
This observation is relevant to my study because my
judges will be sitting while making their observations.
This will not allow them to experience kinesthetically
the movements which they are scoring.
It is unknown how
this will effect the reliability of the judges.
The results of James' study also indicate that it
is difficult to evaluate the meaning various judges place
/.
on a posture.
These conclusions support the assumption
of this study - that the judges employed must be welltrained, so that their own personal preferences do not
interfere with their making of movement observations.
Both Maginnis and Kalish's studies showed the necessity for judge reliability and how that reliability could
be established.
They developed tools which would measure
movement both quantitatively and objectively.
The metho-
dology to be used in my study follows theirs, but the
instrument used may be more subjectively oriented than
either that of Kalish or Maginnis.
It is the position
of this investigator, however, that with less training, a
19
' .
a judge can be taught to use the score sheet as a sensitizing and insight-gaining device.
I
-
~
~-----·-
CHAPTER III
Method
Experimental Design
The experimental design of this study consists of
three components.
These sub-experiments will test for
either inter or intra-judge reliability.
Inter-judge
reliability will be the amount of agreement between
observers.
Intra-judge reliability will be the amount of
agreement with self, or the ability to make repeated observations of the same movement routine with a high degree
of consistency or agreement.
The purpose of Experiment I will be to test for interjudge reliability.
It will consist of a series of ten to
fifteen minute observations, which will be made of each
child at play by two judges.
The two judges will be
looking at the same child for the same time period, but
will not discuss their observations.
Two teams of two
judges each will be employed.
The purpose of Experiment II is also to test for
inter-judge reliability.
The same four judges will be
involved in Experiment II as were involved in Experiment I.
Experiment II will consist of a series of ten to fifteen
minute observations which will be made of each child by
four observers.
All four judges will be looking at the
same child for the same time period, but will not discuss
20
.21
their observations.
This experiment will take place one
week after Experiment I.
The children observed in Experi-
ment II will be different than those
ob~erved
in
E~periment
I.
Experiment III will consist of observations of a
three-part movement routine which will be scored separately
by each of the four judges.
These routines will be la-
belled routine a, b, and c.
One hour later the three rou-
tines will be repeated and scored again.
The data from
Experiment III will be dealt with in two ways.
The first
method (called part 1) of Experiment III will compare the
observations of three movement routines with the observations of an exact replication of these one hour later,
in order tq test for intra-judge reliability.
The compari-
son will determine the number of changes made by the judge
in the scoring of the same routine one hour later.
The
second method in which the data will be analyzed (called
part 2) of Experiment III will consist of a comparison
of observations made by all four of the judges of the
identical movement routines one hour apart.
This experi-
ment will test for inter-judge reliability.
Selection of Subjects:
Experiments I and II
Subjects selected were students from a normal fourth
grade elementary school classroom.
Seven boys and seven
girls were picked randomly from twenty-six children,
playing or watching the game that the whole class was
;;"
____
-------------
22
involved with on the playground.
Their play hour was from
1:30 to 2:30 in the afternoon.
Selection of Subject:
Experiment III
I selected myself as the subject fur Experiment III.
This methodological decision is based on this investiga-
.
tor's extensive professional training in movement and
movement analysis.
This insures reproduction of the same
movement routines with a high degree of consistency between
repetitions.
This consistency was necessary in order to
insure that the changes the judges made in the scoring of
the routines was due to a lack in their reliability,
rather than to a change in the actual routine which they
observed.
Selection of Judges
Four judges 'vere selected from a list of all first
or second year graduate students in Dance/Movement Therapy
at a college of national prestige.
The four judges have
had approximately the same amount of experience in Effort/
Shape analysis and observation techniques, trained under
a certified Effort/Shape instructor.
Instrumentation
The instrument devised for the study was a score sheet
drawn from a notation system developed primarily by Rudolf
Laban, his associates, and students.
The score sheet
consisted of twenty items to be circled when the observer
23
saw the child demonstrate that movement.
Space was pro-
vided for comments, but these comments were not included
in the twenty i terns. Comments were used bi the exp.e.rirnenter
only as insight about the judges.
The following are the terms included on the instrument
or score sheet which the judges used while observing the
movement behavior of children during play, and the movement
patterns demonstrated in the three routines of Experiment
II I.
Quantitative aspects of movement are mechanically
oriented, concerned with degrees and measurable amounts.
They can be described in terms of how much time, degree
of strength, and the measurable number of steps or changes
in direction._,
It tells one what has been done.
qualitative aspects of movement are concerned with the
I
inner content which colours what is done.
They can be des-
cribed as "the moving person's attitude towards" (North,
1975, p. 230).
Effort,to Laban,is stress.
It is expressive behavior.
It is the intensity of discharge of kinetic energy, the
inner drive, impulse of irnpac.t of movement" (Davis, 1973,
p. 36).
Effort is the "inner quality and intensity of
emotion, the liveliness of movement and the readiness to
change 11 (Davis, 1973, 32-33).
"A person's efforts are vi-
sibly expressed in the rhythms of his bodily motion" (Laban
& Lawrence,
1947, xi).
24
Effort graph consists of motion factors and movement
elements which can be written in symbols.
The symbol "/ 1,1
in the notation means some effort is present.
Motion factors are space, weight, time and flow.
Each motion factor is divided into two elements or parts
called movement elements.
space
time
weight
flow
Movement elements are
~
/
-
/
of the motion factor symbol,
depending upon whether the movement "yields-to or resists
space, weight, time or force" (North, 1975, p. 232).
bound
~/_~._._
direct/
free
__;/"
flexible )
light
strong
k
r
sustained
sudden
I
-<.:···
- "'~-- --
·---
-
-·
-
25
flexible
light
free
bound
sustained
sudden
strong
Flow is how you are.
"It is the eas.e or restraint
of the action" (North, p. 231).
"The motion factor of
flow is associated with precision, emotional feeling and
relationships" (North, p. 241).
flow:
free flow, and bound flow.
There are two types of
The degree of flow
depends upon the amount of simultaneous contraction of antagonistic muscles (Kestenberg, p. 522).
Free flow is the quality of yielding, abandon, fluency.
It can be characterized by behaviors such as a wild
slashing in anger or a loose
~imbed,
free swinging walk.
Bound flow is the quality of control, restraint, or
caution.
It can be characterized by behavior such as
threading a needle or a cautious and reluctant walk towards
a feared person.
Space is an attitude or state of being (or not-being)
attentive.
It is the intensity of awareness or readiness
_l
2()
to interact.
"It is concerned with the kind of concen tra-
tion or focus in space" (Dell, p. 30).
involve motion into an environment.
It does not always
Space- can be used
with flexibility or directness.
Flexibility is an indulging or indirect use of space.
It is characterized by behaviors such as "body twisting
in embarrassment, or avoidance, wringing a cloth, or a
meandering walk" (North, p. 233).
Directness can be described as the body resisting
using space.
It is characterized by behaviors such as
"precise placement of an object, aiming at a target, or a
penetrating look" (North, p. 233).
Weight or force is a quality of overcoming or giving
in to the
~eight
of the body.
Weight can be used with
lightness or with strength.
/.
Lightness_is a lighter use of tension in the body, a
fine or sensitive touch.
It can be seen in behaviors such
as handling delicate objects, caressing, or small tapping
movement.
Strength is a quality of firmness.
muscular tension and :force.
It uses more
It can be seen i!f.SUch actions
as pushing, punching, and defiance.
Time is the speed or rhythm of one's movements.
Time
can either be sustained or sudden.
Sustainment is an indulging 1n time.
It is a sense
that it is not important when an action is completed (such
as a gradual smile).
27
Suddenness is a fighting time.
seen in behaviors such as
It is quick and can be
excitement~
nervous twitches,
and fast finger picking of a guitar.
Body or inner attitude is a combination of two movement elements, such as a movement which is both quick
and direct.
Complete efforts or drives are a combination of
three elements.
It may or may not have flow, but in order
for it to be a complete effort it must have weight, space,
and time.
Characteristic effort pattern is the combination of
movement elements, inner attitudes, or drives
which predominate and are seen most often.
Phrase is a movement sentence.- The pattern of movement
occurrences, the·order in which sequences of elements,
/.
inner attitudes and/or drives appear.
Even phrase is a phrase with no accents, and with
no change from beginning to end.
Impact phrase is a phrase which begins slow and ends
fast.
Impulse phrase is a phrase which begins fast and ends
slow.
Swing phrase is a phrase which has its accent in the
middle.
Body part initiation is the limb or area of the body
which repeatedly begins a phrase.
The body part which
28
leads the rest of the body in the effort.
Body shape is the predominant posture or carriage
of the body.
The body can be shaped high, wide,
cu~ved
or twisted.
High shape 1s pin-like, an erect carriage, lifted.
Wide shape is wall-like, enjoys leaning.
The person
uses his body in an expansive way.
Curved shape is when the body resembles the letter C.
It occurs when the upper or lower torso curves forward or
back and gives a feeling of roundness.
Twist shape is when the body resembles the letter S.
The torso is twisted so that the upper and lower torso
face different direction.
Shaping "is the form of the movement; it contains
effort and eff,ort flow" (Davis, p. 34).
It establishes
relationships between oneself and the environment, showing
the "way an individual copes with and adapts to inner
states and outer stimuli" (Davis, p. 34).
"It is concerned
with the pathways and forms the body parts create in space"
1)Dell, p. 30).
Body planes are the directions possible for whole
body movement in space (not just limbs).
The three planes
are vertical, horizontal and saggital.
Vertical plane is associated with predominant use of
weight.
It is cartwheel like.
Movements are up and down.
It can be thought of as conveying the
non~verbal
message
29
of presentation and intention to do.
Horizontal plane is associated with movements which
are predominantly side to side.
Table-like, it can be
thought of·as attention, opening and erasing.
Saggital plane movements are forward and back in
space.
Locomotion, summersalt-like, this plane is asso-
ciated with operation and decision making.
/
..
::.
")
Sample Score Sheet
Name of Child
Efforts - circle when seen
.-=-*
r
/
Body Part Initiation
I
I
f
-
/
-
Planes
verticle
horizontal
saggital
Bodr sha.Ee
high
wide
curved
twist
II
Comments:
'
I
I
1-- .
Date
characteristic. phrase
pattern
even
impact
impulse
swing
Name of judge
31
The following procedures list all steps which were
taken for the collection of data.
The purpose of this
technique was to facilitate a clearer understandi11g of hhe
experimental design which consisted of _three experimental
components - two involving observations of childr.en during
play, and one involving observations of structured movement
routines performed by the investigator.
The three experi-
ments were designed to either test for inter or intra-judge
reliability.
A repetition of these steps by another
investigator would make it possible to duplicate the study.
Procedure for Experiments I and II
1.
Judges were instructed in the use of the score sheet
and specific terms were reviewed with them.
2.
The instructions were:
Circle all items which you see
the child demQnstrate, even if the quality was not predominant in the child's movement behavior.
3.
Judges were given the score sheets, and told not to
discuss their observations of the movement behavior of the
children.
4.
The children began to play either "prisoner" or "kick-
ball."
These are both team games, involving gross motor
movement in a playground environment.
5.
Both are activities.
Each child to be observed was selected by one of the
judges.
The selection was random, except that care was
taken not to repeat observations of a child already observed
either the week before or by one of the other three judges.
'i:"'-
____
y'~-
32
Judges were simply asked before the observation began
whether that child had been observed by anyone on a previous
occasion.
6.
The child was watched for 10 to 15 minutes indepen-
dently by each judge.
7.
Scoring took place throughout the time period.
8.
No more scoring took place after the time period was
over.
Even though a judge may have seen a particular
child move at a later time, those extra movements were
not to be recorded.
9.
The next child to be observed was selected and proce-
dures 5-8 were repeated.
10.
The total observation period was one hour per day,
once per week, for two weeks.
11.
Since the data collected in Experiment I was not com/.
pared to the data collected from Experiment II, feedback
(based on a review of each judge's observations made in
Experiment I) was given to the judges in order to reduce
misunderstandings, difficulties, or errors in the use
of the instrument.
Procedure for Experiment III
1.
Judges were instructed in the use of the score sheet
and specific terms were reviewed.
2.
Judges were given the score sheet and told not to dis-
cuss their observations.
3.
Judges observed a movement routine executed by the
33
investigator three times.
The routine was a simple com-
bination of movements, designed to limit the number of
movement qualities which were seen.
4.
Scoring occurred after each repetition.
5.
Procedures 3 and 4 were repeated for routines "b" and
"c."
These routines also used different movement qualities,
efforts or elements so that there would be little similarity between the three routines.
6.
The score sheets were collected.
7.
Judges were instructed not to talk about or think
about the routines and their scoring of the routines.
8.
The judges observed the children at play for one hour.
9.
Procedures 2 through 6 were repeated.
Data Collection Techniques
For Experiment I the observations of each pair of
judges were compared.
They were compared on the basis of
agreement or disagreement.
Agreement occurred when both
judges circled the same movement item.
Disagreement
occurred when one judge circled the item and the other judge
did not.
A value of "1" was given to all items where
there was agreement among judges; a value of "0" was given
tb all items where there was disagreement.
An example for Experiment I might be:
In watching
child #1, Judge A observed that this child showed quickness
in his movements.
score sheet.
Judge A therefore circled quick on the
If Judge B, while watching the same child,
~·-
- - - - - - - - - - - _ _:!i:'-
---
34
did not see this child move quickly, then she would not
circle the item quick on the score sheet.
In this example
the Judges did not agree; therefore they received-a "0"
for that item.
If, however, Judge B had seen quickness,
and she had circled it on the score sheet, then the judges
would have agreed, and would have received a "1" for that
item.
For Experiment II the observations of all four judges
wer,e compared.
The number of items in which all four of
the observers agreed was noted; the number of items in
which three out of four judges agreed was noted; the number
of i terns where 2 of the judges saw one thing while the ,)
other two did not see it was noted.
A "1" was given to
all items that were circled or seen by the judge.
A "0"
was given to items
that were not circled.
/.
An example to more clearly illustrate this evaluation method is:
Judge
Movement Item - quick
A
a score of 1
=
quick movement was seen and circled
Judge B
a score of 1
=
quick movement was seen and circled
c
a score of 0
circled
=
quick movement was not seen or
Judge D
a score of 0
circled
=
quick movement was not seen or
Ju~ge
Judge
This example illustrates that a 2:2 type of agreement has
occurred.
Judges A
&B
saw and circled quickness.
Judges
35
C
&D
did not see the quality in the child's movement,
and therefore did not circle quickness on the instrument.
For :E!xperimen t
I I_I the observations each j udg_e made
of three identical movement routines before and after a
one-hour break were compared on the basis of:
(1) The judge's own first and second scoring of the
routines, and (2) all of the other judges' scoring of that
same routine.
The same number system as was used in Ex-
periment II was used in both parts of Experiment III.
Statistical Procedures
1.
A random mean for any two-choice test filled in by
two observers was calculated.
2.
The mean of the experimental sample was calculated
for Experiment I.
3.
A'random mean for any two-choice test filled in by
four observers was calculated.
4.
The mean of the experimental sample was calculated for
Experiment II.
5.
The mean for the first part of Experiment III was
calculated.
6.
The mean for the second part of Experiment III was
calculated.
The rationale for using quantitative statistics for
qualitative data is based on the purpose of the study to measure degree of agreement among independent judges,
rather than the quality that might be inherent in the
assessment of movement by the judges.
CHAPTER IV
Results
Statistical Considerations for Treatment of Data:
Experiment I
Experiment I consisted of a comparison of the observations made by two judges of the movement behavior of
children during play.
being one pair, and C
Each pair of judges ( A
&D
&B
being the other pair) made
ten to fifteen minute observations for each child.
A
&B
observed different children than did judges C
Judges
& D.
The judges worked independently once the child to be
observed was selected.
The judges did not discuss their
observations while the scoring was taking place.
On the ssore sheet used throughout this study the
judges had only two choices.
The first choice was to cir-
cle a particular item if the quality was seen.
choice was not to circle the item.
The second
For example I would·
'
circle the item "quick" if I saw that quality, but I
would not circle it if I did not see that quality.
Agreement on the score sheet indicated, therefore,
that either both observers circled the item or both did
not circle the item.
Disagreement on the score sheet occurred when one
of the observers circled the item and the other one did not
circle it.
36
37
Statistically, for any two-choice test filled ·in by two
observers, a purely random result would show:
jud~es ~
1.
Agreement by both
of the time.
2.
Disagreement by both judges
~
of the time.
The standard formula for calculating a random mean is
.)-\ = £..x · f (x), where the mean 'X' equals the sum of all
values of x times the probability f(x) of x occurring.
By assigning the value:
1 for items where there is agreement
0 for items where there is disagreement
the mean score of each set of two observations can be
calculated. For random results the expected mean is SO% of
the total number of choices.
mean
'~',
Comparison of the expected
with the actually observed mean X (calculated
from the data) will indicate the value of the scoring
procedure employed.
The probable error in the random mean (or standard
deviation) is given by the formula
Q"'
=ln. p. q,
where
n = the number in the sample
p = the probability of success
q = the probability of failure
For the data of Experiment I tr=
-
tfio. 5. 5 =
Js=2. 236
Statistical Considerations for Treatment of Data in Results
of Experiment II
The data from Experiment II was statistically treated
38
38
differently. This change \vas necessary because the conditions of
this experiment are different.
by
In Experiment II, observations made
all four judges were compared rather than observations made by only
two of the judges.
The objective was to test for inter-judge reliability"
among four people.
Statistical analysis show that for any two choice test filled in
by
four
observers~
a purely random result would be:
1} Agreement on a 2:2 basis 3/8 of the time.
{This means that
two observers agree on one choice and two observers agree on the
other choice.
i.e.
Judgf~S
A & C circle the 'item quickness and Judges
B & D do not ci'rcle the item quickness).
2) Agreement by three observers 1/2 of the time.
A,B~C
(i.e. Judges
all circle quickness but Judges D does not).
3) Agreement by a'll four observers l/8 of the time.
(i.e. all
four Judges circle the item quickness).
The random mean can again be calculated using the standard for·mula; as follows:
11
Assigning the x values
0 for the 2:2 type of agreement;
2) 1 for the 3:1 type of agreement;
3) 2 for the 4:0 type of agreement.
As discussed aboves the probability that an x value of 0 \'Jill be obtained is 3/8; for a value of 1 it is 1/2; and for a value of 2 it
is l/8.
39
The random mean
for a two-choice test filled in
by four observers is given by..f-t=
= 3/4.
co~3/8)
+ (1~~)
+
(2· 1/8)
A random-result therefore is 75% agreement--over
the total number of choices.
Results of Experiment I
(Judges A
&B
independently scored the movement
behavior of each child for ten to fifteen minutes.
"Items" refers to the Laban terms which were selected
for use in the instrument or score sheet used by Judges
for their observations.
Refer back to Chapter II, dis-
cussion of the instrument, for review of the twenty items.
Comparison of Observations Made by Judges A
&B
Child
#items in
agreement
# items in
disagreement
1
17
3
Z9
2
14
6
20
3
14
6
20
4
17
3
20
The observed mean
Judges A
&B
X is
15.5.
Total
# items
Therefore on an average,
agreed on 15.5 out of 20 items.
40
Comparison of Observations Made by Judges C
Child
# items in
agreement
# items in
disagreement
&D
Total #
of items
5
14
6
20
6
13
7
20
7
14
6
20
8
15
5
20
9
14
6
20
10
16
4
20
The mean score for Judges C
&D
that Judges C
items.
&D
is 14.3.
This indicates
agreed on an average of 14.3 out of 20
The total mean for both pairs of observers was
14. 9.
/
Discussion of Results of Experiment I
~
In analyzing the results of Experiment I (where the
scoring of one child by two judges was compared) the following was found:
The expected mean
'~"
for a two-choice test consisting
of 20 items is 10.
The observed mean, X is 15.5 for )udges A
for Judges C
&D
& Band
14.3
with the average mean of 14.9 for both
parts.
For the data of Experiment I,
T=~
=[5=
The difference between the expected mean and the
2.236.
41
observed mean is 14.9 - 10
4.9 divided by 2.236
=
=
4.9.
2.19 standard deviations
above the expected mean.
In a table of the normal curve it was found that
there is a 98.5% probability that 14.9 is a statistically
significant result on a random expectation of 10.
Therefore, the odds that the observed sample is not
random are 98 to 2.
A comparison of the mean score received by each team
of judges simultaneously observing different children
indicates that there is only a 1.2 point difference between
their mean number of times that agreement occurred.
This
implies that there is a fairly consistent level in the
ability of all of the judges to observe and score movement
qualities.
/
An overview of the amount of agreement and disagreement
on each item by each pair of observers showed that 3 out
of 20 items had a high degree of disagreement.
Three
items which showed the highest amounts of disagreement were
even phrase, impulse phrase, and body part initiation.
These items were not more difficult to identify.
This
indicates a misunderstanding of how those items should be
scored or exactly what those terms mean.
An overview of all of the results also brought out
observer specific difficulties in seeing certain items.
It was then possible to give feedback to the judges so
42
that they would be more aware of those difficulties
when making their next set of observations.
Results of Experiment II (a comparison of the observations
made of each child by all four judges)
The children observed in this experiment were not the
same ones that were observed in Experiment I.
Child
Total 4:0
Agreement
(all 4 )
Total 3:1
Agreement
( 3 of 4)
TOtal 2:2
Agreement
(2 saw,
2 did not)
Total
rtems
11
'I"'
12
1
20
12
10
10
0
20
13
10 l)
8
2
20
14
8
10
2
20
/.
43
For child 1:
X
= [_x ·f (x) =
X
=
sample mean
( 0. 1) = (1-12) + ( 2. 7) = 26
20
zcr20
20
X = ( 0· 0) = (1·10)
For child 2:
20
For child 3 :
20
= c2 ·1-o)
20
=
1.3
= 30 = 1.5
20
X = (_Q:_l) + (1· 10) = (2·10) = 30 = 1.5
For Child 4:
X
=
----zo
20
20
(O· 2) = (1•10) = (2·8)
20
20
20
20
= 26 = 1.3
20
Total (over 80 possible choices)
Type of agreement
4:0 = 35
3:1 = 40
2:2 =
X
=
5
(0· 5)
=
80
(1· 40)
80
=
(2· 35)
80
=
110 = 1. 375
80
/.
random mean= .75
Discussion of Results of Experiment II
The results of Experiment II (where observations made
of each child by all four judges were compared) showed that
the mean score was 1.375.
Compared to an expected (random)
mean of .75.
In order to interpret the statistical significance of
the observed mean, a measure of probable error, "g-" the
standard deviation must be
calcu~ated.
Because the test of Experiment II is more complicated
than that in Experiment I, involving four judges rather
44
than two, a new formula for standard deviation must be intraduced:
lJ =
{)" =
g- =
p-=
J (x
J
-)1_)
2
• f (x)
2
2
(0 - 3/4) • 3/8 + (1-3/4) • -1/2 + (2 -
3/4)~ ~
v'9/16 • 3/8 + 1/16 .. ~ + 25/16 .. 1/8
vh/16
• 56/8
=
7
4
=
.66
The error in the expected mean (a purely random mean)
therefore, is .66.
The difference between the expected mean, .75 and
the observed mean is .625.
When .625 is divided by .66 the result is .95, which
is almsot one standard deviation above the expected mean.
In a table of the normal curve for this type of test,
.95
=
32.89% above the mean or middle of a normal curve.
Therefore, 50%
=
32.89%
= an
82.89% probability that
1/375 is significantly different from .75.
Therefore, the odds that the observed sample is different than a random sample are 83 td 17.
The data seem to indicate that the four judges could,
at an 83% confidence level, reliably observe the same
movement patterns which were demonstrated by each child.
This implies that the judges all understood the instrument
and were able to use it effectively in recording their
observations.
45
These results could be important if an-assessment program of children was being planned.
An 83% reliability is
high enough so that a recreation program director might
place value in the assessment made of each child, based on
his movement behavior, and therefore include this method
in his program.
Review and Justification for Experiment III
The same four judges were used in Experiment III as
were used in Experiments I and II.
The purpose of Experi-
ment III was to test both for intra and inter-judge reliability.
The four judges observed and scored three movement
tines.
rou~ .:t
The routines were performed by the investigator,
so that both the number and quality of the movements
/.
observed could be controlled.
A one-hour break was taken
in which the judges were requested not to talk or think
about the routines.
After one hour, the three routines
were repeated and the four judges observed and scored
them a second time.
The data collected for Experiment III was analyzed in
two different ways.
The first way (called Experiment III,
part 1) was designed to test for intra-judge reliability.
This test was necessary in order to test if each judge
could reliably repeat a scoring of the same movements at
a different time.
46
The second way (called Experiment III, part 2) was
designed to test for inter-judge reliability.
The data
collected from the scoring done by all of_the judges for
each routine was analyzed in the same manner as was the
data from Experiment II.
Results from Experiment III, part 1 (a comparison of the
observations each judge made of the same routines at
separate times)
When the first and second scoring of each routine were
compared:
Judge
Routine #
A
B
c
D
Items agreed
(repeated same
score)
Items disagreed
(changed score)
Total
Items
a
19
1
20
b
19
1
20
c
19
1
20
a
13
7
20
b
18
2
20
c
19
1
20
a
16
4
20
b
18
2
20
c
17
3
20
a
18
2
20
b
19
c
18
1
2
20
20
Total mean for all judges is 17.5 out of 20.
47
Discussion of results of Experiment III, part 1
The results of the first part of Experiment III indicate that the judges s.cored identical i tenis the same way
an average of 17.5 out of 20 times.
The results show that
two of the judges were more consistent than the other two.
This then might be an experiment which should be done
prior to those which
inv~lve
observations of children, in
order to select the most internally consistent judges.
The data indicate that no one item showed any significantly higher
fre~uency
of error than any other item.
In looking at the total mean for each judge separately,
in the scoring of three routines consisting of twenty
items, we would expect a mean of 30 if the results were
random.
The observed mean for the most consistent judge,
I-
Judge A, was 57.
~ =
The error in the expected mean is:
tJn· p .q
n
=
p
= .S(probability of success)
60(number in sample)
q •
.S(probability of failure)
tr =
J6 o· s · s = Jl5
=
3.
a7 3
The difference between the expected mean and observed
mean is 27.
27 divided by 3.87
=
6.9 standard deviations
above the expected mean.
The observed mean for the
B, was 50.
~east
consistent judge, Judge
48
The difference between the expected mean and this second
observed mean is 20.
20 divided by 3.87 = 5.16 standard
deviations above-the expected mean.
In either case the probability that the results were
random is less than .01% or over 99.9% probability that
the observed results were not random.
Results of Experiment III, part 1 (judges observed and
scored three movement routines, observed children for one
hour, and then observed and scored the same three routines
a second time.
The observations made by all four judges
were compared in order to determine inter-judge reliability)
The data will be analyzed using the same formula and
methods as were used in Experiment II
Comparison? of Observations of Three Structured
Movement
Routine #
Routines:
total 4:0
agreement
total 3:1
agreement
total 2:2
agreement
total
items
First
Observation
a
b
c
Second
Observation
13
11
13
a
8
b
10
14
c
3
7
4
4
2
20
3
20
10
7
2
20
3
20
5
1
20
20
49
Routine "a, II first observation:
x=
x=
=
(0·4) + ( 1· 3) + (2 •13)
zo·
20
20
29
20
=-
1. 4 5
1.45.
Routine "b,ll first observation:
= ( 0. 2)
(1•7)
29
TO
=
1. 45
X = (0•3) + (1·4) + (2·13) = 30
20
20
20
20
=
1.5
X
+
=
(2 ·11)
TO
TO
X
+
20
=
1. 45
Routine "c, II first observation:
X = 1.5
Routine "a, II second observation:
x=
(Q:__?)
+ ( 1 ·1:0)
20
+
(2 • 8) = 26 = 1.3
20
20
20
X = 1.3'
Routine "b," second observation:
X = (0·3) + (1· 7) + (2 ·10)
TO
20
20
=
27
20
=
1. 35
X = 1. 35
Routine "c," second observation:
X = ( 0. 1) + (1· 5) + ( 2 ·14) = 32
20
20
20
20
=
1. 65
X = 1.65
Random mean
=
. 75
Sample mean for 4:0 type of agreement = 1.15
Sample mean for 3:1 type of agreement
Average mean for sample
=
1. 4 75
=
1.8
50
Discussion of Results of Experiment III, part 2
The results of the second part of Experiment III
indicate that the-re was a mean of 1. 15 for- the 4: 0 --type of
agreement, and a mean of 1.8 for the 3:1 type of agreement.
These are both higher than a random mean which would be .75.
An average of the two means for the experimental sample is
1.475.
The
=
expected mean was 1.475.
the error in the
expected.mean~s
Tnis~equaills
.66.
The difference between the expected and observed means
is .725.
(1.475 - .75 =.725)
.725 divided by .66 equals 1.09 standard deviations
above the expected mean.
1.09
=
36.2% above the mean of a normal curve.
Therefore, / . 36.2% + 50%
=
86.2% probability that 1/475
is a significant difference from .75.
Therefore the odds
that the experimental sample is not random are 86 to 14.
The mean number of times in which agreement occurred
consistently falls between one and two standard deviations
above the mean of a random sample.
This appears to indi-
cate that the results are not· random.
It is believed that the results of this experiment are
better than the results achieved in Experiment II, because
the situation was more controlled.
The change was not due
to any increased learning on the part of the judges.
A review of the data indicates that for all cases
where there was a 3:1 type of agreement, no one judge
consistently disagreed with the other three judges.
51
Summary Tables
Experiment I
Child
'Judges
1-4
team
A &B
5-10
Average #
Items Agreed
team
C &D
Total Items
15.5
20
14.3
20
Experiment II
Child
Judges
11-14
4:0 Type
Agreement
A,B,C,D
35
3:1 Type
Agreement
2:2 Type
Agreement
40
5
Total
Items
80
I.·
~xperiment III, part 1
P"udge
Average agreement with self
on 1st & 2nd observations of
Routines a, b, c.
Total
Possible
A
19
20
B
16.6
20
c
17
20
D
18.5
20
52
Experiment III, part 2
Routine #
Judges
1st & 2nd
observations:
a, b, c
A,.:.:B,
c,
;.
D
Total
4:0
Agreement
69
Total
3:1
Agreement
36
Total --!TOtal
2:2
Possible
Agreement
15
120
53
In summary, the first two experiments- conducted seemed
to indicate that movement behavior of children during play
could reliably be observed by independent judges
probability level of 83% to 98.5%.
a~-a
Experiment III showed
that intra judge reliability could be established through
a comparison of before and after observations of structured movement routines at a 99.9% probability level, and
that inter-judge reliability increased from 83% in Experiment II to 86% when the content of the movements observed
was controlled in Experiment III.
/
CHAPTER V
Conclusion
In conclusion, the results of the data facilitates a
rejection of the null hypothesis.
All three of the experi-
ments conducted showed significant differences between the
observed results and random results.
Summary of the Entire Study
The study was begun with the intention of testing the
reliability of independent judges to observe the movement
behavior of children during play.
A score sheet of twenty
items was devised from a notation system created by Rudolf
Laban and others.
The score sheet functioned as a focusing
device so that all four of the judges were looking for
/.
the same specific movement efforts, phrasing patterns,
body shape, and use of space.
The review of selected literature demonstrated that
movement preferences can be observed through a variety of
means, depending upon the frame of reference of each
investigator.
The results of. the North and Kestenberg
studies supported the proposed conceptual framework in
concluding that movement may reflect personality, and that
preferred movement styles may reflect the manner in which
that individual copes with both inner and outer stimuli.
54
_j
-
55
Three experiments were conducted in order to test the
hypothesis that the observed results would differ significantly from results obtained if observations were random.
Experiment I consisted of a comparison of the scores given
to ten children, one child at a time, by two observers.
Experiment II consisted of a comparison of the observations
made of four children, scored one at a time by all four
observers.
Experiment III
co~sisted
of a comparison of
observations made of three short structured movement routines with observations of the same routines one hour later.
The data was analyzed in two ways.
Method #1 (called
part 1) consisted of a comparison of the observations each
judge made of the same movement routines at separate times.
The purpose was to test for intra-judge reliability.
Method 2 (called part 2) consisted of an inter-judge com/
parison of the scoring done by all four judges of the same
routines.
Analysis of the data showed that the judges were able
to consistently and repeatedly see the same movement behavior with the odds ranging from 99.9% probability that
the observed results were not· random to 83% probability that
the observed results were not random.
This shows high
probability that the observed results were biased (not
random).
It was therefore concluded that the null hypo-
thesis must be rejected.
Finally, future studies will be suggested with the
56
hope that the information gathered in this ·study will be
expanded upon by persons who are interested in the relamove~ent
tionship of
piefeiences and behavior to
emotional health, physical fitness, sex or age.
l~~ining,
Implications
The implications which can be made from this study,
for a variety of fields, are many.
For the teacher,
recreator, or therapist, the results imply that one can
learn to be more sensitive to the non-verbal behavior of
children.
This heightened sensitivity can be used in
order to help the child become a happier and more successful adult.
Knowledge of motor preferences may also aid
the professional in planning programs which are individually designed to meet the special needs of each child.
Since recreation allows for voluntary involvement in
activities, the results of this study might remind the
recreation practitioner (1) not to standardize activities
for children, and (2) to design programs which provide
opportunities for less structure.
Recommendations for Future Study
The question now is:
~enerated
Of what value is the information
in this study?
This study has shown that it is possible for independent judges to reliably identify and record movement
qualities.
It is suggested that, for the field of
57
therapeutic recreation, a training program .in the use of
the instrument employed in this study might make it possible
for professional5 to increase their abilities of identifying personal preferences in the movement behavior of
their clients.
The training program could practically be
completed during a two-day seminar.
This seminar would
not qualify the trainee to do professional movement behavior assessments, but it would serve to sensitize him to
an area of behavior which would increase his professional
effectiveness.
It is suggested that this study be repeated in order
to establish the validity of the research.
More judges
should be used, and a larger sample of children, so that
more observations could be done.
More time should have
been allotted to the judges for practice sessions with
/.
the test before data was collected, and additional time
should be given for the observations, so that a more accurate assessment of the child's movement repertoire could
be achieved.
It is suggested that a study be done exploring the
relationship between having a. wider range of movement
and full functioning or optimal functioning of the individual.
This might be done through a comparison of a
movement assessment with a psychological assessment for
each child.
The range of movement would be defined as
the total number of circled items.
Those children who
58
received a higher number of circled items would be defined
as having a wider range of movement.
If the judges were instructed to only_ circle the predominant movement patterns rather than all the qualities
which they saw each child demonstrate, a comparison could
then be made with other psychological assessment tools
in order to further explore the correlation between certain
movement patterns and particular personality characteristics.
A series of correlation studies could be done comparing the movement assessment of each child with:
scores, (2) ability to learn novel tasks,
(1)
I.Q.
(3) movement as-
sessments of children of a variety of ages, and (4) ability
to learn and perform new motor or intellectual skills.
These studies could potentially discover methods for predicting the success each child might have in a variety of
areas, both ac'ademic and physical.
It might also produce
methods of establishing diagnostic techniques, so that
problems could more effectively be identified, and then
alleviated through special programs.
A study comparing a large number of movement assessments of normal children to a large number of movement
assessments of atypical children might be done.
Through
this study a norm might be established which could be used
in the development of treatment plans for those falling
below the norm, and gifted programs for those scoring above
the norm.
This study might stimulate the production of
59
data which would prove whether or not there is such a thing
as a normal movement behavior or an optimal movement
behavior range.
A study of the differences between-the scores of boys
and girls might be done.
It has been hypothesized that
the movements of boys and girls are different in areas such
as the use of impact or impulse phrasi:.ng.
An actual com-
parison of each item on the score sheet, relative to the
sex of the observed child, might test this hypothesis.
I hope a correlation study will be done between movement assessments of children observed during play and movement assessments of children observed in other structured
settings or environments.
This study might help increase
the body of knowledge accumulating on how play behavior is
different than other forms of behavior.
I
Finally, I suggest a before and after study of children (who have first been observed to have low scores),
who have undergone special training in movement or play.
The training should expose the children who have deficits
in their abilities to physically respond in a rich or varied
manner to more options, or ways of moving, living, and
being by themselves and with others.
60
BIBLIOGRAPHY
1.
Axline, Virginia M.
lantine, 1947.
2.
Birdwhistell, ~ay L. Introduction to Kinesics.
Louisville: University of Louisville, 1952.
3.
Bruner, Jerome S., and Jolly, Alison and Sylva, Kathy.
Development and Evolution. New York: Basic Books,
Inc., 1976.
4.
Caplan,Theresa, and Caplan Frank.
New York: Anchor, 1974.
5.
Cratty, Bryant J. Movement, Perception and Thought The Use of Total Body Movement as a Learning Modality.
Palo Alto: Peek Publications, 1970.
6.
Learning.
Play Therapy.
New York:
Bal-
The Power of Play.
Movement Behavior and Motor
Philadelph1a: Lea & Febiger, 1973.
7.
Davis, Martha (ed.). Research AEproaches to Movement
and Personality. New York: Arno, 1973.
8.
Dell, Cec~ly. A Primer for Movement Description.
New York: Dance Notation Bureau, Inc., 1970.
9.
Ellis, Michael J.
10.
Erikson, Erik H. Childhood and Society.
W. W. Norton & Co., 1950.
11.
Gesell, Arnold and Ilg, Frances L. The First Five
Years of Life. New York: Harper & Bros., 1940.
Why People Play.
New York:
New York:
12.
Hutchinson, Ann. Labano'tation.
Arts Books, 1970.
Theatre
13.
James, William. "A study of the expression of bodily
posture." Journal of General Psychology, Vol. 7:
405-435, 1931.
14.
Kalish, Beth Isaacs. "Body movement scale for autistic and other atypical children.- an exploratory
study using a normal group and an atypical group."
Unpublished dissertation, Bryn Mawr College, January,
1976.
61
15.
Kestenberg, Judith S., M.D.
"The role of movement
patterns in development. Rhythms of movement."
Psychoanalytic Quarterly.
New York:
January, 1965,
1-36
16-.
, "The Role of Movement Patterns in Development,
Flow of Tension and Effort, 11 Psychoanalytic Quarterly,
New York: March 1965, 517-562.
17.
Laban, Rudolf. Mastery of Movement.
Plays Inc., 1975.
18.
_ _ _ _ ,Lawrence, F.C., Effort. London: MacDonald,l947,
19.
Loizos, Caroline.
Zoology
20.
Maginnis, Maria C.
"A study of autistic gestures in
elementary school age children." An Unpublished
thesis.
University of California, Los Angeles,
August 1966.
21.
Maison, Kathleen, (ed.): Dance Therapy,Focus on Dance
#7. Washington D. C.: American Association for Health,
Physical Education and Recreation, 1974.
22.
Millar, Susanna.
The Psychology of Play.
England:
Penguin Books Ltd., 1968.
23.
Miller, Irwin and Freund, John E.
Probability and
Statistics for Engines.
New Jersey:
Prentice-Hall
Inc., 1965.
24.
Moran, Joan, and Kalakian, Leonard. Movement Experiences for the Mentally Retarded or Emotionally
Disturbed Child. Minneapolis, Minnesota: Burgess Co.,
1974.
25.
Neal, Larry L. (ed.).
Leisure Today:
Selected Readings.
Washington D.C.: AAHPER Publications, 1976
26.
North, Marion.
ment.
Boston:
27.
Piaget, Jean.
Play, Dreams and Imitation in Childhood.
New York: W. W. Norton & Co., 1950.
28.
Piers, Maria W. (ed.).
Play and Development.
York: W. W. Norton & Cd., 1972.
Play in-Mammals"
Boston:
Symposium on
Middlesex,
1-
Personality Assessment Through MovePlays Inc., 1975.
New
62
29.
Singer, Robert N.
mance. New York:
30.
Slovenk, Ralph, and Knight, James A. _Motivations in
Play, Games and Sports. Springfield: Charles C.
Thomas, 1967.
31.
Weitz, Shirley, (ed.). Nonverbal Communication,
Readings with Commentary. New York: Oxford University Press, 1974.
32.
White, Elissa. "Effort-shape, its importance to
dance therapy and movement research." Dance Therapy
Focus on Dance #7. Washington, D. C.: American
Association for Health, Physical Education, and
Recreation, 1974.
Motor Learning and Human PerforMacMillan, 1975.
t-0
1.0
EXPERIMENT I
Ju<iges A&: B
PL.
.-l
iS
~
c5 &i ~
j ~ J iJ~ ~. § i
.-l
:~
Judge .
t:
J
Q)
Q)
Child
'0
~
j
~ ~
~
Cl)
I
~
Q)
J.<
"'
Q)
~
tl
~
):
0
{'i
.....
~
tl
~ ~
p.,
'0
Q)
~
Q)
rl
';d
"t=0
Key:
.-l
J.<
~ ~ ~ TOTAL AGREEMENT TOTAL DISAGREEMENT
Jud.lte A
Child 1
1
1
1
1
1
1
1
1
1
1
1
1
0
0
1
1
1
1
0
1
JudJ:te B
, Child 1
1
1
1
1
1
1
1
1
1
1
1
1
0
0
1
1
1
1
0
1
JUdge A
Child 2
1
1
1
1
0
1
1
1
0
1
0
1
1
0
1
0
1
0
1
1
Jud.lte B
Child 2
1
1
1
1
0
1
1
1
0
1
0
1
1
0
1
0
1
0
1
1
.Jud.lte A
Child 3
1
1
1
1
1
1
1
1
0
0
0
1
0
1
1
0
1
0
1
1
2
1
1
1
1
1
1
1
1
0
0
0
1
1
0
1
0
1
1
Judge B
Child
0
1
0 = Disagreement
"'
.....tl ..... ~
~ ·~
~
J.<
1 = Agreement
'
Judge A
Child 4
1
1
1
1
1
1
1
1
1
1
0
1
1
1
1
0
1
0
1
1
Judsl:e B
Child 4
1
1
1
1
1
1
1
1
1
1
0
1
1
1
1
0
1
0
1
1
17
3
14
6
14
6
17
3
-
-.::t
\{)
EXPER.ll<!ENT I
Jud.ges C & D
~
'til :g) ~
31 ~
., 0<11 )
0
&!88
..-1
X
.,
o
~
"
.,
~
~
~ t:
~
~
o
~
111
~
&i~~~~~i ~ ~
~
.,
~
~
p
.
~
~
~
';;l
~ ~ ~
o
o
N
~
·M
"'
:>
0
:X:
0)
oI
1I 1
·rl
~
t
~
Child 5
I 1 I 1 I 1 I 0 I 1I 0 I 1 I 1I
Child 5
I I I I I I I I I I I I I I I I 1 I I ct I I 1 I
JudP:e C
Chi_ld 6
JudP:e D
Child 6
D
I
1
1
1
0
Io I 1I
oI
IoI 1 I
oI
1
0
1
1
1I 1I
0
0
1
1 I 1I 1I 1 I 1 I
oI
1I o I
1 I 1 I 1I 1I 1I
oI
1I
oI
1I 1I 1I 1I
0
oI
o I 1I 1 I 1 I 1I 1I o I 1
oI oI oI
1
1
1
1I 1 I 1 I 1 I 1 I
oI
6
I
I
7
13
1
- - - - J r - - - - - r - - - - t - - - 1 - - t - - t - t -t--t--t--1--t--t--t--t--:--1--t-t"-t-~---'-----t-------
_,!uQB.e C
6
14
__J.J!dg_e,._.~o~...~.....~~~..L.-J._:..L...!...l~L.!...L:..L...!...t.__!..JL.!.~~..!...t.~~.I...;~.!..!~L!.~
f.-
I--
I
A~?reement
14
1
1
1 =
2 = D1.sagreement
Total Agreement ~otal Disagreement
JudP:e C
oI oI
Key:
_
1-- -
-~u~e=Q_-=r Q_l,lj.;.Q_a
~e__D__ . [child 8
1 1
1
1
1
1
,J..r-1-.
1
1_
"'-- -"'"''
'- .""'-'"-"
Ju e D
Chilg.-9_
~
~
~
~
~
Child
J--l.U~
f-
~
Judge
!?
t
ll ~
~
,:.:;:
·----J~
r ;.:,~,~ r,.
D _ L- Child 1Q
1_~ 1 1.JL~e-L
o
o
o~~
1111111
o
o
o
1
1
1 !1 !1 !1
,f't~·
'+ll~"
1 ...2hlililo
1
1
1
1
1
1
1
1
1
1
L-1 L.J__ . .!.
Q
o
1
,_[;
u
,
1
1
____ _
1
~ ~
;
1
1
o
o
1
-
15
I
1
·JliTr~n
o_J]..l--1-'-~-1
o
1
1
1
1
_
...:!...r-!
1
5
14
I
6
--·--
-·-
·----:;---~-- ~-
.!.-'1-'----------..l-
·-----
If)
\0
~
EXPERIMENT II
s,
....,
,_
QJ
Judge
Child
QJ
,_
Q)
LL.
"0
t:
::>
0
"'
.,_, "'..a
t: ·~
+'
u
,_
,_0
"' .,_, u::" 0
::::;
-C
Q)
(/')
....,
QJ
3
0
~
"'
"""u
t:
::>
>
·~
Q)
cr w
QJ
"' c:
"'!c. !c.::> "':;::
u
·~
(/')
QJ
"'
~
"0
0..
~
0
"'
Q)
-C
QJ
"'
:;:
"0
·~
:;:
.,_,
,_> "'
u
~
:::1
';;;
...,
u
0t:
·~
N
.,_,
,_
'(
Q)
0
>
:I:
~
.,_,
"'
·~
"'"'
"'"'
Judge A
Child 11
1 1 1 1 0
Judge B
Child 11
1 1 1 1 1 1 1
Judge C
Child 11
1 1 1 0 1
Judge D
Child 11
1 1 0 1 1 1 1 1 1 1 1 0 0 1 1 0 0 1 1 1
Judge A
Child 12
1 1 1 1 1 1 0 1 0 0 1 0 0 0 0 0 0 0 0 1
Judge B
Child 12
1 1 1 1 1 1 1 1 1 0 1 0 1 0 0 0 1 1 1 1
Judge C
Child 12
1 1' 1 1 1 1 1 1 0 0 1 0 0 1 0 0 0 1 1 1
Judge D
Child 12
1 1 1 1 1 1 1 1 0 1 1 1 0 0 0 1 0 1 1 1
Judqe A
Child 13
1 1 1 0 11
1 11
Judae B
Odlri
1~
'1
11 11 11 11
.lilting r
rhil ti 11
!1
1 il In h
.l11nn" n
rhnti ,,
0 1 1 1 1 1 1
11
Judge A
Child 14
1 1 1 1
p p p
Judge B
Child 14
1 1
Judge C
Child 14
1
Judge 0
Child 14
1 h
4:0 Type
Agreement
3:1 Type
Agreement
2:2 Type
Agreement
Tota 1
Items
~
1 1 1 0 0 0 0 0 0 0 1 0 0 0 1
1 1 0 1 0 1 0 1 0
1 1 1
1 0 1 0 0
1 1 1 1
7
1 1 0 0
12
1
20
0
20
1 1 1
I
10
10
I
p
1
lo
lo lo
lo 11 lo lo In
In h
11
In In 11
11 11 In In h
h
In
In In 11 In lo 11
1 1 1
11
p 1 p
0
In
n
In 1 ~ ~.
b
b
1
p p p
1 1
b b
p p
8
2
20
8
10
2
20
p b b
b h h b b h b b
h h h
10
h
b h b b h h
I
I
I
!
1.0
\0
EXPERIMENT II I
Judges A & B
Part 1
QJ
QJ
Q)
'-
u..
ludoe A
-o
<::
:::0
0
co
~ ....
.... "'
<::
';( u
0
'- 5
'....
"'
--'
~ c; ;:;:;
..<:
QJ
QJ
V)
-"'
u
:;
<::
Q)
cr ..<i
....u
"'
0.
.5
QJ
Vl
:::0
0.
~
....'c..
<:: "'
"'
!-&
i -~ :I: ~
3:
Vl
03
Key:
'";;;
QJ
-o
~
u
....
<::
0
N
QJ
....
>
':::0
'- '[
3: QJ 0
1- >
::c
u
"'
-~
·~
1
0
~
...."'
=
=
i tern was ci rc 1ed
item was not circled·
·c,
Ol
"'
Vl
Routine "a" First Observation
1 0 1 0 1 1 0 0 0 0 0 0 0 1 0 0 1 1 0 0
Routine "a" Second Observation
1 0 1 0 1 1 1 0 0 0 0
Routine "b" First Observation
0
Routine "b" Second Observation
0 0
Routine "c" First Observation
0
1 0 0 0 1 1 0 1 0 0 0 0 0 0 1 0 0 0 1
Routine "c" Second Observation
0
1 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 1
Agreement_
Disagreement
qg
1
19
1
19
1
0 0 1 0 0 1 1 0 0
0 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0 0 1 0
1 0 0 1 0 1 0 0 0
0
0
0
1 0 0 0 1 0
i
I
Judge B
&
I
Routine "a"
First Observation
0
Routine "a"
Second Observation
1 0 0 1 0 0 1 0 'l
0 1
13
7
Routine "b"
First Observation
Routine "b"
Second Observation
1 0 1 0 0 1 0 1 0 0 1 0 1 1 0 0 0 1 1
~
1 0 0 1 1 1 0 1 0 0 1 0 1 1 0 iD 0 1 1 p
18
2
Routine "c"
First Observation
Routine "c"
Second Observation
1 1 0 1 1 1 0 1 0 0 0 1 1 0 0 1 1 1 1
1 0 0
1
p
1
pp
1
p
1 1 ID
1
..
1-
0 0 0 1 l
1 0 1 1 u
1
p
0
p
1
p
pp p
-
0 0 1 l
Pr
0 1 0
p
p p
p p~
---·~
0
I
-
19
1
I
~~
r--..
\0
EXPERIMENT II I
Judges C & D
Part 1
~
OJ
OJ
.l .. ~no
'u..
r
...,
'""" .s::..., "',_0<:: §>< OJu
'..... OJ ;;
0
:::;"' Vl
~
"'
:::1
-""
u
~ :; c:
OJ
;;::; cr a:t
...,
'..., "'
u "' "'"<:: "'
0..
~
!"'
:::1
~
3 ~
V)
tB
.s::
en
u"'
.,"' '""'>'-" ..., ....':;: u
·~
Vl
~
::t:
::l
~
OJ
>
';;;
...,
<::
Key:
~
"'
0
...,
N
·~
·;::
0
::t:
"'
"'
"'
V)
Routine "a" ·First Observation
0 1 1 0 1 1 1 0 1 0 0 0 1 1 0 0 1 1 1 0
Routine "a" Second Observation
0 1 1 0 1 1 1 0 1 0 0 0 1 0 l 0 1 1 0 1
Routine "b" First Observation
0 1 1 0 0 1 0 1 0 1 0 0 1 0 1 0 0 1 1 0
Routine "b" Second Observation
0
Routine "c" First Observation
0 1 0 0 0 1 1 0 0 0 0 1 1 0
0
1 0 0 0
Routine "c" Second Observation
0
1 1 0 0 1 1 0 1 0 0 0 1 0
0
1 0 0
1 1 0 0 1 0 1 0 0 1 0 1 0 1 0 0 1 1 0
1
0
=
item was circled
was not circled
= item
Agreement
Disagreement
I
I
16
4
18
2
17
3
18
2
19
1
18
2
1
0 1
Judae 0
·Routine "a"
First Observation
1 0
1 0 1 1 1 0 1 0 0 0 1 1 0 0 1 1 0 0
Routine "a" Second Observation
1 0 1 0 1 1 1 0 1 0 0 0
Routine "b"
0
0
Routine "b" Second Observation
0
0 1
Routine "c" First Observation
Routine "c" Second Observation
0
-·-·-
First Observation
0
1
0
0
0
1
0
0
0
1 0 1 0
1 0
0
1 0 1 0 0 0 1 0
0
0
1 0 1 0 1 0
0
0
0
1
0
0
0
1
0
0 1 0
0
0 1 1 0 0 1 0 0 0 0
0
1
0
0
0
1
1
0
0 1 1 0 1 0
0 0
0
1 0 0 0 1
1 0
0
ID
0
~
00
1.0
{;,
EXPERH1ENT I I I
Part 2
v
~
<-
<lJ
~ !] 1~ 1~ t~ lc51~ Ia; ~ ~~ lg'l:~..c: ~ 1~
Ol
Routine "a" First Observation
,.,
.....
S...O•r-+'
~
m
~
~
.....
r-r-:::J>
o
~
o
w
c..
E
~
~
o..
•r-
3
V')
"C
o
co
s...
0'1
.-
:X:
r-
:::1
u
'~
.....,.~
.....,.
VI
•r-
3
f-
S<lJ
>
Key:
.....
s:::
fi:
s...
4:0
0
::c
1 10 11 IO 11 11 10 IO IO IO IO lo lo 11 IO IO 11 11 10 IO
Judge B
0 11 11 IO 11 11 11 IO 11 IO IO IO 11 11 IO IO 11 11 11 11
Judge C
0 11 11 IO 11 11 11 IO 11 IO IO IO 11 11 IO IO 11 11 11 IO
Judqe D
1 lo 11 IO 11 11 11 lo IJ lo lo lo
In In
h h
= item
= item
was circled
was not circled
•r-
Judge A
11 h
1
0
3:1 I 2:21
t0ta1
13
3
4
20
11
7
2
20
i3
4
3
20
In In
Routine "b" First Observation
Judge A
o I o I o I o I o 11 I o ll I o I o I o I o Io I o 11 I o Io I o 11 IO
Judge B
1 IO 11 IO IO 11 IO 11 IO IO 11 lo 11 11 IO IO Ia 11 11 IO
Judge C
0 11 11 10 IO 11 IO 11 IO 11 IO lo 11 IO 11 IO lo 11 11 IO
Judqe D
o lo 11 lo lo 11 lo 11 lo 11 lo lo
h
In
h
In In In
1
1
In
\?
'
Routine "c" First Observation
Judge A
o 11 lo lo Ia 11 11 lo 11 Ia lo lo Ia Ia lo 11 Ia lo lo 11
Judge B
1 IO IO 11 IO 11 11 IO 11 11 10 lo 11 IO IO 11 IO IO 10 11
Judge C
0 11 IO IO 10 11 11 IO IO IO IO 11 11 IO IO 11 IO IO IO 11
Judqe D
o 11 10 lo 10
h h
In In
h
In In In In In
h
In In In h
I
C'l
\0
EXPERmENT I II
Part 2
-c .....,
Routine "a" Second Ob.servation
0
X
<II
QJ
3:
~
<II
~
..
rtf
E
c.
3:
o
I
S...r-
::s
3:
>:X:!/)
pp
lo n 11 P P
p 11 p p p
1 1o 11 to 11 11 11 to lo lo lo lo ~
1 to IO ll IO lo 11 lo 11 IO IO
Judge C
o 11 11 Jo 11 11 11 Jo 11 Jo Jo
Judge 0
_1
was not circled
Io .11 .lo._lLJ.J._h.J.n_.lLJn..Jo_b_b
3:1
2:21
10
2
20
7
3
20
b
8
h h h
nh~P,..v~tinn
Judge A
OIOI1IOIOI1IOI1IOIOIOIOIOIOI1IOIOIOI1IO
Judge B
11 Ol Ol 11 o I 11 o I 11 o I o 11 I o 11 11 I o I o I o 11 11 IO
Judge C
Judqe 0
I I, I011 I I0I, I0rI rI0I 11 I, Io·
011111 0 0
0
0
1 0
0
0 1 0 J .0
1 0 0 0
0
0
0
J 0 0 0
I
10
J 0
...
Routine "c" .Second Observation
Judge A
0111 Ol O!Of 1lliOIOIOIOIOIOIO!OI1IO IO IO 11
Judge B
1IOIOIJIOI1lliOI1IOIOIOI1IOIOI1IOIOIOI1
Judge 0
= item
OJ_,_,
VlCO:X:3:Uft-
Judge B
Judge C
1 = item was circled
0
C:~...C:aJ>V1
C..:::S·r-~Cl
t-t
Key:
"'
iii 4:0
~11~ ·~ ~I"
I
teta1
~ 1 ,.~~ ~-~ '::.::"~-1----:----,....--~--------­
UVIO'I
Judge A
Routine "b"s.,cond
~
s:: ,... .u.
0'1~<11,
~Is FE I~ ;;I~ :6I~ lc:
C
QJ
....s..
14
o
o
o 1,lolol,lololor·
olooo1ooo1
5
20