The Effect of Speaking Rate and Experience on Cued Speech Transliterator Accuracy
Session 0665
Poster 45
Katherine Pelley, Dana Husaim, Morgan Tessler, Jessica Lindsay, and Jean Krause
[email protected], [email protected], [email protected], [email protected], [email protected]
-Previous work-
–
Handshapes convey consonants
Placements convey vowels
Consonants
Vowels
Monophthongs
5
1
Mouth
FR<
Diphthongs
Side
Side-throat
&MX\
YN5
the caves
Welsh
7
3
TJU
6IF<
rehearse
thug Joe
4
8
K«²‹²
fir tree
¡‹Q#
Aloha
C+C7
time out
Chin
Throat
Chin-throat
Wn'
too tall
DÀP
by when
• Correct cues had the highest frequency of occurrence (49%)
• Omissions were the most frequent type of error (33%)
Production category
• Moreover, most previous research tends to be qualitative
• Qualitative ratings provide important accuracy data, often
miffed
6
2
American Sign Language (ASL)
• Yet, 95% of classroom sign interpreters and transliterators
use an English-based sign system in their jobs (Jones et al., 1997)
Æ
Research is needed on interpreters using English-based
communication modes, including Cued Speech transliterators
-Effect of Experience-
Frequency of production
Correct cues
49%
Omissions
33%
n+G+
Oy vay!
73+
look at it
on a Likert scale (1-5), but finer resolution is needed to determine the relationship between accuracy and intelligibility
Æ
Quantitative research needed
Substitutions
18%
Insertions
6%
• Factors that are likely to affect Cued Speech transliterator
• Experience: increased accuracy would be expected with
Visit booth # 1328 in the Exhibit Hall for more information on Cued Speech
-Effect of Speaking Rate-
• How does the accuracy of Cued Speech transliterators,
• Many deaf children are mainstreamed and use an interpreter in school settings.
However, few guidelines are implemented in the field of interpreting, and the ones
that are in effect have no basis in research (Kluwin and Stewart, 2001)
measured by percent-correct cues produced, vary with:
• Speaking rate: slow, normal, fast
• Experience: novice, veteran
• Effect of lag time will be analyzed in follow-up work
-Accuracy vs. Intelligibility• When a deaf child uses an interpreter in a classroom, clarity of the “visual signal”
that the student receives is a necessary element of successful classroom communication. That is, ...
• The visual signal must be clear and received correctly (analogous to clarity of
“speech” for spoken languages)
• After the visual signal is correctly received, the student’s language skills will
play an important role in whether the meaning of the message is understood
• The clarity of the visual signal depends on the two channels in the communication
pathway:
1. The amount of information
preserved by the interpreter
2. The amount of information
accessible to the student
Accuracy
Intelligibility
-MethodsParticipants
• Six Cued Speech transliterators (CSTs), assigned to one of
two categories based on level of experience
• Two “novice” transliterators - had minimal or no certification and less than (the equivalent of ) one full-time year of
experience
• Four “veteran” transliterators - had the highest level of
certification and/or more than five years of experience
• On average, across all experience levels...
70
Interpreter
Student
• Accuracy: percent of message correctly produced by the interpreter/
• A common misconception is that intelligibility is equivalent to
accuracy:
Intelligibility
transliterator
• Intelligibility: percent of message correctly received by the deaf student
100
–
80
–
60
• The 50% intelligibility point depends on various factors
type of materials
amount of context
speaking rate
type of degradation
slow: 88 wpm (recording expanded by a factor of 1.25)
normal: 109 wpm (original)
fast: 137 wpm (recording compressed by a factor of 0.8)
• Speaking rate counterbalanced across segments
20
0
0
20 40 60 80 100
Accuracy
Intelligibility
(e.g. French & Steinberg, 1947; Miller & Nicely, 1955)
–
40
• However, research in other modalities suggests that such a
relationship is unlikely…
• In speech, for example, the relationship is sigmoidal
participant transliterated materials at three different speaking
rates
• Transliterators were presented with audio recordings of an
8th grade “lecture”
• The lecture was presented in three segments, each at a
different speaking rate
Procedures
• Transliterations were viewed in slow motion using Adobe Pre-
100
80
60
40
20
0
0
20 40 60 80 100
Available auditory
information
miere Pro 1.5, and each cue produced was classified in one of
four production categories
• Correct cues
• Omissions
• Substitutions
• Insertions
• Accuracy declines more sharply for novices than for veterans
• Increases in experience level are generally associated with increases in accuracy
N • Both novices
performed similarly:
o ~45%
correct
v ~50% omissions
~3% substitutions
i
Omitted large
c Æ
chunks of message
e Æ
Highly accurate
s
when cueing
3%
3%
Correct Cues
41%
48%
Substitutions
49%
Omissions
56%
CST3
• Three veterans
24%
28%
28%
57%
59%
V
e
t
e
r
a
n
s
60%
17%
12%
15%
CST2
CST5
CST6
V
e
t
e
r
a
n
s
• Remaining veteran (CST4)
had characteristics of both
groups:
• Similar to veterans in
substitutions (22%)
• Similar to novices in
omissions (50%)
50
40
22%
29%
performed similarly:
~60% correct
~25% substitutions
~15% omissions
Æ
Omitted only cues
within words or
shorter sequences,
not large chunks
Æ
Cued vast majority
of message
• However, some veterans (e.g. CST4) may not follow this pattern
• “Practice makes permanent,” and all transliterators should be monitored for
accuracy, regardless of experience level
• Nature of errors differs with experience level:
• Novices have a higher number of omissions
– Large chunks of the message are missing (from the mouth as well as the hands)
– Apart from these omissions, cueing is very accurate
• Veterans produce more substitutions
– Omissions tend to be confined to cues within words or shorter sequences
– Substitutions reduce overall accuracy, but vast majority of message is transliterated
Accuracy vs. Intelligibility
• The bad news: Accuracy of “typical” CSTs is substantially lower than 100%
• Some highly experienced veterans are likely to be much more accurate than
the veterans examined in this study
• However, these transliterators were randomly selected and are representative
of at least some segment of “typical,” working transliterators
Æ
Increased transliterator training and professional development opportunities
should be created to address these issues in working transliterators
• The good news: Intelligibility is likely to be somewhat higher than accuracy
1. Accuracy scores reported here are conservative
– Many substitutions are likely to be partially correct (e.g. right handshape, wrong
placement), and no partial credit was awarded
2. When words are mouthed, this information is available even if cues are wrong
3. Accuracy could be higher for key words than non-key words
4. Transliterators could have preserved message content via use of paraphrase
-Future Work• Collect additional accuracy data (more transliterators, more experience levels)
49%
• Conduct further analysis of accuracy results (e.g. partial credit for substitutions)
CST4
• Evaluate intelligibility of videos that were analyzed for accuracy in this study
(i.e. present to deaf persons who use CS transliterators)
Rate x Experience Interaction
30
• Are novices and veterans affected differently by
20
speaking rate?
• Slope analysis: calculated the slope of each
10
transliterator’s performance as a function of speaking rate, measured in words per minute
0
Correct Cues
Omissions
Substitutions
Insertions
Æ
The decline in accuracy was mostly caused by an increase
in omissions
• In addition, the negatively relationship between accuracy and
speaking rate was exhibited by all transliterators
CST1
CST2
CST3
CST4
CST5
CST6
80
Correct cue frequency (%)
Teacher
• Clarity of the visual signal is easily quantifiable
N
o
v
i
c
e
s
Slow
Normal
Fast
60
Materials
• Video recordings of the cued messages produced when each
• Increases in speaking rate have a negative effect on accuracy
• Were more accurate than novices (60% vs. 45%)
• Produced fewer omissions than novices (15% vs. 50%)
• Correct cues: negative relationship with speaking rate
• Omissions: positive relationship with speaking rate
• Substitutions and insertions: no effect of speaking rate
Frequency of production (%)
-Purpose of Study-
• Most veterans (one exception)…
V
e
t
e
r
a
n
s
1986)
• Deaf people often use interpreters to facilitate communication. Types of interpreters vary with communication mode:
• American Sign Language (ASL) interpreter
• Signed English transliterator
• Cued Speech transliterator
Summary of Results
CST1
accuracy include
• Rate: in ASL (direct communication), rate of signing is
inversely proportional to comprehension (Fischer and Reed, 1999)
• Lag time: for ASL interpreters, less lag time between the
speaker and the interpreter results in more miscues (Cokely,
increased level of experience
L0V5
young church
-Conclusions-
Overall Effect of Experience
• All veterans produced more frequent substitutions than novices (22% vs. 3%)
(e.g. Strong and Rudser, 1986)
OHV
deep azure
• Of the limited research available, most studies focus on
-Overall Results• On average, across all speaking rates and experience levels...
Average
• As speaking rate increased,...
• Correct cues: declined more steeply for novices
than for veterans (-0.6% vs. –0.4% per wpm)
• Omissions: increased more steeply for novices
• Substitutions
– novices: remained steady or declined
–
60
50
40
most veterans: generally increased
(+0.05, +0.17, and +0.20% per wpm), but CST4 more like
novices (-0.07% per wpm)
• Insertions: Given the low incidence of insertions,
no pattern was apparent
30
20
10
0
slow
normal
Speaking Rate
fast
• Repeat experiments for other communication options (SEE, CASE, ASL) so that
80
• Accessibility can be ensured for all deaf children who use interpreters in the
70
classroom, regardless of preferred communication mode
60
• Training and testing of interpreters/transliterators can be fine-tuned based on
evidence collected (e.g. most appropriate accuracy and/or speed requirements)
50
40
• Compare psychometric functions (key-word intelligibility vs. accuracy) across
communication options in order to
• Increase understanding of intelligibility of visual signals
• Gain insight into modality-independent aspects of perception
30
20
10
0
than for veterans (+0.6% vs. +0.3% per wpm)
(-0.01 and -0.05% per wpm)
70
Frequency of Omissions (%)
–
• Few previous studies are available on interpreter accuracy
slow
CST1
Frequency of Substitutions (%)
-BackgroundCued Speech is a visual communication system that uses hand “cues” along with
the mouth movements of speech to make the phonemes of a spoken language clear.
• Cues are synchronized with mouth movements
• Each cue corresponds to a consonant-vowel combination:
Communication Sciences and Disorders
normal
CST2
CST3
CST4
fast
CST5
-References-
CST6
1. Cokely, D. (1990). “The effects of lag time on interpreter errors,” Sign Language Studies, 53, 341-375.
2. Fischer, S.D., Delhorne, L.A., & Reed, C.M. (1999). “Effects of rate of presentation on the reception of American Sign
Language,” J. of Speech & Hearing Research, 42, pp.568-582.
3. French, N.R. & Steinberg, J.C. (1947). “Factors governing the intelligibility of speech sounds,” J. Acoust. Soc. Amer.,
19:1, pp. 90-119.
4. Jones, B., Clark, G., & Soltz, D. (1997). “Characteristics and practices of sign language interpreters in inclusive education programs,” Exceptional Children, 63, 257-68.
5. Kluwin, T.N. & Stewart, D.A. (2001). “Interpreting in schools: A look at research.” Odyssey, 2, 15-17.
6. Miller, G.A. & Nicely, P.E. (1955). “An analysis of perceptual confusions among some English consonants,” J. Acoust.
Soc. Amer., 27:2, pp. 338-352.
7. Strong, M. & Frisch Rudser, S. (1986). “The subjective assessment of sign language interpreters,” Sign Lang. Studies,
49, 343-362.
fast
The authors wish to thank Wendy Park for her helpful tutoring in the fundamentals of the Cued Speech system.
Financial support for this work was provided in part by a grant from the National Institute on Deafness and
Other Communication Disorders (NIH Grant No. 5 R03 DC 007355).
35
30
25
20
15
10
5
-Acknowledgments-
0
slow
normal
Speaking Rate