A Research Agenda for Transforming Pedagogy and Enhancing Inclusive
Learning through Synchronised Multimedia Captioned Using Speech
Recognition
Mike Wald
School of Electronics and Computer Science
University of Southampton
United Kingdom
[email protected]
Abstract: Many exciting opportunities for learning occur in environments where communication
occurs spontaneously in real-time through speech, gestures, writing, drawing or actions. Some
students can find it difficult to learn and take notes in these situations and this can be especially
problematic for disabled students and non-native speakers. This paper will discuss an agenda for
researching how multimedia captioned using speech recognition can make learning more
interactive, inclusive, personalised, flexible and productive for students at any stage or phase of
learning through the synchronised text supporting sensory and memory demands of spoken
language and enabling the audio visual material to be manipulated by searching and annotating.
Introduction
This paper discusses a proposed agenda for researching how synchronised captured spontaneous multimedia
captioned using speech recognition (SR) can transform and enhance inclusive learning for all students at any stage
or phase of learning. The research would develop proof of concept technology prototypes and knowledge and
understanding of how to create and exploit SR and related technologies to transform pedagogy and make learning
more interactive, inclusive, personalised, flexible and productive. An interdisciplinary conceptual framework also
needs to be developed to understand the relationship between technology and pedagogy.
Many exciting opportunities for learning occur in environments where communication occurs spontaneously in realtime through speech, gestures, writing, drawing or actions. Students can find it difficult to learn and take notes in
these situations (Piolat et al. 2004) and this can be especially problematic for disabled students (Fuller et al. 2004a)
and non-native speakers (Barbier and Piolat 2005). A systematic analysis of barriers to learning experienced by
disabled students in higher education (Fuller et al. 2004b) showed that of the 10% of undergraduates who had
declared a disability nearly half found problems with listening or watching and making notes. Students who do not
declare a disability receive less support and so can have even more problems.
Currently some educational institutions are digitally recording classes to provide students with revision material or
substitute learning experiences (Mediasite 2006). The recordings can include iPod videos (Kiernan 2006) with slides
and instructor annotations (Blackboard 2006) downloadable onto students’ personal devices to support their memory
through allowing them to replay the material at preferred times and locations. However these approaches can
encourage passive rather than active learning if opportunities are not provided for students to interact with the
content, teachers or each other (e.g. annotate, search, question, discuss or collaborate etc.). Disabled students are
greatly disadvantaged by the lack of captioning of these multimedia recordings with the very high costs of manual
captioning often cited as the reason for non-compliance with guidelines and Disability Discrimination Legislation
(WAI 1999, SENDA 2001). Taking notes can be an active learning experience (Kress 2003) and this can be
facilitated by capturing personal ideas, annotations and questions synchronised with recordings to allow interaction,
review and collaboration (Mühlhäuser & Trompler 2002).
The provision of multimedia consisting of text captions synchronised with recorded speech and images enables their
communication qualities and strengths to be available for different contexts, content, tasks, learning styles and
preferences. Text can reduce the memory demands of spoken language, speech can better express subtle emotions,
while images can communicate moods, relationships and complex information holistically. The audio visual
material can also be manipulated through searching and annotating the synchronised text. Since there is little
evidence that students’ preferred learning media can be predicted reliably through learning style instruments
(Coffield et al. 2004), ensuring every student has a full personal choice of media can only be achieved by always
providing text captions. Some students may find the colloquial style of text transcribed verbatim from spontaneous
speech intended to be listened to by an audience easier to follow than academic text written with the intention of it
being read silently.
Anticipated learning outcomes for disabled and non-disabled students include enhanced learning, motivation,
engagement, concentration, note-taking, participation, interaction, confidence and self esteem. Anticipated outcomes
for teachers include enhanced teaching and communication facilitated by easier reflection and review of teaching,
and new methods of interactive teaching, learning and assessment due to students’ improved learning and
interaction.
Research Background
The potential for learning to be enhanced through interacting with captioned synchronised multimedia is supported
and informed by the results of Wald’s detailed review of relevant research (Wald 2006a):
 Disabled and non-disabled students and teachers provided with SR spontaneous speech captions
automatically formatted by pause length and viewed live or synchronised with recorded speech and slides
on browsers or SMIL players found teaching and learning improved and developed strategies to cope with
recognition errors. Standard SR software was found to be unsuitable as it required dictation of punctuation;
 An interactive user interface using synchronised multimedia that best communicates the information and
encourages the user to actively process the information can improve a person's ability to learn and
remember;
 Multimedia should be designed to provide tools for annotation and collation of notes to stimulate learner
engagement and should accommodate the different abilities and styles of learners, allowing learners to
focus attention on one single media resource at a time if preferred;
 Time synchronising handwritten notes of teachers and/or students with recorded lecture audio, video and
slides facilitates interaction with the material;
 Playback at up to twice normal speed allows faster scanning through familiar material;
 SR offers the opportunity for cost effective automatic captioning to facilitate keyword searching or
phonetic searching (which can find words not known to the system);
 Recordings encourage reviewing of lectures shortly after they occurred and also later for revision without
negatively impacting on attendance;
 A linear screen display is not as easy as paper for most people to interact with and so can benefit from
navigation and searching tools;
 The optimum display methods for hand held personal display systems may depend on the size of the screen
and so user options should be available to cope with a variety of devices displaying captions.
Research into SR captioned multimedia is synergistic with research into ways to store, access, search and use
multimedia information (Taylor et al. 2004) and related to the fulfilment of Vannevar Bush's 1945 Memex vision
(Bush 1945, Memories for Life 2006, MyLifeBits 2006).
Universal Design is interpreted by some as designing for ‘all users’ and by others as designing for ‘the widest range
of people’ (Vanderheiden 1996). Universal Instructional Design (UID) is defined as the flexible design of
instructional materials and activities that make the learning goals achievable by individuals with wide differences in
their abilities (Burgstahler, 2001). For some, ‘flexible design’ means adopting a proactive approach where student
diversity is anticipated and an integrative approach is adopted in preference to multiple separate solutions (Scott et
al. 2003) while for others it means being prepared to make accommodations, modifications or exceptions (Pliner and
Johnson 2004, Ouellett 2004).
Technologies to be Researched and Developed
Required technologies need to be able to digitally record and replay multimedia content through multimodal
interfaces automatically creating an indexed SR transcript of spoken language displayable on different devices and
synchronised with audio, video, still images, whiteboard, slides, on-screen graphics, teacher and student notes,
annotations and bookmarks.
The commercial market does not currently provide the required SR captioning technologies for education because of
readability and accuracy limitations and speech signal quality issues (i.e. while it is possible to obtain dictation
accuracies greater than 95% when reading from a script using a headset noise cancelling microphone, people do not
dictate punctuation or pronounce words clearly when talking spontaneously and the signal to noise ratio decreases
when using lapel or handheld microphones). Lower accuracy SR (e.g. less than 70%) can however still facilitate
searching and interaction and in addition to addressing accuracy and speech quality issues research should also
indicate how pedagogy can be transformed and learning enhanced as SR technologies improve in the future.
interfaces and display devices
edit
search, index, summarise
ANNOTATION
share and collaborate
SR CAPTIONS
annotate/associate/hyperlink
VIDEO & IMAGES
Personal:
Students
Control
AUDIO
Public:
Teachers
control
SCREEN
Synchronise
spontaneous speech, writing, drawing, activities
Technologies should be able to work with live and recorded multimedia. Users should be able to adjust replay speed
while maintaining audio pitch; create their own structures using ‘tags’ and hyperlinks to other resources; search
phonetically or by word and edit and collaborate. The speaker and any sounds being made (e.g. laughter, music) can
be indicated in the same way as TV subtitling/captioning. Student and teacher created tags, folksonomies, metadata,
taxonomies and ontologies could support the structuring of recorded multimedia and its reuse as learning objects.
The research should also consider interoperability standards, accessibility, usability, new forms of multimodal
interface, intellectual property and privacy. The importance of SR punctuation, utterance segmentation and errors on
readability should also be studied (Jones et al. 2003). Usability studies can involve a range of techniques including
using real time editing (Wald 2006b) or stenography to compare perfect transcription with differing error rates.
Through GPS enabled wireless personal devices coupled with absolute time recordings and individual’s calendars,
personal histories and locations can be integrated with public recordings of learning events to note where and when
words were said and so help put them in context. A networked laptop, tablet or handheld computer would allow for
a personal customised interaction and display of the synchronised captioned transcript with annotations, notes and
drawings. Smaller and more limited devices (e.g. mobile phone text messaging, keypad voting systems) could also
be used for interaction. An overview of the integration of the technologies to be researched is visualised in Figure 1.
NOTES
Figure 1. Technologies to be researched
Beneficiaries of Research
The research could benefit all students and teachers in all subjects and at any stage or phase of learning and if the
technologies were always available support for disabled students could be ubiquitous and not dependent on self
disclosure of disability. For example, when SR technology was used to provide both live and archived verbatim
transcriptions of lecturers’ speech for all students in classes (Leitch & MacMillan. 2003) over 90% of the students
who did not have disabilities also found it was useful and improved their notetaking and the disabled students
appreciated they were not the only ones to benefit. The research could help transform pedagogy by allowing
students to concentrate more on thinking, questioning and actively learning rather than using all their mental effort
attempting to record everything through notetaking.
Inclusion
Enhancing inclusion can include providing greater accessibility to learning technologies and motivating excluded
learners through more engaging learning experiences. This can occur through: a) providing captioning of speech for
deaf learners, or any learner when speech is not available or suitable (including assisting non-native speakers to
understand or manually or automatically translate spoken words they cannot recognise or understand); b) supporting
those who find notetaking difficult for cognitive, physical or sensory reasons or unable to attend classes for medical
or mental health reasons; c) helping blind, visually impaired or dyslexic learners to use and search learning material
more easily by augmenting synthetic speech with natural recorded real speech; d) helping engage disaffected
learners through accessible and personalised learning focusing on individual needs.
Personalisation and Flexibility
Enhancing personalisation and flexibility can include: a) creating personally meaningful and rewarding learning
experiences; b) achieving a better match with learners’ needs and dispositions; c) enabling learners to make
informed choices about their own learning and to learn where, when and with whom they want, in ways that suit
their approaches to learning; d) helping to match the needs, abilities, aspirations, and circumstances of learners to
achieve a higher order of learning outcome. This can occur through: a) personalised use in formal and informal
settings whether face-to-face or virtual online (e.g. lectures, tutorials, seminars, workshops, field trips, work-based
learning) by integrating and synchronising personal searchable notes/annotations (e.g. typing, writing, drawing,
mindmaps); b) individual use (e.g. recording verbalised thoughts accompanying reading, writing, drawing); c)
collaborative use (e.g. discussions, support for community memories); d) choice of communication modality and
format and interaction style to support preferred learning styles.
Productivity
Enhancing productivity can include: a) achieving higher quality and more effective learning in affordable and
acceptable ways; b) facilitating reuse, collaborative development and sharing of technology enhanced learning
resources; c) improving productivity of practitioners’ and learners’ time through providing evidence for assessment
and E-portfolios (e.g. through selections of relevant sections) and through efficient creation of synchronized editable
multimedia resources for learning that can be structured, reused (e.g. as learning objects) and searched and managed.
Assessment
Students’ synchronised annotations and questions can provide teachers with more detailed feedback about
understanding and difficulties encountered than keypad electronic voting systems and so also facilitate appropriate
responses by teachers to individuals and groups. Technologies could support and facilitate innovative assessment
approaches (e.g. students submitting reflective annotated links to multimedia clips of participation in seminars as
evidence for demonstrating required learning outcomes). The technologies could also support teachers and students
describing and redescribing conceptions and reflecting on action and interaction (Laurillard 1993) and the
opportunities for reflection and review will also improve teaching and spoken communication and encourage
understanding and creativity.
Evaluation
James and Brown (2005) discuss the importance of explaining how an approach or intervention influences learning
as well as describing the effect and James et al. (2005) note that for ‘users’ and ‘researchers’ to have confidence in
the results of research requires their continuous collaboration and that conclusions reached from the evidence from
empirical, theoretical, cumulative and user bases need to be better than any possible alternative conclusions.
Interdisciplinary research therefore can benefit from subjective judgments by teachers and students ascertained
through interviews, questionnaires and focus groups, triangulated with objective measures in usability studies and on
actual courses e.g.:
 Analysis of the use made by students of the technologies through data capture techniques such as key and
mouse logging and screen capture as well as measurement of time the resources were used and how they
were used and times and errors when completing set tasks;
 Analysis of grades/marks and quality to evaluate how the learning on parts of a course using the technology
compare to learning on those parts when the technology wasn’t used.
Conclusion
This paper has proposed and discussed an agenda for researching how synchronised captured spontaneous
multimedia captioned using speech recognition can transform and enhance inclusive learning for all students at any
stage or phase of learning. Although there is much supportive background research and many of the required
technologies have been identified, in depth objective and subjective studies in authentic learning contexts will be
required to inform further research and show how and to what extent the technology can enhance learning and
transform pedagogy.
References
Barbier, M. L., Piolat, A. (2005). L1 and L2 cognitive effort of notetaking and writing. In L. Alla, & J. Dolz (Eds.).
Proceedings at the SIG Writing conference 2004, Geneva, Switzerland.
Blackboard (2006) Retrieved April 8, 2006, from
http://www.blackboard.net/company/press/release.aspx?id=800330
Burgstahler, S. (2002) Distance learning: universal Design, universal Access, Educational Technology Review, 10,
1, Online. Retrieved October 5 2005 from: http://www.aace.org/pubs/etr/issue2/burgstahler.cfm
Bush (1945) As We May Think, The Atlantic Monthly, 176(1), 101-108
Retrieved April 8, 2006, from http://ccat.sas.upenn.edu/~jod/texts/vannevar.bush.html
Coffield, F., Moseley, D., Hall, E., Ecclestone, K. (2004) Learning styles and pedagogy in post-16 learning: A
systematic and critical review, Learning and Skills Research Centre
Fuller, M., Bradley, A., Healey. M. (2004a) “Incorporating disabled students within an inclusive higher education
environment” Disability & Society Vol. 19, No. 5: 455-468
Fuller, M., Healey. M., Bradley, A., Hall, T. (2004b) "Barriers to Learning: a systematic study of the experience of
disabled students in one university", Studies in Higher Education, Vol. 29, No. 3: 303-318
James, M., Brown, S. (2005) Grasping the TLRP nettle: preliminary analysis and some enduring issues surrounding
the improvement of learning outcomes. The Curriculum Journal 16(1) pp. 7-30
James, M., Pollard, A., Rees, G., Taylor, C. (2005) Researching learning outcomes: building confidence in our
conclusions. The Curriculum Journal 16(1) pp. 109-122
Jones, D., Wolf, F., Gibson, E., Williams, E., Fedorenko, F., Reynolds, D. A., Zissman, M. (2003). Measuring the
Readability of Automatic Speech-to-Text Transcripts, Proc. Eurospeech, Geneva, Switzerland
Kiernan, V (2006) Retrieved April 8, 2006 from
http://www.tegrity.com/Articles/Tegrity_The%20Chronicle%20of%20Higher%20Education_22406.pdf
Kress, G. (2003) Literacy in the New Media Age,London:Routledge
Laurillard. D. (1993), Rethinking university teaching – a framework for the effective use of educational technology,
London: Routledge
Leitch, D., MacMillan, T. (2003). Liberated Learning Initiative Innovative Technology and Inclusion: Current Issues
and Future Directions for Liberated Learning Research. Saint Mary's University, Nova Scotia. Retrieved April 10,
2006, from http://www.liberatedlearning.com/
Lifeblog (2006) Retrieved April 8, 2006, from www.nokia.com/lifeblog
Mediasite 2006 Retrieved April 8, 2006, from http://www.sonicfoundry.com/
Memories for life (2006) Retrieved April 8, 2006, from http://www.memoriesforlife.org/
Mühlhäuser, M., Trompler, C. (2002). Digital Lecture Halls Keep Teachers in the Mood and Learners in the Loop.
World Conference on E-Learning in Corp., Govt., Health & Higher Ed. 2002, Montreal, Canada. 714-721
MyLifeBits (2006) Retrieved April 8, 2006, from
http://research.microsoft.com/barc/mediapresence/mylifebits.aspx
Ouellett, M.L. (2004) Faculty development and universal instructional design, Equity & Excellence in Education,
37,135-144.
Piolat, A., Olive, T., Kellogg, R.T. (2004). Cognitive effort of note taking. Applied Cognitive Psychology, 18, 1-22
Pliner, S.M. and Johnson, J.R. (2004) Historical, theoretical and foundational principles of universal instructional
design, Equity & Excellence in Education, 37, 105-113.
Scott, S.S., McGuire, J.M. and Foley, T.E. (2003) Universal design for instruction: a framework for anticipating and
responding to disability and other diverse learning needs in the college classroom. Equity and Excellence in
Education, 36, 1, 40-49.
SENDA (2001). Retrieved April 12, 2006, from http://www.opsi.gov.uk/acts/acts2001/20010010.htm
Taylor, J. de Roeck, A. Anderson, A. Sharples, M. Tom Boyle, T. Hall, W. Rodden, T. Scott, D.
(2004) A Grand Challenge for Computing: Learning for Life Retrieved April 8, 2006, from
http://www.bcs.org/upload/pdf/learninglife.pdf
Vanderheiden, G. (1996) Universal design. What it is and what it isn’t. Retrieved October 9, 2006, from
http://trace.wisc.edu/docs/whats_ud/whats_ud.htm
WAI (1999). Retrieved April 12, 2006, from http://www.w3.org/WAI
Wald, M. (2006a) Learning Through Multimedia: Automatic Speech Recognition Enabling Accessibility and
Interaction. Proceedings of ED-MEDIA 2006: World Conference on Educational Multimedia, Hypermedia &
Telecommunications, Montreal, AACE pp. 2965-2976.
Wald, M. (2006b) Creating Accessible Educational Multimedia through Editing Automatic Speech Recognition
Captioning in Real Time. International Journal of Interactive Technology and Smart Education: Smarter Use of
Technology in Education 3(2) pp. 131-142.