Teaching and learning analysis
using modern automatic speech
recognition with smartphones
Tuomas Virtanen, Jouni Viiri, Sami Lehesvuori, Pasi Pertilä,
Roberto Araya, Hanna Kronholm, Andre Mansikkaniemi
Academy of Finland and CONICYT, Chile
New Learning Environments and Technologies
Introduction
 Collecting educational data is getting easier and easier in modern
 Automatic analysis of classroom interaction would be a step
forwards when tracing distinct features from a large set of data
 It would not replace human as a researcher rather it unleashes
further resources on the topic being explored/investigated
 Instant feedback for professional development of teachers
Department of Teacher Education
technology-oriented society
 Data collected in educational settings
 Analyzing classroom interaction, for example, is extremely time
consuming and requires scholarly resources
Dialogic talk – Automatic speech recognition
research focusing on the dialogic teaching and features of talk
supporting this approach
 The automatisation of analysis is developed in collaboration
between University of Jyväskylä and Technical University of
Tampere
Department of Teacher Education
 The research aim of this study originates from our previous
Dialogic talk vs. Authoritative talk
Communicative approaches
Authoritative
(Focus on science view)
Interactive
question & answer routine
IRF
IRE
Non-interactive
teacher instruction
lecturing
probing
Dialogic
supporting
(Alternative views are considered)
elaborating
IRFRFRF
review
Department of Teacher Education
(Mortmer & Scott, 2003)
Dialogic indicators – What? Why? How?
 The thin line between dialogic and authoritative comes to finding
out how something is said in addition to what:
JOO ↑ or ↓
D
A
= OKAY
Department of Teacher Education
EXAMPLE:
Research questions
1.
2.
3.
What kind of dialogic features can be recognised from teacher
talk?
What forms of talk do students use (e.g., during group work)
How do the appearance of concepts change temporally in both
teacher and student talk? How the concepts are linked with each
other?
Department of Teacher Education
The main objective is to develop automatic speech recognition for
educational settings. This objective can be presented in following
research questions:
Types of talk and indicators
 Exploratory vs. factual (Sahin & Kulm, 2008)
• Indicators: Key words, intonation, wait time (e.g., Skidmore
& Murakami, 2010)
 RQ2: Forms of student talk
 Categorization in to Explorative, Cumulative and Disputative talk
(Mercer, 2004)
• Indicators: Key words, intonation, how many students are
verbally participating
 RQ3: Appearance of concepts
• Indicators: Detecting key concepts and temporal linking
Department of Teacher Education
 RQ1: Types of questions and teacher feedback
1.
2.
3.
4.
5.
6.
7.
Data capture
CIAE, TUT, JYU
Enhancement and Diarization
TUT
Keyword & name detection
TUT, CIAE , JYU
Concept coverage
CIAE, JYU
Teaching analysis
CIAE, JYU
Teacher and student support systems
CIAE, JUY
Comparison of teaching, teacher education research JYU, CIAE
Koulutuksen tutkimuslaitos - Finnish Institute for Educational Research
Department of Teacher Education
Work package overview
Paralinguistic features
 Is it possible to automatically detect paralinguistic features
Fundamental frequency contour of example question vs. nonquestion sentences
Department of Teacher Education
indicating dialogicity such as:
 Intonation
AUTOMATIC ANALYSIS: Paralinguistic features
 Is it possible to automatically detect paralinguistic features
Department of Teacher Education
indicating dialogicity such as:
 Wait time
Speech recognition
….power...
Why...
 Can characterize both the topic covered, and type of talk
Department of Teacher Education
 Automatic keyword detection
Dialogue analysis
 Who is talking?
 How frequently one is talking?
student 2
student 1 student 1
teacher
teacher
student 3
teacher
teacher
Department of Teacher Education
 How many people are participating verbally? Etc.
Data collection
(e.g., Quality of Instruction in Physics = QuIP)
 Collecting audio data via smartphones
 Questions to be solved
• How is this done?
• In what circumstances is this possible? (Privacy policy, Law
& Ethical issues)
Department of Teacher Education
 Use of existing video/audio data from previous research projects
Audio
• Annotations
• Transcriptions
• Teacher talk style
• Content information
Koulutuksen tutkimuslaitos - Finnish Institute for Educational Research
Department of Teacher Education
TUT – Audio analysis
• Pre-processing and analysis
• Audio capture and enhancement
• Diarization (who spoke when)
• Development of automatic extraction of indicators
• Prosodic features (intonation, stress, pitch, etc.)
• Keyword detection using ASR
Koulutuksen tutkimuslaitos - Finnish Institute for Educational Research
Department of Teacher Education
TUT – Audio analysis
Development of automatic analysis, using the
collected audio and annotations for
1. Concept coverage
2. Teacher talk style
3. (Teacher support system)
4. (Student support system)
Koulutuksen tutkimuslaitos - Finnish Institute for Educational Research
Department of Teacher Education
TUT – Audio analysis
Thank you
for your interest!
www.jyu.fi/edu/laitokset/okl/en
www.tut.fi/en/about-tut/departments/signal-processing/