Providing Natural Means for Teacher in Smart
Classroom for Tele-education *
Yuanchun Shi, Weikai Xie, Guangyou Xu, Enyi Chen, Yanhua Mao, Fang Liu
Department of Computer Science and Technology
The State Key Laboratory on Intelligence System and Technology
Tsinghua University, Beijing 100084
[email protected]
Abstract Smart Classroom is an ongoing project dedicated to develop a Smart
Space augmented classroom in which Tele-education can be seamlessly
integrated with face-to-face education. In this paper, we describe the features of
Smart Classroom, especially those may help to get a better user experience.
Pen-based UI on the projected display of the teaching slides provides just the
same experience as using a real blackboard or whiteboard for writing. Laserpointer To Cursor together with the Voice Command helps the teacher to
communicate with students and control the content of presentations. Virtual
Assistant can enhance the speech interaction between the teacher and the
system. And Smart Cameraman module can automatically capture the proper
view of the classroom according to the context of the class as the video feed for
the remote students.
Keywords: Smart Spaces, Smart Classroom, Natural User Interface
1 Introduction
A promising application domain of UbiComp is the education. The Classroom
2000 project [1] at GIT has been devoted into the automated classroom experience
capturing. Besides, the Smart Classroom project at our institute is focused on Teleeducation. Most currently deployed real-time Tele-education systems are desktopbased, in which the teacher’s experience is totally different from teaching in a real
classroom. For example, he/she should remain stationary in front of a desktop
computer instead of moving freely in the classroom. The goal of Smart Classroom
project is to eliminate the discrepancy of a teacher’s experience in Tele-education and
that in the traditional classroom education, as well as enable the integration of these
two currently separated education practices. Our approach was to move the legacy
real-time Tele-education system off the desktop, extending its user interface into the
3D space of an augmented classroom (called Smart Classroom) so that in this
*
The related research is supported by NSFC, “863” High-tech Plan, The Ministry of Education,
China and IBM China Research Lab.
classroom the teacher could interact with the remote students with multiple natural
modalities just like interacting with the local students.
Smart Classroom
Camera
Camera
Display
Camera
Server
Fig. 1. Smart Classroom System
Fig.1 shows the general idea of Smart Classroom. A real classroom is instrumented
and turned into a natural user-interface of the Web-based tele-education software for
the teacher side, where teachers can move freely and use the conventional teaching
metaphors that they are familiar with to give classes to remote students, instead of the
cumbersome desktop-based metaphors used in most of current real-time teaching
systems. Since the teacher is in a real classroom, we can accommodate local students
presenting in the classroom at the same time. This way we can eliminate the
difference of on-campus learning and tele-education activities, because teachers can
give classes to local students and remote students simultaneously, which can reduce
the required workforce of teachers as compared to separate operation of on-campus
education and tele-education. Furthermore, the process of the lecture will be recorded
as a hypermedia courseware for playback after class.
The rest of the paper will be organized as following: First, we give a scenario of
Smart Classroom as a teacher’s experience. Then, we describe the setting of the
classroom, user interface technologies and the smart cameraman facility respectively.
After a short statement of the software infrastructure which integrate the functional
modules, we finally give a conclusion.
2. Scenario
When Prof. Shi walks into the Smart Classroom, her biometrics features are
detected automatically and the room recognizes her. So the classroom says “Good
morning, Prof. Shi, you can give your lecture now.” She goes to the whiteboard. Then
she finds her lecture, the lecture for Digital Logic course, displaying on the
whiteboard just as where she stops in the last class and finds the representing image of
remote students, who attend her lecture by the Internet, are also displayed on a wall.
Thus Prof. Shi begins her lecture.
When she gives her presentation, just like in a normal classroom, she can make
annotations on the blackboard with electronic chalk. Besides, she can also control the
display of the lecture conveniently by her speech or a laser pen on her hand. When the
remote students meet some difficulties on understanding her lecture, they can consult
Prof. Shi, just like the local students. For example, the classroom says, “Tom has a
question now.” Prof. Shi replies “Tom, go ahead.” So Tom’s audio and video will be
presented in the classroom. He can also sketch his question on same board where
Prof. Shi presents her slides and adds annotations.
The remote students can get the live video and audio of the classroom with
different focus according to what the teacher is illustrating. For example, when Prof.
SHI makes annotations on the blackboard, the video focus on blackboard will be fed
to them; when she explains a realia in her hand, the video focus on the realia will be
fed to them; when she interprets a concept on the dais, an overview scene of the
classroom will be fed to them.
Actually this scenario can be accessed as a video demo on our Website[2].
3 Room Setting
According to the scenario, besides distance learning technologies, teachers can
have the similar experience in Smart Classroom as in a traditional face-to-face
classroom which is most familiar and efficient.
Smart Classroom is inspired by the research filed of Smart Space. A Smart Space
is a richly instrumented physical environment where people can get transparent access
to information and assistance from computers while performing their ordinary tasks in
this space.[3]
There are many research institutes carrying out research on Smart Spaces with
different emphases. Interactive Workspaces in Stanford [4] is the multi-device, multiuser environments for people to work together in technology-rich spaces with
computing and interaction devices on many different scales. Intelligent Rooms in
MIT [5] have cameras, microphone array, speech command to try to interpret what
people are doing in order to help them. Easy Living in Microsoft Research [6] is an
intelligent environment for people to live easily. Dream Space in IBM Research [7]
allow humans to interact with the real world in natural ways, using the common skills
of speaking, gesturing, glancing, moving around, reaching out. Aware Home in
Georgia Tech [8] builds a home, which provides services to its residents to enhance
their quality of life or help them to maintain independence as they age. Despite of
having different purpose, all of them must take consideration of interaction with real
world.
Smart Classroom is just such an effort to turn an ordinarily classroom into a Smart
Space for Tele-education.[9] We augment an ordinary classroom with wall-sized
displays, sensors, cameras and the associated software modules such as gesturetracking module and speech-recognition module so as to allow the teacher presenting
in the classroom to access the tele-education software transparently, rather than
appeal to a desktop computer. Through Smart Classroom, we actually extend the user
interface of the teacher’s interface from a desktop computer into the 3-D space of the
classroom.
The space of a classroom usually can be divided into two parts - the teaching area
that mostly used and occupied by the teacher, and the audience area where local
students reside. The most significant physical instrumentations in a Smart Classroom,
compared to an ordinary one, take place in the teaching area, as illustrated in Fig. 2.
They are two large projector screens. The one on the front wall is a touch-sensitive
screen, replacing the usual blackboard or whiteboard found in a classroom. It’s called
Mediaboard to display teaching materials. Remote students can view the content of
this board on their client program (which is only desktop-based). Another screen
called Student Board is on the sidewall. It is the window to remote students, on which
the image of remote students participating a session will be displayed and the video
and audio of the remote student who takes the floor will be played here too. The
teacher and local students become aware of the presence of remote students through
this facility
Remote Students
Studentboard
Mediaboard
Virtual
Assistant
Fig. 2. A snap shot of the Smart Classroom
Besides these two obvious facilities, there are about a half-dozen cameras, each
with different usage, installed at different places in the classroom. Some are used to
recognize the action of the teacher and the result is interpreted as an interaction
command to the underlying tele-education software. Others are used to capture the
live video of the classroom for broadcasting to the remote students. In addition, the
teacher wears a wireless microphone for capturing his speech.
4. Natural Teaching Experience
In Smart Classroom, in order to give classes to remote students, the teacher no longer
needs to remain stationary in front of a desktop computer and, for most of common
tasks involved in a class, the teacher no longer needs to use keyboard and mouse
either. To this end, the following technologies are developed and integrated in the
Smart Classroom.
4.1. Pen-based UI
As mentioned above, the Mediaboard is displayed on a touch-sensitive screen,
which is actually a commercial product called SmartBoardTM. Most functions of a
mouse found in a desktop setting can be accomplished by operating directly on this
board. For example teachers can control the display of the slides. Moreover, using the
provided pens and erasers, teacher can write comments and scribbles on the slides or
wipe the strokes, as illustrated in Fig.2, which provides just the same experience as
using a real blackboard or whiteboard. And the remote students will get to see the
presentations and writings on the board, as shown in Fig.3.
List of
participating
students
Mediaboard
and the
slides
displayed on
it
Video from
Teacher
side
Fig. 3. Remote Students’ Client
At the beginning of the project, we ever decided to develop a mechanism that can
have the same function as a mouse but do not ask the teacher approach to a desktop
computer.[10] Therefore we developed the Virtual Mouse module, which is based on
a hand-tracking algorithm. The teacher’s hand movement on the vertical plane was
tracked when standing steps away from the Mediaboard and turned into the
movement of the cursor on the board or strokes on the slides. However, the user
experience turned out to be frustrating for two reasons: 1) The skin-color modal and
appearance modal of hand used to locate the teacher’s hand is inevitable unreliable in
a setting like Smart Classroom where front-projection projector is used, because the
environment illumination changes as the projected content changes. The case is even
worse when the light of the projector shot on the teacher’s hand directly. 2) While the
metaphor here, using teacher’s finger as an extended pointer device, is natural for
driving the cursor, but not for draw scribbles on the slides, for one is difficult to keep
his/her hand movement smooth without a firm support point of his/her arm.
That’s why we gave up the idea of Virtual Mouse but developed other two
mechanisms which together can achieve the same goal but more reliable. First, to
allow the teacher stably draw scribbles on the slides, we adopted SmartBoard. Second
mechanism is the Laser Pointer To Cursor module we developed.
4.2. Laser Pointer To Cursor
Laser pointers are widely used as a tool for indicating the focus nowadays. Besides
this common utility, in Smart Classroom the teacher can even use a laser pointer as an
interactive tool. The function is provided by a computer vision module that can track
the movement of the spot of the laser pointer and recognize its certain movement
patterns. This recognition result is interpreted differently according to whether the
spot is on the Mediaboard or the Studentboard.
If on the Mediaboard, the movement of the spot is interpreted as the mouse
movement event and circling the spot around a point is interpreted as the mouse-click
event, so that the teacher can select and open a link on the HTML page by laser
pointer. Although the same task can be completed by manipulating on the Mediaboard
directly, this is nevertheless useful, for the teacher do not need to approach the
Mediaboard every time he/she need to open a link. As the laser pointer can be
tracked, its position can be always recognized as the cursor’s position which can also
help to catch the remote students’ attention just like it works in the classroom for the
local students.
If the spot is on the Studentboard, the recognition result is interpreted as the current
selection of a remote student, and as an indication, this remote student’ representing
image will be highlighted. Together with voice command, it provides a convenient
method for the teacher to control the floor between remote students. For example, the
teacher can point the laser pointer on a remote student’s representing image and say
“Go ahead” (as shown in Fig.4), which will result the system to give the floor to this
remote student.
Go ahead
The representing image
of a remote student is
highlighted as the
teacher pointing the laser
pointer on it
Fig. 4. Give the floor to a remote student by a laser pointer plus voice command
4.3. Avatar as Virtual Assistant
The Smart Classroom incorporates a speech-recognition module as well as a text-tospeech module. Therefore the teacher can complete several common tasks involved in
a class by voice command such as “Give the floor to Tom” or “Jump to the previous
page”. The system can also use the synthesized voice to notify the teacher of certain
events. For example when a remote student named Peter asks for the floor, the system
will alert the teacher “Peter is asking for the floor.” However interact with a dummy
room with voice will seem funny in some sense, so we introduced a Virtual Assistant
figure into the Smart Classroom to impersonate the classroom against the teacher. The
Virtual Assistant shows a face of a virtual person, which are displayed on the
StudentBoard as illustrated in Fig. 2 as well as in Fig. 5, and its face expression and
lips movement is synchronized with the synthesize voice of the system. Thus the
teacher can see a vivid virtual assistant that can understand her/his voice command
and give notifications or feedbacks through speech, just as if there is a real assistant.
4.4. Login to The Classroom Based on Biometric Characteristic
Since a Smart Classroom is a public space, each teacher who gives a class in it needs
to identify him/herself to the system in order to be authorized to use the facilities in
the classroom. The common way of authentication on a desktop computer is inputting
one’s ID and Password with keyboard. In Smart Classroom, we use a combination of
face-recognition and speaker-verification technology to automatically identify the
teacher, providing the teachers an undeterred experience while login to the classroom.
As the teacher enters the Smart Classroom, he first should show up in front of a
mirror (behind which a camera for capturing the teacher’s face is installed) and speak
out his name, if both the face-recognition and speaker-verification processes are
passed, the Virtual Assistant will greet him, indicating the Smart Classroom is now
ready to serve him. This procedure is showed in Fig. 5. Moreover, the system will use
the teacher’s identification information to load the right voice model for the teacher
into the speech-recognition module if the teacher have trained such a model in
advance, which is beneficial for the accuracy rate of the recognition result.
Virtual
Assistant
Your Name,
Please.
A camera for FaceRecognition is
installed behind the
mirror
I’m Weikai
Fig. 5. A teacher login to the Smart Classroom
5 Smart Cameraman
When taking a class in a real classroom, the students will change the focus of their
sights as the context of the class changes. For example, when the teacher are writing a
formula on the blackboard, the students will focus their sight on the formula, while
when the teacher is showing a model in his hand, the students will focus their sight on
the model. However, in most of current real-time teaching systems, the students can
only get the teacher-side video with a fixed scene no mater how the context of the
class changes, which significantly decreases the efficiency of understanding of the
teacher’s instruction.
(a) The teacher was writing
on the Mediaboard
(b) The teacher was
showing a model
(c) The teacher was discussing
with local students
Fig.6. Different Scenes Remote Students Get According to Context of The Class
To overcome with this problem, a facility called Smart Cameraman is introduced
into Smart Classroom, which can distinguish among several kinds of context in a
class by observing some clues in the classroom, and then select a camera with proper
view according the context from an array of available ones as the source of the live
video to remote students. Currently, this module can successfully distinguish the
following three kinds of context:
1)
Teacher Writing on Mediaboard, where the teacher is writing or scribbling
on the Mediaboard. In this case, the camera which focus on the Mediaboard
will be selected as Fig.6 (a) shows.
2) Teacher Showing A Model, where the teacher is holding a model in his
hand. In this case, the camera which follows the teacher’s hand will be
selected as Fig.6 (b) shows.
3) Others, for all other situations. In this case, the camera with a overview of
the whole classroom will be selected as Fig.6 (c) shows.
The cues used by the Smart Cameraman to estimate the current context includes
the output of a Person-Tracking module which track the position of the teacher, a
Gesture-Recognition module that can decide whether the teacher is holding something
in his hand and the touch event reported by the SmartBoard.
6 The Software Infrastructure
As we have seen, besides the tele-education software, there are many other
modules running in the Smart Classroom that provides periphery services, including
speech-recognition module, text-to-speech module, tracking modules and so on.
Totally we have about a dozen of modules running on eight distributed computers and
we developed a multi-agent system (as shown in Fig.7) called Smart Platform [11] to
interconnect and coordinate these modules. Here each module is running in a separate
process and communicate with each other through predefined a XML format
messages. For example, the module in charge of the laser pointer tracking will
periodically send a message to the module in charge of the Mediaboard to update the
location of the cursor while detecting a laser pointer spot.
A
A
Context
Unification
Agent
MediaBoard Agent
A
A
A
Virtual Mouse Laser Pointer
User Profile
Agent
Tracking Agent
Agent
A
Teacher Action
Recognition
Agent
A
Speaker
Recognition
Agent
A
Avatar & TTS
Agent
A
StudentBoard
Agent
Face Recognition
Agent
A
A
Speech
Recognition
Agent
Fig.7. Software Platform for Smart Classroom
Since the Smart Classroom is a fairly complicated system, its reliability is an
important issue that needs careful consideration. Besides each composing modules are
carefully designed and developed, the interconnection of modules takes a loosecoupling policy. First, we use message passing instead of RPC so that one module
will not block for the completion of a communication request, which reduce the
possibility of a halting module to block other module which is communicating with it.
Second, we use a mediated communication scheme in that all the modules only need
to maintain a connection with a centralized message dispatcher module and all the
inter-modules communication are send to and forwarded by this dispatcher. This
structure reduces the recovery work need to do when a module is restarted after
failure.
7 Conclusions
Currently we have made concrete achievements on each part of the project and the
prototype system runs quite well in initial informal evaluations. We are planning a
formal user study of this system in the near future. We are also cooperating with the
Distance Learning School of Tsinghua University for the large-scope user
experiments 2003 fall.
With the development of computing and network technology, the portable
computing device will be proliferated. The teacher may take a portable computer or
PDA into the Smart Classroom, it is interesting how to configure interface depending
on the set of devices available. How to seamlessly add computing device into the
Smart Classroom is another research topic we are now paying more attention on.
Reference
1.
Gregory D. Abowd. Classroom 2000: An Experiment with the Instrumentation of a Living
Educational Environment. IBM Systems Journal, Special issue on Pervasive Computing,
Vol38, No4: 508-530
2.
3.
4.
5.
6.
7.
8.
9.
Smart Classroom Project. http://media.cs.tsinghua.edu.cn/~pervasive
NIST. http://www.nist.gov/smartspace
Interactive Workspaces in Stanford, http://graphics.stanford.edu/projects/iwork/
Intelligent Rooms in MIT, http://vismod.www.media.mit.edu/vismod/demos/smartroom/
Easyliving in Microsoft Research, http://www.research.microsoft.com/easyliving/
Dream Spaces in IBM Research, http://www.research.ibm.com/natural/dreamspace/
Aware Home in Georgia Tech, http://www.cc.gatech.edu/fce/ahri/
Xie, W.K., Shi, Y.C. & Xu G.Y. Smart Classroom - an Intelligent Environment for Teleeducation. In Proceedings of The Second Pacific-Rim Conference on Multimedia (PCM
2001), Beijing, China. Springer LNCS2195, 662-668.
Jiang Changhao, Shi Yuanchun, Xu Guangyou, Xie Weikai. Classroom in the Era of
Ubiquitous Computing - Smart Classroom. ICWLHN2001, Dec 5-7, 2001, Singapore
Yanhua Mao, Weikai Xie, Yuanchun Shi, Guangyou Xu. Building the Software
Infrastructure –From OAA to Smart Platform. 526-533. PCM2002, Dec, Taiwan
LNCS2532
10.
11.