Proceedings of the 2001 IEEE
International Conference on Robotics & Automation
Seoul, Korea • May 21-26, 2001
Pressure Distribution Image Based Human Motion Tracking System
Using Skeleton and Surface Integration Model
Tatsuya Harada, Tomomasa Sato and Taketoshi Mori
The University of Tokyo
7-3-1 Bunkyo-ku, Hongo, Tokyo 113-8656 JAPAN
fharada,
g
tomo, tmori @ics.t.u-tokyo.ac.jp
Abstract
In this paper, a lying person's motion tracking system by using a pressure distribution image and a full
body model is proposed. The full body model consists
of a skeleton and a surface model to cope with a variety of body shapes. BVH les are used as the skeleton
model that describes a hierarchy of joints and links.
Wavefront Object les are used as the surface model
that describes geometry of the surface. The bed has
210 pressure sensors that are under the mattress. It
can measure pressure distribution image of a lying person. The lying person's motion is tracked by considering potential energy, momentum and a dierence between the measured pressure distribution image and a
pressure distribution image that is calculated by the full
body model. Experimental results reveal that the realized system can track not only horizontal motions such
as opening and closing legs but also vertical motions
such as raising the upper body.
1
Introduction
Moving the body is very important for humans to
live. It makes humans take adaptive behavior to the
outside world. It is said that physical conditions and
mental conditions are buried in the body movements,
because humans often move their bodies when they are
in good health, but move rarely their bodies when they
are in bad health. Therefore it is thought that physical
and mental conditions can be estimated by measuring
the body movements.
Humans spend one third or a quarter of life for being
in bed. Especially people who need care usually spend
the whole day for being in bed. If their movements
are measured and evaluated quantitatively, these data
can be used for a health monitoring and evaluation of
rehabilitation progress.
Supine and lateral postures [1], body parts' posi-
0-7803-6475-9/01/$10.00 © 2001 IEEE
3201
tions [2] and infant's status (e.g., crying and sleeping)
[3] can be recognized by using a pressure sensors distribution bed. However these systems cannot track motions such as twisted motions of the upper body. Static
Charge Sensitive Bed (SCSB) [4] is famous for monitoring the body movements in bed. SCSB can measure
respiration, a heart rate and twitch movements. Temperature sensors distribution bed [5] can measure gross
movements such as body turns. However, these beds
cannot measure articular motions. AMI made "Actigraph" is widely known as a body movement measuring system that sensors are attached to the body. The
size of the Actigraph is about as large as a wristwatch.
It can count body movements. There is the system
that can recognize postures using acceleration sensors
[6]. Although these sensors become small, many sensors must be attached to the body to measure articular
movements. Since attaching many sensors to the body
restricts person's activities, it is diÆcult to measure
unaected body movements.
There are many researches to track human motions
by using video camera. O'Rourke [7] proposed the
body motion tracking algorithm by using human model
constraints about 20 years ago. Rehg [8] studied ngers
and palm tracking based on the kinematics constraints.
Wren [9] realized three-dimensional upper body motion tracking system by using "blobs". However these
systems cannot track the subtle upper body's twisting
motions, which seem same postures as video images. It
is diÆcult for these systems to extract face and hand
features because the body is lost of sight in a quilt.
There is the possibility that monitoring the lying person by video cameras invades her or his privacy.
In our research, we developed lying person's motion
tracking system. This system realizes unrestraint sensing by using a pressure sensors distribution bed and
realizes motion tracking such as raising and twisting
the upper body and opening and closing legs thanks to
a skeleton and a surface integration model.
2
Motion Tracking System
2.1
Required Functions
In order to track bed-ridden person's motions, three
functions described below are required.
Attaching sensors to the body
produces mental and physical burdens for persons. In order to measure unaected body motions, sensors need to be attached not to the body
but to an environmental side as a bed. In this
paper, a pressure sensors distribution bed is introduced to measure body motions unrestraintly.
Unrestraint Sensing:
The shapes of the bodies are various for each person. If the shape of the body
changes, the pressure distribution, which is measured by the bed, also changes. In this paper, in
order to cope with a personal variety of the shape
of the body, a surface model, which ts the bedridden person, is imported to the tracking system. By using the imported surface model, the
personal tting function is thought to be realized.
Personal Fitting:
In order to track human
vertical motions such as raising the upper body,
a skeleton and a surface integration model is introduced. By using this integration model, the
shape of the surface can be changed according to
the joint angles of the skeleton. A pressure distribution image is constructed based on this integration model and is compared with the measured pressure distribution image. Considering
with potential energy, momentum and a dierence between the model based and measured images, the vertical motion tracking function is realized.
Vertical Motion Tracking:
2.2
System Overview
A human motion tracking system that satises with
required functions described in section 2.1 is realized.
This system consists of a pressure sensors distribution
bed (Figure 1 (a)), pressure sensors control box (Figure 1 (b)), a pressure distribution measuring computer
(Figure 1 (c)) and a human motion tracking computer
(Figure 1 (d)).
The pressure distribution measuring computer measures the pressure distributions of the bed-ridden person and transmits them to the human motion tracking
computer. This computer tracks human motion by using following algorithm.
1. Importing a skeleton model and a surface model
3202
Figure 1: Hardware conguration
2. Manual adjustment of the skeleton model and the
surface model
3. Measuring a pressure distribution as a motion
tracking target at the frame t
4. Construction of a pressure distribution image by
using a full body model
5. Calculation of joint and translation parameters
considering a dierence between measured and
model based pressure distribution, gravity and
momentum
6. Updating joint and translation parameters of the
full body model and re-calculating a model based
pressure distribution image
7. If dierence between measured and model based
pressure distribution converges, then decide parameters at frame t and return to a pressure distribution image measuring at frame t + 1. Otherwise return to the joints and translation parameters calculation for searching better parameters
at frame t.
Details of this system are described in the following
sections.
2.3
Pressure Sensors Distribution Bed
The pressure sensors distribution bed has 210 arrayed pressure sensors. Figure 1 (e) shows the bed.
software it is easy to modify the structure of the skeleton. 2) Importing and exporting BVH les is very
easy because BVH le is described as ASCII format.
3) Tracking results can be analyzed anywhere if these
results are exported as the BVH format.
As one of the examples of BVH les, Figure 3 shows
the structure of the skeleton and joints' name, which
is included in Biovision motion collections. The root
of this skeleton is the hip. The number of joints is 18.
Each joint has three degree of freedom. The hip has
six degree of freedom. The total number of degree of
freedom is 57.
Figure 2: Pressure distribution image
Force Sensing Register (FSR) is used as the pressure
sensor. The FSR is a thin lm sensor, which is made
from piezoresistive polymer. The resister value of the
FSR is reduced in proportion to applied force. The
size of the pressure sensors distribution bed is 1920 760 17 [mm]. A distance between pressure sensors
is 78 [mm]. About 50[mm] thickness futon mattress is
spread over this sensor bed.
The pressure sensors control box can select one of
the pressure sensors and read the selected pressure sensor's value. The pressure sensors control box scans
pressure sensors one by one by using multiplexers, reads
one of pressure sensors' value and transmits this sensor's value to the pressure distribution measuring computer. This computer measures pressure distribution
as a pressure distribution image. Sampling frequency
of the image is about 10 [Hz]. Its resolution is 12 bit.
Figure 2(a) shows the lying person's picture and Figure 2(b) shows the pressure distribution image. The
dark color indicates a high pressure point.
2.4
Full Body Model
A full body model is constructed by integrating a
skeleton model and a surface model. The following
sections explain the skeleton and surface model and an
integration method of them.
2.4.1
Skeleton Model
BVH les are used as skeleton models. BVH le can
be divided into a hierarchy part and a motion part.
The hierarchy part describes a structure of joints and
links. The motion part describes time series of joints
and translation parameters. The system utilizes the
hierarchy part of BVH le to construct the skeleton
model.
Merits of using BVH les are: 1) Since much software can import and export BVH les, by using these
3203
Figure 3: BVH File from Biovision Motion Collection
2.4.2
Surface Model
In order to realize the personal tting functions, a
surface model is adopted. Although there are many
formats, which describe surface models, in this paper Object le of Wavefront's Advanced Visualizer is
adopted. Objects les dene the geometry and other
properties for objects. Object les can be in ASCII
format (*.obj) or binary format (*.mod). Since it is
easy to handle the ASCII format les, the obj le is
adopted.
Merits of using obj les as the surface models are:
1) Since much software can import and export to obj
les, by using these software it is easy to modify the
structure of the surface. 2) It is possible to correspond
to the dierent shape of the body only by changing obj
les. 3) There is high possibility of utilizing formats
that are dierent from obj les because there are many
converters to obj format. Figure 4 shows the surface
model which is used in this research.
The obj le is not the solid model, but is the hollow
model. The reason why the hollow model is adopted
is described below. There is an elastic body as a futon
mattress between the human body and the pressure
sensors distribution bed. Reaction force is thought to
Figure 5: Integration of Skeleton and Surface
Figure 4: Surface model
be produced according to a displacement. The relationship between the human and the mattress is given
by: mg + F = f (x), where m is the mass of the human,
x is the displacement of the mattress, F is the force
produced by the muscle and f (x) is the force produced
by the displacement of the mattress. If the force applying to the pressure sensors is dened as P , the relationship between the mattress and the pressure sensor
is given by: P = f (x). The equation eliminating f (x)
from above equations is given by: mg + F = P . This
equation expresses that the force applying to the pressure sensors can be calculated, if we know the mass
of the body and the force produced by the muscle.
However, it is very complicated to construct the model
including the mass and the force. On the other hand,
from the equation: P = f (x), the force applying to
the pressure sensor is expressed by the function of the
distance between the pressure sensor and the human
body. The force applying to the pressure sensor can
be calculated by the knowledge of the distance of the
pressure sensor and the human body. Therefore, in order to construct the model based pressure distribution
image, it is much easier to use the hollow model than
the solid model.
2.4.3
skeleton's joints coincide with the surface's joints.
The surface model is assigned automatically to the
skeleton model after manual adjustment of the skeleton
model. The assignment algorithm is described below.
First, one of the vertexes, which construct the surface
model, is selected. Secondly, a distance between the
selected vertex and all the links of the skeleton is calculated. Thirdly, the vertex is assigned to the link whose
distance is the shortest. These assign procedures are
operated for all the vertex. After this assignment, the
surface model can change according to the skeleton's
joint and translation parameters. Figure 5 shows this
assignment procedure.
Integration of Skeleton and Surface
After importing the skeleton model and the surface
model, these models are integrated into the initial full
body model. Before integration, the system scales the
skeleton model and the surface model automatically
to the bed-ridden person's height, which is previously
inputted. It is necessary to adjust the skeleton model
and the surface model manually, because there is no
relationship between the BVH le and the obj le. The
skeleton model is moved and deformed in order that the
3204
Figure 6: Model based pressure distribution
2.5
Model Based Pressure Distribution
It can be thought that pressures depend on the distance between the pressure sensors and the surface as
described in section 2.4.2. We assume that the shortest
distance between the pressure sensor and the surface
inuences the pressure of this sensor.
We consider the relationship between the distance
and the pressure. If the force applying to the mattress is small, the displacement of the mattress is in
proportion to the force. However if the force increases,
the rate of the displacement becomes small. Therefore,
the force function of the mattress is non-linear whose
characteristics are: 1) The force is 0, if the surface is
far from the sensor. 2) If the surface is close to the
sensor, the pressure is very high. One of the simplest
functions, which contain these characteristics, is the
following function.
Pa
wi =
(ri + Pb )2
(1)
where wi is the pressure of the ith sensor, ri is the
shortest distance between the ith sensor and the surface, Pa and Pb are coeÆcients. A set of wi (1 i N )
is the pressure distribution image where N is the number of the pressure sensors. This function is applied to
construct the model based pressure distribution image.
Pa and Pb are decided by comparing with the measured pressure distribution images and are xed irrespective of the dierent shape of the body. Assuming
that meshes of the surface model are very small, ri is
calculated by:
ri = minfjjvl si jjg
(2)
l
where v l is the lth position vector of the vertex of the
surface, si is the position vector of the ith sensor. Figure 6 shows the model based pressure distribution image construction process.
2.6
Tracking Method
In order to track the human motion, joint and translation parameters are determined to minimize the difference between the measured pressure distribution image and the model based pressure distribution image.
This dierence is given by:
Ep =
XN wi
i=1
(
mi )2
(3)
where mi is the measured pressure of ith sensor. A set
of mi (1 i N ) is the measured pressure distribution
image.
The body is inuenced by the gravity. If no force
is applied to the body, the body falls to the bed. The
potential energy of the gravity is given by:
3205
Eg =
X gzj
Nnode
(4)
j=1
where zj is the height of the skeleton's j th node from
the bed, Nnode is the number of the nodes of the skeleton and g is the gravity acceleration. As an evaluation
function for tracking the human motion, the sum of
the square of the dierence and the potential energy is
adopted. The evaluation function Eall is given by:
Eall = p Ep + g Eg
(5)
where p is weight for the dierence and g is weight for
the potential energy. The skeleton's joint and translation parameters are determined to minimize this evaluation function. We assume that the parameters are
searched sequentially. This process is given by:
dxk
@Eall
=
(6)
dt
@xk
where xk is the k th skeleton's joint and translation pa-
rameter. Considering with equation (3) and equation
(4), the equation (6) can be rewrote as:
X
dxk
= k f2p (wi
dt
i
mi )
X
@wi
@z
+
g j g (7)
@xk g j @xk
where k is added as weight for the dierential of xk .
Then the dierential of xk is transferred to discretetime expression as 4xk (t; n). Where t is the frame
number of the measured pressure distribution image
and n is the calculation time for convergence. In order
to accelerate convergence, momentum term is added to
4xk (t; n). The 4xk (t; n) can be rewrote as:
4xk (t; n) = 4xk (t; n) + 4xk (t; n
0
0
1)
(8)
where (0 < < 1) is weight for momentum term.
Then the skeleton's parameters are given by:
xk (t; n) = xk (t; n 1) + 4xk (t; n)
0
(9)
These processes are calculated until the evaluation
function converges. At the initial state of frame 1,
skeleton's joints and translation parameters (xk (1; 1))
are adjusted to match the lying person's posture manually at the moment.
3
Experiment
Acknowledgments
A subject is a male whose height is 176 cm and
weight is 61 kg. The minimum number of the backbone joints is one and its degree of freedom is three for
expressing the upper body upward and downward motion or twist motion. Therefore, we use the BVH le
that is included in the Biovision motion collection as
mentioned Figure 3, although the degree of freedom of
this model's backbone is fewer than the degree of freedom of the human's backbone. We create the surface
model by using the Poser3 (MetaCreation) and export
it to the obj le. The number of vertexes of this model
is 2404 and the number of polygons is 4124. By using
this skeleton and the surface integration model, we experimented a human motion tracking. Target motions
are: 1) Opening and closing the legs. 2) Moving up
and down the legs. 3) Moving up and down the upper
body. 4) Twisting the body. 5) Bending the knees.
Figure 7 shows these experimental results. The top
gures are video images, the middle gures are the
measured pressure distribution images and the bottom gures are tracking results of the full body model.
These experimental results show that human motions
mentioned above can be tracked correctly by using the
proposed algorithm. However the model's lower arms
and hands positions does not coincide with the video
images slightly. That is because the lower arms and
hands do not appear on the measured pressure distribution images.
4
Conclusion
In this paper, the unrestraint human motion tracking system by using the skeleton and the surface integration model and the pressure sensors distribution
bed was developed. By using the pressure sensors
distribution bed, the lying person's unaected body
movement can be measured. Thanks to the integration of the surface model and the skeleton model, the
pressure distribution image can be calculated. Considering with the dierence between the model based pressure distribution image and the measured pressure distribution image, potential energy and the momentum
of the body movement, motions such as upward and
downward motion and twist motions can be tracked
correctly. This system is thought to be used for analyzing the body movement, a health monitoring and
evaluation of rehabilitation progress and so on. Automatic initial position adjustment, dierent shape of
body experiments and clarifying the limitation of the
tracking are future works.
3206
The authors would like to thank Mr. Yoshimi for
developing the pressure sensors distribution bed. This
research is supported by JSPS Grant-in-Aid for Scientic Research No.12-08887 and No.11555069.
References
[1] Y. Nishida, M. Takeda, T. Mori, H. Mizoguchi and T.
Sato, "Monitoring Patient Respiration and Posture Using Human Symbiosis System, in Proc. of Int. Conf.
on Intelligent Robots and Systems, Vol.2, pp.632-639,
1997.
[2] T. Harada, T. Mori, Y. Nishida, T. Yoshimi and T.
Sato, "Body Parts Positions and Posture Estimation
System Based on Pressure Distribution Image," Proc.
of IEEE Int. Conf. on Robotics and Automation, Vol.2,
pp.968-975, 1999.
[3] T. Harada, A. Saito, T. Sato and T. Mori, "Infant Behavior Recognition System Based on Pressure Distribution Image," Proc. of IEEE Int. Conf. on Robotics
and Automation, Vol.4, pp.4083-4089, 2000.
[4] Alihanka J, Vaahtoranta K, Saarikivi I, "A New Longterm Monitoring of Ballistocardiogram, Heart Rate,
and Respiration," AM J Physiol., Vol.240, pp.384-392,
1981.
[5] T. Tamura, J. Zhou, H. Mizukami and T. Togawa,
"A System for Monitoring Temperature Distribution
in Ded and Its Application to The Assessment of Body
Movement," Physiol. Meas., Vol.14, pp.33-41, 1993.
[6] A. K. Nakahara, D. L. Jae, and E. E. Sabelman, "Development of a Second Generation Wearable Accelerometric Motion Analysis System," Proc. of 2nd National Rehabilitation Research and Development Service Meeting, 2000.
[7] J. O'Rourke and N. I. Badler, "Model-Based Image
Analysis of Human Motion Using Constraint Propagation," IEEE Trans. Pattern Analysis and Machine
Intelligence, Vol. 2, No.6, pp.522-536, 1980.
[8] J. Rehg and T. Kanade, "Visual Tracking of High
DOF Articulated Structures: An Application to Human Hand Tracking," Proc. of Third European Conf.
on Computer Vision, Vol. II, pp. 35-46, 1994.
[9] C. R. Wren, B. P. Clarkson, and A. P. Pentland, "Understanding Purposeful Human Motion," Proc. of the
Fourth IEEE Int. Conf. on Automatic Face and Gesture
Recognition, Grenoble, France, March 26-30, 2000.
Figure 7: Experimental results. Top gures: Video images of lying person. Middle gures: Pressure distribution
images. Bottom gures: Tracking motions of the full body model.
3207