Proceedings of the 2002 IEEE/RSJ
Intl. Conference on Intelligent Robots and Systems
EPFL, Lausanne, Switzerland • October 2002
GAG(Gaining Angle of Gaze) Strategy
for Ball Tracking and Catching Task
R y o s u k e Mori
Fumio Miyazaki
Graduate School of Engineering Science, Osaka University
1-3 Machikaneyama, Toyonaka, Osaka 560-8531, Japan
{ ryosuke, miyazaki} @robotics. me. es. osaka-u, ac.jp
Abstract
monocular vision system.
In this paper, we propose a ball tracking and catching strategy called GA G (short for "gaining angle of
gaze") that enables a mobile robot to track and catch
a ball flying in three-dimensional space. The GAG
strategy does not require the information of the ball's
position in three-dimensional space, but instead requires only the angle of elevation of gaze captured
as two-dimensional visual information. We also derived a visual feedback control scheme based on GAG
and demonstrated that the proposed scheme enables
the robot to track and catch a ball flying in threedimensional space by using only a monocular vision
system.
1
2
o f Ball C a t c h i n g S t r a t e g y
Good baseball outfielders certainly catch the ball,
beginning to run toward the dropping point of ball
to the ground immediately after launching the ball
and moving with appropriate path and speed. Various hypotheses have been proposed to explain this
ability so far. Among them, the following two hypotheses are highly suggestive in the sense that they
take the human's weakness of spatial perception into
consideration.
(1)P. McLeod and Z. Dienes's hypothesis (OAC)[2]
(2)M. McBeath, D. Shaffer and M. Kaiser's hypothesis (LOT) [3]
In either case, the information of the ball's position
in three-dimensional space obtained from stereo disparity, accommodation, image expansion rates and
so on, is considered to be unavailable because of its
incorrectness until the ball reaches in the vicinity of
a dropping point. Instead, the tracking path and
speed are decided successively, by using the information of the ball's position in three-dimensional space
captured from vision system.
Introduction
For the purpose of understanding human behavior
from the motion of robots designed to imitate human facilities, we have examined motion strategies
for the human ball catching task by using a mobile
robot. As a result, we derived a visual feedback control scheme for mobile robots to realize the motion
strategy called LOT proposed in the field of human
behavior (the details of LOT will be explained in
the subsequent section) and demonstrated that it is
possible for a robot to catch a ball flying in threedimensional space by using only a monocular vision
system[I].
OAC(Optical Acceleration Cancellation) is a ball
tracking and catching strategy applicable in the case
that the fielder exists in the ball trajectory plane (a
vertical plane that includes the ball trajectory) containing the home plate. The model of strategy is
that the fielder tracks and catches the ball, keeping
d ( t a n a ) / d t constant (where a is the angle of gaze
from fielder to ball). When fielders run backwards or
forwards to catch a ball, they run at a speed which
keeps d2(tana)/dt 2 zero. If it drops down to the
ground in front of him, a = 0 °. If it goes over his
head, a = 90 °. Hence, he can reach the ball before it
hits the ground if he keeps 0 ° < a < 90 ° throughout
the flight. The schematic of this catching strategy is
shown in Figure. 1.
However, the motion strategy above has the following problems:
(1)In addition to the information about the flying
ball, it always requires the ball's initial optical location (home plate) relative to the robot.
(2)It is not applicable in the case that the robot exists in the same vertical plane as the ball trajectory.
In this paper, we propose a new motion strategy
for the ball catching task that solves these problems
and derive the corresponding visual feedback control
scheme for mobile robots. This scheme also enables
a robot to track and catch a ball by using only a
U.S. Government work not protected by U.S. copyright
Model
In the meantime, LOT(Linear Optical Trajectory) is
a ball tracking and catching strategy applicable in
281
44".4
"'(I)'...
"4...
0°
...... "4..
< o~ < 9 0 °
"'44.4 .
"'4
modeling the ball trajectory affected by the air drag,
spin, wind and so on, by means of adjusting the
tracking path according to the motion of the ball.
However, these strategies also have the following
problems:
(a)OAC is applicable in the case that the fielder always exists in the ball trajectory plane only. On the
other hand, LOT is applicable in the case that the
fielder exists out of the ball trajectory plane only.
Each of them is not applicable in all cases.
(b) Although LOT covers the two-dimensional tracking, it is not clear how to modify the direction and
path at the time visual information is updated.
(c)It is not clear what conditions are required for
these strategies to work properly.
4444
"'Ig)~
"'~)
",\
a
--0 °
a
--90
°
Figure 1: Schematic of OA C model
In this paper, we propose a ball tracking and catching strategy that has the same features of OAC and
LOT and solves the above-mentioned problems. The
proposed strategy pays attention to the tangent of
the angle of gaze c~, tan c~ similarly to OAC and
guarantees that the fielder tracks and catches the
ball, keeping tan ct gaining in a finite rate-of-change.
If the fielder keeps tan ct gaining continuously, he
can reach the ball before it hits the ground because
0 ° < ct < 90 ° is satisfied throughout the flight. This
strategy enables the two-dimensional tracking similarly to LOT, using the angle of gaze only. Besides,
we derive a visual feedback control scheme for mobile robots with monocular vision and make clear the
condition that the proposed strategy works properly.
Figure 2: Schematic of L O T model
3
Model
of mobile
robot
In this section, we describe a model of mobile robot
to realize the robot ball catching task (shown in Figure. 3). In this figure, we set the world coordinate
frame fixed at floor face Nw and the robot coordinate frame NR, and assume the origin OR is a center
point of the robot, the translation direction is XR
and the cross direction is YR. In the world coordinate frame Ew, the coordinate of the center of wheels
is (xw, yw), the coordinate of the center of the robot
is (x,y) and the angle between XR and X ~ is 0.
The thickness of wheel is assumed to be negligible,
hence, the distance between the wheel and the center of wheels is d, and the radius of wheel is r. The
touchdown velocities VR,VL are given by
the case that the fielder exists out of the ball trajectory plane. Adding to the vertical optical angle
ct used in OAC, the lateral optical angle ~ between
the ball and the ball's initial optical location (home
plate) from the view of fielder is used. The model of
strategy is that the fielder tracks and catches the ball,
keeping tan/~ changing in proportion to the change
of tan ct (that is, keeping tan ct/tan/~ constant). The
schematic of this catching strategy is shown in Figure. 2. This strategy adapts to the balls hit at various angles to either side and fielders can catch the
ball converting the temporal problem to a spatial
one.
vR - r b R ,
Compared to the conventional methods, such as going at a stroke toward the estimated ball catching
point, both OAC and LOT have the following features:
(1)These strategies do not require the information
of the ball's position in three-dimensional space, but
the vertical and lateral optical angle captured as twodimensional visual information only.
(2)Fielders can track and catch the ball without
(1)
(2)
where CR,¢L are the rotational velocities, and the
translation velocity of the center of wheels (body) v
and the angular velocity c~ are given by
1
- ~(v~ + v~),
1
c~ - ~ (vR - v~).
282
(3)
(4)
Y~
~
CatchingPart
'-...~
Y~
CCDCamepc
~
MobileBase
;;;~
Ow
|
/
Xw
/
X
Xw
~ X
Figure ~" Ball catching task
Figure 3: Model of mobile robot
of robot, to derive a visual feedback scheme. The
schematic of coordinate frame for derivation of Jacobian matrix is shown in Figure. 5.
The relations between the rate of change in position/orientation of body and the translation/angular
velocity are given by
~
v cos 0,
(5)
~)~ - v sin 0,
(6)
0 -
(7)
-
~.
In this figure, the left one shows the relation of robot
and ball. The right one shows the position of ball
on the image plane. In the world coordinate frame,
the coordinate of the center of robot is (xR, yR), the
orientation of robot is 0, the ball location is (xb, yb).
In the robot coordinate frame, the ball location is
(Xb,,Yb,). In the image coordinate frame, the ball
location is (xbc, y~c).
Besides, the relations between (x, y) and (xw, yw) are
given by
x = x~ + scos0,
(8)
y = y~ + s sin0,
(9)
The relations between the rate of change in position/orientation of body and the rotation angles of
wheel CR, eL are given by
where s is the offset distance between the center of
wheels and the body center.
4
Trajectory
task
control
for
ball
r
2d
(10)
Considering the coordinate transformation between
/
Yb
In this figure, a specifies the angle of elevation of gaze
from robot to ball. If the GAG model is applied to
the ball catching task, by keeping the value of tan a
gaining in a finite rate-of change, it is resultantly
possible for the robot to track and catch the ball. In
order to realize this strategy, we pay attention to the
differential of the output function t a n a depending
on the location of the robot and the ball.
Derivation
~R
sin 0
catching
The observed value that is required for the GAG
model is the vertical optical angle only. Thus, in
this paper, we assume to extract the angle from the
feature in the image plane of one CCD camera with
fish lens mounted on the autonomous mobile robot.
The schematic of this task is shown in Figure. 4.
4.1
Y cos 0
T'
Y
sin 0
__
of Jacobian
~
.......
/
./
YR..................................
~
XR
/../
///"
XR X b
PP~'rXW
matrix
Figure 5: Schematic of coordinate frames
In this section, we derive Jacobian matrix that relates between the output function and the motion
283
Xc
the world coordinate frame and the robot coordinate
frame, we can obtain the following equation:
[ XbR ] Ybn
[ COS0
sin 0
-
sin0 ]
cos 0
-
Xb--XR
Yv - YR
.
According to the GAG model, the output function
of GAG is given by
(11)
F(Xb~, Yb~) -- tan ct -- tan
Differentiating Eq.(ll) with respect to time leads to
[ iCbR ]
-
[--cosO
~)bR
--sinO
-- cos 0
sin 0
cos 0
--sin0
+
YbR
--XbR
sin 0
cos0
~/R
-+- Jcl'
0
_ (Xb~COS 20b~ + Xbc ~ dXb~
_
\ r b ~ f sin 20b~
Xbc -- f " x?~b . tan-1 ~b
z '
Ybc -- f " ~ " tan -1 ~b.
=
-[
where, x - XbR, Y -- YbR, Z -- ZbR, rb -- V/x 2 + y2 and
f is the focal distance. Differentiating each equation
with respect to time yields
[(1 x2)
dXbc
f
dt
rb
r3
b
tan-1
-xy
rb
~tan-l--+ z
fx
dz
z 2 + r b2 dt
AdX
dy
dz
dt + B--d-[ + E d--['
+f
_
dybc
dt
(--xy
-
f
~
z
b( z ~ +
r2
dt
b)
--
xyz
) dy
r~(z ~ + r e
b
b) dt
xyz
dt
y~z
~ z 2 + r b)
~
b(
dy
dt
JFJc2JclJR
-(15)
4.2
J
D
/~
+
fj
+ Jc2,i.
F
ybc]
+ Jc2'ib.
+ JFJc2Jcl'
:bb
(21)
Visual
~R
JFJc2JclJR
feedback
__
eL I - J u .
control
(22)
scheme
JFJc2JclJR
×
(16)
JxB+JyD
rXbR
2d
1)
]
rXbR
2d
+1)
-[J1
{r[1
+ Jc2Jcl'
eL
[ JxA+JyC
Substituting Eq.(12) into Eq.(16) yields
J c2 J cl
~y~ ]. (20)
-J~
The Jacobian J is rewritten as
Xbc
Xbc
yb~
The matrix of the first term in the right-hand side of
Eq.(21) J F J c 2 J c l J R ( = J ) is the Jacobian between
the rotational velocities of the driving wheel of the
mobile robot and the time derivative of the output
function. Neglecting the second and third terms, we
have the following differential motion model for control design.
) dx
y~ )
rb
r ba t a n - l - -z + r
C
Jy ]
+ J F J c 2 ' Zb"
(14)
fy
dz
z 2 + r b2 dt
dx
dy
dz
C - ~ + D-~- + F d--[
-
Jx
Accordingly, from Eq.(10), Eq.(17) and Eq.(20), this
equation is rewritten as
Expressing Eq.(14) and Eq.(15) in matrix form leads
to
~)bc
(19)
dx
r__b -4- r 2
tan- --z + r~ ( z + r~
1
rb
+f
--
l rb
dt
Expressing Eq.(19) in matrix form leads to
(13)
z
r---~f/
( yb~ COS2 Obc + Y b~ ~ dyb~
r---~f / dt
\ rb~ f sin 20b~
dXbc
dyb~
Jx--~-+Jy
at .
(12)
"
Considering the relation between the robot coordinate frame and the image coordinate frame, we have
?~b
The time
d F • dXbc + d F • dybc
dXb~
dt
dyb~
dt
dF
dt
Xb ]
~)b
fib
(18)
%
JOb
0
7r _ Ob~
where, Ob~ -- rb~/f, rb~ -- v/X2bc + yb2~.
derivative of Eq.(18) is given by
-Jcl
) _ ~cos0~
sin, Obc
(•
×
25
~]b
~
11
0 0
r
+~
V1
-V1
v2
-v2
,(23)
where V1 - --YbR,V2 - XbR. Choosing a feedback input u so as to make the time derivative of the output
(17)
284
xcl
[
mobile robot are 300[mini and the maximum velocity
is 1.2[m/sec]. At the initial time (t = 0), the initial
position of the center of the robot is at the origin
(0, 0) of the world coordinate frame and the initial
orientation of the robot is the same direction as the
x-axis. We consider two ball trajectories, passing
a catching point (0.4, 0.4, 0.55) at the catchin time
(t = 1) from the initial positions (0, 0.4, 0.55) and
(0.4, 0, 0.55), which are given by
(Zbc,Ybc)
o
xb(t) - 0.4t,
yb(t) - 0.4,
zb (t) - 0.55 + 4.9t - 4.9t 2
Figure 6: Ball catchable region
xb (t) - 0.4,
yb (t) - 0.4t,
zb (t) - 0.55 + 4.9t - 4.9t 2
r [ &+-}(&VI+J1V2) ]
m
2~
- J 2 --[- d(J2V1 --[- JIV2)
/kF, (24)
where A is a constant gain and A F = F n o m - F is
the deviation of the output function from its nominal
value Fnom measured at a certain time just after the
ball is launched. In fact, substituting Eq.(24) into
Eq.(22) yields
r2
---~
P
1
J1 + ~(JlVl + &v2)
J1 - ~(JlVl + &v2)
T
Corresponding variations of the output function F
defined in Eq.(18) are shown in Figure. 9 and Figure. 10. From these figures, we can see that F
steadily increases until just before the ball is caught,
which reflects the GAG strategy. However, F rapidly
decreases just before the ball is caught because the
ignored factors in deriving Eq.(22) affect F due to
the rapid change in the angle of elevation of gaze.
J1V2) ]
Av
J2 + 2 l(J2Vl
1
- J 2 Av ~(J2V1 Av J1V2)
/kF
r2
-2-~a(v~J~ + 2VlJiJ~ + V~J~)~XF
-2-5 a[J1
AF,
v1
(25)
Finally, we examine the ball catchable region from
these results. There is a clear difference in the movement of robot between Figure.7 and Figure.8. In
Figure.8, the robot moves to the ball catching point,
tracking the movement of the ball launched. While in
Figure. 7, robot moves backward once and then moves
to the ball catching point, tracking the movement of
the ball. This movement of the robot demonstrates
that the robot first moves backward to go into the
ball catchable region and then continues to track the
ball.
where
½ v1
½ ½
XbR
--YbR
--YbR
XbR
(26)
and A F keeps its value negative until the ball falls to
the ground (correctly speaking, until F becomes less
than Fnom). If XbR > 0 and XbR--2y2bR > 0 when A > 0,
the matrix Eq.(26) is positive definite a n d / ~ > 0 for
any J1, J2. Accordingly, in the case that the ball
exists in the shaded region of the horizontal plane
(the ball catchable region) as shown in Figure. 6,
F > 0 is satisfied by choosing an appropriate sign of
5
5.1
(0_<t_< 1). (28)
If the center of the robot in X Y plane can reach the
catching position (0.4, 0.4) at the catching time, the
ball catching task succeds. Simulation results are
shown in Figure. 7 and Figure. 8. In the figures,
the progression of solid lines represent the path of
robot, dots represent the path of the center of the
body and small circles represent the ball trajectory
plotted every 0.1[sec]. The 10-cm-square around the
center of the body represents the area that can catch
the ball. Judging from these results, at the catching
time, we can see that the robot catches the ball in
the vicinity of the center of its body.
function positive definite with respect to J1 and J2,
we have
u
(0_<t_< 1), (27)
5.2
Supplementary explanation
• We verified through various ball trajectory sireulations that the GAG strategy provides the robot
with a large catchable area of the ball (The catchable
area means the set of all landing points for the ball
that can be successfully caught by the mobile robot).
These simulations were performed assuming the use
of a monocular vision system (constant nominal values are employed for (XbR, YbR, ZbR) in Jcl and Jc2).
S i m u l a t i o n s o f ball c a t c h i n g t a s k
Simulation results
Conditions for the ball catching simulation are described as follows" The total length and width of the
285
0.6 ~
0.4
I
'
I
'
I
'
I
'
'
I
I
'
I
'
I
'
I
'
10
[
F n o 1TI -
~ 0.2
0
0.2
0.4
0.6
Time [sec]
0.8
0
Figure 9: Variation of output function
-0.2
-0.2
0
0.2
X[m]
0.4
0.6
'
I
'
I
'
I
'
I
F
Fnom
Figure 7: Path of mobile robot
I
'
I
'
I
'
10
'
I
'
0
I
0.6
0.4
0.2
0.4
0.6
Time [sec]
0.8
Figure 10: Variation of output function
~ 0.2
a real r o b o t system[4]. However, t h e y d e m o n s t r a t e d
the catching t a s k using t h e balloon to slow down this
high-speed task. H i g h - p e r f o r m a n c e h a r d w a r e is required to validate their m e t h o d , whereas our simple
m e t h o d is suitable for this task. I m p l e m e n t i n g t h e
p r o p o s e d control scheme with a real r o b o t is our fut u r e goal.
0
-0.2
I
-0.2
I
0
0.2
X[m]
0.4
0.6
References
[1] R. Mori and F. Miyazaki: "Examination of Human
Ball Catching Strategy through Autonomous Mobile
Robot", proceedings 2002 IEEE International Conference on Robotics and Automation, pages 4236-41,
2002.
Figure 8: Path of mobile robot
• T h e simulation results in this p a p e r are o b t a i n e d by
r u n n i n g simulations in which it is a s s u m e d t h a t the
image from vision s y s t e m is e x t r a c t e d every 10[msec].
We also verified t h e effectiveness of the p r o p o s e d
m e t h o d in the case t h a t t h e frame r a t e is 33[msec]
c o r r e s p o n d i n g to t h e real robot.
6
[2] P. McLeod and Z. Dienes: "Running
Ball", Nature, 362, page 23C1993.
to Catch the
[3] M. K. McBeath, D. M. Shaffer and M. K. Kaiser:
"How Baseball Outfielders Determine Where to Run
to Catch Fly Balls", Science, 268, pages 569-573,
1995.
Conclusion
[4] A. Suluh, K. Mundhra, T. Sugar and M. McBeath:
"Spatial Interception for Mobile Robots", proceedings 2002 IEEE International Conference on Robotics
and Automation, pages 4263-68, 2002.
In this paper, we p r o p o s e d the G A G s t r a t e g y t h a t
has the same features of OAC and L O T and solves
their problems. Moreover, based on GAG, we derived a visual feedback control scheme for mobile
r o b o t s with m o n o c u l a r vision s y s t e m a n d verified
t h a t this scheme enables t h e r o b o t to t r a c k a n d catch
a ball launched in various directions.
[5] E. Aboufadel:
"A Mathematician Catches a Baseball", American Mathematics Monitor, 103, pages
870-878, 1996.
[6] C. Samson, M. L. Borgne and B. Espiau: "Robot
Control : The Task Function Approach", Oxford Science Publications, 1991.
A. Suluh et al. i m p r o v e d OAC a n d L O T models a n d
validated these strategies t h r o u g h e x p e r i m e n t s using
286