Fast Arc Detection Algorithm for Play Field Registration in Soccer
Video Mining
Fei Wang, Lifeng Sun, BoYang, Shiqiang Yang
Abstract— This paper presents an LSF-based framework for
detecting arcs in broadcast soccer video. The successful
identification of the arcs will evidently facilitate the soccer
video analysis. The existing methods are not available for all
playfield arcs including both the middle field circle and penalty
box arcs. A new algorithm is proposed in the paper improved
from LSF, called ALSF (Advanced Least Square Fitting),
which can be used to detect arcs even though they are only 1/4
of eclipses (the same as penalty box arcs). With the
improvement, we proposed a new framework to detect the arcs
in broadcast soccer video. The proposed method first removes
the points of straight lines. Then, all points of each connective
area are transformed into a new reference frame to use LSF to
get the equations of arcs. Experiments on more than 3 hours
broadcast soccer video show the proposed method is effective
with above 98% precision and 70% recall.
I. INTRODUCTION
With the development of the interactive video review,
automatic soccer video analysis give fans a fast and
convenient way to access the match rather than watch the
game themselves. Much effort done in the field, the subject
remains an open issue. The reason is that the structure
information is scarce and the usable information (the lines,
the arcs, the persons and the ball, etc) is difficult to be
extracted preciously and quickly, which harms the accuracy
and practicality of the application.
Playfield arcs detection can significantly facilitate the
soccer video mining, such as playfield registration as well as
the camera calibration [1-3]. As we know, the arcs are
unique in one playfield. For example, the left orientation arc
with ends on a line must be the left penalty box arc, and the
whole eclipse or arc with ends on the edge of screen must be
the middle field circle. So if we can get an arc in one frame,
the position will be known. With the detection results, more
feature points (the points on both a straight line and an arc)
can be gotten for calibration. More positions, such as the
positions near the penalty box, can be restored.
The dominative algorithms of arc (eclipse) detection can
be divided into three categories: Eclipse Hough
Transformation and its variations [4-14], Least Squared
This work is supported by the National Natural Science Foundation of
China under Grant No. 60573167 and Intel Cooperation Project: Personal
Desktop Video Mining System.
Fei Wang is with Department of Computer Science and Technology,
Tsinghua University ([email protected]).
Bo Yang is with Department of Computer Science and Technology,
Tsinghua University.
Lifeng Sun is with faculty of Department of Computer Science and
Technology, Tsinghua University.
Shiqiang Ynag is with faculty of Department of Computer Science and
Technology, Tsinghua University
Fitting [15], and Invariant Pattern Filter [16].
Nowadays, the EHT is widely used to detect the eclipses
in soccer video. The SEHT [6] (standard eclipse Hough
transformation) uses each edge points to vote for all possible
eclipse equations, which needs large computation and
storage consumption (because of 5 variants and the
complexity is O(n5)). The Randomized Eclipse Hough
Transformation [7] improves SEHT, but the aberration of the
arcs in soccer video will make it fail. To overcome the
shortcomings of the SEHT, some research work has been
proposed for detecting eclipses in global views of soccer
videos [3-5]. These algorithms can only detect almost whole
eclipses or eclipses more than their half in the frames with
the middle lines, which is very restricted and cannot take
effect on the arcs of penalty boxes.
LSF [15] and Invariant Pattern Filter [16] are faster and
more widely available than above algorithms, which may be
able to detect the penalty box arcs, but they also fail in the
soccer video analysis. The key obstacle why they are not
available for soccer video is the arcs will have errors while
shooting. The two algorithms need a lot of points to
eliminate them. But in broadcast soccer video, we usually
can not get enough right points, because the line point
extraction is not able to get all the points of lines in one
frame. More seriously, the person or other objects may
occult some parts of the lines, and then the arcs will become
to short line-lets whose points are not enough for LSF.
Another possible situation is the arcs and other objects are
connective, so we can not tell points of the arcs from points
are of the objects. The LSF is more robust than Invariant
Pattern Filter when we cannot get enough points, but there is
one problem of LSF if we use it in the soccer video. In the
LSF algorithm, we must solve a set of linear equations for
get the equation of arcs. When there are a lot of points in one
part, the linear equations will become ill-conditioned,
because some parameters of the parameter matrix will be
much bigger than others. So the correct results cannot be
achieved in the broadcast soccer video.
To overcome the shortcomings of LSF, two problems
must be solved. Firstly, we must separate the points of arcs
from others. This process will decrease error introduced by
points which are not contained by the target arc. The
difficult is that other objects removed, the arcs will be
affected, even though they will be segmented into several
parts which have no enough points. Secondly, we must use
some measures to decrease the gap among the parameters of
the parameters matrix of linear equations in the process.
Then the correct results can be achieved.
In this paper, we propose a new framework to detect the
arcs including both the middle field circles and penalty box
arcs. Basing on LSF, it will be much faster than the EHT
algorithm, and is able to find most of arcs. The framework
has five parts in the whole process: the playfield segment,
the edge point extraction, the straight lines removal, ALSF
(Advanced Least Square Fitting) to get the equations of
connective areas, and the post-process to check the results.
We use the color to segment the playfield. After that, we
use the Laplace-Gauss filter to extract edges. Then, Hough
Transformation is used to detect straight lines, which will be
removed from the binary images. Next, we use seed-growing
to get connective areas, and put their points into ALSF to
achieve arc equations. The results will be checked because
the parameters of arcs must be in a range at last.
The algorithm complexity of time and storage of the
whole process are only O(mn) (the same as the magnitude of
the straight line detection), while the SEHT is O(n5). And it
can work well for detecting the penalty box arc which can
not be detected by most of algorithms before.
The rest of the paper is structured as follow: section 2
presents the algorithm, section 3 shows the experiment result
and section 4 is the conclusion.
II. FRAMEWORK FOR ARC DETECTION
The figure 1 shows the framework of arcs detection for
given frames of broadcast soccer video. In the rest of the part,
we will describe its components in turn.
Figure 1 the diagram of framework for detecting arcs of frames in soccer
video
2.1 Playfield Segment
Before the arcs detection process, the playfield segment is
needed, because we only detect the arcs in global view, and
the lines out of the playfield must not be the playfield lines.
Playfield is segmented by the methods proposed by [18]
both accurately and effectively. Several frames will be
chosen randomly to statistic the pixel distribution in HSI
color space. Then we can get a range of grass color in the
video. Furthermore, we use the point number of grass areas
to filter parts which may be false. An area will be filtered if
it is smaller than an adaptive threshold.
Then, we can remove the area of parts which is out of the
playfield. The points in the grass range with the largest
height in their column will be considered as the top grass.
The points above them will be removed. After the process,
the view type will also be known, because we can know the
whether a frame is a global view through their grass-ratio
with high precision.
2.2 Edge Extraction
For the fast detection, the line point extraction must be
simple and effective. There are many methods for object
extraction in computer vision. As we know, the lines in the
playfield are white. But we can not extract the points by only
using the color in many videos because the lines are green in
videos, and the lines in shadow may not be as bright as the
field. And more, when the module becomes simpler, the
extracted binary image will become more complex.
As a result, we choose the 5×5 Laplace-Gauss filter to
balance the two requirements, and use the OTSU method to
get the adaptive threshold for the binary image.
This module can work well in most of broadcast soccer
video, even if the situations are very different from each
other as we test.
2.4 Points Classification
After extracting edge points, we must classify all the
points, and put them into different groups for getting the
correct equations of them. The process is very important for
ALSF. If there are too many noise points in a lines group,
the ALSF will fail. In fact, if two different lines are put into
one group, the ALSF cannot take effect, too. It is a pity most
arcs are connected to straight lines in real soccer video, so
the straight lines must be removed. The problem is one arc
may be divided into several parts when the straight lines
removed. These parts may be too small to be used in ALSF
because all methods basing on LSF have to get enough
points for eliminating errors.
The Hough Transformation is used to detect all the
straight lines. This algorithm is the standard method for
straight lines detection which is very robust and effective.
This Process will take most of time and storage consumption
of the framework as we test. The complexity of algorithm is
O(mn), while other methods are O(n5).
Getting all straight lines, we can remove them from the
binary images. As we known, the width of lines is not fixed.
To simplify the problem, points near the lines are all set 0.
⎛0 0
⎜
⎜ 0 −1
⎜2 0
C =⎜
⎜0 0
⎜0 0
⎜
⎜0 0
⎝
The binary image without straight lines includes several
unattached parts. Some are the points of persons or balls, and
others may be the points of arcs.
2.5 ALSF Algorithm
Using seed-growing algorithm to gain all the connective
areas, the candidate point groups are able to gotten now. We
can use the number each parts to remove groups which must
not be an arc. Parts with points less than a threshold will be
removed.
Because the arcs are all transformed from partial circles,
they must be a part of eclipses. As a result, the
eclipse-specific fitting algorithm may be available. But we
are not able to use the LSF directly, because when the center
point of the eclipse is not near the (0, 0), or the semimajor is
too long, which is very common in soccer video, the
equations will become very unstable and can’t get correct
results. So the LSF has to be improved.
To overcome the limitation of the LSF, we can find the
max and min values in both x and y coordinates of each
connective areas, and then transform all their points into
[0,2]. This progress is as follow:
(1)
sx = 2 /( xmax − xmin )
sy = 2 /( y max − y min )
(2)
x' = ( x − xmin ) × sx
y ' = ( y − y min ) × sy
F(X,α) = a1x' +a2 x' y'+a3 y' +a4 x'+a5 y'+a6 = 0
2
2
(5)
Where
(10)
As we known, the constrained fitting problem can be
written as:
min E = Dα
(11)
And subject to
α T Cα = 1
(12)
The equations can be achieved through Lagrange
Multiplier:
(13)
2 D T Dα − 2λCα = 0
T
(14)
α Cα = 1
T
The S = D D is a 6×6 positively-defined symmetric
matrix. This system can be solved by considering the
generalized eigenvector of (13). If (λi , u i ) solves (13) then
so does (λi , µui ) for any µ .
(3)
(4)
After the transformation, we can represent a general conic
by an implicit second order polynomial:
2 0 0 0⎞
⎟
0 0 0 0⎟
0 0 0 0⎟
⎟
0 0 0 0⎟
0 0 0 0 ⎟⎟
0 0 0 0 ⎟⎠
To solve the equations, we can set
E = S −1 × C
(15)
There may be 6 real solutions as known. The solution will
be chosen with the lowest residual α i S α i =
T
S
is
a
positively-defined
λ i . But the
symmetric
matrix,
with α i Sα i > 0 . Therefore, there is a real solution, if and
T
α = (a1 , a2 , a3 , a4 , a5 , a6 )T
(6)
And
X = ( x ' 2 , x' y ' , y ' 2 , x' , y ' ,1) T
(7)
The F ( X ,α ) is called the “algebraic distance” of a
point (x, y) to the conic F ( X ,α ) = 0 . The fitting of a
general conic may be approached by minimizing the sum of
N
squared algebraic distances
∑F
i =1
2
( X i ,α ) . The arc is a part
of eclipse, so that we can set [6]
4a1a3 − a2 = 1
2
(8)
If we set
D = ( X '2 X 'Y ' Y '2 X ' Y ' 1)
(9)
only if λi > 0 .
The A Cholesky decomposition shows S = Q . Then the
sign of eigenvalues of E will be the same as C. Because
there is only a positive eigenvalue of C, the positive
eigenvalue of E is unique. Hence, the whole equations have
only one solution [16].
If we have not transformed the coordinates of points into
the new reference frame, the S66 would be 1 and the other
data would be very large, and then, the equations would
become ill-conditioned.
Solve the equations, and the equations of new curves can
be gotten. Apparently, we have to change the equation
parameters back in the screen coordinate system then.
If we set the transform matrix as:
2
⎛ sx×sx
⎜
⎜ 0
⎜ 0
T =⎜
⎜2sx×xmin
⎜ 0
⎜
⎜x ×x
⎝ min min
0
sx×sy
0
0
0
0
0 0⎞
0 0
0
sy×sy
0
0 0
sx×ymin
sx 0 0
0
sy×xmin 2sy×ymin 0 sy 0
(16)
xmin×ymin ymin×ymin xmin ymin 1⎠
Then, we can get the conic equation of the detecting arc in
the screen reference frame:
(17)
R = Tα
2.4 Post-process to filter the arc equations
After solving the equations, there may be some false arcs,
such as the curves of person edges. As we know, the
eccentricity of playfield arcs will not be too big. For the
disciplinarians of projection transformation, the revolved
angle will not be too big, too. But we are not easily to know
the two parameters of the arcs while their equations are the
forms of Formula (5).
So we can transform F ( X ,α ) = 0 into the parameter
equations to know whether the detected curves are the real
arcs in the playfield. If we set a as the semimajor, b as the
semiminor and θ as the revolved angle, the equation
of eclipses can be written as:
⎛ x ⎞ ⎛ cosθ
⎜⎜ ⎟⎟ = ⎜⎜
⎝ y ⎠ ⎝ sin θ
− sin θ ⎞⎛ a cos ϕ ⎞
⎟⎜
⎟
cosθ ⎟⎠⎜⎝ b sin ϕ ⎟⎠
(18)
Two rules can be gotten to check the arcs:
⎧a > b > a / 10
⎨
⎩θ < π / 4
Figure 2 some result images
(19)
We use the two rules to remove the arcs which are
evidently not real arcs in field. To add the precision, the
average of squared algebraic distances can be achieved if we
put all points of arcs left back to formula (5).
III. EXPERIMENTS AND RESULTS
We wrote a program using Visual C++ 2003 and
MATLAB, which can automatic extract the frames from
video and mark the arcs. The test was conducted on 2002
WC China VS Turkey and Portal VS USA. The sequences
are MPEG-1 recorded from TV. Our program automatically
draws one frame per 25 frames out from the test sequences
for detecting arcs, and marks the whole eclipses including
them onto the pictures. In the evaluation, a frame will be
considered to have an arc if there is an arc more than 1/4 of
an eclipse (the whole arc of a penalty box).
As the figure 2 shows, the program can work well for
many situations event if there is only a small arc without the
middle line of playfields which cannot be detected by most
of existing algorithms.
The arc in the first image cannot be detected because there
are many persons near the arc, and the seed-growing don’t
work correctly.
TABLE 1 THE RESULT OF THE ALGORITHM WORK ON REAL VIDEO FOR
DETECTING THE ARCS
Name
Arcs
number
Precision
Recall
2002
WC
CHI Vs TUR
(first half)
2002 WC Chi
Vs
TUR
(second half)
2002
WC
USA
Vs
POR(first
half)
2002
WC
USA
Vs
POR(second
half)
210
99.5%
85.2%
89
100%
86.6%
242
98.8%
79.8%
116
98.3%
79.3%
As the results shown in Table 1, the algorithm is effective
in detecting arcs including both the penalty box arcs and the
middle field circles, with high precision. Though the arcs
may be small, it will give a satisfied result.
The factors influencing the results are concentrated on
two aspects. The first is when many persons on the arcs, the
algorithm may not take effect, because the persons will cut
the arcs into many short line-lets. We cannot get enough
points to eliminate the error introduced by shoot. The other
is the objects may connect with the arc. Then when we use
the ALSF, the equations will change a lot. And the results
are not able to be achieved.
IV. CONCLUSION AND FUTURE WORK
We develop an arc detection framework basing on the
ALSF algorithm in broadcast soccer video, which is fast way
to detect the arcs. More importantly, it is available for more
situations, giving us a way to get the penalty box arcs which
are very important for soccer video analysis.
This proposed method has several contributions as follow
paragraphs shows.
First, it is a very fast way to detect the arcs and exploit the
domain knowledge of soccer video. When the resolution of
video becomes higher, the advantage will become more
significant. The low consumption will make the broadcast
video analysis used more widely than ever.
Second, it can detect the arcs less than 1/2 of an eclipse,
so that more information can be gained. The improvement
will let us know the meanings of some straight lines and
points (the joint of arcs and lines).More information will be
gotten. For example, we can know which line is the penalty
box line in the screen. The penalty box line is very important
in fouls analysis (the defense fouls in the penalty boxes leads
a penalty kick).
Third, the results can improve the precision of registration
of positions, because some line meanings determined will
largely eliminate the ambiguity of positions, and advance the
precision of the playfield segment.
Last, we can get more point relations between the screen
and the real world. More position, such as position front of
penalty box, can be restored. It will make the 3D-restruction
technology used in more situations.
In the future, the algorithm performance can be improved
by using the line-lets rather than points groups, which may
be used when there are many persons.
REFERENCE
[1]
[2]
[3]
X. Yu, C. Xu, H. W. Leong, Q. Tian, Q. Tang and K. W. Wan.
Trajectory-based ball detection and tracking with applications to
semantic analysis of broadcast soccer video, ACM MM 2003, Nov. 2-8,
2003, Berkley, CA, USA.
Xinguo Yu. *Hon Wai Leong, Joo-Hwee Lim, Qi Tian, and Zhenyan
Jiang, Team Possession Analysis for Broadcast Soccer Video Based
on Ball Trajectory IEEE PCM2003
Xinguo Yu and Xin Yan, 3D Reconstruction and Enrichment of
Broadcast Soccer Video, ACM MM’04
[4] Xinguo Yu, and Hon Wai Leong. A ROBUST HOUGH-BASED
ALGORITHM FOR PARTIAL ELLIPSE DETECTION IN
BROADCAST SOCCER VIDEO, IEEE ICME 2004
[5] Xinguo Yu, Hon Wai Leong, and Xu Changshen, A Robust and
Accumulator-Free Ellipse Hough Transform. ACM MM’04
[6] N. Bennett, R. Burridge and N. Saito. A method to detect and
characterize ellipse using the Hough transform, IEEE PRAMI, Vol. 21
(1999), 652-657.
[7] Samuel Inverso, Ellipse Detection Using Randomized Hough
Transform,
2002,
http://www-igm.univ-mlv.fr/~vnozick/ENSEIGNEMENTS/PROJET_
TURORE/PROJET_TUTORE_2003_2004/IRCGN/ARTICLES/Ellips
eDetection.pdf
[8] M. A. Fischler, R. C. Bolles. Random sample consensus: a paradigm
for model fitting with applications to image analysis and automated
cartography, Commun. ACM 24 (1981, 6), 381-395
[9] K. Hansen, J. D. Anderson. Understanding the Hough transform:
Hough cell support and its utilization, Image and Vision Computing,
Vol. 15 (1997), 20-218.
[10] J. Illingworth and J. Kittler. A survey of the Hough transform, CVGIP,
Vol. 44, 1988, 87-116.
[11] A. Imiya, T. Hada, and K. Tatara. The Hough transform without the
accumulators. SSPR & SPR 2002, INCS 2396, 823-832.
[12] N. Kiyati, H. Kälviäinen, S. Alaoutinen. Randomized or probabilistic
Hough transform: unified performance evaluation, Vol. 21(2000),
157-1164.
[13] C. F. Olson. Improving the generalized Hough transform through
imperfect grouping, IVC, Vol. 16 (1998), 627-634.
[14] H. K. Yuen, J. Illingworth and J. Kittler. Detecting partially occluded
ellipses using the Hough transform, Image and Vision Computing, Vol.
7(1989), 31-37.
[15] Andrew W. Fitzgibbon Maurizio Pilu Robert B. Fisher, Direct Least
Squares Fitting of Ellipses, ICPR96
[16] T. I. Atherlon and D. I. Kerbysan. Size invariant circle detection,
Image and Vision Computing, Vol17 1999
[17] F. Bookstein. Fitting conic sections to scattered data.
ComputerGraphics and Image Processing, (9):56–71, 1979.
[18] M.Luo, Y.F.Ma, H.J.Zhang, “Pyramidwise Structuring for Soccer
Highlight Extraction”, PCM 2003