Video Analysis & Coding/Computer
Vision. Lecture 6
Jenny Benois-Pineau
LABRI UMR 5800/Université Bordeaux
Summary
➔ Large magnitude OF. Farneback’s method. Modeling of
global motion. Motion descriptors
2
octobre 11, 2015
Groupe projet collège sciences et technologies
Large Magnitude OF.
Méthode de Farenback(1)
➔ Problem : estimate OF, seek for large magnitude flow
➔ Principle : use polynomial (quadratic) expantion of image
signal in the neighbourhood of a given point x .
➔ 
f ( x ) = x T A x + bT x + c
➔ The coefficients A, b, c are estimated by a least square fit in
the neighbourhood of the point x.
3
June 24,2014
Video Analysis and Indexing for Medical Applications
➔ Let us suppose a translationnal motion
➔ In the l’image I1
f1 ( x ) = x T A1x + b1T x + c1
➔ In the image I 2 (using brightness conservation hypothesis)
T
➔ 
f2 ( x ) = f1 ( x + d ) = ( x + d ) A1 ( x + d ) + b1T ( x + d ) + c1
➔ =
T
x T A1x + ( b1 + 2A1d ) x + d T A1d + b1T d + c1 = x T A2 x + bT2 x + c 2
4
June 24,2014
Video Analysis and Indexing for Medical Applications
Méthode de Farenback(2)
➔ The coefficients of the two polynomials are equal
A2 = A1
b 2 = b1 + 2A1d
c 2 = d T A1d + b1T d + c1
➔ Hence we can solve wrt
d
1
d = A1−1 ( b 2 − b1 )
2
➔ Nevertheless
›  Some errors occure
›  The displacement is translationnal only in small patches (remind
block-based motion estimation)
5
June 24,2014
Video Analysis and Indexing for Medical Applications
Méthode de Farenback(3)
➔ Let us denote
A(x) =
Δb ( x ) =
A1 (x) + A 2 (x)
2
1
(b2 (x) − b1 (x))
2
➔ Finally
A ( x ) d ( x ) = Δb ( x )
➔ The solution point – by point is noisy. Hence the problem is
formulated for the neighbourhood
∑
2
w ( Δx ) A ( x + Δx ) d ( x ) − Δb ( x + Δx ) → min
Δx∈Ω
6
June 24,2014
Video Analysis and Indexing for Medical Applications
➔ Finally
d(x) =
−1
(∑ wA A) − ∑ wA Δb
T
T
➔ In case of large magnitude motion, the polynomial
expansion is realised around the displaced point
➔ 
A1 (x) + A 2 (x )
2
1
Δb ( x ) = ( b 2 ( x ) − b1 ( x )) + A ( x ) d ( x )
2
A(x) =
➔ with
7
x = x + d ( x )
June 24,2014
Video Analysis and Indexing for Medical Applications
➔ Estimation method
➔ - iterative : teh displacement estimated at th eprevisous
step are used as initialisation for the next step
➔ - multi-scale( pyramid) – initialisation with 0 vectors at the
apex.
8
June 24,2014
Video Analysis and Indexing for Medical Applications
Results on Yosemit sequence
https://www.youtube.com/watch?v=0xRDFf7UtQE
9
J.L. Barron, D.J. Fleet and S.S. Beauchemin, “Performance of Optical
FlowTechniques”, Computer Vision and Pattern Recognition, 1992. Proceedings
CVPR '92., 1992 IEEE Computer Society Conference on
June 24,2014
Video Analysis and Indexing for Medical Applications
Results on Yosemit sequence
Sky region was excluded, average angular error
10
June 24,2014
Video Analysis and Indexing for Medical Applications
Global Motion characterisation
➔ 3D – 2D relation
Y
G2
G1
X
Z
➔  The displacement of a rigid (non-deformable) body in 3D space in the
coordinate system associated with its center of mass can be expressed as
XG2 = T + RXG1
➔  Here
XG1 = X 1-G1 , (XG2 = X 2 - G1 )
11
June 24,2014
Video Analysis and Indexing for Medical Applications
Relation 3D-2D
T = (Tx,Ty,Tz)T is the translation vector
R = RzRyRx
- Is the rotation matrix
RzRyRx =
⎛ cosθ z − sin θ z
⎜
⎜ sin θ z cosθ z
⎜ 0
0
⎝
0 ⎞⎛ cosθ y 0 − sin θ y ⎞⎛ 1
0
⎟⎜
⎟⎜
0 ⎟⎜ 0
1
0 ⎟⎜ 0 cosθ x
1 ⎟⎠⎜⎝ sin θ y 0 cosθ y ⎟⎠⎜⎝ 0 sin θ x
Small rotation angles sinθ = θ and cosθ = 1 ,
Finally :
⎛ X 2 − X g1 ⎞ ⎛ Tx ⎞ ⎛ 1 − θ z − θy ⎞⎛ X 1 − X g1 ⎞
⎜
⎟ ⎜ ⎟ ⎜
⎟⎜
⎟
1
− θx ⎟⎜ Y1 − Yg1 ⎟
⎜ Y2 − Yg1 ⎟ = ⎜ Ty ⎟ + ⎜ θ z
⎜ Z − Z ⎟ ⎜ T ⎟ ⎜θ
⎟⎜ Z − Z ⎟
θ
1
g1 ⎠ ⎝ z ⎠ ⎝ y
x
g1 ⎠
⎝ 2
⎠⎝ 1
11/10/15
0 ⎞
⎟
− sin θ x ⎟
cosθ x ⎟⎠
Relation 3D-2D
Vector equation :
!
!
!
!
!
!
G1 P2 = T + RzRyRxG1 P1 (1)
G1 P2 = G1 P1 + P1 P2
!
!
! !
(1) and (2)⇒
P1 P2 = T + P1G1 + RzRyRxG1 P1 (3)
(2)
! !
Si P1=G1 alors P2=G2 et T = G1G2 (4)
!
!
!
P1G1 + G1G2 = P1G2 (5)
!
!
!
(3), (4) et (5)⇒
P1 P2 = P1G2 + RzRyRxG1 P1
⎛ X 2 − X 1 ⎞ ⎛⎜ X g 2 − X 1 ⎞⎟ ⎛ 1
⎜
⎟
⎜
⎜ Y2 − Y1 ⎟ = ⎜ Yg 2 − Y1 ⎟ + ⎜θ z
⎜ Z − Z ⎟ ⎜ Z − Z ⎟ ⎜ 0
⎝ 2
1 ⎠ ⎝ g 2
1 ⎠ ⎝
If θx=0 and θ y= 0
11/10/15
−θ z
1
0
0 ⎞⎛ X 1 − X g1 ⎞
⎟ (6)
⎟⎜
0 ⎟⎜ Y1 − Yg1 ⎟
1 ⎟⎠⎜⎝ Z1 − Z g1 ⎟⎠
Perspective projection
Y
Z
y
⎧ x = f X
⎪
Z
⎨
Y
⎪ y = f
⎩
Z
o
(7)
x
O
Hypothesis :
X
- Moving object is a plane surface orthogonal to the axis OZ : Z(x,y)=const
inside the object
- f=1;
ΔZ
Z1
ΔZ (8)
-Expressiong Z = Z + ΔZ
2
1
, Z << 1 ⇒ Z ≈ 1 − Z
2
1
from
(6),(7) and (8)⇒
11/10/15
4-parameter affine model
⎧ dx = x2 − x1 = t x + k (x1 − x g1 ) − θ (y1 − y g1 )
⎨
⎩dy = y2 − y1 = t y + θ (x1 − x g1 ) + k (y1 − y g1 )
avec
t x = x g 2 − x g1
t y = y g 2 − y g1
- Parameters of rotatio
θ =θz
k=−
- Parameters of translation
ΔZ
Z1
11/10/15
- Zoom parameter
(9)
Motion models
!
Developing the function d = (dx, dy )T in Taylor
series around (x g , y g ) up to 1st order
! ⎛ dx ⎞ ⎛ a0 ⎞ ⎛ a1 a2 ⎞⎛ x − x g ⎞
⎟⎟
d = ⎜ ⎟ = ⎜ ⎟ + ⎜
⎟⎜⎜
⎝ dy ⎠ ⎝ b0 ⎠ ⎝ b1 b2 ⎠⎝ y − y g ⎠
here
∂dx
a1 =
∂x
(10)
M
∂dx
a2 =
∂y
∂dy
b1 =
∂x
Θaff = (a0 , b0 , a1 , a2 , b1 , b2 )T
11/10/15
∂dy
b2 =
∂y
Affine 6-parameter model
Afficne models
Expressing
1
1
1
T
M = (traceM )I + M − M + M + M T − (traceM )I
2
2
2 !
∂dx ∂dy
traceM = a1 + b2 =
+
= div(d )
∂x ∂y
1
1 ⎛ ∂dy ∂dx ⎞ ⎡0 − 1⎤
T
( M − M ) = ⎜
−
⎟ ⎢
2
2 ⎝ ∂x ∂y ⎠ ⎣1 0 ⎥⎦
!
rot z (d )
⎛ − b2 + a1 b1 + a2 ⎞
1
T
⎟
M + M − (traceM )I = ⎜
⎝ b1 + a2 b2 − a1 ⎠
2
∂dy ∂dx
∂dx ∂dy
hyp 2 =
+
hyp1 =
−
∂x ∂y
∂x ∂y
⎛ 1 0 ⎞ 1
⎛ 0 1 ⎞
1 ⎛ 1 0 ⎞ 1 ⎛ 0 − 1⎞ 1
M = div⎜
⎟ + rot ⎜
⎟ + hyp1⎜
⎟ + hyp2⎜
⎟
2 ⎝ 0 11/10/15
1 ⎠ 2 ⎝ 1 0 ⎠ 2
⎝ 0 − 1⎠ 2
⎝ 1 0 ⎠
) [
(
[
]
]
Hierarchy of Affine models
⎧dx t x
⎨ = t
⎩dy y
⎧dx = t x + k ( x − x g )
⎨dy = t + k y − y
y
g
⎩
(
)
⎧ dx = x2 − x1 = t x + k
⎨dy = y − y = t + θ
2
1
y
⎩
(x1 − xg1 )− θ (y1 − yg1 )
(x1 − xg1 )+ k (y1 − yg1 )
⎧dx ⎛ a0 ⎞ ⎛ a1 a2 ⎞⎛ x − x g ⎞
⎟
⎟⎟⎜⎜
⎨ = ⎜⎜ ⎟⎟ + ⎜⎜
⎟
y
−
y
g ⎠
⎩dy ⎝ b0 ⎠ ⎝ b1 b2 ⎠⎝
⎧dx ⎛ a0 ⎞ 1 ⎛ div ⋅ ( x − x g ) − rot ⋅ ( y − y g ) + hyp1 ⋅ ( x − x g ) + hyp2 ⋅ ( y − y g ) ⎞
⎨ = ⎜⎜ ⎟⎟ + ⎜⎜ div ⋅ ( y − y ) + rot ⋅ ( x − x ) − hyp1 ⋅ ( y − y ) + hyp2( x − x ) ⎟⎟
g
g
g
g ⎠
⎩dy ⎝ b0 ⎠ 2 ⎝
2
⎧dx ⎛ a0 ⎞ ⎛ a1 a2 ⎞⎛ x − xg ⎞ ⎛ a3 a4 ⎞⎛⎜ x − xg ⎞⎟ ⎛ a5 ⎞
⎟ + ⎜⎜
⎟⎟
⎟⎟⎜⎜
+ ⎜⎜ ⎟⎟ x − xg y − y g
⎨ = ⎜⎜ ⎟⎟ + ⎜⎜
⎟
2
⎜
⎟
y
−
y
b
b
b
b
b
dy
g ⎠ ⎝ 3
⎩
⎝ 0 ⎠ ⎝ 1 2 ⎠⎝
4 ⎠⎝ y − y g ⎠ ⎝ b5 ⎠
(
(
)
)
(
)(
)
Robust global motion estimation from MB OF
➔ 6- parametr global motion model
⎧⎪dx(x, y)=a1 +a2 x+a3 y
⎨
⎪⎩dy(x, y)=a4+a5 x+a6 y
(dx(x, y),dy(x, y))T
- Displacement vector
- coordinates, of macro-block
center
(x, y)
11/10/15
Formulation in the sense of least squares
Z =Hθ +V
- Model equation
(
Z = dx1 ,dx2,...dxN,dy1 ,dy2,...dyN
T
θ = ( a1, a2 , a3, a4 , a5, a6 )⎛ 1 x1 y1 0 0 0 ⎞
⎜...... ......... ... ⎟
⎜ 1 x y 0 0 0 ⎟
H =⎜ N N
⎟
0
0
0
1
x
y
1
1
⎜
⎟
...
...
...
...
...
...
⎜
⎟
⎝ 0 0 0 1 xN yN ⎠
V
11/10/15
)T
- Vector of measures (2N)
Vector of model parameters
- Observation matrix (2Nx6)
- Observation noise
Solution by least square method
−1
θˆ = ( H T H ) H T Z
- Solution in the sense of least
squares
- Estimated vector of
measures
ˆ Hθˆ
Z=
- Residual error
ri=zi−zˆi
11/10/15
RANSAC Algorithm
For motion estimation – when noisy fields
Random Sample Consensus
➔ Priniciple : iterative algorithm which allows for discarding
outliers
➔ 1. Randomly choose a set of measures
➔ 2. Estimate the model for the set of selected measures(MC)
➔ 3. Find the outliers amongst all available value and discard
them (consensus set)
d(θ , x) − d 0 (x) ≤ Th
➔ 4. Repete 1 – 3 NbrIt times
22
June 24,2014
Video Analysis and Indexing for Medical Applications