1
ONLINE MULTI-OBJECT TRACKING
VIA STRUCTURAL CONSTRAINT
EVENT AGGREGATION
Ju Hong Yoon
Chang-Ryeol Lee
Ming-Hsuan Yang
Kuk-Jin Yoon
KETI
CV Lab., GIST
UC Merced
CV Lab., GIST
In CVPR 2016
2
OUTLINE
• Introduction
• Structural Constraint Event Aggregation
• Structural constraint cost function
• Event aggregation
• Assignment event initialization and reduction
• Two-Step Online MOT via SCEA
• Experiments
• Conclusion
3
OUTLINE
• Introduction
• Structural Constraint Event Aggregation
• Structural constraint cost function
• Event aggregation
• Assignment event initialization and reduction
• Two-Step Online MOT via SCEA
• Experiments
• Conclusion
4
INTRODUCTION
Data association
Detections-to-detections
Multi-object tracking
(MOT)
Detections-to-tracklets
Tracklets-to-tracklets
Object appearances
Similar objects ?
5
INTRODUCTION
6
INTRODUCTION
• A new data association method :
1. The structural motion constraints between objects
Location , Velocity
2. Event aggregation : Assignment ambiguities
7
INTRODUCTION
8
INTRODUCTION
• Two-step online 2D MOT framework :
1. Structural constraint event aggregation
2. Infer and recover the missing objects
9
OUTLINE
• Introduction
• Structural Constraint Event Aggregation
• Structural constraint cost function
• Event aggregation
• Assignment event initialization and reduction
• Two-Step Online MOT via SCEA
• Experiments
• Conclusion
10
STRUCTURAL CONSTRAINT EVENT
AGGREGATION
The state of an object 𝑖 at frame 𝑡 :
Position
Velocity
Size
Structural motion constraint between two objects :
11
STRUCTURAL CONSTRAINT EVENT
AGGREGATION
12
OUTLINE
• Introduction
• Structural Constraint Event Aggregation
• Structural constraint cost function
• Event aggregation
• Assignment event initialization and reduction
• Two-Step Online MOT via SCEA
• Experiments
• Conclusion
13
STRUCTURAL CONSTRAINT COST
FUNCTION
• The MOT task can be considered as a data association problem
𝑖
1
2
3
.
.
.
N
𝑘
1
2
3
.
.
.
M
0
14
STRUCTURAL CONSTRAINT COST
FUNCTION
• The MOT task can be considered as a data association problem
• If the detection 𝑘 is assigned to the object 𝑖,
• Otherwise,
• The best assignment event is then estimated by minimizing total
assignment costs
15
STRUCTURAL CONSTRAINT COST
FUNCTION
A detection 𝑘 at frame 𝑡 :
16
STRUCTURAL CONSTRAINT COST
FUNCTION
anchor assignment
structural constraint
17
STRUCTURAL CONSTRAINT COST
FUNCTION
Size
Appearance
18
STRUCTURAL CONSTRAINT COST
FUNCTION
19
20
OUTLINE
• Introduction
• Structural Constraint Event Aggregation
• Structural constraint cost function
• Event aggregation
• Assignment event initialization and reduction
• Two-Step Online MOT via SCEA
• Experiments
• Conclusion
21
EVENT AGGREGATION
22
OUTLINE
• Introduction
• Structural Constraint Event Aggregation
• Structural constraint cost function
• Event aggregation
• Assignment event initialization and reduction
• Two-Step Online MOT via SCEA
• Experiments
• Conclusion
23
ASSIGNMENT EVENT INITIALIZATION
AND REDUCTION
24
ASSIGNMENT EVENT INITIALIZATION
AND REDUCTION
25
26
OUTLINE
• Introduction
• Structural Constraint Event Aggregation
• Structural constraint cost function
• Event aggregation
• Assignment event initialization and reduction
• Two-Step Online MOT via SCEA
• Experiments
• Conclusion
27
28
TWO-STEP ONLINE MOT VIA SCEA
29
TWO-STEP ONLINE MOT VIA SCEA
30
TWO-STEP ONLINE MOT VIA SCEA
31
TWO-STEP ONLINE MOT VIA SCEA
32
TWO-STEP ONLINE MOT VIA SCEA
Hungarian algorithm
33
TWO-STEP ONLINE MOT VIA SCEA
• Update final tracking result with Kalman filter for smoothing :
34
TWO-STEP ONLINE MOT VIA SCEA
• Structural constraint update :
35
TWO-STEP ONLINE MOT VIA SCEA
• Object management :
• Add new objects (velocity = 0)
 The distances and the appearance between a
detection in the current frame and unassociated
detections in the past a few frames are smaller than a
certain threshold
• Delete objects
 If they are not associated with any detections for two
frames
36
OUTLINE
• Introduction
• Structural Constraint Event Aggregation
• Structural constraint cost function
• Event aggregation
• Assignment event initialization and reduction
• Two-Step Online MOT via SCEA
• Experiments
• Conclusion
37
EXPERIMENTS
• Data association evaluation
• Efficiency of the event reduction
• Comparisons with State-of-the-Art Methods
38
DATA ASSOCIATION EVALUATION
• RMN
• LM
-
• SCNN -
Relative Motion Network [29]
Linear Motion (Baseline)
(without the structural constraints or event aggregation)
Structural Constraint Nearest Neighbor
(without event aggregation)
[29] J. H. Yoon, M.-H. Yang, J. Lim, and K.-J. Yoon. Bayesian multi-object tracking
using motion context from multiple objects. In WACV, 2015
39
DATA ASSOCIATION EVALUATION
ETH sequences (Bahnhof, Sunnyday, and Jelmoli sequences) [8]
[8] A. Ess, B. Leibe, K. Schindler, and L. V. Gool. A mobile vision system for robust
multi-person tracking. In CVPR, 2008
40
EFFICIENCY OF THE EVENT REDUCTION
41
COMPARISONS WITH STATE-OFTHE-ART METHODS
• State-of-the-art
• MDP [26]
• TC ODAL [1]
• RMOT [29]
• NOMT-HM [5]
• ODAMOT [11]
[1] S.-H. Bae and K.-J. Yoon. Robust online multi-object tracking based on tracklet confidence and online discriminative
appearance learning. In CVPR, 2014
[5] W. Choi. Near-online multi-target tracking with aggregated local flow descriptor. In ICCV, 2015
[11] A. Gaidon and E. Vig. Online Domain Adaptation for Multi-Object Tracking. In BMVC, 2015
[26] Y. Xiang, A. Alahi, and S. Savarese. Learning to track:Online multi-object tracking by decision making. In ICCV,2015
[29] J. H. Yoon, M.-H. Yang, J. Lim, and K.-J. Yoon. Bayesian multi-object tracking using motion context from multiple
objects.In WACV, 2015
42
COMPARISONS WITH STATE-OFTHE-ART METHODS
• Evaluation metrics :
• MOTA - Multiple Object Tracking Accuracy
• MOTP - Multiple Object Tracking Precision
• MT
- the number of mostly tracked
• ML
- the number of mostly lost
• FG
- the fragment
• ID
- the identity switch
• Rec
- the Recall
• Prec - the Precision
• sec/Hz - the runtime
• AR
- the average ranking
43
COMPARISONS WITH STATE-OFTHE-ART METHODS
• Benchmark dataset :
• KITTI dataset [12] : 29 sequences
• Detections : DPM [10], regionlet [24]
• MOT Challenge dataset [17] : 22 sequences
[10] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with
discriminatively trained part based models. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 32(9):1627–1645, 2010
[24] X.Wang, M. Yang, S. Zhu, and Y. Lin. Regionlets for generic object detection. In ICCV, 2013
[12] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. IJRR, 2013
[17] L. Leal-Taix´e, A. Milan, I. Reid, S. Roth, and K. Schindler. Motchallenge 2015: Towards a benchmark
for multi-target tracking. In arXiv:1504.01942, 2015
44
45
COMPARISONS WITH STATE-OFTHE-ART METHODS
46
COMPARISONS WITH STATE-OFTHE-ART METHODS
47
OUTLINE
• Introduction
• Structural Constraint Event Aggregation
• Structural constraint cost function
• Event aggregation
• Assignment event initialization and reduction
• Two-Step Online MOT via SCEA
• Experiments
• Conclusion
48
CONCLUSION
• Structural motion constraints - Large camera motion
• Event aggregation - Assignment ambiguities
• Two-step algorithm - Recover missing objects
49
THANKS FOR LISTENING!