Visual Event
Detection & Recognition
Filiz Bunyak Ersoy, Ph.D. student
Smart Engineering Systems Lab
Research Interest



Recognition of visual events from video sequences.
Use of visual event information for intelligent video
surveillance and semantic video indexing & retrieval.
Incorporation of learning to event modeling and
recognition.
Very few event detection systems that are currently available are not flexible,
work for a very limited domain for very limited number of predefined, mostly
hand-coded events. They are not designed to be extended or modified.
Learning will enable adaptability and extensibility of an event detection
system.
Content of Video Data
Low Level
Visual Features
 Color
 Texture
 Shape
 Motion
 Shot Boundaries




Mid Level
Semantic
Content
People/Objects
Location
Actions
Time
High Level
Semantic Content



Story
Concept
Event
Challenges
1-Representation:
3-Indexing:
 Modeling visual features
 Efficient indexing algorithms
 Image/video object
for high-dimensional feature
representation
space
 Description of spatio Robust, scalable indexing
temporal relationships
algorithms for spatio-temporal
 High-level event
queries
representation
4-Summarization:
2-Analysis:
 Automated summarization of
 Segmentation of video
visual content
objects
 Visualization of content at
 Adaptive grouping of
different levels
features & objects
 Compressed-domain feature
extraction
Proposed Framework
Context
Object, Scene &
Event Libraries
Object Information
Object
Classification
Feature
Extraction
Event Inference
Motion Analysis
Events
Objects
Relationships
Events
Object Trajectories &
Spatio-temporal
relationships
Context is any a priori information
provided to the system.
Motion Analysis
Tracking
Feature
Extraction
Moving Object Detection
 Temporal differencing
 Background subtraction
 Optical Flow
Moving Object
Detection
Correspondence
Analysis
Context
Prediction




Tracking
Region-based methods

Contour-based methods

Feature-based methods
Model-based methods
Update
Object States
Single-view methods (single
camera)
Multi-view methods (multiple
cameras)
Event Inference
Spatio-temporal
Relationships
Information
About
Objects
Event Inference
Methods
A priori Information
About the Application,
Goal, Scene, Objects.
Event descriptions.
EVENTS
Possible Event Inference Methods
 Rule based
 Logical Formalisms
 Temporal Logic
 Event Logic
 Fuzzy Logic
 Bayesian belief network
 Hidden Markov model
 Petri-net
 Grammar
Applications of Event
Detection/Recognition

Surveillance and Monitoring
 Traffic (track vehicle movements and
annotate action in traffic scenarios.)
 Detection of accidents, traffic
violations, congestions.
 Gather statistics about human
activities, road utilization etc.
 Surveillance of public places / shops
/ offices etc.
 Detection of atypical incidents, theft,
vandalism, shoplifting, abandoning
(possibly dangerous) objects.



Indexing of Broadcast Video
 Sports video indexing for
newscasters or trainers.
 Semantic indexing for
automated annotation for
content retrieval.
Interactive environments:
environment that respond to the
activity of occupants.
Robotic collaboration: creating
robots that can effectively navigate
their environment and interact
with other people and robots.
Some Interesting Systems
from the Industry-1
Realtime Video Analysis Group @
Honeywell Laboratories:
Siemens Corporate Research
Cooperative Camera Network (CCN):
Subway monitoring System
Indoor
 Real time segmentation of people
 Reports the presence of visually tagged
in subway platforms for the
individual throughout a building
purpose of congestion (crowding)
structure.
detection
 Meant to be used for monitoring
potential shoplifters in department
stores.
Detection of Events for Threat
Evaluation & Recognition
(DETER): Outdoor
 Monitor large open spaces like parking
lots and reports unusual moving
patterns by pedestrians & vehicles.
Some Interesting Systems
from the Industry-2
Mitsubishi Electric Research
Laboratories
Applications
 Detecting accidents through analysis of
traffic surveillance video.
 Detection of traffic jams using our MPEG-7
motion activity descriptor.
 Extraction of semantic features from lowlevel features of soccer games.
http://www.merl.com/projects/event-detection/
Some Interesting Systems
from the Industry-3
ASCOM: INVIS Traffic
Detect
 Traffic speed and traffic flow
density
 Congestion and slow traffic
 Stationary vehicles (possibly an
accident)
Wrong Way Early Warning System
 Recognizes vehicle patterns and
compares subsequent images to
determine vehicle and direction.
 Thus it can detect any car driving the
wrong way into a lane against the
flow.
Copyright © 2002 Ascom www.ascom.com