Personalized Abstraction of
Broadcasted American Football
Video by Highlight Selection
Noboru Babaguchi (Professor at Osaka Univ.)
Yoshihiko Kawai and Takehiro Ogura (NHK)
Tadahiro Kitahashi (Professor at Kwansei Gakuin Univ.)
IEEE Transactions on Multimedia, 2004
Outline






Introduction
Related Work
Method of detecting significant events in
video stream
Method of generating video abstracts
Experimental results
Conclusion
Introduction

Video abstract


Creating shorter video clips or video posters from an
original video stream
Two schemes of video abstraction

Temporally compressing the amount of the video data


Smith et al., Lienhart et al., He et al., Oh et al., Babaguchi
Provide image keyframe layouts representing the whole
video contents

Yeung and Yeo, Uchihashi et al., Chang et al., Toklu et al.
Introduction

This method of abstracting sports video





Specifically broadcasted TV programs of
American football
Take personalization into consideration
Belong to first scheme of video abstraction
Abstraction based on highlights that are closely
related to semantical video contents
Detecting significant events like score events
Introduction

How to detect events




Image analysis is very difficult
This method’s solution is to make use of external metadata,
called gamestats
Linking video segments with descriptions of the gamestats
Personalization



Extensively attempted in a variety of application fields
Emphasize it because the significance of scenes vary
according to preferences and interests
Provide a profile to collect personal preferences
Related work – time compression

Smith et al.


Lienhart et al.


Extracted significant information from video such as
keywords, specific objects, camera motions and scene
breaks with integrating language, audio, and image
analyzes
Assemble and edit scenes of significant events in action
movies, focusing the on actor/actress’s closeup, text, and
sound of gunfire and explosion
These two method are based on surface features of
the video rather than on its semantical contents.
Related work – time compression

Oh et al.



Babaguchi



Abstracting video using user selected interesting scenes
Automatically uncover the remaining interesting scenes in
the video by choosing some interesting scenes
Video abstraction based on its semantical content in the
sports domain
To select highlights of a game, an impact factor for a
significant event in two-team sports was proposed
He et al.


Create summaries for online audio-video presentations
Use pitch and pause in audio signals, slide transition points
in the presentation, and users’ access patterns
Related work – spatial expansion

Goal


Yeung and Yeo


Automatically create a set of video posters (keyframe
layouts) by the dominance value of each shot
Uchihashi et al.


Visualize the whole contents of the video
Making video posters whose size can be changed
according to the importance measure
Chang et al.

Make shot-level summaries of time ordered shot sequence
or hierarchical keyframe clusters, as well as program-level
summaries
Detection of significant events

Detect significant events in the original video stream
according to the description in the gamestats
Identification of event frames



An event occurs in the shot including the event
frame
Attempt to recognize text expressing the game time
in the overlay, and then to identify an event frame
To identify the event frame, an overlay model is
employed
Detection of event shots


A shot is defined as consecutive image frames at a
single camera view
Classify the event shots into four types

live-play, replay, pre-play, and post-play shot
Generation of personalized video abstract



Generating video abstracts from the detected
significant events
Select highlights of the game from all the events,
considering profile descriptions
The generating rules for the video abstract:
Profile


A video abstract has to be personalized because
significance of events could change individually
Provide a profile to collect personal preferences and
interests, the items are:




Favorite teams
Favorite players
Events to want to see
Specifications


Range of the video stream to be abstracted
Length of the abstract
Significance degree of events


The highlights of the game depend on the significance of each event,
and significance can be estimated in terms of event rank, event
occurrence time, and the profile
Event rank






State change event (SCE): a score event can change the current state into a
different state
Rank 1: SCE’s.
Rank 2: not SCE’s, but exceptional score events
Rank 3: events closely related to score events
Rank 4: all other event that are not Rank 1 to 3
Rank based significance degree of event Ir:
where ri denotes the rank of the ith event Ei, αis a coefficient to consider how
large the difference of the rank affects the significance
Significance degree of events

Event occurrence time


The score events occurring at the latter or final stage of the game largely affect the
result that should have great significance
Occurrence time based significance degree of event It:
where N is the number of all events, β is a coefficient to consider how large the
occurrence time affects the significance

Profile


Comparing the descriptions of the profile and the occurring event
Profile based significance degree of event Ip:
where l denotes the number of descriptions that don’t coincide with each other, γ is a
coefficient to consider how large the profile affects the significance

Significance degree of an event I:
Selection of highlights


To determine highlights, we concentrate on both
priority order of shots and significance degree of
events
Priority order of each shot segment








Motion live shot
Still live shot
Motion replay shot
Still replay shot
Motion pre-play shot
Still pre-play shot
Motion post-play shot
Still post-play shot
Selection of highlights
Experimental results
Experimental results
Experimental results

Two measures to evaluate the
quality of the generated
abstract
where N denote the number of
highlights included in abstracts
Experimental results – effect of
personalization

Inclusion ratio: the ratio of the length of shots which are
concerned with the specified team to the total length
Experimental results – effect of
personalization
4-symbol string in the cells of table
represents each condition of the preplay, live, replay, and post-play shots for
the event
Experimental results – effect of
personalization
Experimental results – effect of
personalization
Conclusion




Based on the detected significant events by
recognizing the textual overlays
Link the video contents with useful external
metadata by using the gamestats
Three sorts of significance degrees play a central
role in highlight selection
Remaining problems



The method is for different two-team sports
A tailoring mechanism for shots, adjusting for the total
abstract
Seek for a sophisticated way of refining the profile