Keyword Extraction and Image
Annotation Games to Enhance the
Cultural Database Creation
Virach Sornlertlamvanich
and Thatsanee Charoenporn
[email protected], [email protected]
National Electronics and Computer Technology Center, Thailand
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Motivation
• Cultural Knowledge Creation
• Image and object labeling
– Keyword and semantic relation extraction
– Image as a focal point
• Cultural Knowledge Services
– Service platform
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
3 Steps in Digital Cultural Communication
Step 1: Cultural knowledge curation
– Reuse
– Standardization
Step 2: Cultural image annotation
– Keyword extraction
– Semantic relation acquisition
– Image annotation games
Step 3: Cultural knowledge service
– Cultural knowledge platform for application service
development
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
CULTURAL KNOWLEDGE CURATION
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Community Co-Creation Cultural Knowledge Base
Institution
Curation and Presentation
Cultural knowledge curation
Standardized Annotated
Cultural Knowledge Base
- Search
- Category
- Statistics
Community
Citation
• Museum
• Museum archive
• Other departments
Community Co-Creation
- Input
- GPS data
- Tag
- Invitation, registration, approval
Audience
Provincial Cultural Knowledge Base Service
- Search
- Text
- Filter
- Similarity (color, structure, role, mood, image)
- Presentation
- Location, category
- Statistics
- Relation
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Cultural Knowledge Portal Creation


Scope of Collection
Cultural Personnel/Organization
- Artist
- Scholar
- Religious Monument
- Writer/Author
- Society/Association
- Cultural Network
- Cultural Unit

Cultural Artifact
- Archaeological Objects
- Artwork
- Visual Art
- Book/Press
- Audiovisual Media
- Utensil
- Costume

Way of Life
- Ethnic
- Religion and Belief
- Tradition and Rite
- Language and Literature
- Local Wisdom
- Performing Art and Music

Cultural Site
- Archaeological Site
- Historical Park
- Historical Site
- Architecture
- Religious Place
- Museum
- Library
- Archive
- Monument
- Theatre
- Tourism spot
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Cultural Databank
http://www.m-culture.in.th/
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Cultural Databank
http://www.m-culture.in.th/
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
CULTURAL IMAGE ANNOTATION
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Keyword Extraction
• Some keywords are readily available in the set
tags, but many of them are still missing.
• Our task is to extract those missing keywords
from the description and title.
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Keyword Extraction
• Some keywords can be linked to external
pages, e.g. Wikipedia.
• Our task is to find appropriate articles
corresponding to those keywords.
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Method for KW Extraction
• Chunking model (Uchimoto et al., 2004) for
keyword extraction
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Training Data Preparation
• Generate a keyword list from tags and titles that are not
shorter than 5 characters and not longer than 30 characters
• Segment descriptions using a state-of-the-art Thai word
segmentation algorithm (Kruengkrai et al., 2009)
• Note that the word segmentation algorithm was trained using
ORCHID corpus and TCL’s lexicon (contents of ORCHID corpus
and our current data are quite different)
• Label the segmented descriptions with the keyword list
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Training Data
• Description
ผ้าซิ่นลายมัดหมี่บา้ นปทุมแก้ว เป็ นงานฝี มือพื ้นบ้ าน …..
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Labeling
• Apply BIO tagging
– B: beginning position of a keyword
– I: intermediate (or end) position of a keyword
– O: other words
• If several matches are possible, select the
longest one (like in the previous example)
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Training Data
• Description
ผ้าซิ่นลายมัดหมี่บา้ นปทุมแก้ว เป็ นงานฝี มือพื ้นบ้ าน …..
• Segmented/Tagged/Labeled Description
Word
POS tag
Label
ผ้ าซิ่น
N
B-K
• Keyword List
ลายมัดหมี่บ้านปทุมแก้ ว
N
I-K
<space>
เป็ น
งานฝี มือพื ้นบ้ าน
……
P
V
N
O
O
O
…..
…..
(extracted from tag and title)
…..
ผ้ า
…..
ผ้ าซิน่
ผ้ าซิน่ ลายมัดหมี่บ้านปทุมแก้ ว
…..
…..
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Chunking Model
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Preliminary Experiment Result
• 3000 examples for training, 500 examples for
testing
• Based on Margin Infused Relaxed Algorithm
(MIRA), Crammer et al., 2005
– Baseline features (Unigram and Bigram) +
– 3 character prefix/suffix of current word +
– 3 consecutive POS tags
• Recall=0.8256, Precision=0.9061, F1=0.8640
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Semantic Relation Acquisition
• Extract commons syntactic patterns between
two nouns
• Our task is to acquire triples (ei , rij , ej ), where
– ei and ej are entities (keywords)
– rij is a relationship between them
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Extract Common Syntactic Pattern of a Predicate
between Two Keywords
• Example
Predicate
Anchored keyword
Title: วัดทุ่ง
Description: วัดทุ่ง มีอายุราว 500 ปี สันนิฐานว่าสร้างขึ้นในสมัยกรุงสุโขทัย
Title: วัดตราชู
Description: วัดตราชู สร้างขึ้นในสมัย กรุงศรี อยุธยาตอนต้ น ราว
พ.ศ.2076
Title: หลวงพ่อขาว
Description: เป็ นพระพุทธรูปเก่าแก่เนื ้อหินทรายปางสมาธิ ขนาดหน้ าตักกว้ าง ๒
ศอก
ประดิษฐานอยู่ในวิหารวัดหลวงวัดสันนิฐานว่าสร้างขึ้นในสมัยอยุธยา
Title: พระพุทธรูปปางมารวิชยั
Description: สร้างขึ้นในสมัยรัตนโกสินทร์ ตอนต้ น
Title: วิหารวัดโยธานิมิต
Description:
สร้าConference
งขึ้นในสมั
พระบาทสมเด็
จพระเจ้US.,
าตากสิ
นมหาราช
PNC 2012 Annual
andยJoint
Meetings, UC Berkeley,
December
7-9, 2012
Extract Common Syntactic Pattern of a Predicate
between Two Keywords
• Example
(วัดทุง่ , สร้างขึ้นในสมัย, กรุงสุโขทัย)
(วัดตราชู, สร้างขึ้นในสมัย, กรุงศรี อยุธยาตอนต้ น)
(หลวงพ่อขาว, สร้างขึ้นในสมัย, อยุธยา)
(พระพุทธรูปปางมารวิชยั , สร้างขึ้นในสมัย, รัตนโกสินทร์ ตอนต้ น)
(วิหารวัดโยธานิมิต, สร้างขึ้นในสมัย, พระบาทสมเด็จพระเจ้ าตากสิน
มหาราช)
(ei,
BUILT_IN,
ej)
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Extract Common Syntactic Pattern of a Predicate
between Two Keywords
• Example
Predicate
Anchored keyword
Title: กระโจมไฟบ้ านโรงถ่าน
Description: สร้างโดย อพท. เมื่อปี พ.ศ.2550 เป็ นท่าเทียบเรี อสาหรับเรื อ
ท่องเที่ยว
Title: ศาลเจ้ าตากสิน วัดบ้ านค่าย
Description: ศาลปูนขนาดกลาง สร้างโดยพระครูพิพฒ
ั น์ชยาภรณ์
Title: วัดทุ่งโฮ้ งใต้
Description: สร้ างขึ ้นเมื่อ พ.ศ.2370 จากตานานเล่าว่าสร้างโดยกลุม่ ชาวลาว
พวน
Title: ศาลพระพรหม
Description: ตังอยู
้ ่บริ เวณสวนตุงโคม ตาบลเวียงอาเภอเมืองเชียงรายจัดสร้าง
โดยเทศบาลนครเชียงราย
Title:วงเวียนนิมิตร
Description:
วงเวียนนิมิตรหรื อวงเวียนม้ าน ้าก่อสร้างโดยเทศบาลนครภูเก็ตในปี
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Extract Common Syntactic Pattern of a Predicate
between Two Keywords
• Example
(กระโจมไฟบ้ านโรงถ่าน, สร้างโดย, อพท.)
(ศาลเจ้ าตากสิน วัดบ้ านค่าย, สร้างโดย, พระครูพิพฒ
ั น์ชยาภรณ์)
(วัดทุง่ โฮ้ งใต้ , สร้างโดย, กลุม่ ชาวลาวพวน)
(ศาลพระพรหม, สร้างโดย, เทศบาลนครเชียงราย)
(วงเวียนนิมิตร, สร้างโดย, เทศบาลนครภูเก็ต)
(ei,
BUILT_BY,
ej)
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
ESP Game and Peekaboom
proposed by Luis von Ahn, May 25, 2006 by Pete Cashmore
• ESP Game – In the ESP Game, the two players are shown an image
and asked to enter a word that describes it. The players can’t see each
other’s guesses. The aim is to enter the same word as your partner in the
shortest possible time. But there’s an ulterior motive here: much of the
data is recorded, and could be used to power image search engines in the
future. What’s cheaper – paying thousands of Mechanical Turkers to label
all the images on the web, or tricking people into doing it for free?
• Peekaboom – Peekaboom takes the ESP Game to the next level.
Unlike the ESP Game, it’s asymmetrical. To start, one user is shown an
image and the other sees an empty black space. The first user is given a
word relating to the image, and the aim is to communicate that word to
the other player by revealing portions of the image. So if the word is “eye”
and the image is a face, you reveal the eye to your partner. But the real
aim here is to build a better image search engine: one that could identify
individual items within an image.
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
ESP Game
• Two players are shown
an image
• asked to enter a word
that describes it.
• The aim is to enter the
same word as your
partner in the shortest
possible time.
Twitter
Angry bird
Bird
To name the image
Bird
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Bird
Mohawk
Peekaboom
• One user is shown a
named image and show
the part of the image
according to the name
• Another user gives a
word relating to the
image
• The aim is to enter the
same word as it is
named in the shortest
possible time.
Bird
Squirrel
Flying fish
Bird
To label the object
in the image
Bird
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Extended Peekaboom
• One user is shown a named
image and show the part of
the image according to the
name
• Another user gives a word
relating to the image
• The aim is to enter the same
word as it is named in the
shortest possible time.
• A word from the Synset can be
matched.
• Once a synset is selected cross
language matching can be
determined.
Bird
Squirrel
Flying fish
Bird
AWN
Bird
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Demo
• ESP-like game
– http://m-culture.in.th/game/esp_game
– Play mode
• Single player mode: play against history
• Two-player mode: guess to match each other
• Extended Peekaboom game
– http://m-culture.in.th/game/peekaboom
– Play mode
• Single player mode: play against history
• Two-player mode: guess to match each other
– For Thai language, use AWN to support synonym, hypernym,
hyponym, meronym, and holonym
– For other languages, use AWN to support synonym only
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Preliminary Experiment
• 18 images played by 19 persons. For each image,
we allow 60 seconds to guess a proper word.
Exact
Syn
Hyper
Hypo
Mero
Holo
229
32
16
1
7
11
• AWN can expand the matching in 67 cases or
increase 22% of matching ratio.
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
CULTURAL KNOWLEDGE SERVICE
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Shop Database
Title shop
Snippet description
Tags A, B, C
Cultural Database
A
B
C
D
Product Database
Title culture
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
Title culture
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
Title culture
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
Title shop
Snippet description
Tags A, B, C
Title shop
Snippet description
Tags A, B, C
Maker Database
Title maker
Snippet description
Tags A, B, C
Title maker
Snippet description
Tags A, B, C
Title maker
Snippet description
Tags A, B, C
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
To find a related Product from Culture information
Title culture
Snippet description
Tags A, B, C
Title culture
Snippet description
Tags A, B, C
Title culture
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
Title culture
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
To find the background Culture information from a Product
Title product
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
Title culture
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Product and Culture information relation
Title culture
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
Title culture
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
Title culture
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
Title culture
Snippet description
Tags A, B, C
Title product
Snippet description
Tags A, B, C
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Summary
• From this ESP-like game, we successfully named
the images or at least obtained a list of
candidates for labeling the object in the image to
be used in the next extended Peekaboom game.
• Synonym, hypernym, hyponym, meronym,
holonym from AWN can help expanding the
matching ratio.
• Cross language image labeling is realized by AWN
synonym.
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012
Future Work
• Enhancing keyword extraction to find more
term candidate for image matching
• Call for participation of the extended ESP and
Peekaboom games for image labeling
PNC 2012 Annual Conference and Joint Meetings, UC Berkeley, US., December 7-9, 2012