Human Computation
Play a Game to Develop an Ontology
Peyman Nasirifard
p+e+y+m+a+b-b+n dot sin(arcsin(lastname)) @ deri.org
1
Agenda
Introduction

CAPTCHA
Games with a purpose



ESP game
Peekaboom
Verbosity
Possible game for developing simple ontologies
Play a game
Conclusion
2
Introduction
Human-based computation is a technique when a computational
process performs its function via outsourcing certain steps to
humans.
3
Back to History
Yahoo! and Gmail are not interested to
enable a bot to create thousands accounts
per day for sending spam

They use CAPTCHA to prevent it
plus
4
CAPTCHA
Stands for “Completely Automated Public
Turing test to tell Computers and Humans
Apart”
Luis von Ahn et al. coined the term in
2000
A Program that can tell
whether a user is a human
or a computer
Many different techniques
5
Some Examples
6
Dog or Cat?
Human: mmm… dog
Computer: mmmmmmmmmmmmmmm…
7
Human Computation
If we use people to break CAPTCHA, we
are doing human computation


In some countries, some companies hire
people to break CAPTCHA and send spam
Some companies cleverly use humans to
break CAPTCHA and send spam
How?
8
Clever spammers
9
Clever Spammers
Free Nude Photos
Type the word in the box if you want
to see the next picture
10
Really?!
Jan 2004: world without spam by 2006!


Huge amount of investment
Bill Gates receives 4 million spams per day
11
Nice Quote
Luis von Ahn: Instead of hiring people and
pay them to solve our problems, we can
design games and people will pay us to
play our games and solve our large-scale
problems!
12
The ESP game
Object of the game: type the same word
Only thing in common is: an image
Players


Do not know each other (randomly paired)
Can not communicate
Advantages:





Two different sources labels the image
enjoyable
labels all images on Google image in a short time
Help to improve English!
There are many people that play over 20 hours a week
13
The ESP game
Player 1
Player 2
• CAR
• WOMAN
• GIRL
• CAR
• TREE
Agree: CAR
Get points
14
Taboo words
Taboo words

More difficult, but more fun
• CAR
• WOMAN
15
Single version of ESP game
• CAR
• WOMAN
• GIRL
• CAR
• TREE
• The engine records everything from previous players
• A single player will actually play with another player, but not at the same time
16
Cheating and Repetition
Problem: Agreement on cheating

Let’s label all images with “dog”
Solution: At random, system gets players
test images to check whether they play
honestly or not

If they do not play honestly, the system will let
them play, but nothing will be recorded
For certainty, only labels which at least N
pairs agreed upon will be stored
17
The Limitations of ESP
The ESP Game can label images (and
consequently tell you what’s in them), but
it cannot:


Find the objects being labelled
Determine the way in which the object
appears – does the label “car”
refer to the text “car” or an actual
car in the image?
18
The place of objects in an image

Such information would be extremely useful
for computer vision research
man
dog
19
The Guesser guesses:
•Flower
•Petal
•Butterfly
The Revealer clicks on parts
of the image and shows them
to the Guesser.
Server: Correct, Butterfly
20
Hints
The hints
label help
“car” distinguish
is ambiguous
the--manner
in which the label “car” appears:
this is “car”
this is the object “car”
this is also “car”
this is the text “car”
21
Verbosity
Collect common-sense facts



Water quenches thirst
Sky is blue
Lions eat meat
We as human know hundreds of millions
common sense facts

Computers do not know
If know, potentially make them more intelligent
(e.g. search better)
22
Common sense fact samples
• It is liquid
• It is white
Milk
• it has lactose
• cereal is eaten with it
23
Verbosity
Narrator
Guesser
MILK
is typically near cereal
is a liquid
24
Verbosity
Narrator
Guesser
MILK
is typically near cereal
is a liquid
MILK
25
Verbosity
Narrator
Guesser
Object
Common sense facts about
the object
26
Verbosity
Narrator
Guesser
Object
Common sense facts about
the object
Object
27
Templates
• ___ is a kind of ___. Allows for hierarchical
categorization.
• ___ is used for ___. Provides information about
the purpose of a word.
• ___ is typically near/in/on ___ (three
templates). Provide spatial data.
• ___ is the opposite of ___ / ___ is related to
___ (two templates). Provide data about basic
relations between words.
• ___. In the game, this is a “wildcard” that
collects related words.
28
Symmetric vs. Asymmetric
Verbosity is a asymmetric game, whereas ESP game is a symmetric
game.
Symmetric games: constraint is number of outputs per input
Asymmetric games: constraint is number of inputs that produces the
same output
29
Possible game to build an ontology
Several game should work together


Images come from ESP game
Not always: only those images are selected
which have one object in it
i.e. car, bike, monitor, mouse, house

These images are input to next game which
tries to catch the properties of objects
car has colour, car has wheels, car has
manufacture, car has owner, car has building year,
etc.
30
Possible game to build an ontology
Cardinality will be caught by templates, as
soon as we have properties.


Car has four wheels
Car has one plaque
These sentences will be transferred to
OWL representation using a mediator.
The more pairs play the game, the more
complex the ontology will be
31
Contact me if you are interested to work on it
32
This page intentionally left blank
33
Guess what!
It has usually four wheels
It has usually one seat
It is kind of vehicle
It operates with human power
It operates with batteries
It has a break system
It is a kind of chair
34
Answer
35
Conclusion
Games are enjoyable!
More than 9 billion Human-hours of
solitaire are played each year
We may cleverly using humans to solve
large-scale problems by designing
interesting games
Many people play word-guessing games
to improve their English
Go and play to promote science!
36
References
[1] Verbosity: A Game for Collecting Common-Sense Facts, http://www.cs.cmu.edu/~biglou/Verbosity.pdf
[2] Peekaboom: A Game for Locating Objects in Images, http://www.cs.cmu.edu/~biglou/Peekaboom.pdf
[3] Labeling Images with a Computer Game, http://www.cs.cmu.edu/~biglou/ESP.pdf
[4] Games with a Purpose, http://www.cs.cmu.edu/~biglou/ieee-gwap.pdf
[5] Wikipedia, http://en.wikipedia.org/wiki/Human-based_computation
[6] We'll End Spam Within 2 Years,
http://www.connectedhomemag.com/Networking/Articles/Index.cfm?ArticleID=41587
http://news.bbc.co.uk/2/hi/business/3426367.stm
[7] CAPTCHA,
http://en.wikipedia.org/wiki/Captcha,
http://www.captcha.net
[8] ESP game, www.espgame.org
[9] Peekaboom game, http://www.peekaboom.org/
[10] Verbosity game, www.peekaboom.org/verbosity/
[11] Presentation, http://isandtcolloq.gsfc.nasa.gov/fall2006/presentations/Ahn.ppt
[12] Presentation, www.aladdin.cs.cmu.edu/workshops/lamps05/Slides/Peekaboom.ppt
37
Game Over
p+e+y+m+a+b-b+n dot sin(arcsin(lastname)) @ deri.org
38