Watson: Jeopardy! and Beyond
Murthy Devarakonda, Ph.D.
Research Scientist and Team Manager
Watson Technologies
IBM Research
© 2013 IBM Corporation
The Jeopardy! Challenge
A palpable, compelling and notable way to drive the technology of Question Answering
along Key Dimensions
Broad/Open
Domain
Complex
Language
$200
$400
The juice of this bog
fruit is sometimes
used to treat urinary
tract infections
This Bavarian city
traces its origins to a
Benedictine monastery
at Tegernsee
$1000
Most modern
supercomputers run on
this operating system
developed by Linus
Torvalds
High
Precision
Accurate
Confidence
High
Speed
$600
$800
In cell division, mitosis
splits the nucleus &
cytokinesis splits this
liquid cushioning the
nucleus
Grace Murray Hopper
is credited with
applying this 3-letter
term to a mysterious
computer problem
© 2013 IBM Corporation
© 2013 IBM Corporation
Watson answers by finding, reading, scoring and combining evidence
IN 1698, THIS COMET
DISCOVERER TOOK A
SHIP CALLED THE
PARAMOUR PINK ON THE
FIRST PURELY
SCIENTIFIC SEA VOYAGE
Question
Analysis
Important Terms: 1698, comet,
paramour, pink, …
AnswerTypes: comet discoverer
Date(1698),
Took(discoverer, ship)
Called(ship, Paramour Pink)
…
Content
(Structured & Unstructured)
Primary
Search
Candidate Answer/Hypothesis Generation
Isaac Newton
High-Speed
Evidence
Retrieval
100’s of NLP Scoring
Algorithms
…
[0.58 0.5 -1.3 … 0.97]
Wilhelm Tempel
[0.71 1
13.4 … 0.60]
HMS Paramour
[0.42 0
2.0 … 0.90]
Christiaan Huygens
[0.84 0.5 10.6 … 0.88]
Halley’s Comet
[0.33 0
6.3 … 0.83]
Edmond Halley
[0.21 1
11.1 … 0.92]
Pink Panther
[0.91 0
-8.2 … 0.31]
Peter Sellers
[0.91 0
-1.7 … -.20]
…
Diverse and Extensible Evidence
Scoring
1)
2)
3)
4)
Edmond Halley (0.85)
Christiaan Huygens (0.20)
Peter Sellers (0.05)
…
Merging &
Ranking Based on
Statistical Machine
Learning
© 2013 IBM Corporation
DeepQA: The architecture underlying Inside Watson
Using a highly parallelized architecture generates many hypotheses, collects a wide range
of evidence and balances the combined confidences of over 100 different analytics that
analyze the evidence form different dimensions
Question
Evidenc
e
Sources
Answer
Sources
Primary
Search
Learned Models
help combine and
weigh the
Balance Evidence
Candidate
Answer
Generation
100’s Possible
Answer
Scoring
& Combine
Models
Models
Models
Models
Models
Models
Deep
Evidence
Evidence
Retrieval 100,000’s Scores from
Scoring
1000’s of
Pieces of Evidence
many Deep Analysis
Algorithms
Answers
Multiple
Interpretations
Question &
Topic
Analysis
100’s
sources
Question
Decomposition
Hypothesis
Generation
Hypothesis
Generation
Hypothesis and Evidence
Scoring
Hypothesis and Evidence
Scoring
...
Synthesis
Final Confidence
Merging &
Ranking
Answer &
Confidence
© 2013 IBM Corporation
Watson Learns Relationships by “Reading”
Volumes of Text
Syntactic Frames
Semantic Frames
Inventors patent inventions (.8)
Officials Submit Resignations (.7)
People earn degrees at schools (0.9)
Fluid is a liquid (.6)
Liquid is a fluid (.5)
Vessels Sink (0.7)
People sink 8-balls (0.5) (in pool/0.8)
© 2013 IBM Corporation
Evaluating Possibilities and Their Evidence
In cell division, mitosis splits the nucleus & cytokinesis
splits this liquid cushioning the nucleus.
Organelle
Vacuole
Cytoplasm
Plasma
Mitochondria
Blood …
Many candidate answers (CAs) are generated from many different searches
Each possibility is evaluated according to different dimensions of evidence.
Just One piece of evidence is if the CA is of the right type. In this case a “liquid”.
Is(“Cytoplasm”, “liquid”) = 0.2↑
“Cytoplasm is a fluid surrounding the nucleus…”
Is(“organelle”, “liquid”) = 0.1
Is(“vacuole”, “liquid”) = 0.2
Is(“plasma”, “liquid”) = 0.7
Wordnet
Learned
Is_a(Fluid, Liquid)
Is_a(Fluid, Liquid)
?
yes.
© 2013 IBM Corporation
Different Types of Evidence: Keyword Evidence
In May 1898 Portugal celebrated
the 400th anniversary of this
explorer’s arrival in India.
In May, Gary arrived in
India after he celebrated his
anniversary in Portugal.
arrived in
celebrated
In May
1898
Keyword Matching
400th
anniversary
Evidence suggests
“Gary” is the answer
BUT the system must
learn that keyword
matching may be
weak relative to other
types of evidence
Keyword Matching
Portugal
celebrated
In May
Keyword Matching
anniversary
Keyword Matching
in Portugal
arrival in
India
explorer
Keyword Matching
India
Gary
© 2013 IBM Corporation
Different Types of Evidence: Deeper Evidence
In May 1898 Portugal celebrated
the 400th anniversary of this
explorer’s arrival in India.
On27th
27thMay
May1498,
1498,Vasco
Vascoda
daGama
Gama
On
Onlanded
27th May
1498,
Vasco
da
Gama
th of
Kappad
Beach Vasco da
Onlanded
the 27inin
MayBeach
1498,
Kappad
landed in Kappad Beach
Gama landed in Kappad Beach
Search Far and Wide
Explore many hypotheses
celebrated
Find Judge Evidence
landed in
Portugal
May 1898
Stronger
evidence can
be much
harder to find
and score.
400th anniversary
arrival
in
Many inference algorithms
Temporal
Reasoning
Statistical
Paraphrasing
27th May 1498
Date
Math
Paraphrases
India
GeoSpatial
Reasoning
Kappad Beach
GeoKB
Vasco da Gama
explorer
The evidence is still not 100% certain.
© 2013 IBM Corporation
With Precision, Accurate Confidence and Speed, the rest was History
© 2013 IBM Corporation
Watson is ushering in a new era of computing . . .
System
Intelligence
Cognitive
Programmatic
Tabulation
Punch cards
Time card readers
1900
Search
Deterministic
Enterprise data
Machine language
Simple outputs
1950
Discovery
Probabilistic
Big Data
Natural language
Intelligent options
1
2011
. . .enabling new opportunities and outcomes
11
© 2013 International Business Machines Corporation
Watson takes on medical applications…
Page 12
© 2013 IBM Corporation
Started on American College of Physician’s Doctor’s Dilemma questions
such as: Diagnosis for fever, adenopathy, and arthralgia following drug exposure
Correct Response : Lemierre’s Syndrome
100%
Accuracy
80%
60%
40%
20%
0%
Jeopardy Watson
Medical Watson
3+ years of progress
© 2013 IBM Corporation
Teaching Watson Medical Concepts
Diseases
Medications
Symptoms
Modifiers
Watson Uses US National Library of Medicine’s UMLS (Unified Medical
Language System) for Medical Vocabulary
© 2013 IBM Corporation
Answering some questions requires finding the missing link…
© 2013 IBM Corporation
Missing Links
Buttons
Category: Common Bonds
Shirts,
TV remote controls, Telephones
Mt
Everest
Edmund
Hillary
He was first
On hearing of the discovery of George Mallory's body, he told reporters he still thinks he was first.
© 2013 IBM Corporation
Consider this “missing-link” USMLE question…
A 63-year-old patient is sent to the neurologist with a clinical picture of
resting tremor that began 2 years ago. At first it was only on the left hand
but now it compromises the whole arm. At physical exam, the patient has
an unexpressive face and difficulty in walking, and a continuous movement
of the tip of the first digit over the tip of the second digit of the left hand is
seen at rest.
What part of his nervous system is most likely affected?
Choices:
A. Cerebellum
B. Pons
C. Substantia Nigra
D. Caudate Nucleus
© 2013 IBM Corporation
Multi-Step
Ineferencing
© 2013 IBM Corporation
A More Detailed View of the Analysis
© 2013 IBM Corporation
Thank You
© 2013 IBM Corporation