Natural Language Processing
Giuseppe Attardi
Introduction to Probability, Language Modeling
IP notice: some slides from: Dan Jurafsky, Jim Martin, Sandiway Fong, Dan Klein
Outline

Probability
 Basic probability
 Conditional probability

Language Modeling (N-grams)




N-gram Intro
The Chain Rule
The Shannon Visualization Method
Evaluation:
• Perplexity
 Smoothing:
• Laplace (Add-1)
• Add-prior
1. Introduction to Probability

Experiment (trial)
 Repeatable procedure with well-defined possible
outcomes

Sample Space (S)
• the set of all possible outcomes
• finite or infinite
 Example
• coin toss experiment
• possible outcomes: S = {heads, tails}
 Example
• die toss experiment
• possible outcomes: S = {1,2,3,4,5,6}
Slides from Sandiway Fong
Introduction to Probability

Definition of sample space depends on what we
are asking
 Sample Space (S): the set of all possible outcomes
 Example
• die toss experiment for whether the number is even or odd
• possible outcomes: {even,odd}
• not {1,2,3,4,5,6}
More definitions

Events
 an event is any subset of outcomes from the sample space

Example
 die toss experiment
 let A represent the event such that the outcome of the die toss
experiment is divisible by 3
 A = {3,6}
 A is a subset of the sample space S= {1,2,3,4,5,6}

Example
 Draw a card from a deck
• suppose sample space S = {heart,spade,club,diamond} (four suits)
 let A represent the event of drawing a heart
 let B represent the event of drawing a red card
 A = {heart}
 B = {heart,diamond}
Introduction to Probability

Some definitions
 Counting
• suppose operation oi can be performed in ni ways,
then
• a sequence of k operations o1o2...ok
• can be performed in n1  n2  ...  nk ways
 Example
• die toss experiment, 6 possible outcomes
• two dice are thrown at the same time
• number of sample points in sample space = 6  6 =
36
Definition of Probability





The probability law assigns to an event a
nonnegative number
Called P(A)
Also called the probability A
That encodes our knowledge or belief about the
collective likelihood of all the elements of A
Probability law must satisfy certain properties
Probability Axioms

Nonnegativity
 P(A) >= 0, for every event A

Additivity
 If A and B are two disjoint events, then the probability
of their union satisfies:
 P(A U B) = P(A) + P(B)

Normalization
 The probability of the entire sample space S is equal to
1, i.e. P(S) = 1.
An example





An experiment involving a single coin toss
There are two possible outcomes, H and T
Sample space S is {H,T}
If coin is fair, should assign equal probabilities to 2
outcomes
Since they have to sum to 1
P({H}) = 0.5
P({T}) = 0.5
P({H,T}) = P({H}) + P({T}) = 1.0
Another example



Experiment involving 3 coin tosses
Outcome is a 3-long string of H or T
S ={HHH,HHT,HTH,HTT,THH,THT,TTH,TTT}
Assume each outcome is equiprobable
 “Uniform distribution”

What is probability of the event that exactly 2 heads
occur?
A = {HHT,HTH,THH}
P(A) = P({HHT})+P({HTH})+P({THH})
= 1/8 + 1/8 + 1/8
=3/8
Probability definitions

In summary:
P( E ) 

number of outcomes correspond ing to event E
total number of outcomes
Probability of drawing a spade from 52 wellshuffled playing cards:
13 1
  0.25
52 4
Probabilities of two events
If two events A and B are independent
 Then

 P(A and B) = P(A) · P(B)

If flip a fair coin twice
 What is the probability that they are both heads?

If draw a card from a deck, then put it back, draw
a card from the deck again
 What is the probability that both drawn cards are
hearts?
How about non-uniform probabilities? An example

A biased coin,
 twice as likely to come up tails as heads,
 is tossed twice
What is the probability that at least one head
occurs?
 Sample space = {hh, ht, th, tt} (h = heads, t = tails)
 Sample points/probability for the event:

 ht 1/3 x 2/3 = 2/9
 th 2/3 x 1/3 = 2/9

hh 1/3 x 1/3= 1/9
tt 2/3 x 2/3 = 4/9
Answer: 5/9 = 0.56 (sum of weights in red)
Moving toward language

What’s the probability of drawing a 2 from a deck
of 52 cards with four 2s?
4
1
P(drawing a two) 
  .077
52 13

What’s the probability of a random word (from a
random dictionary page) being a verb?

# of ways to get a verb
P(drawing a verb) 
all words
Probability and part of speech tags

What’s the probability of a random word (from a
random dictionary page) being a verb?
P(drawing a verb) 
# of ways to get a verb
all words
How to compute each of these
 All words = just count all the words in the dictionary

 # of ways to get a verb: number of words which are verbs!
 If a dictionary has 50,000 entries, and 10,000 are verbs….
P(V) is 10000/50000 = 1/5 = .20

Conditional Probability

A way to reason about the outcome of an
experiment based on partial information
 In a word guessing game the first letter for the word is
a “t”. What is the likelihood that the second letter is
an “h”?
 How likely is it that a person has a disease given that a
medical test was negative?
 A spot shows up on a radar screen. How likely is it that
it corresponds to an aircraft?
More precisely
Given an experiment, a corresponding sample
space S, and a probability law
 Suppose we know that the outcome is within
some given event B
 We want to quantify the likelihood that the
outcome also belongs to some other given event
A
 We need a new probability law that gives us the
conditional probability of A given B
P(A|B)

An intuition







A is “it’s raining now”
P(A) in Tuscany is .01
B is “it was raining ten minutes ago”
P(A|B) means “what is the probability of it raining now if it
was raining 10 minutes ago”
P(A|B) is probably way higher than P(A)
Perhaps P(A|B) is .10
Intuition: The knowledge about B should change our
estimate of the probability of A.
Conditional probability
One of the following 30 items is chosen at
random
 What is P(X), the probability that it is an X?
 What is P(X|red), the probability that it is an X
given that it is red?

Conditional Probability



let A and B be events
P(B|A) = the probability of event B occurring given event
A occurs
definition: P(B|A) = P(A  B) / P(A)
S
Conditional probability
P(A|B) = P(A  B)/P(B)
or
P( A, B)
P( A | B) 
P( B)
Note: P(A,B)=P(A|B) · P(B)
Also: P(A,B) = P(B,A)
A
A,B B
Independence

What is P(A, B) if A and B are independent?
P(A,B) = P(A) · P(B) iff A, B independent.
P(heads, tails) = P(heads) · P(tails) = .5 · .5 = .25
Note: P(A|B) = P(A) iff A, B independent
Also: P(B|A) = P(B) iff A, B independent
Summary
Probability
 Conditional Probability
 Independence

Language Modeling

We want to compute
P(w1,w2,w3,w4,w5…wn) = P(W)
= the probability of a sequence

Alternatively we want to compute
P(w5|w1,w2,w3,w4)
= the probability of a word given some previous words



The model that computes
P(W) or
P(wn|w1,w2…wn-1)
is called the language model.
A better term for this would be “The Grammar”
But “Language model” or LM is standard
Computing P(W)

How to compute this joint probability:
P(“the” , ”other” , ”day” , ”I” , ”was” , ”walking” ,
”along”, ”and”,”saw”,”a”,”lizard”)

Intuition: let’s rely on the Chain Rule of Probability
The Chain Rule
Recall the definition of conditional probabilities
P( A  B)
P( A | B) 
P( B)
 Rewriting:

P( A  B)  P( A | B) P( B)
More generally
P(A,B,C,D) = P(A)P(B|A)P(C|A,B)P(D|A,B,C)
 In general

P(x1,x2,x3,…xn) = P(x1)P(x2|x1)P(x3|x1,x2)…P(xn|x1…xn-1)
The Chain Rule applied to joint probability of words in
sentence
P(“the big red dog was”) =
P(the) * P(big|the) * P(red|the big) *
P(dog|the big red) * P(was|the big red dog)
Very easy estimate:

How to estimate?
P(the | its water is so transparent that)
P(the | its water is so transparent that) =
C(its water is so transparent that the)
____________________________________________________________________________________________
C(its water is so transparent that)
Unfortunately

There are a lot of possible sentences

We’ll never be able to get enough data to compute
the statistics for those long prefixes
P(lizard|the,other,day,I,was,walking,along,and,saw,a)
or
P(the|its water is so transparent that)
Markov Assumption

Make the simplifying assumption
P(lizard|the,other,day,I,was,walking,along,and,saw,a) =
P(lizard|a)

or maybe
P(lizard|the,other,day,I,was,walking,along,and,saw,a) =
P(lizard|saw,a)
Markov Assumption
So for each component in the product replace with
the approximation (assuming a prefix of N)
n1
1
P(wn | w
)  P(wn | w
n1
nN 1
Bigram version
n1
1
P(w n | w
)  P(w n | w n1 )
)
Estimating bigram probabilities

The Maximum Likelihood Estimate
count(wi1,wi )
P(wi | wi1) 
count(wi1 )
c(wi1,wi )
P(wi | wi1) 
c(wi1)
An example




<s> I am Sam </s>
<s> Sam I am </s>
<s> I do not like green eggs and ham </s>
This is the Maximum Likelihood Estimate, because it is the one which
maximizes P(Training set|Model)
Maximum Likelihood Estimates

The maximum likelihood estimate of some parameter of a
model M from a training set T
 Is the estimate
 that maximizes the likelihood of the training set T given the model M



Suppose the word Chinese occurs 400 times in a corpus of a
million words (Brown corpus)
What is the probability that a random word from some other
text will be “Chinese”
MLE estimate is 400/1000000 = .004
 This may be a bad estimate for some other corpus

But it is the estimate that makes it most likely that “Chinese”
will occur 400 times in a million word corpus.
More examples: Berkeley Restaurant Project
sentences






can you tell me about any good cantonese
restaurants close by
mid priced thai food is what i’m looking for
tell me about chez panisse
can you give me a listing of the kinds of food that
are available
i’m looking for a good place to eat breakfast
when is caffe venezia open during the day
Raw bigram counts

Out of 9222 sentences
Raw bigram probabilities

Normalize by unigrams:

Result:
Bigram estimates of sentence probabilities
P(<s> I want english food </s>) =
P(i|<s>) x
P(want|I) x
P(english|want) x
P(food|english) x
P(</s>|food)
=.000031
What kinds of knowledge?
P(english|want) = .0011
P(chinese|want) = .0065
P(to|want) = .66
P(eat | to) = .28
P(food | to) = 0
P(want | spend) = 0
P (i | <s>) = .25
The Shannon Visualization Method
Generate random sentences:
 Choose a random bigram <s>, w according to its
probability
 Now choose a random bigram (w, x) according to its
probability
 And so on until we choose </s>
 Then string the words together

<s> I
I want
want to
to eat
eat Chinese
Chinese food
food </s>
Approximating Shakespeare

Shakespeare as corpus
N=884,647 tokens, V=29,066
 Shakespeare produced 300,000 bigram
types out of V2= 844 million possible
bigrams: so, 99.96% of the possible
bigrams were never seen (have zero entries
in the table)
 Quadrigrams worse: What's coming out
looks like Shakespeare because it is
Shakespeare

The wall street journal is not shakespeare
(no offense)

Lesson 1: the perils of overfitting

N-grams only work well for word prediction if the
test corpus looks like the training corpus
 In real life, it often doesn’t
 We need to train robust models, adapt to test
set, etc
Lesson 2: zeros or not?

Zipf’s Law:





A small number of events occur with high frequency
A large number of events occur with low frequency
You can quickly collect statistics on the high frequency events
You might have to wait an arbitrarily long time to get valid
statistics on low frequency events
Result:
 Our estimates are sparse! no counts at all for the vast bulk of
things we want to estimate!
 Some of the zeroes in the table are really zeros But others are
simply low frequency events you haven't seen yet. After all,
ANYTHING CAN HAPPEN!
 How to address?

Answer:
 Estimate the likelihood of unseen N-grams!
Slide adapted from Bonnie Dorr and Julia Hirschberg
Zipf's law

decompressor
are needed t o see t his pict ure.
f  1/r
(f proportional to 1/r)
there is a constant k such that
fr=k
Zipf's Law for the Brown Corpus
Zipf law: interpretation

Principle of least effort: both the speaker and the
hearer in communication try to minimize effort:
 Speakers tend to use a small vocabulary of common
(shorter) words
 Hearers prefer a large vocabulary of rarer less ambiguous
words
 Zipf's law is the result of this compromise

Other laws …
 Number of meanings m of a word obeys the law: m  1/f
 Inverse relationship between frequency and length
Smoothing is like Robin Hood:
Steal from the rich and give to the poor (in probability mass)
Slide from Dan Klein
Laplace smoothing
Also called add-one smoothing
 Just add one to all the counts!
 Very simple
 MLE estimate:


Laplace estimate:

Reconstructed counts:
Laplace smoothed bigram counts
Laplace-smoothed bigrams
Reconstituted counts
Note big change to counts



C(count to) went from 608 to 238!
P(to|want) from .66 to .26!
Discount d= c*/c
 d for “chinese food” =.10!!! A 10x reduction
 So in general, Laplace is a blunt instrument


But Laplace smoothing not used for N-grams, as we have
much better methods
Despite its flaws Laplace (add-k) is however still used to
smooth other probabilistic models in NLP, especially
 For pilot studies
 in domains where the number of zeros isn’t so huge.
Add-k

Add a small fraction instead of 1
Even better: Bayesian unigram prior smoothing
for bigrams

Maximum Likelihood Estimation
C(w1,w2 )
P(w2 | w1) 
C(w1)

C(w1,w2 ) 1
PLaplace(w2 | w1) 
C(w1 )  vocab



Laplace Smoothing
Bayesian Prior Smoothing
C(w1,w2 )  P(w2 )
PPrior(w2 | w1) 
C(w1 ) 1
Practical Issues

We do everything in log space
 Avoid underflow
 (also adding is faster than multiplying)
Language Modeling Toolkits

SRILM
http://www.speech.sri.com/projects/srilm/
 IRSTLM
 Ken LM
Google N-Gram Release
Google N-Gram Release










serve
serve
serve
serve
serve
serve
serve
serve
serve
serve
as
as
as
as
as
as
as
as
as
as
the
the
the
the
the
the
the
the
the
the
incoming 92
incubator 99
independent 794
index 223
indication 72
indicator 120
indicators 45
indispensable 111
indispensible 40
individual 234
http://googleresearch.blogspot.com/2006/08/all-our-n-gram-are-belong-to-you.html
Evaluation







We train parameters of our model on a training set.
How do we evaluate how well our model works?
We look at the models performance on some new data
This is what happens in the real world; we want to know
how our model performs on data we haven’t seen
So a test set. A dataset which is different than our training
set
Then we need an evaluation metric to tell us how well our
model is doing on the test set.
One such metric is perplexity (to be introduced below)
Evaluating N-gram models
 Best evaluation for an N-gram
 Put model A in a task (language
identification, speech recognizer,
machine translation system)
 Run the task, get an accuracy for A (how
many lgs identified corrrectly, or Word
Error Rate, or etc)
 Put model B in task, get accuracy for B
 Compare accuracy for A and B
 Extrinsic evaluation
Difficulty of extrinsic (in-vivo) evaluation of Ngram models

Extrinsic evaluation
 This is really time-consuming
 Can take days to run an experiment

So
 As a temporary solution, in order to run experiments
 To evaluate N-grams we often use an intrinsic
evaluation, an approximation called perplexity
 But perplexity is a poor approximation unless the test
data looks just like the training data
 So is generally only useful in pilot experiments
(generally is not sufficient to publish)
 But is helpful to think about.
Perplexity

Perplexity is the probability of the test set
(assigned by the language model), normalized
by the number of words:

Chain rule:

For bigrams:
Minimizing perplexity is the same as maximizing probability
The best language model is one that best predicts an
unseen test set
A totally different perplexity Intuition



How hard is the task of recognizing digits
‘0,1,2,3,4,5,6,7,8,9,oh’: easy, perplexity 11 (or if we ignore
‘oh’, perplexity 10)
How hard is recognizing (30,000) names at Microsoft.
Hard: perplexity = 30,000
If a system has to recognize






Operator (1 in 4)
Sales (1 in 4)
Technical Support (1 in 4)
30,000 names (1 in 120,000 each)
Perplexity is 54
Perplexity is weighted equivalent branching factor
Slide from Josh Goodman
Perplexity as branching factor
Lower perplexity = better model

Training 38 million words, test 1.5 million words,
WSJ
Unknown words: Open versus closed vocabulary
tasks

If we know all the words in advance
 Vocabulary V is fixed
 Closed vocabulary task

Often we don’t know this
 Out Of Vocabulary = OOV words
 Open vocabulary task

Instead: create an unknown word token <UNK>
 Training of <UNK> probabilities
• Create a fixed lexicon L of size V
• At text normalization phase, any training word not in L changed to
<UNK>
• Now we train its probabilities like a normal word
 At decoding time
• If text input: Use UNK probabilities for any word not in training
Advanced LM stuff

Current best smoothing algorithm
 Kneser-Ney smoothing

Other stuff




Interpolation
Backoff
Variable-length n-grams
Class-based n-grams
• Clustering
• Hand-built classes





Cache LMs
Topic-based LMs
Sentence mixture models
Skipping LMs
Parser-based LMs
Backoff and Interpolation
Another really useful source of knowledge
 If we are estimating:

 Trigram P(z|xy)
 but C(xyz) is zero

Use info from:
 Bigram P(z|y)

Or even:
 Unigram P(z)

How to combine the trigram/bigram/unigram
info?
Backoff versus interpolation
Backoff: use trigram if you have it,
otherwise bigram, otherwise unigram
 Interpolation: mix all three

Interpolation

Simple interpolation

Lambdas conditional on context:
How to set the lambdas?

Use a held-out corpus
Training Data

Held-Out Data
Test
Data
Choose lambdas which maximize the probability
of data
 i.e. fix the N-gram probabilities
 Then search for lambda values
 That when plugged into previous equation
 Give largest probability for held-out set
 Can use EM to do this search
Intuition of backoff+discounting

How much probability to assign to all the zero
trigrams?
 Use GT or other discounting algorithm to tell us

How to divide that probability mass among
different contexts?
 Use the N-1 gram estimates to tell us

What do we do for the unigram words not seen in
training?
 Out Of Vocabulary = OOV words
ARPA format
Summary

Probability
 Basic probability
 Conditional probability

Language Modeling (N-grams)




N-gram Intro
The Chain Rule
The Shannon Visualization Method
Evaluation:
• Perplexity
 Smoothing:
• Laplace (Add-1)
• Add-k
• Add-prior