Fitting Generalized
Linear Fixed Effects
Models in R
David Reitter, Informatics, University of Edinburgh
[email protected]
What linear models can do for you
• Factor analysis (cf. ANOVA)
• Regression (continuous response)
• Continuous predictors (covariates)
• Unbalanced designs (observational studies!)
• Non-normal response variables (GLM)
• Repeated measures / time series
(random effects) (GLMM)
The Titanic Dataset
• Survival data with factors
Sex, Age (Child/Adult), Cabin-Class, Crew
• Provided with R, but we’ll use a more
complete dataset
Exploratory Data Analysis
# Dead
Prob(Dead|Age)
A first linear model: Anova
ANOVA and Linear Models
assume
• ANOVAs
Normality of response
•
•
•
•
Linearity
Homogeneity of variances
IID sampling
• ANOVA as special case of the general LM:
•
•
•
y = β0 + β1 x1
β0: intercept (baseline)
β1: between-group variation
... compared to the within-group error
•
Non-balanced data
• Experimental Designs are often balanced
• controls error across conditions
• continuous variables binned
•
Information about nature of effect missing! E.g. decay
of preactivation in priming is log-linear.
• ANOVA compatible
• Naturalistic data is usually unbalanced
Factors and Predictors
• Age is a continuous variable
• binned for ANOVA: I(Age>16)
• do not discretize continuous variables!
• information loss
• bias through (arbitrary?) bins (thresholds)
• LMs can deal with continuous variables
...however
• Are the ANOVA / LM assumptions met?
• Observations are not IID
• spatial correlation (Boat)
• Response normally distributed?
• compare visually with qqnorm, qqplot
• apply Kolmogorov-Smirnov (ks.test) and/
or Shapiro-Wilks (shapiro.test)
• “Binning” may be needed:
Transformations
• Generalized Linear Models (GLM) perform a
transformation of the response via a link
function
• No need for a manual transform!
• Link functions for glm include
• binomial (logit link): dichotomous response
• poisson: count data
Titanic GLM
... a simple model
... produces a huge model with mostly
insignificant interactions
... a model with the interaction
Does Class:Age help?
Age matters when you’re a stewart
Actual Prediction
• A 49 year-old second-class passenger - how
likely did he survive?
y!
= log y − log(1 − y)
y
)
log(
1−y
y!
=
y
=
!
ey
1 + ey!
Random Effects
• Most designs, both experimental and
observational, involve some dependence
between samples:
• time series data
• repeated measures
• spatial correlation of samples
• Mixed Effects Models include random effects
and allow grouping of interdependent
samples.
GLMM
• Generalized Mixed-Effects Model
• Specification:
• fixed effects formula
formula = target ~ log(time) * primed
• random effects formula:
random = ~ 1 | speaker
• nested F1/F2 effects:
random = ~ 1 | subject/item
•
•
Library: nlme load with library(nlme)
Library: MASS
load with library(nlme)
Functions: lme, nlme
Function: glmmPQL
Contrasts
• To estimate effect sizes under different
combinations of factors, use “within” formula
notation: Survived ~ Age/Class
• “regresses out” Age before estimating
effect of Class
• Use
intervals with glmmPQL models to get
confidence intervals for bar charts
Reporting results
Utts CP MAPT
Time PP SWBD
Time CP SWBD
Time PP MAPT
0.012
Utts PP MAPT
p(prime=target|target,distance)
Utts PP SWBD
Utts CP SWBD
0.014
0.016
lme / lmer models need Markov-Chain Montecarlo Sampling to estimate p and
confidence intervals
0.010
•
Switchboard
PP
Switchboard
CP
Map Task
PP
els on separately sampled datasets.
0.008
coefficients (βi ) Std. Error
Map
Intercept
-3.778 0.025 *** Task CP
ln(D ISTT ime )
-0.057 0.015 **
0.00 - 0.05 - 0.10 - 0.15 - 0.20 - 0.25 - 0.30 - 0.35
4
6
10
12
14
ln(F REQ) 2
0.5388 0.190
***
between 0.010
prime and***
target (seconds)
IST) : ln(F REQdistance:
)) Temporal Distance
0.083
Figure 2: Priming effect sizes ( ln(D IST)) under ln(D
different
ROLE and S OURCE situations. Prime-target distance
by numln(D IST
) : (ROLE = CP )
-0.031 0.012 *
ber of utterances
(Exp.
95%
CI. Ef-= MFigure
3: Decaying repetition
probability
estimates
ln(D
IST3)
) :and
(Rseconds
OLE = (Exp.
P P )5).
: (S
OURCE
apT ask)
-0.050
0.014
** dependfects estimated from separately fitted nested regression moding
on
the
increasing
distance
between
prime
and
target,
conln(D IST) : (ROLE = CP ) : (S OURCE = M apT ask)
-0.137 0.018 ***
Time CP MAPT
trasting different ROLE and S OURCE situations. (Exp. 5)
0.014
p(prime=target|target,distance)
between ROLE and D IST (p = 0.92).
tual linguistic
To someexperimental
extent, corpora
This activity.
finding confirms
resultscan
by help
Bock make
and
that distinction.
Griffin (2000) and Branigan et al. (1999), who find syntacThe differences
between
and task-oriented
tic priming over
longer conversational
distances, even though
the effect deeffect of R
OLE(Experiment
on bias may be3)related
to speakeron
dialoguecays.
that(The
we pointed
out
are founded
idiosyncracies,
i.e. more
chance repetition
within
speakers.)
the correlation
of distance
between
prime and
target
and repTo determine whether there is a significant influence of dietition likelihood.
This correlation is likely to be sensitive to
alogue type on priming, comparing the effects we have seen
the scalein of
D ISTANCE
. 2,Aswean
alternative,
we described
can use in
the
experiments
1 and
built
a further model,
delay between
the left boundaries of the priming and target
the next section.
phrases as the relevant predictor.
Exp. 3:
Comparing
corpora
The models discussed
measure
the distance
between prime
With
their Interactive
Model, (Pickering
Garand target
in utterances.
In Alignment
this experiment,
we fittedand
a second
rod,
2004)
argue
that
the
situation-model
alignment
of
speakregression model, estimating decay over time.
ers is due to lower-level priming effects. In task-oriented diaTo compare
the two (obviously interrelated) predictors
logue, and in the task carried out by participants in Map Task,
D ISTT ime
and D
ISTto
, we
estimated
two simple
linear
reU tts
speakers
need
align
in order
to successfully
complete
their
gressiontasks.
models,
for time,
thepredict
otherthat
onesyntactic
for number
Thus,one
the theory
would
primingof
between
speakers (CP)
is greater
in task-oriented
dialogue.
utterances
as predictor.
Such
regression
models can,
as opWe
test
this produce
hypothesisabymeaningful
fitting a model
the joint dataIn
posed to GLMMs,
R2of measure.
set with we
S OURCE
as a binary
factor, indicating whether
a repthese models,
include
the maximum-likelihood
estimate
etition stems from Map Task (task-oriented) or Switchboard
of the number
of chance repetitions,
which
calculated
from
(not task-oriented).
From Map Task,
only is
dialogues
in which
the overall
frequency
of each
rule
(thisincluded.
is in addition
interlocutors
could
not seesyntactic
one another
where
to the covariates discussed before). The response variable
0.012
As seen in the previous experiments, it can make a difference
whether a speaker primes themself or is primed by their in-
0.010
Results
PP SWBD
Interestingly,
gapPriming
between CPover
and PPtime
priming
to verify the hypothesis using time as the relevant decay cor- terlocutor. Utts
Exp.the5:
Once again we find that repetition is more likely the shorter
is substantially
affected by the choice of corpus (last two inrelate. We
do so in Experiments 4 and 5.
CP SWBD
While Utts
timeand utterance-based models fit their respective
the distance between prime and target utterances is. Unlike in
teractions in Table 1). In both corpora, we find a positive PP
data similarly
well, in
time
is Task,
a theoretically
attractive measure
Switchboard, interlocutors repeat each other’s syntactic strucUtts PP However,
MAPT
effect.
Map
CP and PP priming
Exp. 4:tures
Pre-activation
decay:
over
orrepeat
with priming
more readily and more
similarly
to thetime,
way they
of distance,
in particular
because
utterance
cannot
be distinguished
(cf. Experiment
2),the
while
in Switch-is difficult to
Utts CP MAPT
their own structures.
board,
there
is
little
CP
priming
(cf.
Experiment
1).
Figdelineate in the context of speech.
each utterance?
PP bars)
SWBD provides the resulting priming strength
The model showed a reliable effect of ln(D IST)
ure 2 The
(firstTime
four
methodology
of this experiment is as in Experiment 3,
(t =
− 71.2,experiments
p < 0.005) . have shown that repetition estimates Time
While the
previous
theSWBD
factorial combinations of ROLE and
CP
exceptfor
that
Dfour
IST ime is the distance predictor, instead of the
had asoon
reliable
constant
effect on repetition
rates
probabilityROLE
decays
after
any stimulus,
it is unclear
S OURCE at increasing T
distance. Also, priming is stronger for
ISTUTime
used
PP
MAPTpreviously.
− 11.0, p < diminishes
0.0001), but there
no or
interaction
ttsrules.
whether(tthe=pre-activation
with was
time,
with ac- lessDfrequent
0.008
with the set of factors and predictors.
0.016
Table 1: The regression model for the joint data set of Switchboard and Map Task (Exp. 5). This is the minimal model without
Again,
a GLMM*was
to correlate
condition
insignificant
covariates.
p <built
0.01,
** p < priming
0.005, ***
p < 0.0001.
Results
Switchboard
PP
Switchboard
CP
Map Task
PP
Map Task
For Switchboard,
Time CP MAPT the model estimates a higher coeffiCP
Results
cient
for ln(D IST), suggesting that there was faster decayThe
in Map
Task (Baseline
effect
of -LN
(D- 0.20
ISTpriming
):- 0.25
βlnDist
=- 0.35 found in
interaction
of corpus
type
decay
0.00
0.15and
- 0.05
- 0.10
- 0.30
2
4
6
8
10
12
14
−0.092,
p < 0.0001;
βlnDist:CP
= 0.083,isp stronger
< 0.0001;in task-oriented
Experiment
3 holds.
CP priming
βlnDist:M
=
−0.044,
p =sizes
0.05;( ln(D IST)) under different
distance: Temporal Distance between prime and target (seconds)
apT ask
Figure
2:Table
Priming
effect
dialogue.
1 contains
the estimated model.
βlnDist:CP :M apT ask = −0.140, p < 0.0001).
R
OLE
and
S
OURCE
situations.
Prime-target
distance
by
numThe model
based on temporal
Frequency
is negatively
correlateddistance
with makes
decay essentially
ber of utterances
(Exp. 3) and
(Exp.has
5). 95%
CI. EfFigure
comparable
Theseconds
S OURCE
an interaction
ef- 3: Decaying repetition probability estimates depend(βlnDist:lnF
0.049, p < 0.0001).
req =predictions.
estimated
from
separately
fittedIST
nested
regression
mod-priming
ing on the increasing distance between prime and target, confectfects
onthe
themarked
priming
decaybetween
ln(D
),andboth
for CP
Finding
difference
CP
PP primelsalso
on separately
trasting different ROLE and S OURCE situations. (Exp. 5)
ing,(βand
a clear
PPsampled
priming
effect
in spontaneous
con= datasets.
−
0.137,
t = − 7.6,
p < 0.0001)
lnDist:CP
:M apT
ask
versation,
Dubey et
(2005), who do not find reliand forextends
PP priming
(βal.
lnDist:P P :M apT ask = − 0.050,
able evidence of adaptation within speakers in Switchboard
t = Again,
−syntactic
3.7,
p rules
< 0.0005).
2,priming
3 provide
the predica GLMM
was
built toFigures
correlate
condition
Results
for selected
in coordinate
structures.
with
set four
of factors
and predictors.
tions
forthe
the
combinations
of ROLEthat
and
S OURCE. As seen in the previous experiments, it can make a difference
Thus,
the
data
is consistent
with the hypothesis
semanwhether a speaker primes themself or is primed by their intic alignment
Resultsin dialogue is based on lower-level (syntactic)
terlocutor. Interestingly, the gap between CP and PP priming
priming. However, when comparing
data across corpora, we
Discussion
again we
find that
is more
likely
is substantially affected by the choice of corpus (last two inneed toOnce
be careful
to ensure
thatrepetition
differences
in genre
andthe
an-shorter
Both
corpora
of
spoken
dialogue
we
investigated
showed
an
theare
distance
utterances
is. The
Unlike
in
teractions in Table 1). In both corpora, we find a positive PP
notation
not thebetween
primaryprime
causeand
of target
the effect
at hand.
effect
offordistance
between
prime
and
target
in syntactic
Switchboard,
interlocutors
repeat
each
other’s
struc- repepriming effect. However, in Map Task, CP and PP priming
coefficient
pre-activation
decay
is sensitive
to syntactic
utterance
tures
more
readilyanand
more
the utterances
way they
repeat effect
tition,
thus
providing
evidence
for ato
structural
priming
cannot be distinguished (cf. Experiment 2), while in Switchlength,
which
becomes
issue
forsimilarly
instance
when
own structures.
board, there is little CP priming (cf. Experiment 1). Figarefor
nottheir
consistently
marked orrules.
if decay
time and
arbitrary
syntactic
Inoccurs
both over
corpora,
we also found
modelIndeed,
showed
a utterances
reliable effect
of ln(D IST)
ure 2 (first four bars) provides the resulting priming strength
notreliable
with The
utterances.
most
in Switchboard
effects
of
production-production
(PP) priming
(t = dialogue
− 71.2,turns,
p <both
0.005)
. the genre, they are usuestimates for the four factorial combinations of ROLE and
are(self-priming)
actually
and
given
and comprehension-production-priming.
But
ROLE
constant
effect iton
repetition
S OURCE at increasing distance. Also, priming is stronger for
ally longer
than had
thosea inreliable
Map Task.
Therefore,
makes
sense rates
only
a corpusbutofthere
task-oriented
dialogue less
did frequent rules.
(t in
= the
− Map
11.0, Task,
p < 0.0001),
was no interaction
between ROLE and D IST (p = 0.92).
Model Checking
• Use plot(model) to show diagnostic graphs
•
shows four diagnostics, incl. Q-Q and residuals vs. fitted
• Interpretation: see Crawley 2005
ideal case!
Count data, with linear model
(no transformation)
(Model does not
show effects!)
i <- runif(60); j <- runif(50); p <- rbind(data.frame(r=as.integer(rpois(60,40)*(i*3)),
t='a', i=i), data.frame(r=as.integer(rpois(50,45)*(j*2.5)), t='b', i=j))
Count data, with Poisson GLM
(i.e. log(1+y) transform)
(Model shows two main
effects and the interaction)
i <- runif(60); j <- runif(50); p <- rbind(data.frame(r=as.integer(rpois(60,40)*(i*3)),
t='a', i=i), data.frame(r=as.integer(rpois(50,45)*(j*2.5)), t='b', i=j))
To read
•
•
H. Baayen: Practical Data Analysis for the Language
Sciences with R. To Appear.
S. Vasishth: The foundations of statistics: A simulationbased approach. In Prep.
Download: http://www.ling.uni-potsdam.de/~vasishth/SFLS.html
•
•
M. J. Crawley: Statistics. An Introduction using R, Wiley 2005
W.N.Venables & B.D. Ripley: Modern Applied Statistics with S.
Springer 2002
GLMs in R: Don’t panic!
Picture: http://www.solarnavigator.net