ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Multivariate Responses
In the general mean-variance specification
E (Yj | x) = f (xj , β) ,
var ( Yj | xj ) = σ 2 g (β, θ, xj )2 ,
we have assumed that the responses Y1 , Y2 , . . . , Yn are conditionally
independent, conditioning on x1 , x2 , . . . , xn .
In many situations, this assumption may fail:
clusters of observations, such as pups born to mother rats;
serial correlation in repeated measurements on each
experimental unit.
1 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Recall Example 1.7: Developmental toxicology studies
Developmental toxicology studies in rodents are used in testing and
regulation of potentially toxic substances that may pose danger to
developing fetuses.
A total of m pregnant rats are exposed to different doses of a toxic
agent, and each mother rat gives birth to ni pups.
The response Yi,j , i = 1, 2, . . . , ni is birthweight, and the objective is
to characterize the effect on birthweight of different doses of the
agent across the population of all exposed mothers and their pups.
2 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Recall Example 1.8: Pharmacokinetics of theophylline
Subject i receives an oral dose Di of theophylline.
Response Yi,j is the subject’s level of the drug at time ti,j after
administration, j = 1, 2, . . . , ni .
For a given subject, a pharmacokinetic model may explain the
time-variation in the response.
A broader objective is to understand pharmacokinetic behavior in the
entire population of subjects.
3 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Why worry?
If we ignore dependence:
parameter estimates are generally inefficient;
standard errors are generally wrong, hence inferences (confidence
intervals, hypothesis tests) do not have nominal properties
(coverage probability, size);
statistical framework may be inappropriate for scientific
objectives.
Inefficiency may not be important, invalidity is always important, but
relevance to the science is paramount.
4 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
We limit discussion to situations where groups of observations may
unambiguously be assumed to be independent:
m response vectors Yi , i = 1, 2, . . . , m;
ni observations on subject i


Yi,1
 Yi,2 


Yi =  ..  .
 . 
Yi,ni
5 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Covariates
Within-individual covariates:
describe conditions under which Yi,j was observed;
needed even if inference were restricted to individual i;
e.g., ti,j = time of j th observation on individual i.
Among-individual covariates:
same value for all observations on individual i;
e.g., treatment assigned to this individual, or individual
characteristics such as gender.
6 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Covariate notation
Within-individual covariate vector zi,j ;
Stacked:



zi = 

zi,1
zi,2
..
.





zi,ni
Among-individual covariate vector ai .
Combined:
xi =
7 / 37
zi
ai
.
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Sources of Dependence
Dependence simply means that
fi (Yi | xi ) 6=
ni
Y
fi,j (Yi,j | xi ) .
j=1
Very general, hence difficult to specify.
It is helpful to distinguish:
“individual-level” sources;
“population-level” sources.
8 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Individual-Level Sources of Dependence
For example, suppose we model repeated measurements of a
subject’s blood pressure using a within-individual linear regression:
Yi,j = β0,i + β1,i ti,j + ei,j ,
where ti,j is the time of the j th measurement on the i th subject.
The linear trend
β0,i + β1,i t
represents the mean response for that subject.
9 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
The subject’s actual blood pressure at time t at the time of testing is
β0,i + β1,i t + eP (t)
where eP (t) is random variation around that mean response, perhaps
a stationary stochastic process.
If measurement error eM,i,j is non-negligible, then
ei,j = eP (ti,j ) + eM,i,j .
10 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Then
var (Yi,j ) = var (eP,i,j ) + var (eM,i,j )
and, for j 0 6= j,
cov (Yi,j , Yi,j 0 ) = cov (eP,i,j , eP,i,j 0 ) .
We would need to specify a model for these variances and
covariances in order to make inferences about β0,i and β1,i .
11 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
The conceptual representation says that the response vector for a
single individual is intermittent observations on a stochastic process,
whose realizations fluctuate to some extent about a smooth inherent
trend, possibly subject to additional measurement error.
Here the frame of inference is the individual subject.
12 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Population-Level Sources of Dependence
If the subjects are themselves a random sample from some
population, then the “parameters”
β0,i
βi =
β1,i
associated with the i th subject are a random sample from the
corresponding population of parameter vectors.
We shall be interested in the mean and dispersion in this population,
which describe the average across subjects and the variation among
subjects.
Here the frame of inference is the population of subjects.
13 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Subject-Specific Modeling
Example: Theophylline concentration-time profiles
14 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Pharmacokinetics suggests the subject-specific model
β1,i ti,j
− β
−β
t
3,i
i,j
Di β3,i e 2,i − e
E (Yi,j | zi,j , β i ) =
.
(β2,i β3,i − β1,i )
where
zi,j contains Di = dose for i th subject, and ti,j = time of j th
measurement for i th subject (Di is the same for all
measurements, but is included in zi,j instead of ai for
convenience);
the vector


β1,i
β i =  β2,i 
β3,i
consists of parameters specific to subject i.
15 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
This may be written
E (Yi,j | zi,j , β i ) = f (zi,j , β i ) .
Because β i is associated with the randomly selected i th subject, it is
a random variable, and the model for that subject is conditional on
the value of β i .
We also need some assumptions about how β i varies from subject to
subject.
We might assume β i ∼ N(β, D), or equivalently
β i = β + bi ,
where bi ∼ N(0, D).
16 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
If the subjects were in two groups, e.g. smokers vs non-smokers, the
mean might depend on the group to which the subject belongs:
β i = β (0) + δi β (1) + bi
β (0)
= I δi I
+ bi
β (1)
where δi is an indicator variable for smokers.
More generally:
β i = Ai β + bi ,
where Ai is a subject-specific design matrix, which is a function of
the individual-level (or among-individual) covariate vector ai , and β
is the vector of all relevant parameters.
17 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
For instance, for the theophylline data,




β1,i
1 wi ci 0 0



β2,i
0 0 0 1 wi
βi =
=
β3,i
0 0 0 0 0



0 0



0 0


1 wi


βCl,0
βCl,w
βCl,c
βV,0
βV,w
βka,0
βka,w





 + bi .




Here wi = i th subject’s body weight and ci = i th subject’s creatinine
clearance rate (a measure of kidney function), both of which are
components of ai .
The 7 βs have pharmacokinetic interpretation.
18 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Summary of two-stage modeling
Stage 1: Individual model.
E (Yi,j | zi,j , β i ) = f (zi,j , β i ) .
Stage 2: Population model.
β i = Ai β + bi .
19 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Within-individual variation
Need to specify var (Yi | zi , β i ).
Following the earlier discussion,
var (Yi,j |zi,j , β i ) = var (eP,i,j |zi,j , β i ) + var (eM,i,j |zi,j , β i )
= σP2 + σ 2 g (β i , θ, zi,j )2 ,
where:
σP2 represents natural variation in the true concentration;
σ 2 g (β i , θ, zi,j )2 represents measurement error.
20 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
In the context of pharmacokinetics, it is often thought that the
measurement error due to the assay is the predominant source of
variation, whereas biological fluctuations about the trajectory are very
small by comparison.
This would lead to the familiar variance model
var(Yi,j |zi,j , β i ) = σ 2 g (β i , θ, zi,j )2 .
21 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Also need to specify correlations in var (Yi | zi , β i ).
We might assume that the biological variations have some correlation
matrix Γi (α, zi ), and that measurement errors are uncorrelated.
Then
var ( Yi | zi , β i ) = σP2 Γi (α, zi ) + σ 2 W (β i , θ, zi,j )−1 ,
where
W (β i , θ, zi,j )
(
= diag
22 / 37
1
1
1
,
2,
2,...,
g (β i , θ, zi,1 ) g (β i , θ, zi,2 )
g (β i , θ, zi,ni )2
Multivariate Responses
)
.
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
A general subject-specific model
Stage 1: Individual model
E (Yi | zi , β i ) = fi (zi , β i ) .
With β i = Ai β + bi , and ai the among-subject covariates on which
Ai depends, we may also write
E (Yi | zi , ai , bi ) = E ( Yi | xi , bi ) = fi (xi , β, bi ) .
For the variance:
var (Yi | zi , ai , bi ) = Ri (β i , γ, zi ) = Ri (β, γ, xi , bi ) .
23 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Stage 2: Population model
β i = Ai β + bi
where b1 , b2 , . . . , bm , called the random effects, are usually assumed
to be i.i.d., independent of xi , with
E (bi ) = 0,
var (bi ) = D.
24 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Remarks
Note that observations in the same data vector Yi share the same
random effect bi . Thus, the model takes into account naturally the
population-level phenomenon that observations on the same
individual tend to be “more alike” than observations from different
individuals.
This model is ideally suited to the scientific objectives stated in the
case of pharmacokinetics: β and D characterize mean and variation
in the population of parameters, which is of direct scientific interest.
Known as nonlinear mixed effects models – “mixed effects” to
recognize the presence of both fixed parameters (β and D) and
random effects (bi ) in the model.
25 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Remarks on moments
The subject-specific model tells us about E (Yi | xi , bi ) and
var ( Yi | xi , bi ). But how about E (Yi | xi ) and var ( Yi | xi )?
The 2-stage model implies that
E (Yi | xi ) = E {E (Yi | xi , bi )| xi }
= E {fi (xi , β, bi )| xi } .
If fi (·) is linear in bi , this is just fi (xi , β, 0).
That is, for a (sub)population of subjects all with the same covariates
xi , the mean response across the population satisfies the same model
as the mean response for a single subject, with the population
average values of the parameters.
26 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
When fi is nonlinear, E ( Yi | xi ) cannot be written in closed form.
Similarly, var ( Yi | xi ) is complicated.
Alternative approach
Write down a model for the marginal conditional mean and
covariance matrix E ( Yi | xi ) and var ( Yi | xi ) directly.
27 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Population-averaged (marginal) modeling
Example
Six-Cities study
Yi,j = indicator of wheezing in i th child at j th observation time,
ti,j .
within-individual covariates: δi,j = indicator of mother’s smoking
status at time ti,j .
among-individual covariates: location, gender, ...
28 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
We could use a logistic regression model:
T
E (Yi,j | xi ) =
e β0 +β1 ti,j +β2 δi,j +ai β3
.
T
1 + e β0 +β1 ti,j +β2 δi,j +ai β3
Since Yi,j is Bernouilli,
var ( Yi,j | xi ) = E ( Yi,j | xi ) {1 − E (Yi,j | xi )} ,
and we need only to specify correlations.
29 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
The objective is to understand the “typical (average) response
vector” as a function of covariates, i.e., the dependence of E ( Yi | xi )
on xi . In the above example, it is the probability of wheezing at each
age as a function of covariates that is of scientific interest.
This application is an example of a cohort study, in which a group of
individuals is followed and information is recorded on each, with the
objective of understanding behavior over time.
Such studies are common in epidemiology, which seeks to understand
the interplay between different potential risk factors and outcomes
that are important to public health.
30 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
On the basis of observation of associations among risk factors and
responses, epidemiologists would like to make public policy
recommendations.
In such circumstances, individual behavior is not the focus; rather,
the objective is to understand the phenomenon of interest at the
population level, so that broad recommendations can be made. This
is different from the situation in pharmacokinetics.
Thus, it is not routine to build a model for individual behavior, but to
model the average behavior E ( Yi | xi ) directly.
31 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
A general form
For the mean:
E (Yi | xi ) = fi (xi , β)
(ni × 1)
Note: fi (·) and β do not have the same interpretation as in a
subject-specific model.
For the variances and covariances:
var ( Yi | xi ) = Vi (β, ξ, xi )
(ni × ni )
Specify models for the variances and correlations separately.
Failure to take correlation into account might result in inefficient
and potentially misleading inferences.
32 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Subject-Specific versus Population-Averaged
models
Subject-specific approach
Parameters often have direct subject-matter interpretation.
Interest typically focuses on the distribution of particular aspects
of individual behavior.
Interest may well focus on the population of responses. The
approach may be adopted solely as a mechanism for modeling
correlation: the random effects induce a correlation structure.
Population-averaged approach
Fitting and making inferences about population-averaged models
are direct extensions of univariate methods.
33 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
The two approaches may or may not lead to the same model.
If f is linear, the two give the same models:
E (Yi | xi ) = E {E (Yi | xi , bi )| xi }
= E {fi (xi , β, bi )| xi }
= E(Xi β i |xi )
= E(Xi β + Xi bi |xi )
= Xi β.
34 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
The interpretations are still different:
From the subject-specific perspective, β has the interpretation
as the mean of the population of individual regression
parameters β i that dictate individual-specific mean models.
Thus, β may be interpreted as the “typical parameter value.”
From the population-averaged perspective, β has the
interpretation as the parameter producing the “typical response
vector.”
If f is nonlinear:
the two give different models;
which approach is preferred usually depends on the application
and scientific objective.
35 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Specifying Variance-Covariance Structure
Variances may be suggested by the nature of the response, or by
exploration as in the univariate case.
Covariance structure is usually completed by specifying correlations:
unstructured;
compound symmetry;
m-dependent;
AR(1);
exponential (simplifies to AR(1) for equally-spaced times);
Gaussian.
36 / 37
Multivariate Responses
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Remarks
For many types of response, the correlation matrix need only be
nonnegative definite.
For others, notably binary (Bernouilli) responses, the means may
impose constraints on the correlations.
We often ignore these constraints, and may in fact use a “working”
variance-covariance structure that is impossible for the type of
response.
37 / 37
Multivariate Responses