1
Chapter 13.2: Regression with Transformed Variables
Instructor: Dr. Arnab Maity
2
So far we have learned about simple linear regression model (y = β0 + β1 x + e), where we
assume that the relationship between response y and a single predictor x is linear. However,
in many situations, this may not be the case.
Example: (Book 13.4) Data on x = length of a scamp and y = mercury content were
measured. The following output are from StatCrunch.
Simple linear regression results:
Dependent Variable: Mercury_Content
Independent Variable: Scamp_Length
Mercury_Content = -0.3073 + 0.0010 Scamp_Length
Sample size: 18
R (correlation coefficient) = 0.8261
R-sq = 0.6825
Estimate of error standard deviation: 0.0893
Parameter estimates:
Parameter
Estimate
Intercept
-0.30733211
Slope
0.0010416969
Std. Err.
0.09051024
0.0001776061
DF
16
16
95% L. Limit
-0.49920524
0.00066518876
95% U. Limit
-0.11545897
0.001418205
Notice that
• From the scatterplot, while the linear model explains 68% variability, it seems there
might be a non-linear relationship between length and mercury content.
• The rate of change in scamp length for lower mercury levels seems different from that
for the higher mercury levels, suggesting perhaps a logarithmic relationship.
Let us re-run the analysis after we transform y to ln(y). See output below.
Simple linear regression results (w/ transformation):
Dependent Variable: log(Mercury_Content)
Independent Variable: Scamp_Length
log(Mercury_Content) = -4.6857 + 0.0057 Scamp_Length
Sample size: 18
R (correlation coefficient) = 0.8945
R-sq = 0.8002
Estimate of error standard deviation: 0.3612
Parameter estimates:
Parameter
Estimate
Intercept
-4.6857475
Slope
0.0057463624
Std. Err.
DF
0.36573356
16
0.00071767031 16
95% L. Limit
-5.461068
0.0042249693
95% U. Limit
-3.910427
0.0072677555
3
Based on the analysis using transformed response ln(y), notice that
• The value of R2 increased from 68.25% to 80.02%, implying the the linear relationship
between ln(y) and x is much stronger than the one between just y and x.
• The estimated regression line for the new analysis is log(Mercury Content) = −4.6857+
0.0057(Scamp Length). Thus the interpretation of the slope estimate is different than
the original regression. Can you give the interpretation of slope in this case?
• In the scale of the original y values, the model becomes
Mercury Content = exp{−4.6857 + 0.0057(Scamp Length)}.
• What is the predicted value of y, the mercury content for say x = 500?
• The 95% Prediction interval for the transformed model for x = 500 is : (−2.6026, −1.0246).
How can we get a prediction interval for a new y in the original scale with x = 500?
Here are the fitted curves (in the original scale of y for both the models:
Regression of y vs. x
Regression of ln(y) vs. x
4
Analyzing transformed data
Definition: A relationship between y and x is called intrinsically linear if, by means of
transformation on x and/or y, the relationship can be described as Y 0 = β0 + β1 x0 , where x0
and y 0 are the transformed version of x and y.
Examples: Some useful intrinsically linear functions are
True relationship
Exponential: y = αeβx
Power: y = αxβ
Reciprocal: y = α + β/x
y = α + β ln(x)
Transformation to linearize
y 0 = ln(y)
y 0 = ln(y), x0 = ln(x)
Linear form
y 0 = ln(α) + βx
Keep in mind:
1. Interpretations of the estimated parameters in the transformed model must be done
carefully. This will depend on what transformation was used before fitting the linear
model.
2. If the chosen model is not intrinsically linear (that is, the transformation does not
ensure strong linear relationship) then this method cannot be used. Instead, we need
to apply other (advanced) techniques such as non-linear least squares etc.
3. To do inference (testing, confidence interval and prediction interval etc), we still need
to check whether the residuals follow the assumptions of normality and equal variance.
5
Logistic Regression
The simple linear regression (with or without transformation) is appropriate when the response variable y is quantitative. However, in many situations this is not the case, the y
could be binary (with or without a disease, failed or did not fail etc.). In such cases, we can
assume y can take only two values, 0 and 1, depending on the two possible outcomes. The
value of the predictor the relates to the probability of y taking a specific value.
Example: (Book Exercise 13.24) Kyphosis refers to severe flexion of the spine following
corrective spinal surgery. To determine whether a patients age at the time of surgery alters
the risk of getting Kyphosis, 40 patients were monitored and their age (in months) at surgery
(x) and whether they had Kyphosis (y) were measured. The scatterplot is given below.
Kyphosis (yes/no) vs. Age
Boxplot of Age grouped by Kyphosis (yes/no)
Question 1: Does age at surgery have any association to the risk of getting Kyphosis?
Question 2: Say a patient was of 100 months of age at the time of surgery. What is the
chance of them suffering from Kyphosis? What are the odds of them getting Kyphosis?
Model: Suppose we have data (y1 , x1 ), . . . (yn , xn ), where x is quantitative and y can take
the values 0 or 1. In logistic regression, we model the probability
p(yi = 1) =
for i = 1, . . . , n.
exp(β0 + β1 xi )
,
1 + exp(β0 + β1 xi )
6
Notice that we are modeling a probability, rather than the actual y itself. This is a major
change from simple linear regression model.
1. Since we are modeling the probability p(yi = 1), we should ensure this value is between
0 and 1 for any value of x, β0 and β1 . Does the logistic model ensure this already?
2. There are two parameters in the model: β0 and β1 . Which of these parameters represents the association between x and y? Explain.
3. What is the estimated chance of a patient getting Kyphosis is their age at surgery is
x = 100 months?
Estimation of parameters: Unlike the simple linear regression model, there is no closed
form expression for estimators of β0 and β1 . We need to use a software to do logistic regression. The estimation is done by a technique we already learned in class: the maximum
likelihood estimation (MLE). However, this needs to be done numerically, in this case.
Logistic regression output for the Kyphosis data is shown below:
Logistic regression results
Dependent Variable: Kyphosis (Success = 1)
Independent Variable(s): Age at surgery
Parameter estimates:
Variable
Intercept
Age at surgery
Estimate
-0.5726
0.004295
Std. Err.
0.6023
0.005849
Question: How to interpret βb1 ?
Question: How to test for β1 ?
Zstat
-0.9506
0.7344
P-value
0.3418
0.4627
Odds Ratio
95% Low. Lim.
95% Up. Lim.
1.004
0.9928
1.0158
7
Odds and Odds Ratio: For a given value of x = x∗ , the estimated probability of y = 1 is
pb(x∗ ) = pb(y = 1 when x = x∗ ) =
exp(βb0 + βb1 x∗ )
,
1 + exp(βb0 + βb1 x∗ )
that is, just take the original logistic model and plug-in the estimated values of β0 and β1 ,
and set x = x∗ .
The odds is then defined as
[odds at (x = x∗ )] =
p(x∗ )
chance of getting Kyphosis at age x = x∗
=
1 − p(x∗ )
chance of not getting Kyphosis at age x = x∗
In the Kyphosis example, we asked: if a patient was of 100 months of age at the time of
surgery, what is the chance of them suffering from Kyphosis? What are the odds of them
getting Kyphosis? The estimated chance of them getting Kyphosis at x = 100 is
Their odds of getting Kyphosis at x = 100 is
Interpret the odds:
The odds ratio is defined as the change in odds due to 1 unit increase in x:
odds at (x + 1)
odds at x
For logistic regression it can be shown that
• odds ratio is simply exp(β1 ), and thus β1 = ln(odds ratio)
• odds ratio does not depend on x value
• In linear regression, the slope β1 was change in y due to 1 unit increase in x. In logistic
regression, the parameter β1 is change in log-odds due to 1 unit increase in x.
8
Interpret the odds ratio: With each additional increase in age (x), we estimate that the
odds of Kyphosis will ANSWER GOES HERE by a factor of ANSWER GOES HERE, or
equivalently, will ANSWER GOES HERE by ANSWER GOES HERE percent.
In general, odds ratio close to 1.00 means
Confidence intervals: A 100(1 − α)% confidence interval for β1 is
βb1 ± zα/2 SE(βb1 )
A 100(1 − α)% confidence interval for the odds ratio, eβ1 is
(eβ1 −zα/2 SE(β1 ) , eβ1 +zα/2 SE(β1 ) ),
b
b
b
b
that is, exponentiate the interval for β1 .
Testing for β1 : To test for H0 : β1 = 0, that is, no association between x and y, we usually
use a (large sample) z-test. The test statistic is
Z = βb1 /SE(βb1 ),
where the SE for βb1 is computed by the software. The rejection region and corresponding
p-values are computed as in the standard z-test.
Prediction of y for a new x = x∗ : First compute the estimated chance pb(x∗ ). Prediction
of new y would be
ypred = 1 if pb(x∗ ) > 0.5;
ypred = 0 if pb(x∗ ) ≤ 0.5.