fariSample mean 𝑋̅ =
𝑋1 +⋯+𝑋𝑛
𝑛
𝑋1 +⋯+𝑋𝑛 −𝑛𝜇
Central Limit Thm: 𝑃 {
Step 4: check if observed TS falls into region
Critical Region is set of all samples that will lead to
rejection of 𝐻0
P-value Prob of observing evidence as much/more in
favor of 𝐻1
Step 1: Set up null and alt hypotheses
Step 2: Find TS 𝑇 with known dist under 𝐻0
Step 3: Calculate p-value
< 𝑥} ≈ 𝑃{𝑍 < 𝑥}
𝜎√𝑛
̅)2
(∑𝑛
𝑖=1(𝑋𝑖 −𝑋 )
Sample Variance: 𝑆 2 =
, 𝐸[𝑆 2 ] = 𝜎 2
𝑛−1
Use 𝑛 − 1 to make E[sample var] = population var.
Chi Square Use to find 𝑆 2 from normal.
𝑍1 … 𝑍𝑛 are iid standard normal
𝑋 = 𝑍12 + ⋯ + 𝑍𝑛2 , 𝑋 is chi-square with 𝑛 deg freedom Χ𝑛2
𝑋̅−𝜇
𝑥̅ −𝜇
𝑥̅ −𝜇
2
2
Χ𝑚
+ Χ𝑛2 = Χ𝑚+𝑛
𝑝 = 𝑃 {| ⁄ 0| ≥ | ⁄ 0 |} = 2 (1 − Φ (| ⁄ 0 |))
𝜎 √𝑛
𝜎 √𝑛
𝜎 √𝑛
𝜎2
Sampling from a Normal Population: 𝑋̅~𝑁 (𝜇, )
Step 4: check if 𝑝 is small enough
𝑛
𝑋1 + ⋯ + 𝑋𝑛 ~𝑁(𝑛𝜇, 𝑛𝜎 2 ), 𝑋̅ ⊥ 𝑆 2
If 𝑝 is smaller than your 𝛼, reject 𝐻0
𝑝 tells us how cautious/easily convinced we should be
T-Distribution If 𝑍 ⊥ 𝑋𝑛2, 𝑇 = 𝑍/√𝑋𝑛2 /𝑛 , 𝑡𝑛
𝑋̅−𝜇
Decision Errors Type I: Reject 𝐻0 when it’s actually true
~𝑡𝑛−1
𝑆/√𝑛
“False alarm” error 𝛼
𝑚]
Method of Moments 𝐸[𝑋 = 𝑔𝑘 (𝜃1 … 𝜃𝑘 )
Type II: Accept 𝐻0 when it’s not true. Miss.
𝑚
𝑚
𝑚 = 𝑋1 +⋯+𝑋𝑛
̅̅̅̅
Operating characteristic (OC) curve: 𝛽(𝜇) =
If 𝑘 unknown parameters, let 𝑋
𝑛
𝑃𝜇 {𝑎𝑐𝑐𝑒𝑝𝑡 𝐻0 }
̅̅̅̅𝑘
𝑔𝑘 (𝜃1 … 𝜃𝑘 ) = 𝑋
1 − 𝛽(𝜇) = 𝑃𝜇 {𝑟𝑒𝑗𝑒𝑐𝑡 𝐻0 } is Power Function
Maximum Likelihood Estimator
One-Sided Test 𝐻0 = 𝜇0 , 𝐻1 = 𝜇 > 𝜇0
Likelihood 𝐿(𝜃) = 𝑓(𝑥1 … , 𝑥𝑛 ; 𝜃)
𝑋̅−𝜇0
𝑥̅ −𝜇
𝑥̅ −𝜇
𝑓(𝑥1 … ; 𝜃) = 𝑓𝑥 1 (𝑥1 ; 𝜃) ∗ 𝑓𝑥𝑛 (𝑥𝑛 : 𝜃)
TS: 𝑍 =
, 𝑝 = 𝑃 {𝑍 ≥ ⁄ 0 } = 1 − Φ ( ⁄ 0 )
𝜎/√𝑛
𝜎 √𝑛
𝜎 √𝑛
Maximize the log likelihood log 𝐿(𝜃) = 𝑙(𝜃)
Alternately 𝐻0 : 𝜇 ≤ 𝜇0 , composite hypothesis
Estimate Difference in Means
𝑥̅ −𝜇
𝑥̅ −𝜇
TS same as above, 𝑝 = 𝑃 {𝑍 ≥ ⁄ 0 } ≤ 1 − Φ( ⁄ 0 )
2 +(𝑚−1)𝑆 2
(𝑛−1)𝑆
𝜎 √𝑛
𝜎 √𝑛
1
2
Get pooled sample variance: 𝑆𝑝2 = (𝑛−1)+(𝑚−1)
(the bell curve only has one region that matters)
𝑋̅−𝜇
Like a weighted average, weights as degrees of freedom
Unknown Variance 𝐻0 : 𝜇 = 𝜇0 , TS 𝑇 = 𝑆⁄ 0 ~𝑡𝑛−1
√𝑛
Evaluating a point estimator
𝑥̅ − 𝜇0
𝑥̅ − 𝜇0
Biased: shooting off-target
𝑝 = 𝑃{|𝑇| ≥ |
})
| = 2 (1 − 𝑃 {𝑡𝑛−1 ≤
𝑏(𝜃̂) = 𝐸[𝜃̂(𝑋1 … 𝑋𝑛 ) − 𝜃
𝑆⁄√𝑛
𝑆⁄√𝑛
2
Equality of Means 𝜎𝑥2 and 𝜎𝑦2 are known, 𝑋, 𝑌 are normal
̂
Mean Squared Error: 𝐸 [(𝜃̂ − 𝜃) ] = 𝑉𝑎𝑟(𝜃̂) + 𝑏2 (𝜃)
2 sets of indep samples
2
2
=𝐸 [(𝜃̂ − 𝜃) ] − (𝐸[𝜃̂] − 𝜃)
𝐻0 : 𝜇𝑥 = 𝜇𝑦 𝑜𝑟 (𝜇𝑥 ≤ 𝜇𝑦 )
Hard to assess credibility of a point estimate by itself
TS: if 𝜇𝑥 = 𝜇𝑦 , |𝑋̅ − 𝑌̅| is more likely to be small.
Precision: measure spread
𝜎2
𝜎2
𝑋̅ − 𝑌̅~𝑁 (𝜇𝑥 − 𝜇𝑦 , 𝑥 + 𝑦 ) since sum of indep normal is
𝑛
𝑚
Confidence Interval
still normal.
Step 1: find a statistic involving the parameter
𝑋̅−𝑌̅−(𝜇𝑥 −𝜇𝑦 )
𝑋̅−𝜇
Standardize:
=TS. 𝑝 = 𝑃{|𝑇𝑆| ≥ |𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑|}
Mean with known variance:
~𝑁(0,1)
Mean with unknown Var:
𝜎/√𝑛
𝑋̅−𝜇
√𝜎𝑥2 ⁄𝑛+𝜎𝑦2 ⁄𝑚
~𝑡𝑛−1
𝑆/√𝑛
(𝑛−1)𝑆 2
2
mean:
~Χ𝑛−1
𝜎2
=2 (1 − Φ (
Step 3: Find the equivalent interval for the parameter
𝑋̅−𝜇
𝜎
𝜎
−𝑧𝛼⁄2 ≤ ⁄ ≤ 𝑧𝛼⁄2 ⇔ 𝑋̅ − 𝑧𝛼⁄2 ≤ 𝜇 ≤ 𝑋̅ + 𝑧𝛼⁄2
√𝑛
≤ 𝜇 ≤ 𝑋̅ + 𝑧𝛼⁄2
𝜎
√𝑛
√𝑛
}= 1−𝛼
𝜎 √𝑛
𝜎2
𝜎1 , 𝜎2 known, use 𝑋̅ − 𝑌̅ ± 𝑧𝛼/2 √ 1 + 𝜎22 /𝑚
𝑛
𝜎1 , 𝜎2 unknown but equal use 𝑋̅ − 𝑌̅ ±
1
1
( + )(𝑛−1)𝑆12 +(𝑚−1)𝑆22
𝑛 𝑚
𝑡𝛼,𝑛+𝑚−2 √
𝑛+𝑚−2
2
Lower confidence interval
2
2
𝜎
𝜎
𝜎1 , 𝜎2 known, use (−∞, 𝑋̅ − 𝑌̅ + 𝑧𝛼 √ 1 + 2 )
𝑛
𝑚
𝜎1 , 𝜎2 unknown but equal use (−∞, 𝑋̅ − 𝑌̅ +
√𝜎𝑥2 ⁄𝑛+𝜎𝑦2 ⁄𝑚
)) and reject if 𝑝 is small
√𝑆𝑝2 ⁄𝑛+𝑆𝑝2 ⁄𝑚
~𝑡𝑛+𝑚−2 𝑝 = 𝑃 {|𝑡𝑛+𝑚−2 | ≥
Sample correlation coefficient: 𝑟 = 𝑆𝑋𝑌 /√𝑆𝑋𝑋 𝑆𝑌𝑌
So 𝑟 2 = 𝑅 2 , 𝑟 =
𝑥̅ −𝑦̅
√𝑆𝑝2 ⁄𝑛+𝑆𝑝2 ⁄𝑚
traditionally
𝑟 does not measure the slope of regression line
𝑆
Slope is 𝑏̂ = 𝑟 𝑌 , is about 𝑟 if 𝑆𝑌 ≈ 𝑆𝑋
𝑆𝑋
𝑟 ≤ 1 so usually you get regression to the mean
Inferential Linear Regression
𝑌 = 𝛼 + 𝛽𝑥 + 𝑒 where 𝑒 is error, assume 𝐸[𝑒] = 0
𝐸[𝑌|𝑥] = 𝛼 + 𝛽𝑥
For each observation 𝑌𝑖 = 𝛼 + 𝛽𝑥𝑖 + 𝑒𝑖
Assume random errors 𝑒𝑖 are indep
Estimate 𝛼, 𝛽, 𝑎𝑛𝑑 𝜎 2 from the tuples of samples
Estimate 𝛼 and 𝛽 from least squares formulas
𝑆𝑆𝑅 estimates 𝜎 2
∑𝑛 (𝑥 −𝑥̅ )(𝑌𝑖 −𝑌̅)
𝐵 = 𝑖=1𝑛 𝑖
, 𝐴 = 𝑌̅ − 𝐵𝑥̅
2
∑𝑖=1(𝑥𝑖 −𝑥̅ )
𝐵 is normally distributed, unbiased estimator for 𝛽
𝜎2
𝐵~𝑁 (𝛽,
) 𝐴~𝑁 (𝛼,
2
𝜎 2 ∑𝑛
𝑖=1 𝑥𝑖
)
𝑆𝑋𝑋
𝑛𝑆𝑋𝑋
𝑆𝑆𝑅 = ∑𝑛𝑖=1(𝑌𝑖 − 𝐴 − 𝐵𝑥𝑖 )2
𝑆𝑆𝑅
2
~Χ𝑛−2
, lose 2 deg. Free since A and B are linear
𝜎2
𝑆𝑆
𝑆𝑆
combinations of 𝑌1 … 𝑌𝑛 , 𝐸 [ 𝑅 ] = 𝜎 2 , 𝐸 [ 2𝑅 ] =
𝑛−2
𝜎
𝑆𝑆
2 ]
2
𝐸[Χ𝑛−2
= 𝑛 − 2. 2𝑅 ~Χ𝑛−2
indep of A and B
𝜎
Confidence Intervals
𝑆𝑆
𝑅
𝛽: (𝐵 − √(𝑛−2)𝑆
𝑋𝑋
𝛼: 𝐴 ± √
𝑆𝑆
𝑅
𝑡𝛼⁄2,𝑛−2 , 𝐵 + √(𝑛−2)𝑆
)
𝑋𝑋
∑𝑖 𝑥𝑖2 𝑆𝑆𝑅
𝑡
𝑛(𝑛−2)𝑆𝑋𝑋 𝛼⁄2,𝑛−2
1
(𝑥0 −𝑥̅ )2
𝑛
𝑆𝑋𝑋
𝛼 + 𝛽𝑥0 : 𝐴 + 𝐵𝑥0 ± √ +
𝑆𝑆𝑅
𝑡 ⁄
𝑛−2 𝛼 2,𝑛−2
√
Distributional Results
(𝑛−2)𝑆𝑋𝑋
𝛽: √
𝑆𝑆𝑟
(𝐵 − 𝛽)~𝑡𝑛−2
∑𝑖 𝑥𝑖2 𝑆𝑆𝑅
(𝐴 − 𝛼)~𝑡𝑛−2
𝐴+𝐵𝑥0 −𝛼−𝛽𝑥0
̅ )2 𝑆𝑆𝑅
1 (𝑥 −𝑥
√( + 0
)
)(
𝑆𝑋𝑋
Future Response 𝑌 = 𝛼 + 𝛽𝑥0 + 𝑒:
~𝑡𝑛−2
𝑛−2
𝑌−𝐴−𝐵𝑥0
√
̅ )2 𝑆𝑆𝑅
𝑛+1 (𝑥0−𝑥
√
+ 𝑆
𝑛
𝑛−2
𝑋𝑋
~𝑡𝑛−2
Mean Response given 𝑥0 : 𝐸[𝑌|𝑥0 ] = 𝛼 + 𝛽𝑥0
Estimates mean response given input 𝑥0
Unbiased, is a linear combination of indep normal r.v.
}
Linear Regression
Sum of squared residuals: 𝑆𝑆 = ∑𝑛𝑖=1(𝑌𝑖 − 𝑎 − 𝑏𝑥𝑖 )2
Choose 𝑎 and 𝑏 that minimize this
Take partial derivatives w/ respect to a, b
Set to 0, get normal equations 𝑎 + 𝑥̅ 𝑏 = 𝑌̅, 𝑛𝑥̅ 𝑎 +
𝑏(∑𝑛𝑖 𝑥𝑖2 ) = ∑𝑛𝑖 𝑥𝑖 𝑌𝑖
∑𝑛 𝑥 𝑌 −𝑛𝑥̅ 𝑌̅
Solution: 𝑏 = 𝑖 𝑛𝑖 2𝑖
, 𝑎 = 𝑌̅ − 𝑥̅ 𝑏
∑𝑖 𝑥𝑖 −𝑛𝑥̅
∑(𝑥𝑖 −𝑥̅ )(𝑌𝑖 −𝑌̅)
√∑(𝑥𝑖 −𝑥̅ )2 ∑(𝑌𝑖 −𝑌̅)2
𝑛
If Assume not equal replace 𝑆𝑝 with 𝑆𝑥 and 𝑆𝑦
Test Equality of Proportions L18
Identities: ∑𝑛𝑖(𝑥𝑖 − 𝑥̅ )2 = ∑𝑛𝑖 𝑥𝑖2 − 𝑛𝑥̅
∑𝑛𝑖(𝑥𝑖 − 𝑥̅ )(𝑌𝑖 − 𝑌̅) = ∑𝑛𝑖 𝑥𝑖 𝑌𝑖 − 𝑛𝑥̅ 𝑌̅
2
𝑆𝑥𝑌
Mean Response 𝛼 + 𝛽𝑥0 :
1
∑𝑛 (𝑋 − 𝑋̅)2 similar for 𝑆𝑦
𝑆𝑥 =
𝑛−1 𝑖−1 𝑖
Pooled sample variance as weighted average
If 𝐻0 is true, then 𝜇𝑥 − 𝜇𝑦 = 0
𝑥̅ −𝑦̅
Step 4: plug in observed numbers
𝑋̅−𝜇
One-sided upper: {−𝑧𝛼 ≤ ⁄ < ∞} = 1 − 𝛼
|𝑥̅ −𝑦̅|
𝑆𝑋𝑋
𝑆𝑋𝑋 𝑆𝑌𝑌
𝑛(𝑛−2)𝑆𝑋𝑋
Unknown Variance Assumed equal
𝜎 √𝑛
𝑃 {𝑋̅ −
So 𝑅 2 =
𝛼: √
Var with unknown
Step 2: find initial interval with prob. 1 − 𝛼
𝑋̅−𝜇
Two-sided:𝑃 {−𝑧𝛼⁄2 ≤ ⁄ ≤ 𝑧𝛼⁄2 } = 1 − 𝛼
𝜎 √𝑛
𝜎
𝑧𝛼⁄2
√𝑛
2
2
𝑆 𝑆 −𝑆
𝑆𝑆𝑅 = ∑𝑛𝑖=1(𝑌𝑖 − 𝑎̂ − 𝑏̂𝑥𝑖 ) = 𝑋𝑋 𝑌𝑌 𝑥𝑌
Prediction Interval
Find an interval that will contain the response with
certain confidence, NOT the same as confidence interval
Best “point prediction” 𝐴 + 𝐵𝑥0 , unbiased
Analysis of Residuals
Assume independence, constant variance
(homoscedasticity), normality.
Check these assumptions by looking at residuals
𝑌 −(𝐴+𝐵𝑥𝑖 )
Standardized residuals 𝑖
(the denominator
√𝑆𝑆𝑅 /(𝑛−2)
estimates 𝜎
Linearity violated if residuals show discernible pattern
Nonconstant variance shows increasing outliers
Check Normality
Sample quantile of standardized residuals vs standard
normal quantile
𝑛+𝑚−2
Quantiles: points taken at regular intervals from CDF of
𝑆𝑥𝑥 = ∑𝑛𝑖=1(𝑥𝑖 − 𝑥̅ )2 = (𝑛 − 1)𝑠𝑥2
r.v.
Approximate Confidence Interval
𝑆𝑌𝑌 = ∑𝑛𝑖=1(𝑌𝑖 − 𝑌̅)2 = (𝑛 − 1)𝑠𝑌2
Have 1 − 𝛼 CI with length no greater than 2Δ𝑝. What’s
This is TSS, total sum of squares, amount of variance in Q-Q plot: compare 2 distributions by plotting quantiles
the min sample size required?
response variables. Part of this is caused by difference in against each other, to check normality
Transforming to Linearity
(𝑝̂ ± 𝑧𝛼⁄2 √𝑝̂ (1 − 𝑝̂ )⁄𝑛 ) Length: 2𝑧𝛼⁄2√𝑝̂(1−𝑝̂)⁄𝑛 ≤ 2Δ𝑝 input variables
2
2
𝑊(𝑡) ≈ 𝑐𝑒 𝑑𝑡 − 𝑙𝑜𝑔 → 𝑙𝑜𝑔𝑊 ≈ 𝑙𝑜𝑔𝑐 + 𝑑𝑡
𝑆𝑆𝑅 = ∑𝑛𝑖=1(𝑌𝑖 − 𝑌̂𝑖 ) = ∑𝑛𝑖=1(𝑌𝑖 − 𝑎̂ − 𝑏̂𝑥𝑖 )
Sub 𝑝̂ with 𝑝∗ after testing 𝑘 times
Where 𝛼 is 𝑙𝑜𝑔𝑐 and 𝛽𝑥 is 𝑑𝑡 and 𝑙𝑜𝑔𝑊 is Y
Hypothesis Testing
Measures remaining amount of variance in the response
Interpret the regression coefficients:
𝐻0 : presume this, in control, conventional, not effective, variables, after we take into account difference of input
Regression equation 𝑊 = 𝑎̂ + 𝑏̂𝑡
no discrimination
variables, known as RSS, residual sum of squares
average number of defects at 0 is 𝑎̂, and as X increases
𝑛(𝑥
̅
𝐻1 : out of control, new finding, discrimination, more
𝑆𝑥𝑌 = ∑𝑖 𝑖 − 𝑥̅ )(𝑌𝑖 − 𝑌)
1, number of defects increases by 𝑏̂ on average
effective
𝑆𝑌𝑌 − 𝑆𝑆𝑅 is amount of variation explained by input var, by
𝑊(𝑡+1)
≈ 1 + 𝑏̂ by Taylor expansion
Level of significance 𝛼 Critical Region Approach
known as explained sum of squares ESS
𝑊(𝑡)
𝐸𝑆𝑆
𝑅𝑆𝑆
Step 1: set up null and alternative hypotheses
𝑊(1.01𝑡)
Coefficient of determination 𝑅 2 =
=1−
≈ 1 + 0.01𝑏̂ by Taylor expansion
𝑇𝑆𝑆
𝑇𝑆𝑆
𝑊(𝑡)
Step 2: find a statistic 𝑍 with known distribution
This shows how well the model fits the data.
Linear Transformations
Called the test statistic TS, 𝑍(𝑋1 … 𝑋𝑛 ) =
Ex. 𝑅 2 = .75, so 75% of variation of y is from changes in Exponential: 𝑙𝑜𝑔𝑌 ≈ 𝑙𝑜𝑔𝑐 + 𝑑𝑥, 𝑌 ≈ 𝑐𝑒 𝑑𝑥
Step 3: Choose an 𝛼 and find “blue region”
x. The rest is from variance even when x is constant
X increases by 1, Y changes by factor of 100d%
Smaller 𝛼 means smaller region
1 1
𝑛 𝑚
( + )(𝑛−1)𝑆12 +(𝑚−1)𝑆22
𝑡𝛼,𝑛+𝑚−2 √
)
So 𝑏 =
̅
∑𝑛
𝑖 (𝑥𝑖 −𝑥̅ )(𝑌𝑖 −𝑌 )
2
∑𝑛
𝑖 (𝑥𝑖 −𝑥̅ )
= 𝑆𝑥𝑌 /𝑆𝑥𝑥
𝑌(𝑥+1)
𝑌(𝑥)
= 𝑒 𝑥 so
𝑌(𝑥+1)
𝑌
Assumptions 𝑋𝑖𝑗 are independent normal r.v.
≈ 1 + 𝑥 where 𝑥 < 0
Logarithm: 𝑌 ≈ 𝑙𝑜𝑔𝑐 + 𝑑𝑙𝑜𝑔𝑥
X increases b factor of 1%, Y changes by d/100
𝑌(1.01𝑥) − 𝑌(𝑥) = 𝑐𝑙𝑜𝑔(1 + .01)
log(1 + 𝑥) ≈ 𝑥 so 𝑌(1.01𝑥) − 𝑌(𝑥) ≈ .01 ∗ 𝑐
Power: 𝑙𝑜𝑔𝑌 ≈ 𝑙𝑜𝑔𝑐 + 𝑑𝑙𝑜𝑔𝑥, 𝑌 ≈ 𝑐𝑥 𝑑
X increases by factor of 1%, Y changes by factor of d%
𝑌(1.01𝑥)
= (1 + .01)𝑎 , (1 + 𝑥)𝑎 ≈ 1 + 𝑎𝑥
𝑌(𝑥)
Multiple Linear Regression
Least Squares method:
𝐵0
1 𝑥11 𝑥1𝑘
𝐵 = [ … ] = (𝑋 ′ 𝑋)−1 𝑋 ′ 𝑌 where 𝑋 = [1|𝑥21 |𝑥2𝑘 ] and 𝑌 =
𝐵𝑘
1 𝑥𝑛1 𝑥𝑛𝑘
𝑌1
[ ] 𝛽𝑖 is amount of change in response if 𝑥𝑖 increased by
𝑌𝑘
1 keeping others unchanged.
Residuals 𝑟𝑖 = 𝑌𝑖 − 𝐵0 − 𝐵1 𝑥𝑖1 − ⋯ 𝐵𝑘 𝑥𝑖𝑘
Matrix form 𝑟 = 𝑌 − 𝑋𝐵
𝑆𝑆𝑅 = 𝑌 ′ 𝑌 − 𝑌′𝑋𝐵 residual sum of squares
Coefficient of multiple determination
𝑆𝑆
𝑅 2 = 1 − 𝑅 proportion of variation explained by model
𝑆𝑌𝑌
𝑅 is multiple correlation coefficients
As input vars increase, 𝑅 2 goes down, 𝑆𝑆𝑅 goes up
…But this could mean overfitting
(𝑛−1)𝑅2 −𝑘
Adjusted/corrected 𝑅 2 : 𝑅̅2 =
𝑛−𝑘−1
Inference 𝐵 Has multivariate normal dist
𝑁(𝛽, 𝜎 2 (𝑋 ′ 𝑋)−1 ), 𝛽 is unbiased estimator
𝜎 2 (𝑋 ′ 𝑋)−1 is covariance matrix, diagonals give variance
𝜎2
For each 𝑖, 𝐵𝑖 ~𝑁(𝛽𝑖 , )
and
𝑆𝑆𝑅
𝑆𝑖𝑖
1
is 𝑖𝑡ℎ elem in diagonal of (𝑋 ′ 𝑋)−1
𝑆𝑖𝑖
2
= Χ𝑛−𝑘−1
and independent of B
This is an unbiased estimator for 𝜎 2
𝜎2
𝜎2
𝐵𝑖 ~𝑁 (𝛽𝑖 , ) for inference on 𝛽𝑖
𝐵𝑖 −𝛽𝑖
√𝜎
̂ 2 ⁄𝑆𝑖𝑖
𝑆𝑖𝑖
~𝑡𝑛−𝑘−1 and denominator is estimate for sd of 𝐵𝑖 ,
standard error. Test whether 𝑥𝑖 has impact on response
keeping other input vars unchanged 𝐻0 : 𝛽𝑖 = 0
Variances of each are the same
𝜇𝑖𝑗 = 𝜇 + 𝛼𝑖 + 𝛽𝑗 where 𝜇 is overall mean
𝛼𝑖 is correction for the 𝑖th row
𝛽𝑗 is correction for the 𝑗th column
𝜇 + 𝛼𝑖 is mean for 𝑖𝑡ℎ row, use 𝛽𝑗 for mean of 𝑗𝑡ℎ column
Hypotheses 𝐻0𝑟 : row factor has no impact, 𝐻0𝑐 : colun has
no impact
Estimators 𝑋.. is an unbiased estimator for 𝜇, overall
Sample Midterm 2
sample mean→ overall mean
(a) If simple linear regression of Y on X yields a line with
𝑋𝑖. − 𝑋.. is unbiased estimator for 𝛼𝑖
slope equal to 2, then simple linear regression of X on Y
𝑋.𝑗 − 𝑋.. is unbiased estimator for 𝛽𝑗
will yield a line with slope equal to 1/2. False.
Sample mean difference → mean difference
(b) A simple linear regression model (Y on x) has R^2 =
2
Row sum of squares 𝑆𝑆𝑟 = ∑𝑚
𝑖=1 𝑛(𝑋𝑖. − 𝑥.. )
0.7, while an exponential
Variation between rows
model (log(Y) on x) has R^2 = 0.8. This means the
2
Column sum of squares 𝑆𝑆𝑐 = ∑𝑛𝑗=1 𝑚(𝑋.𝑗 − 𝑋.. )
exponential model can explain more variation in Y.
False.
Variation between columns
𝑛
(c) In multiple regression, adding an additional
Error sum of squares 𝑆𝑆𝑒 = ∑𝑚
𝑖=1 ∑𝑗=1(𝑋𝑖𝑗 − 𝑋𝑖. − 𝑋.𝑗 +
2
regressor (input variable) will always increase the
𝑋.. ) Variation due to randomness, subtract mean, 𝛼𝑖 and
multiple R-square, and decrease the adjusted R-square.
𝛽𝑖 from 𝑋𝑖𝑗
False.
𝑆𝑆𝑒
an unbiased estimator for 𝜎 2
(d) In linear regression, a 95% prediction interval for a
(𝑚−1)(𝑛−1)
𝑆𝑆
future response given certain
2
Error: 2𝑒 ~Χ(𝑚−1)(𝑛−1)
𝜎
input level always contains the 95% confidence interval
𝑆𝑆
2
Row: if 𝐻0𝑟 is true, 2𝑟 ~Χ𝑚−1
, indep. Of 𝑆𝑆𝑒
for the mean response given the same input level. True.
𝜎
𝑆𝑆
(e) In one-way ANOVA, the null hypothesis states that
2
Col: 𝐼𝑓 𝐻0𝑐 is true, 2𝑐 ~Χ𝑛−1
, indep. Of 𝑆𝑆𝑒
𝜎
the population mean within each group is equal to zero.
ANOVA Table Let 𝑁 = (𝑚 − 1)(𝑛 − 1)
False.
source
sum of
d.f.
mean
F ratio
p-value
Having done poorly on their math final exams in June,
of var
squares
square
𝑆𝑆𝑟
𝑆𝑆𝑟 ⁄(𝑚 − 1)
row
𝑆𝑆𝑟
𝑚
𝑃{𝐹𝑚−1,𝑁 some students repeat the course in summer school, then
−1
≥ 𝐹 𝑟𝑎𝑡𝑖𝑜} take another exam in August. Assume these students are
𝑚−1
𝑆𝑆𝑒 /𝑁
𝑆𝑆𝑐
𝑆𝑆𝑐 ⁄(𝑛 − 1)
column
𝑆𝑆𝑐
𝑛
𝑃{𝐹𝑛−1,𝑁
representative of all students who might attend summer
−1
≥ 𝐹 𝑟𝑎𝑡𝑖𝑜}
𝑛−1
𝑆𝑆𝑒 /𝑁
school. Given their scores in June and August, explain
Error
𝑆𝑆𝑒
𝑁
𝑆𝑆𝑒 /𝑁
how you can test whether the summer program is
total
𝑆𝑆𝑟
𝑚𝑛
+ 𝑆𝑆𝑐
−1
worthwhile.
+ 𝐸𝐸𝑒
Use paired t-test. 𝜇𝑥 is mean score in June, 𝜇𝑦 mean
2-way ANOVA with Interaction
score in August: 𝐻0 : not effective 𝜇𝑦 = 𝜇𝑥 𝐻1 𝜇𝑦 > 𝜇𝑥
Without interaction, 𝜇1𝑗 − 𝜇2𝑗 is same for all 𝑗
Let Xi and Yi be the scores of ith student, and take the
𝜇𝑖1 − 𝜇𝑖2 same for all 𝑖
̅ = 1 ∑𝑖 𝐷𝑖 , and
difference Di=Yi-Xi. Sample mean 𝐷
Interaction 𝜇 = 𝜇 + 𝛼 + 𝛽 + 𝛾
𝑖𝑗
𝑖
𝑗
𝑛
𝑖𝑗
𝜇 = 𝜇.. : grand mean
𝛼𝑖 = 𝜇𝑖. − 𝜇.. effect of the ith row
Multiple R: 𝑅
𝛽𝑖 = 𝜇.𝑗 − 𝜇.. effect of the jth col
2
R Squared: 𝑅 = 1 − 𝑆𝑆𝑅 ⁄𝑆𝑌𝑌
Adjusted R Squared: 𝑅̅ 2 = ((𝑛 − 1)𝑅 2 − 𝑘)⁄(𝑛 − 𝑘 − 1) 𝛾𝑖𝑗 interaction of ith row and jth column
𝛾𝑖𝑗 = 𝜇𝑖𝑗 − 𝜇 − 𝛼𝑖 − 𝛽𝑗 = 𝜇𝑖𝑗 − 𝜇𝑖. − 𝜇.𝑗 + 𝜇..
Standard Error: 𝜎̂ = √𝑆𝑆𝑅 ⁄(𝑛 − 𝑘 − 1)
Data m rows, n columns, m*n cells, 𝑙 ≥2
Observations: 𝑛
observations/cell 𝑋𝑖𝑗 : sample mean of cell (𝑖,j)
𝑘
1 𝑥1 𝑥
Polynomial Regression 𝐵 = (𝑋 ′ 𝑋)−1 𝑋′𝑌, 𝑥 = [ |𝑥 | 1𝑘 ] 𝑋𝑖𝑗𝑙 denotes an individual number
1 𝑛 𝑥2
Assumptions 𝑋𝑖𝑗𝑘 = 𝜇𝑖𝑗 + 𝜀𝑖𝑗𝑘 = 𝜇 + 𝛼𝑖 + 𝛽𝑗 + 𝛾𝑖𝑗 + 𝜀𝑖𝑗𝑘
ANOVA: Analysis Of Variance
𝜀𝑖𝑗𝑘 Independent random error, ~𝑁(0, 𝜎 2 )
Study the impact of a certain factor
{𝑟,𝑐,𝑖𝑛𝑡}
𝐻0
: no {row,column,interaction}
𝐻0 : all means are equal
Estimation
𝑛𝑟 total number of observation
𝑋… : 𝑔𝑟𝑎𝑛𝑑 𝑚𝑒𝑎𝑛 𝜇
𝑋.. sample mean of all observation
•
𝑋𝑖.. : 𝑟𝑜𝑤 𝑚𝑒𝑎𝑛 𝜇𝑖.
𝑋𝑖𝑗 𝑗th observation in 𝑖th group
•
𝑋.𝑗. : 𝑐𝑜𝑙𝑢𝑚𝑛 𝑚𝑒𝑎𝑛 𝜇.𝑗
𝑋𝑖 . Sample mean of the 𝑖th group
2 𝑋 : 𝑐𝑒𝑙𝑙 𝑚𝑒𝑎𝑛 𝜇
Between-group sum of squares 𝑆𝑆𝑏 = ∑𝑚
𝑖𝑗.
𝑖𝑗
𝑖=1 𝑛𝑖 (𝑋𝑖. − 𝑋•.. )
Amount of variation that can be explained by groups•
Row effect 𝛼𝑖 = 𝜇𝑖. − 𝜇
Within-group sum of squares
•
estimator: 𝛼̂𝑖 = 𝑋𝑖.. − 𝑋…
2
𝑛𝑖
•
Column effect 𝛽𝑗 = 𝜇.𝑗 − 𝜇
𝑆𝑆𝑤 = ∑𝑚
𝑖=1 ∑𝑗=1(𝑋𝑖𝑗 − 𝑋𝑖. ) amount of var that can’t be
𝑆𝑆
•
estimator: 𝛽̂𝑗 = 𝑋.𝑗. − 𝑋…
explained by groups, 2𝑤 ~𝑋𝑛2𝑇 −𝑚 , 𝑆𝑆𝑏 and 𝑆𝑆𝑤 are indep.
𝜎
•
Interaction
𝛾
=
𝜇
−
𝜇 − 𝛼𝑖 − 𝛽𝑗
𝑖𝑗
𝑖𝑗
𝑆𝑆 ⁄(𝑚−1)
TS: 𝐹 = 𝑏⁄
~𝐹𝑚−1,𝑛𝑇 −𝑚 if 𝐻0 is true
•
estimator: 𝛾̂𝑖𝑗 = 𝑋𝑖𝑗. − 𝑋𝑖.. − 𝑋.𝑗. +
𝑆𝑆𝑤 (𝑛𝑡 −𝑚)
𝑋…
𝑝-value=𝑃{𝐹𝑚−1,𝑛𝑇 −𝑚 ≥ 𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑇𝑆}
Error 𝜀𝑖𝑗𝑘 = 𝑋𝑖𝑗𝑘 − 𝜇𝑖𝑗 estimator 𝑋𝑖𝑗𝑘 − 𝑋𝑖𝑗.
ANOVA table for multiple regression 𝐻0 : Y doesn’t •
2
𝑛
𝑙
depend on input
•
Error sum of squares 𝑆𝑆𝑒 = ∑𝑚
𝑖=1 ∑𝑗=1 ∑𝑘=1(𝑋𝑖𝑗𝑘 − 𝑋𝑖𝑗. )
Reading Excel
source
of var
between
groups
within
groups
total
sum of
squares
𝑆𝑆𝑏
𝑆𝑆𝑤
𝑆𝑆𝑏
+ 𝑆𝑆𝑤
d.f.
𝑚
−1
𝑛𝑇
−𝑚
𝑛𝑇
−1
mean
square
𝑆𝑆𝑏
𝑚−1
𝑆𝑆𝑊
𝑛𝑇 − 𝑚
F ratio
p-value
𝑆𝑆𝑒
𝑆𝑆𝑏 ⁄(𝑚 − 1)
𝑆𝑆𝑊 ⁄(𝑛 𝑇 − 𝑚)
𝑃{𝐹𝑘,𝑛−𝑘−1
•
≥ 𝐹 𝑟𝑎𝑡𝑖𝑜}
𝑛
Interaction sum of squares 𝑆𝑆𝑖𝑛𝑡 = ∑𝑚
𝑖=1 ∑𝑗=1 𝑙(𝑋𝑖𝑗. −
•
Two-Factor ANOVA
One-way: single factor
Ex. gas mileage: difference between cars and with
drivers?
Each row/column represents one level
Column averages 𝑋.1 … 𝑋.𝑛
Row averages 𝑋1. … 𝑋𝑚. , overall average 𝑋..
•
•
𝜎2
∼
𝑆𝑆𝑒
2
𝜒𝑚𝑛(𝑙−1)
,
𝑚𝑛(𝑙−1)
2
is an unbiased estimator for 𝜎
𝑆𝑆
2
𝑋𝑖.. − 𝑋.𝑗. + 𝑋… ) , Under 𝐻0𝑖𝑛𝑡 , 𝑖𝑛𝑡
∼ 𝜒(𝑚−1)(𝑛−1)
𝜎2
Row sum of squares
2
•
𝑆𝑆𝑟 = ∑𝑚
𝑖=1 𝑛𝑙(𝑋𝑖.. − 𝑋… )
𝑟 𝑆𝑆𝑟
2
•
Under 𝐻0 , 2 ∼ 𝜒𝑚−1
𝜎
Column sum of squares
2
•
𝑆𝑆𝑐 = ∑𝑛𝑗=1 𝑚𝑙(𝑋.𝑗. − 𝑋… )
•
Under 𝐻0𝑐 ,
𝑆𝑆𝑐
𝜎2
2
∼ 𝜒𝑛−1
2
sample sd 𝑆𝐷 = √
1
𝑛−1
̅)
∑𝑖(𝐷𝑖 − 𝐷
2
̅ /𝑆𝐷 is 𝑡𝑛−1 𝑝 = Pr{𝑡𝑛−1 ≥ 𝑇𝑆}
TS: 𝑇 = √𝑛𝐷
3. In a study of the relationship between the starting
salary (Y) and GMAT score (x)
of n = 100 MBA students, the following results are
obtained: average starting salary
avg(Y) = $90,000, sample standard deviation s(Y) =
$45,000, average GMAT avg(x) = 600,
sample standard deviation s(x) = 100, sample
correlation coefficient r = 0.3.
(a) (10 pts) Find and interpret the slope of the simple
linear regression of Y on x.
𝑟𝑆𝑌
𝑏̂ =
= 135
𝑆𝑥
𝑎̂ = 𝑌̅ − 𝑏̂𝑥̅ =90000-135*60=9000
Y=9000+135x
(b) (10 pts) Find a 95% confidence interval for the
mean salary of people who got 700 in GMAT.
Get 100(1 − 𝛾) C.I. for 𝛼 + 𝛽𝑥0
𝐴 + 𝐵𝑥0 − (𝛼 + 𝛽𝑥0 )
~𝑡𝑛−2
1 (𝑥 − 𝑥̅ )2 √ 𝑆𝑆𝑅
√ + 0
𝑛
𝑆𝑥𝑥
𝑛−2
1
(𝑥0 −𝑥̅ )2
𝑛
𝑆𝑥𝑥
Use 𝐴 + 𝐵𝑥0 ± √ +
√
𝑆𝑆𝑅
𝑡𝛾
𝑛−2 2,𝑛−2
(c) (10 pts) Test at 0.05 level on whether higher GMAT
score means higher salary.
𝐻0 : 𝛽 = 0 mean salary doesn’t increase GMAT
(𝑛−2)𝑆𝑥𝑥
TS is 𝑡𝑛−2 , 𝑇𝑛−2 = √
𝑆𝑆𝑅
𝐵
(𝑛−2)𝑠𝑥𝑥
p-value = 𝑃(𝑇𝑛−2 > √
𝑠𝑠𝑅
𝑏̂
Since 𝑡98 close to standard normal, know p=1-Φ(3)
(d) (15 pts) Construct the ANOVA table for linear
regression.
(e) (10 pts) Simple linear regression of log(Y) on x
yields slope = 0.015%, and that
of Y on log(x) yields slope = 8000. Interpret the results.
If GMAT incrases by 1, starting salary increases by
.015% factor on average.
For slope=8000: if GMAT score increases by factor of
1%, salary increases by 8000*.01=80 on average.
An emergency room physician wanted to know whether
there were any differences in the amount of time it takes
for three different inhaled steroids to clear a mild
asthmatic attack. Over a period of weeks she randomly
administered these steroids to asthma sufferers, and
noted the time it took for the patients’ lungs to become
clear. Afterward, she discovered that 12 patients had
been treated with each type of steroid, with the
following sample means (in minutes) and sample
variances resulting. Construct the ANOVA table to test
on whether the mean time to clear a mild asthma attack
is the same for all three steroids.
Steroid
Sample mean
Sample
variance
A
32
145
B
40
138
C
30
150
𝑚=3, n=12, 𝜇𝑖 is mean time to clear attack for ith
steroid
𝑆𝑆 ⁄(𝑚−1)
|𝐻0 : all means are equal, 𝑏
~𝐹𝑚−1,𝑛𝑚−𝑚
𝑆𝑆𝑤 /(𝑛𝑚−𝑚)
𝑠𝑠𝑏 = 𝑛 ∑𝑚
̅𝑖 − 𝑥̅̅ )2 = 12 ∗ [(32 − 34)2 + (40 − 34)2 + (30 − 34)2 ] =
𝑖=1(𝑥
672
2
𝑛
𝑚
2
𝑠𝑠𝑤 = ∑𝑚
𝑖=1 ∑𝑗=1(𝑥𝑖𝑗 − 𝑥̅𝑖 ) =∑𝑖=1(𝑛 − 1)𝑠𝑖 =
11(145 + 138 + 150) = 34
source
of
varianc
e
betwee
n
groups
within
groups
total
sum of squares
d.f
.
mean
square
F ratio
P-val
672
2
336
𝐹𝑚−1,𝑛𝑚−𝑚
𝑃(𝐹2,33
≥𝐹
− 𝑟𝑎𝑡𝑖𝑜)
33
144.333
3
672+4763=543
5
35