Q1 ‐ 1 pt): First and last name of all group members (up to four members per group): Quiz 04 SOLUTIONS , SOLUTIONS , SOLUTIONS , SOLUTIONS . Show your work and write numeric answers to 4 decimals. Don't round your calculations until the very end. We will deduct points for rounding errors. Please include R/Stata code when showing your work. This quiz/workshop will help you practice with the ideas of likelihood inference in comparison to frequentist methods, particularly one‐sample hypothesis testing. Create an electronic document (pdf, doc, docx, etc.) and email your group's solutions with the subject heading "Quiz 4 ‐ [YOUR NAMES]", e.g. "Quiz 4 ‐ Robert, Laurie, Jeffrey", to me by midnight this Friday (hopefully sooner, the extra time is just in case you need it). Image used w/out permission from http://reliawiki.org/images/thumb/6/6c/Apa_fig3.png/350px‐Apa_fig3.png Q2 ‐ 5 pts) Suppose a hospital has experienced 30 sepsis infections so far this year. Nationwide, comparable hospitals are averaging 36 infections so far this year. Perform a classical hypothesis test of whether this hospital is performing differently than the national average of comparable hospitals. Include a Ho, Ha, TS, RR, p‐value, conclusion, and 95% confidence interval. Ho: μ = 36 Ha: μ ≠ 36 TS: X = 30 When calculating the p‐value straight from the Poisson distribution, the test statistic is simply the observed X. RR: Note that the rejection region depends only on the null hypothesis and the significance level. It does not depend on the observed X at all. Assuming α = 0.05, the RR is the region X would need to land in order to reject at a 5% level. To find it, let's try calculating the p‐value at a some potential values of X. 2*ppois(24,36) = 0.045 ‐‐ Reject Ho 2*ppois(25,36) = 0.069 ‐‐ Don't reject Ho 2*ppois(47,36,lower.tail=F) = 0.064 ‐‐ Don't reject Ho 2*ppois(48,36,lower.tail=F) = 0.045 ‐‐ Reject Ho So the RR is X ≤ 24 and X ≥ 48. p‐value: round( 2*ppois(30,36), 4 ) = 0.3613 Conclusion: At a 5% significance level, we do not reject the hypothesis that the hospitals true rate of infection is 36. Put more naturally, while the hospital's current infection rate is 17% lower than the national average, it remains plausible that the hospital's true rate is the same as the national average. 95% CI: Notice that the CI depends on the data observed and has nothing to do with the null hypothesis, as opposed to the RR which has nothing to do with the data observed and depends completely on the null hypothesis. Rosner's table 8 gives (20.24, 42.83). If we wanted greater precision, we could solve for the 1 bounds using the trial and error approach we used for the exact Binomial CI. However, because of a relationship between the Poisson and the Chi‐squared, there is a simple formula for the exact Poisson CI. The alpha=a level CI for a Poisson with observed X=x is: ( qchisq(a/2,2*x)/2, qchisq(1‐a/2,2*x+2)/2 ). Thus, for X=30 we have: a = 0.05; x = 30; round( c( qchisq(a/2,2*x)/2, qchisq(1‐a/2,2*x+2)/2 ), 4 ) (20.2409, 42.8269) Q3 ‐ 5 pts) Suppose the hospital's goal is for their true rate of infection to be at least 15% lower than the national average. What is the evidence that they are meeting this goal? Include a H1, H2, LR, conclusion, and 1/8th support interval. H1: μ = 36 H2: μ = 0.85*36 = 30.6 LR: L(36)/L(30.6) = P(X=30|μ=36)/P(X=30|μ=30.6) = dpois(30,36)/dpois(30,30.6) = 0.5918914 = 0.5919 (or for H2 vs H1 the LR = 1.6895) Conclusion: There is weak evidence in favor of the true infection rate being 30.6 over 36 by a factor of 1.7. 1/8th SI: The 1/8th SI will be fairly close to the 95% CI, so we know where to focus our search. I'll search in increments of 0.00001 to ensure I get a solution accurate to four decimal places. # Set up the likelihood function # pdf = function(x,mu){ dpois(x,mu) } # likelihood = pdf where x is fixed likelihood = function(mu){ dpois(30,mu) } # Try a bunch of values for mu # the very small delta will give four decimal # places of accuracy, but takes a minute to run delta = 0.00001 mus = seq(15, 50, delta) ls = likelihood(mus) # rescale to maximum likelihood value mls = max(ls) ls = ls/mls plot( mus, ls, typ='l', lwd=3 ) # 1/8th Interval lowerIndex <‐ min(which(ls >= 1/8)) 2 upperIndex <‐ max(which(ls >= 1/8)) lb <‐ mus[lowerIndex] ub <‐ mus[upperIndex] lines( c(lb, ub), c(1/8,1/8), lwd=3, col="blue" ) print( round( c(lb, ub), 4 ) ) (20.1712, 42.5971) Important aside and a hat tip to Min, Emily, and Danielle who proposed a form of this question: What are the regions of moderate and strong evidence for the likelihood inference approach in this setting? In other words, what values of X would have given us LR in the 1/8th range and what values in the 1/32nd range? This is analogous to asking what is the rejection region for a hypothesis test. It depends only on the hypothesized values and doesn't use the observed data at all. It asks what data would you need to observe to have moderate or strong evidence in favor of one hypothesis over the other? I solved for the regions using a quick trial and error approach. > 1/8 [1] 0.125 > x=21; dpois(x,36)/dpois(x,30.6); [1] 0.1370921 ‐‐ still weak evidence for H2 over H1 > x=20; dpois(x,36)/dpois(x,30.6); [1] 0.1165283 ‐‐ now moderate evidence for H2 over H1 > 8 [1] 8 > x=46; dpois(x,36)/dpois(x,30.6); [1] 7.971485 ‐‐ still weak evidence for H1 over H2 > x=47; dpois(x,36)/dpois(x,30.6); [1] 9.378217 ‐‐ now moderate evidence for H1 over H2 > 1/32 [1] 0.03125 3 > x=13; dpois(x,36)/dpois(x,30.6); [1] 0.03735629 ‐‐ still moderate evidence for H2 over H1 > x=12; dpois(x,36)/dpois(x,30.6); [1] 0.03175285 ‐‐ now strong evidence for H2 over H1 > 32 [1] 32 > x=54; dpois(x,36)/dpois(x,30.6); [1] 29.25417 ‐‐ still moderate evidence for H1 over H2 > x=55; dpois(x,36)/dpois(x,30.6); [1] 34.41667 ‐‐ now strong evidence for H1 over H2 So the regions of evidence for these hypothesized values are: Weak: [21, 46] Moderate: [13, 20] and [47, 54] Strong: (‐∞, 12] and [55, ∞) Q4 ‐ 5 pts) Redo Q2 using a normal approximation for the Poisson. Ho: μ = 36 Ha: μ ≠ 36 (exactly the same as Q2) TS, RR, p‐value Solution 1 (chi‐squared formula from Rosner): TS: (30‐36)^2 / 36 = 1 The TS ~ χ21df is Ho is true. The formula is from Rosner and it is based on the square of a standard normal is chi‐squared with 1 degree of freedom. Because we are squaring the difference, we are constructing a test that always has a two‐sided alternative (and only uses the upper tail of the null distribution to calculate the p‐
value). For comparison, we'll calculate the more familiar form of the normal distribution below. RR: Here the rejection region is based on the chi‐squared distribution. Reject if TS ≥ qchisq(1‐0.05,1) = 3.8415 p‐value: pchisq(1,1,lower.tail=F) = 0.3173 (as compared to 0.3613 using the Poisson distribution) TS, RR, p‐value Solution 2 (the Normal approximation in more familiar form): TS: (30‐36) / sqrt(36) = ‐1 This is the more familiar (X‐μ)/SE form. The SE is the standard error under Ho, and for the Poisson assuming μ also assumes the SE, i.e. sqrt(μ). The TS ~ Z if Ho is true. RR: Here the rejection region is based on the Normal distribution. Reject if |TS| ≥ qnorm(1‐0.05/2) = 1.9600. (The absolute value of the TS is used because of the two‐sided alternative.) 4 p‐value: 2*pnorm(1,lower.tail=F) = 0.3173 (Naturally, this is exactly the same as the chi‐squared formulation. The pnorm is only counting the upper tail so we need to multiply by 2 to count both tails.) TS, RR, p‐value Solution 3 (the Normal approximation with a continuity correction): TS: (30.5‐36) / sqrt(36) = ‐0.9166667 To do the continuity correction, we add or subtract 1/2 from the observed value so we essentially capture the full [29.5, 30.5] box in the histogram the Normal is approximating. I don't recommend trying to create a rule to decide whether to add or subtract, but rather to draw the picture and reason it out. Here, the p‐value should include P(X=0)+P(X=1)+…+P(X=30), so we want the left tail of the Normal approximation from 30.5 over. RR: Reject if |TS| ≥ qnorm(1‐0.05/2) = 1.9600. (The continuity correction doesn’t change the RR.) p‐value: 2*pnorm(0.9166667,lower.tail=F) = 0.3593 (as compared to 0.3613 using the Poisson distribution, so the approximation with a continuity correction is pretty good in this case) Solutions 1‐3: Conclusion: At a 5% significance level, we do not reject the hypothesis that the hospitals true rate of infection is 36…. (exactly the same answer as Q2) 95% CI Solution A: 95% CI: Using a framework of (point estimate + Z.025*SE), we get: ( 30 + qnorm(1‐0.025)*sqrt(30) ) ‐‐ notice we use the observed point estimate to estimate the SE, we don't use the null hypothesis at all when calculating the CI. > round( c( 30‐qnorm(1‐0.025)*sqrt(30), 30+qnorm(1‐0.025)*sqrt(30) ), 4) (19.2648, 40.7352) ‐‐ as compared to (20.2409, 42.8269), which is off by about 1 on both bounds but isn't bad 95% CI Solution B: A hat tip to Xue for the idea of using the Normal approximation to estimate the bounds of the exact Poisson CI. Recall the formulae for the exact CI for the Poisson (Rosner's equation 6.23). Applied to our case we have the following. Lower bound ‐ find the μlb that satisfies P(X≥30|μ=μlb) = 0.025 /30! , where f(x) ~ N(μlb, sqrt(μlb)). .
Note the continuity correction takes summing from 30 to ∞ and maps it to integra ng from 29.5 to ∞. This equation is equivalent to solving for the μlb that satisfies (29.5‐μlb)/sqrt(μlb) = Z1‐0.025 = 1.9600. (29.5‐μlb)^2/μlb = Z0.9752 μlb^2 ‐ (59+Z0.9752) μlb + 29.5^2 = 0 mu^2 ‐ (59+qnorm(0.975)^2)*mu + 29.5^2 = 0 5 mu^2 ‐ 62.8414588*mu + 870.25 = 0 This is a job for the quadratic formula; (62.8414588 ‐ sqrt(62.8414588^2‐4*870.25))/2 = 20.6035. Or alternately by WolframAlpha.com, which yields μlb = 20.6035. To solve for the upper bound, follow the same approach while being careful to apply the continuity correction in the other direction, i.e. summing from 0 to 30 maps to integrating from ‐∞ to 30.5. mu^2 ‐ (61+qnorm(0.975)^2)*mu + 30.5^2 = 0 mu^2 ‐ 64.8414588*mu + 930.25 = 0 μub = 43.4141 (20.6035, 43.4141) ‐‐ as compared to (20.2409, 42.8269), which isn't bad. Without applying the continuity correction, we have mu^2 ‐ (60+qnorm(0.975)^2)*mu + 30^2 = 0 mu^2 ‐ 63.8414588*mu + 900 = 0 (21.0151, 42.8264) ‐‐ as compared to (20.2409, 42.8269), which also isn't bad. Q5 ‐ 5 pts) Redo Q3 using a normal approximation for the Poisson. H1: μ = 36 H2: μ = 0.85*36 = 30.6 (no change from Q3) LR: L(36)/L(30.6) = P(X=30|μ=36)/P(X=30|μ=30.6) = dnorm(30,36,sqrt(36))/dnorm(30,30.6,sqrt(30.6)) = 0.5624927 = 0.5624 (or for H2 vs H1 the LR = 1.7778) (as compared to 0.5919 (1.6895) using the Poisson distribution) Solution 2: We could do something similar to the continuity correction approach (hat tip to Drew, Greg, and Amanda for this general idea). I'm neither recommending or discouraging this; just noting it is a neat idea. (pnorm(30.5,36,sqrt(36))‐pnorm(29.5,36,sqrt(36)))/(pnorm(30.5,30.6,sqrt(30.6))‐pnorm(29.5,30.6,sqrt(30.6))) = 0.5632 Conclusion: There is weak evidence in favor of the true infection rate being 30.6 over 36 by a factor of 1.8. (as compared to the 1.7 factor using the Poisson distribution) 1/8th SI: # While the Normal is a two‐parameter model, we are really only varying one parameter, mu: likelihood = function(mu){ dnorm(30,mu,sqrt(mu)) } delta = 0.00001 mus = seq(15, 50, delta) ls = likelihood(mus) 6 # rescale to maximum likelihood value mls = max(ls) ls = ls/mls plot( mus, ls, typ='l', lwd=3 ) # 1/8th Interval lowerIndex <‐ min(which(ls >= 1/8)) upperIndex <‐ max(which(ls >= 1/8)) lb <‐ mus[lowerIndex] ub <‐ mus[upperIndex] lines( c(lb, ub), c(1/8,1/8), lwd=3, col="blue" ) print( round( c(lb, ub), 4 ) ) (20.3834, 42.7382) ‐‐ as compared to (20.1712, 42.5971) using the Poisson distribution Q6 ‐ 5 pts) Suppose the hospital had 30 sepsis infections out of 30,000 patients. Redo Q2 using a Binomial distribution. Ho: θ = 36/30000 = 0.0012 Ha: θ ≠ 36/30000 = 0.0012 TS: X = 30 When calculating the p‐value straight from the Binomial distribution, the test statistic is the observed X. If using a Normal approximation for the Binomial, your TS could be (x/n ‐ θ)/sqrt(θ*(1‐θ)/n). Just x/n doesn't have a distribution you can directly evaluate; that is you either relate the distribution of x/n to the distribution of x and use the binomial, relate x/n to the Normal and standardize it, or relate it to another distribution with some transformation. It's a technical detail, but worth thinking about to understand the purpose of your TS. Your TS is usually not your point estimate for your effect. It has a different purpose, namely determining statistical significance. RR: Assuming α = 0.05, the RR is the region X would need to land in order to reject at a 5% level. To find it, let's try calculating the p‐value at a some potential values of X. 2*pbinom(24,30000,0.0012) = 0.045 ‐‐ Reject Ho 7 2*pbinom(25,30000,0.0012) = 0.069 ‐‐ Don't reject Ho 2*pbinom(47,30000,0.0012,lower.tail=F) = 0.064 ‐‐ Don't reject Ho 2*pbinom(48,30000,0.0012,lower.tail=F) = 0.045 ‐‐ Reject Ho So the RR is X ≤ 24 and X ≥ 48. The RR is exactly the same as it was for the Poisson. p‐value: round( 2*pbinom(30,30000,0.0012), 4 ) = 0.3609 This compares to 0.3613 when using the Poisson. Why are they so close? Important note about Poisson distributions. There is a misnomer about a variable needing a low rate of occurrence to be distributed as Poisson. A Poisson does not need to have a low rate of occurrence. The rate is really relative to the amount of time or patients or whatever observed. A Poisson may have a very large rate, either because it is a frequent event or because a lot of time/patients/whatever have been observed. When a Poisson does have a large rate, it is well approximated by a Normal. However, when a Binomial has a low probability, it is well approximated by a Poisson. Why? Because when the probability is small enough relative to the size of n, the exact value of n becomes relatively inconsequential. Consider doubling or quadrupling the n in this problem. Do you see much change? > 2*pbinom(48,30000,36/30000,lower.tail=F) = 0.04494536 ‐‐ n = 30,000 > 2*pbinom(48,60000,36/60000,lower.tail=F) = 0.04501316 ‐‐ n = 60,000 > 2*pbinom(48,120000,36/120000,lower.tail=F) = 0.04504706 ‐‐ n = 120,000 In fact, this illustration is a little deceptive because the point it's making is very useful. For Binomial distributions where the values of choose(n,x) are too big for the common computer, many programs will automatically employ approximate solutions including ones based on the Poisson. It is likely that R is employing such approximations above. See Loader, Catherine. Fast and Accurate Computation of Binomial Probabilities. 2000. Conclusion: At a 5% significance level, we do not reject the hypothesis that the hospitals true rate of infection is 36. Put more naturally, while the hospital's current infection rate is 17% lower than the national average, it remains plausible that the hospital's true rate is the same as the national average. 95% CI: library(Hmisc) options(scipen=30) round( binconf( 30, 30000, method="exact" ), 4) (0.0007, 0.0014) ‐‐ Note this isn't comparable to the other answers, so let's rescale back to a rate. round( 30000* binconf( 30, 30000, method="exact" ), 4) (20.2438, 42.8177) ‐‐ as compared to (20.2409, 42.8269) when using the Poisson, identical to two decimals Q7 ‐ 5 pts) Suppose the hospital had 30 sepsis infections out of 30,000 patients. Redo Q3 using a Binomial distribution. H1: θ = 36/30000 = 0.0012 H2: θ = 0.85*36/30000 = 0.00102 LR: L(36)/L(30.6) = P(X=30|μ=36)/P(X=30|μ=30.6) = dbinom(30,30000,0.0012)/dbinom(30,30000,0.00102) = 0.5915395 = 0.5915 (or for H2 vs H1 the LR = 1.6905) (as compared to 0.5919 (1.6895) using the Poisson distribution) 8 Conclusion: There is weak evidence in favor of the true infection rate being 30.6 over 36 by a factor of 1.7. 1/8th SI: The 1/8th SI will be fairly close to the 95% CI, so we know where to focus our search. I'll search in increments of 0.00001 to ensure I get a solution accurate to four decimal places. # Solution 1 ‐‐ on the same scale as the other solutions: # Set up the likelihood function likelihood = function(mu){ dbinom(30,30000,mu/30000) } # Try a bunch of values for mu delta = 0.00001 mus = seq(15, 50, delta) ls = likelihood(mus) # rescale to maximum likelihood value mls = max(ls) ls = ls/mls plot( mus, ls, typ='l', lwd=3 ) # 1/8th Interval lowerIndex <‐ min(which(ls >= 1/8)) upperIndex <‐ max(which(ls >= 1/8)) lb <‐ mus[lowerIndex] ub <‐ mus[upperIndex] lines( c(lb, ub), c(1/8,1/8), lwd=3, col="blue" ) print( round( c(lb, ub), 4 ) ) (20.1745, 42.5882) ‐‐ as compared to (20.2409, 42.8269) when using the Poisson, not as close as the normal, but still quite close. # reverting back to the proportion scale round( c(lb, ub)/30000, 4 ) = (0.0007, 0.0014), which is the same as the exact interval to the accuracy shown 9 Fun aside: how do the three likelihood curves compare to each other graphically? # Set common values for the plots # Note, I'd usually have to fix the axis limits with xlim and ylim, but # because the plots all have the same range for X and Y, I can skip that. delta = 0.00001 mus = seq(15, 50, delta) # Poisson likelihood = function(mu){ dpois(30,mu) } ls = likelihood(mus) mls = max(ls) ls = ls/mls plot( mus, ls, typ='l', lwd=3, col="blue" ) lowerIndex <‐ min(which(ls >= 1/8)) upperIndex <‐ max(which(ls >= 1/8)) lb <‐ mus[lowerIndex] ub <‐ mus[upperIndex] lines( c(lb, ub), c(1/8,1/8), lwd=3, col="blue" ) # Normal likelihood = function(mu){ dnorm(30,mu,sqrt(mu)) } ls = likelihood(mus) mls = max(ls) ls = ls/mls par(new=T) plot( mus, ls, typ='l', lwd=3, col="chocolate2" ) lowerIndex <‐ min(which(ls >= 1/8)) upperIndex <‐ max(which(ls >= 1/8)) lb <‐ mus[lowerIndex] ub <‐ mus[upperIndex] lines( c(lb, ub), c(1/8,1/8), lwd=3, col="chocolate2" ) # Binomial likelihood = function(mu){ dbinom(30,30000,mu/30000) } ls = likelihood(mus) mls = max(ls) ls = ls/mls par(new=T) plot( mus, ls, typ='l', lwd=3, col="chartreuse4" ) lowerIndex <‐ min(which(ls >= 1/8)) upperIndex <‐ max(which(ls >= 1/8)) lb <‐ mus[lowerIndex] ub <‐ mus[upperIndex] lines( c(lb, ub), c(1/8,1/8), lwd=3, col="chartreuse4" ) 10 The Binomial and Poisson overlap perfectly (you can only see the Binomial's green line above). I even tried reducing the lwd to 1 and still couldn't distinguish them. The Normal (in chocolate above) is also very close to the others. Learning objectives aside: The way these distributions approximate each other nicely in this setting is fun, but is mainly important when it comes to understanding the common hypothesis tests that use these approximations. Most often when an approximation is used in a test statistic, it will be a Normal approximation of another distribution. In this example, the distributions' approximations agreed nicely. This will not always be the case. Getting comfortable with the common pieces of a hypothesis test (the Ho, Ha, TS, RR, p‐value, conclusion, and CI) will help with employing and understanding the many different tests we use. The side by side comparison of the likelihood inference approach with the hypothesis testing approach should highlight the similarities and differences between these two statistical paradigms. Note that the hypothesis test compares one null value versus everything else. The likelihood approach, as used here, compares two pre‐
specified values. In this example, the decisions made by the hypothesis tests were consistent with the evidence shown by the likelihood inference. This will not always be the case. 11