Appendix E — Problem sets

What follows are exemplar long- and short-answer questions for the material in this course. The questions are synoptic in nature, particularly the long-answer questions, and do not fit neatly into chapter boundaries. This reflects the reality of how statisticians have to operate. Please also note that I normally set exam questions with multiple parts; thus, for example, a ‘single’ short-answer question in your exam may contain multiple questions (e.g., a single question intended to take ten minutes could contain several of the questions below). Please note that you are also given questions throughout this handout at the end of each section, and you are encouraged to make use of those as well. I expect you to focus on these during the problem set session in class, hence answers are not given because I expect you to be speaking with me about them!

E.1 Short answer questions

You run a t-test in R to compare the distributions of two variables (x and y). The output you get is shown below. What can you say to compare the two distributions? Provide evidence to support your answer.

> t.test(x, y)
Welch Two Sample t-test
data: x and y
t = 1.4076, df = 17.709, p-value = 0.1766
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.2501425 1.2622486
sample estimates:
mean of x mean of y
0.8407376 0.3346845

You have conducted a study on the sugar content of Skittles. Below is the output of a multiple regression of a continuous response variable (sugar) against both a continuous explanatory variable (weight) and a categorical/discrete explanatory variable (color). What, to one decimal place, is the mean sugar content of a red Skittle?

> summary(lm(sugar ~ color + weight))
Call:
lm(formula = sugar ~ color + weight)
Residuals:
Min 1Q Median 3Q Max
-0.8027 -0.5091 0.2513 0.4068 0.6345
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.07934 0.28625 0.277 0.7823
colorred 0.41976 0.09952 2.208 0.0296 *
weight 2.54078 0.35643 7.128 1.81e-10 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 0.4828 on 97 degrees of freedom
Multiple R-squared: 0.3453, Adjusted R-squared: 0.3318
F-statistic: 25.58 on 2 and 97 DF, p-value: 1.195e-09

The Akaike Information Criterion, or AIC, is a metric of a model’s performance given the number of parameters it uses. It is mathemetically defined as $AIC = 2(k - \log(likelihood))$. Why is everything in it multiplied by 2? Select one of the following:

There are two terms in AIC (k and the log-likelihood), and so we must scale it accordingly.
Standard errors are two log-likelihood units from the Maximum Likelihood Estimate.
There are two bounds (upper and lower) to the confidence intervals around a model’s Maximum Likelihood Estimate.
Each parameter is calculated with reference to the intercept, and so there are two calculations involved in each parameter.

Which of the following is true about your choice of priors?

Your choice of prior really doesn’t matter at all.
Using a canonical prior will bias your answer.
You should check that your prior doesn’t bias your answer.
If your data don’t match your prior you must change your prior.

The figure below is an MCMC trace of the intercept parameter from a model fit using rstanarm, covering the last 50% of the MCMC. Which of the following statements is true?

The median posterior estimate of this parameter is approximately 0 (SD is approximately 1).
This chain has converged; it is a “hairy caterpillar”.
This chain has not converged; it is not a “hairy caterpillar”.
This chain is too overdispersed for meanginful inference.

E.2 Long answer questions

You make cell count measurements from five petri-dishes with 2, 3, 0, 3, and 1 cells in them. What is the probability of observing such data given a Poisson distribution with a mean of 2? Show your working. Use the probability-density (likelihood) table for various observations ($x$) from a Poisson distribution with a given $\lambda$ below to help you.

	$\lambda=1$	$\lambda=2$	$\lambda=3$	$\lambda=4$	$\lambda=5$
x=0	0.37	0.14	0.05	0.02	0.01
x=1	0.37	0.27	0.15	0.07	0.03
x=2	0.18	0.27	0.22	0.15	0.08
x=3	0.06	0.18	0.22	0.20	0.14
x=4	0.02	0.09	0.17	0.20	0.18
x=5	0.00	0.04	0.10	0.16	0.18

Michael approaches you, saying that five petri-dishes is not enough observations for a publishable study, to which your supervisor responds that they have a strong expectation of an average of 4 cells in each petri-dish. Use the information you have at hand to develop a test statistic for use with these kinds of data, using what you know about the Poisson distribution and its variance. Importantly, justify your statistic: there is no single ‘correct’ answer to this question, and your reasoning matters as much as your response.
Your supervisor calls security and gets Michael removed, after which he suggests that it would be best to model these data as from something called a ‘hurdle’ distribution. In this case, it means you first calculate the probability that any cells will grow at all and then, and then if the cells do grow (they get over the ‘hurdle’), then their counts are drawn from a Poisson distribution. They propose you use a 50:50 chance of the cells in a dish growing. Estimate the likelihood of your data being drawn from such a hurdle model.
AIC is defined as $2k - 2\log(likelihood)$, where $k$ is the number of parameters in a model. Use this equation to calculate the AIC of your models in (a) and (b). Outline which model is best, the relative support for your statement, and your reasoning for it.

A student comes to you for advice on a statistical analysis. They have grown replicates of the same cell culture under all combinations of different temperatures ($5^\circ$, $10^\circ$, $15^\circ$, and $20^\circ$) and pH (5, 6, 7, 8, and 9). They have conducted 5 replicates of each treatment, such that there are $4 \times 5 \times 5 = 100$ replicates in total, and they have counted the number of cells in each culture after 5 days of growth.

The student fits the following model to their data, and tells you that it proves that there is a strong effect of temperature on their data. Do you agree? Why?

> summary(lm(count ~ temp, data=data))

Call:
lm(formula = count ~ temp, data = data)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.6142 -0.9198 -0.1707  1.0537  3.6970 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 14.26791    0.39829   35.82   <2e-16 ***
temp        -0.72069    0.02909  -24.78   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.626 on 98 degrees of freedom
Multiple R-squared:  0.8623,    Adjusted R-squared:  0.8609 
F-statistic: 613.9 on 1 and 98 DF,  p-value: < 2.2e-16

You advise the student to use information-theoretic criteria to better understand their data. They use the R package MuMIn to do so, and generate the following output. They state that it proves that temperature has a strong effect on their cell counts. How do you interpret this output? Provide support for your answer.

> dredge(lm(count ~ temp*ph, data=data, na.action="na.pass"))
Fixed term is "(Intercept)"
Global model call: lm(formula = count ~ temp * ph, data = data, na.action = "na.pass")
---
Model selection table 
     (Int)     ph       tmp  ph:tmp df   logLik  AICc  delta weight
8  0.03086 2.0340  0.008441 -0.1042  5 -133.279 277.2   0.00      1
4  9.14500 0.7318 -0.720700          4 -162.824 334.1  56.87      0
3 14.27000        -0.720700          3 -189.496 385.2 108.05      0
2  0.13640 0.7318                    3 -285.716 577.7 300.48      0
1  5.25900                           2 -288.646 581.4 304.22      0
Models ranked by AICc(x)

The student decides to follow a Bayesian approach, where $y$ is the count of cells and is drawn according to: \[y \sim Normal(\mu, \sigma^2)\]

…their regression equation ($\mu$) is defined according to: \[\mu = a + bx + cz + dxz\]

…where $x$ is the temperature and $z$ is the pH, and the other terms coefficients whose priors are: \[a,b,c,d \sim F(2,2)\]

. The student is unhappy, because their model diagnostics suggest that the model is not converging properly, and the values for the interaction term ($d$) are only ever positive, which is at odds with their frequentist model (see part b). What is wrong with their model specification that is causing this problem? d. The student begs you to outline for them a final, definitive way that they can test for an impact of temperature on their data. Explain to them what to do and, most importantly, justify your approach. e. The student’s supervisor points out that they have now conducted four different kinds of models, and so must account for multiple-testing biases. If such problems affect your answer to part (d) above, outline a solution, and if not, justify why to the supervisor.

# Problem sets What follows are exemplar long- and short-answer questions for the material in this course. The questions are synoptic in nature, particularly the long-answer questions, and do not fit neatly into chapter boundaries. This reflects the reality of how statisticians have to operate. Please also note that I normally set exam questions with multiple parts; thus, for example, a 'single' short-answer question in your exam may contain multiple questions (*e.g.*, a single question intended to take ten minutes could contain several of the questions below). Please note that you are also given questions throughout this handout at the end of each section, and you are encouraged to make use of those as well. I expect you to focus on these during the problem set session in class, hence answers are not given because I expect you to be speaking with me about them! ## Short answer questions 1. You run a t-test in `R` to compare the distributions of two variables (x and y). The output you get is shown below. What can you say to compare the two distributions? Provide evidence to support your answer. ``` > t.test(x, y) Welch Two Sample t-test data: x and y t = 1.4076, df = 17.709, p-value = 0.1766 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -0.2501425 1.2622486 sample estimates: mean of x mean of y 0.8407376 0.3346845 ``` 2. You have conducted a study on the sugar content of Skittles. Below is the output of a multiple regression of a continuous response variable (sugar) against both a continuous explanatory variable (weight) and a categorical/discrete explanatory variable (color). What, to one decimal place, is the mean sugar content of a red Skittle? ``` > summary(lm(sugar ~ color + weight)) Call: lm(formula = sugar ~ color + weight) Residuals: Min 1Q Median 3Q Max -0.8027 -0.5091 0.2513 0.4068 0.6345 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 0.07934 0.28625 0.277 0.7823 colorred 0.41976 0.09952 2.208 0.0296 * weight 2.54078 0.35643 7.128 1.81e-10 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.4828 on 97 degrees of freedom Multiple R-squared: 0.3453, Adjusted R-squared: 0.3318 F-statistic: 25.58 on 2 and 97 DF, p-value: 1.195e-09 ``` 3. The Akaike Information Criterion, or AIC, is a metric of a model's performance given the number of parameters it uses. It is mathemetically defined as $AIC = 2(k - \log(likelihood))$. Why is everything in it multiplied by 2? Select one of the following: a. There are two terms in AIC (k and the log-likelihood), and so we must scale it accordingly. b. Standard errors are two log-likelihood units from the Maximum Likelihood Estimate. c. There are two bounds (upper and lower) to the confidence intervals around a model's Maximum Likelihood Estimate. d. Each parameter is calculated with reference to the intercept, and so there are two calculations involved in each parameter. 4. Which of the following is true about your choice of priors? a. Your choice of prior really doesn't matter at all. b. Using a canonical prior will bias your answer. c. You should check that your prior doesn't bias your answer. d. If your data don't match your prior you must change your prior. 5. The figure below is an MCMC trace of the intercept parameter from a model fit using `rstanarm`, covering the last 50% of the MCMC. Which of the following statements is true? a. The median posterior estimate of this parameter is approximately 0 (SD is approximately 1). b. This chain has converged; it is a "hairy caterpillar". c. This chain has not converged; it is not a "hairy caterpillar". d. This chain is too overdispersed for meanginful inference. ![MCMC trace of the intercept parameter](Figs/practical-mcmc.png) ## Long answer questions 1. a. You make cell count measurements from five petri-dishes with 2, 3, 0, 3, and 1 cells in them. What is the probability of observing such data given a Poisson distribution with a mean of 2? Show your working. Use the probability-density (likelihood) table for various observations ($x$) from a Poisson distribution with a given $\lambda$ below to help you. | | $\lambda=1$ | $\lambda=2$ | $\lambda=3$ | $\lambda=4$ | $\lambda=5$ | |:---|---:|---:|---:|---:|---:| | x=0 | 0.37 | 0.14 | 0.05 | 0.02 | 0.01 | | x=1 | 0.37 | 0.27 | 0.15 | 0.07 | 0.03 | | x=2 | 0.18 | 0.27 | 0.22 | 0.15 | 0.08 | | x=3 | 0.06 | 0.18 | 0.22 | 0.20 | 0.14 | | x=4 | 0.02 | 0.09 | 0.17 | 0.20 | 0.18 | | x=5 | 0.00 | 0.04 | 0.10 | 0.16 | 0.18 | b. Michael approaches you, saying that five petri-dishes is not enough observations for a publishable study, to which your supervisor responds that they have a strong expectation of an average of 4 cells in each petri-dish. Use the information you have at hand to develop a test statistic for use with these kinds of data, using what you know about the Poisson distribution and its variance. Importantly, justify your statistic: there is no single 'correct' answer to this question, and your reasoning matters as much as your response. c. Your supervisor calls security and gets Michael removed, after which he suggests that it would be best to model these data as from something called a 'hurdle' distribution. In this case, it means you first calculate the probability that any cells will grow at all and then, and then if the cells do grow (they get over the 'hurdle'), then their counts are drawn from a Poisson distribution. They propose you use a 50:50 chance of the cells in a dish growing. Estimate the likelihood of your data being drawn from such a hurdle model. d. AIC is defined as $2k - 2\log(likelihood)$, where $k$ is the number of parameters in a model. Use this equation to calculate the AIC of your models in (a) and (b). Outline which model is best, the relative support for your statement, and your reasoning for it. 2. A student comes to you for advice on a statistical analysis. They have grown replicates of the same cell culture under all combinations of different temperatures ($5^\circ$, $10^\circ$, $15^\circ$, and $20^\circ$) and pH (5, 6, 7, 8, and 9). They have conducted 5 replicates of each treatment, such that there are $4 \times 5 \times 5 = 100$ replicates in total, and they have counted the number of cells in each culture after 5 days of growth. a. The student fits the following model to their data, and tells you that it proves that there is a strong effect of temperature on their data. Do you agree? Why? ``` > summary(lm(count ~ temp, data=data)) Call: lm(formula = count ~ temp, data = data) Residuals: Min 1Q Median 3Q Max -4.6142 -0.9198 -0.1707 1.0537 3.6970 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 14.26791 0.39829 35.82 <2e-16 *** temp -0.72069 0.02909 -24.78 <2e-16 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 1.626 on 98 degrees of freedom Multiple R-squared: 0.8623, Adjusted R-squared: 0.8609 F-statistic: 613.9 on 1 and 98 DF, p-value: < 2.2e-16 ``` b. You advise the student to use information-theoretic criteria to better understand their data. They use the `R` package `MuMIn` to do so, and generate the following output. They state that it proves that temperature has a strong effect on their cell counts. How do you interpret this output? Provide support for your answer. ``` > dredge(lm(count ~ temp*ph, data=data, na.action="na.pass")) Fixed term is "(Intercept)" Global model call: lm(formula = count ~ temp * ph, data = data, na.action = "na.pass") --- Model selection table (Int) ph tmp ph:tmp df logLik AICc delta weight 8 0.03086 2.0340 0.008441 -0.1042 5 -133.279 277.2 0.00 1 4 9.14500 0.7318 -0.720700 4 -162.824 334.1 56.87 0 3 14.27000 -0.720700 3 -189.496 385.2 108.05 0 2 0.13640 0.7318 3 -285.716 577.7 300.48 0 1 5.25900 2 -288.646 581.4 304.22 0 Models ranked by AICc(x) ``` c. The student decides to follow a Bayesian approach, where $y$ is the count of cells and is drawn according to: $$y \sim Normal(\mu, \sigma^2)$$ ...their regression equation ($\mu$) is defined according to: $$\mu = a + bx + cz + dxz$$ ...where $x$ is the temperature and $z$ is the pH, and the other terms coefficients whose priors are: $$a,b,c,d \sim F(2,2)$$ . The student is unhappy, because their model diagnostics suggest that the model is not converging properly, and the values for the interaction term ($d$) are only ever positive, which is at odds with their frequentist model (see part b). What is wrong with their model specification that is causing this problem? d. The student begs you to outline for them a final, definitive way that they can test for an impact of temperature on their data. Explain to them what to do and, most importantly, justify your approach. e. The student's supervisor points out that they have now conducted four different kinds of models, and so must account for multiple-testing biases. If such problems affect your answer to part (d) above, outline a solution, and if not, justify why to the supervisor.