Appendix F — Mock exams

What follows are two ‘mock exams’. Each has two short answer questions, and a longer-format question, of the kind you might expect to find in an exam. I expect you to focus on these during your own time if you want extra practice, hence answers are given at the end of this section because I won’t be there to help you with the answers. I have tried to make the page formatting amenable to you treating these as if they were real exam questions, and I encourage you to try taking these under exam conditions.

F.1 Mock Exam 1

F.1.1 Short answer question

You have been given a dataset from an experiment on the impact of calcium and sodium on neural firing rates. Below is the output from a model of firing rate as a function of calcium and sodium. Describe what the experiment has found, using the information below to support your answer.

> summary(lm(firing.rate ~ calcium + sodium))

Call:
lm(formula = firing.rate ~ calcium + sodium)

Residuals:
     Min       1Q   Median       3Q      Max 
-2.09008 -0.55427 -0.04097  0.61000  2.16015 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.14895    0.09403  -1.584    0.116    
calcium      1.10577    0.10005  11.052  < 2e-16 ***
sodium      -0.66157    0.09004  -7.348 6.35e-11 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.9358 on 97 degrees of freedom
Multiple R-squared:  0.6325,    Adjusted R-squared:  0.6249 
F-statistic: 83.48 on 2 and 97 DF,  p-value: < 2.2e-16

F.1.2 Short answer question

Your colleague Michael approaches you with a problem. He has developed a new COVID test with a false-positive rate of 50%: if applied to a patient who does not have COVID, 50% of the time it will come back positive. If the test is applied to four patients who do not have COVID, what is the probability of one test coming back positive? Show your working, either by using the Binomial distribution (defined below and in your handout), or some other means.

We can use the Binomial distribution to calculate the probability of k successes happening out of n trials given a probability of success p which was defined in your handout as:

\[ p(x=k) = \binom{n}{k} p^k (1-p)^{n-k} \]

\[ \binom{n}{k} = \frac{n!}{k!(n-k)!} \]

F.1.3 Long-answer question

You are conducting a study of the impact of various chemical treatments on the expression of a gene – TheMC – in cell cultures.

(A – 25%). You conduct a study of TheMC expression in 10 separate cultures before the addition of a calcium treatment and after calcium addition. You conduct two different analyses of the expression levels (measured in dimensionless units), and on the basis of these tests your supervisor feels there is no evidence of an impact of calcium. Do you agree with your supervisor? Support your answer with reference to the output below.

> t.test(before, after)

    Welch Two Sample t-test

data:  before and after
t = -1.5054, df = 15.596, p-value = 0.1522
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -2.9807504  0.5082676
sample estimates:
mean of x mean of y 
 4.968161  6.204402 

> t.test(before, after, paired=TRUE)

    Paired t-test

data:  before and after
t = -2.6183, df = 9, p-value = 0.02789
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -2.3043251 -0.1681576
sample estimates:
mean of the differences 
              -1.236241

(B – 25%). You conduct an additional experiment where you apply various multipliers of normal human blood-calcium levels (0 = no calcium, 1 = normal calcium, 2=twice normal, 3=three times normal, 4=four times normal, 5=five times normal) either with or without a new potential therapeutical drug. On the basis of the model output below, describe the impact of the new drug and calcium on TheMC expression in this system. Make sure to reference the model output below explicitly, and to be quantitative (i.e., use numbers) in describing the impact of the drug and calcium.

> summary(lm(themc ~ drug * calcium, data=data))

Call:
lm(formula = themc ~ drug * calcium, data = data)

Residuals:
    Min      1Q  Median      3Q     Max 
-1.2476 -1.0820  0.2191  1.0199  1.0906 

Coefficients:
                      Estimate Std. Error t value Pr(>|t|)    
(Intercept)          2.571      2.329e-01  11.038  < 2e-16 ***
drugTRUE             0.001      3.294e-01   0.000  1.00000    
calcium              0.250      7.692e-02   3.250  0.00151 ** 
drugTRUE:calcium     0.249      1.088e-01   2.298  0.02335 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.018 on 116 degrees of freedom
Multiple R-squared:  0.356, Adjusted R-squared:  0.3394 
F-statistic: 21.38 on 3 and 116 DF,  p-value: 4.307e-11

(C – 40%). Your supervisor is concerned that there is too much variation in the base magnitude of cell lines’ TheMC expression, and that this is affecting your results. They ask you to repeat your experiment across a number of different cell lines. Outline how you would go about statistically analysing such data, taking care to describe both the kind of model(s) you would fit (e.g., t-tests), how you would assess support for your hypothesis (which should be the results you find in part B), and your reasoning.

(D – 10%). A colleague suggests that you should re-fit the model in part B in a Bayesian setting because “Bayesian statistics are better”. They also insist that you fit your model with a conservative prior on the intercept [ Normal(0, 0.1) ] and a prior that is strong and at odds with the data [ Normal(100, 0.1) ]. The output from the conservative prior is essentially the same as the output from part B, but the strong prior at odds with the data gives a radically different answer (given below). Your colleague insists that this means your model in part B cannot be trusted, and your results are wrong. What do you think, and why? (Note that the MCMC sampling statements are not given below to save space; assume they contain no indications of model fitting issues).

Model Info:
 function:     stan_glm
 family:       gaussian [identity]
 formula:      themc ~ drug * calcium
 algorithm:    sampling
 sample:       4000 (posterior sample size)
 priors:       see help('prior_summary')
 observations: 120
 predictors:   4

Estimates:
                      mean   sd     10%    50%    90% 
(Intercept)         -421.5    3.1 -425.4 -421.5 -417.5
drugTRUE             100.0    0.1   99.9  100.0  100.1
calcium              100.0    0.1   99.9  100.0  100.1
drugTRUE:calcium     100.0    0.1   99.9  100.0  100.1
sigma                208.4    7.6  198.9  208.2  218.4

Fit Diagnostics:
           mean   sd    10%   50%   90%
mean_PPD   2.7   19.3 -21.6   2.7  27.0

The mean_ppd is the sample average posterior predictive distribution of the outcome variable (for details see help('summary.stanreg')).

MCMC diagnostics
                    mcse Rhat n_eff
(Intercept)         0.0  1.0  5268 
drugTRUE            0.0  1.0  5580 
calcium             0.0  1.0  5983 
drugTRUE:calcium    0.0  1.0  6296 
sigma               0.1  1.0  5528 
mean_PPD            0.3  1.0  4035 
log-posterior       0.0  1.0  1862

F.2 Mock Exam 2

F.2.1 Short answer question

You are given the marks of two cohorts of Bioinformatics MSc students, each capped at exactly 30 students, and asked to see if the second cohort performed better than the first. To begin with, you perform a t-test between the two groups and get the following output:

  > t.test(first, second)

    Welch Two Sample t-test

data:  first and second
t = -1.1623, df = 57.889, p-value = 0.2499
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -2.595297  0.688538
sample estimates:
mean of x mean of y 
64.50909  65.46247

…but then your colleague Michael appears and tells you that a paired t-test would be most appropriate because the first cohort was in 2021 and the second in 2022. You get the following output:

> t.test(first, second, paired=TRUE)

    Paired t-test

data:  first and second
t = -5.5523, df = 29, p-value = 5.473e-06
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.3045604 -0.6021984
sample estimates:
mean of the differences 
             -0.9533794

Do you think there is evidence to suggest that the second cohort performed better than the first? Give your reasoning.

F.2.2 Short Question

Your supervisor asks you to quantify how Calcium injections affect neural firing rates differently in ten cell-lines. But instead of fitting a model to all ten lines at the same time like this:

model <- lm(rate ~ calcium * cellline, data=all.lines)

…or this…

hierarchical.model <- stan_glmer(rate ~ (calcium|celline), data=all.lines)

…your supervisor ask you to fit ten separate models like this:

linea.model <- lm(rate ~ calcium, data=linea)
lineb.model <- lm(rate ~ calcium, data=linen)
linec.model <- lm(rate ~ calcium, data=linec)
lined.model <- lm(rate ~ calcium, data=lined)
linee.model <- lm(rate ~ calcium, data=linee)
linef.model <- lm(rate ~ calcium, data=linef)
lineg.model <- lm(rate ~ calcium, data=lineg)
lineh.model <- lm(rate ~ calcium, data=lineh)
linei.model <- lm(rate ~ calcium, data=linei)
linej.model <- lm(rate ~ calcium, data=linej)

Explain to your supervisor the impact of fitting this many models on (a) Type I errors (‘false positives’) and (b) Type II errors (‘false negatives’). Explain the impact of a Bonferroni correction (or any kind of multiple-testing correction) on Type I and Type II error rates in this case.

F.2.3 Long Question

It is found that all adults have, in varying concentrations, amounts of a chemical called ‘cullen’, which affects the degree to which adults sparkle. Your PhD supervisor has been given cullen concentrations, full sparkle estimates (a continuous variable measured in lumens), quantitative medical history information (e.g., blood tests of whatever kind you think appropriate), and socio-demographic backgrounds from 1000 undergraduates at various London universities.

(A – 35%) Outline how you would test whether there is an association between sparkling and cullen concentrations, or whether the apparent concentration is driven by associations with medical history or socio-demographic background. Use classical frequentist or Bayesian techniques, not supervised machine learning approaches (e.g., regression trees, support vector machines, least-angle regression, or artificial neural networks). Ignore possible issues with sampling and/or pseudo-replication. Note that there are several appropriate ways to address this question; you need only give one, but you must justify and explain it.

(B – 35%) Upon inspecting the data, you become concerned there are issues with its sampling: all of the data are from first-year undergraduates at Imperial College London who live in halls around the city. How would you fit a model to these data that accounted for this pseudoreplication (unequal sampling) where some students are taken from the same hall, and how would you judge whether cullen concentration was a factor in sparkle? Use classical frequentist or Bayesian techniques, not supervised machine learning approaches (e.g., regression trees, support vector machines, least-angle regression, or artificial neural networks). Note that there are several appropriate ways to address this question; you need only give one, but you must justify and explain it.

(C – 20%) You have recorded the rate of cullen secretion from three epidermal cells in a dish, and have found them to have rates of exactly 1, 3, and 5 (ml/min). Your supervisor has a statistical model that gives a probability density function of secretion rates (below; note that likelihood, not log-likelihood, is plotted on the vertical axis). Using this information, what is the probability of observing these three epidermal cells releasing chemicals at these rates?

(D – 10%) Your colleague Michael compares cullen secretion rate between 500,000 people whose surnames begin with the letter ‘A’ and 500,000 people whose surnames begin with the letter ‘B’. He finds that people in the ‘A’ group have a mean secretion rate of 3.1 ml/min and those in the ‘B’ group have a mean secretion rate of 3.1001. A t-test of the two groups is statistically significant (i.e., p < 0.05). Do you think these results are biologically meaningful? For reference, the highest cullen secretion rate ever recorded was 4 ml/min, and the lowest is 0 ml/min. Justify your answer.

Figure to accompany long question of the second exam.

F.3 Answers

F.3.1 Exam 1

F.3.1.1 Short answer question

This question tests whether you can interpret model output. The model shows a positive correlation between calcium and firing rate and a negative correlation between sodium and firing rate. Identification of this from the model would be sufficient for full marks.

F.3.1.2 Short answer question

The students have practised calculations like this in class. The correct solution is:

\[ \binom{n}{k} p^k (1-p)^{(n-k)} = \frac{4!}{(1!3!)} \times \frac{1}{2} \times \left(\frac{1}{2}\right)^3 = 4 \times \left(\frac{1}{2}\right)^4 = \frac{4}{(4\times4)} = \frac{1}{4} \]

Answers would also be marked as correct if they recognise that there are four ‘chances’ to get a positive result, and thus the correct answer is $p(\text{one success}) \times 4 = \left(\frac{1}{2}\right)^4 \times 4 = \frac{1}{4}$.

F.3.1.3 Long answer question

You have had practice interpreting paired t-tests in the course and in your handout, and you are explicitly told that they are the best way to test treatment before/after data. Distinction answers will identify that a paired t-test is the best way to test these data, and will reference the test-statistic value and how it suggests these data are unlikely to be seen under a null hypothesis of no difference among the groups. Very high distinction answers might be particularly well-written, or identify the low sample size as giving poor statistical power (and potentially even identify the high variance in the samples from the output). Merits will get the correct answer, but show limited critical understanding of why. Pass answers may either select the wrong answer, or select the correct answer but struggle to explain why or to interpret the output.
You have practised how to interpret linear model output as contrasts, not absolute values. Distinction answers will correctly identify that the impact of calcium is twice as large in the presence of the drug; the coefficient of the effect of calcium is 0.25 without the drug and 0.5 in the presence of it. The overall intercept is 2.5; there is no effect of the drug on TheMC production on its own, its only effect is mediated through calcium. Merit answers will address most of the above, perhaps missing the fact that it has an effect only on its own, or being overly precise with effect sizes (e.g., claiming the effect in the presence of the drug is exactly 0.499 and not having the common sense to round to 0.5 even given the standard error of the interaction term). Pass answers will correctly identify that there is an interaction but fail to quantitatively identify it.
Essentially any valid answer that accounts for the effect of variation in cell line would be acceptable here. The canonical answer would be to fit a hierarchical model, either a mixed effects model or a Bayesian hierarchical model, that accounts for differences across lines while also incorporating the model structure in part (B). Assessing support for the hypothesis if a mixed effets model is fit would be tricky, because the students are taught in class that such approaches essentially cannot be done outside of information theoretic criteria, but answers that address the challenge of assessing fixed-effect structures and provide a potential way forward that is broadly acceptable can get a Distinction grade. Bayesian methods may also allow for direct posterior distribution tests, and all approaches could involve scaled coefficients as necessary. Distinction answers would be well-reasoned, coherent, and contain no obvious statistical deficiencies. Merit answers would be broadly correct but either unclear in reasoning or precise methodology. Pass answers will either clearly outline approaches that would not be appropriate (e.g., paired t-tests, PCAs, etc.) but are well-described, or correct approaches but in such a way that it is clear they are regurgitating and do not understand the reasoning for their choices.
Specifying an inappropriate prior that is at odds with the data means that the model’s output cannot be trusted. Distinction class answers will recognise this, and outline that the choice of prior is inappropriate and that a prior that is at odds with the data is rarely appropriate and no reasoning has been given for such an absurdly large coefficient value. Merit answers will address some of these concepts but will be confused. Pass answers may struggle to answer the question, or will agree with the colleague and regurgitate valid concerns about the definitions of priors that are irrelevant to the question at hand.

F.3.2 Exam 2

F.3.2.1 Short answer question

Marks in the second cohort are slightly higher than in the first, but the difference is slight (<1 mark) and not statistically significant. The use of a paired t-test is inappropriate because no evidence is given that the students are shared between the groups and the justification given by Michael is inappropriate. A perfect answer would make reference to the above, and also reference some additional aspect of the output from the first t-test model that supports their reasoning (e.g., the p-value). Since, in the lectures, students would have noticed that I am not a fan of using point estimates and p-values unless critically considered, I would accept answers that engage in a reasonable way with the output but seem reticent to report p-values and/or talk about rejecting H0/accepting H1, although I would also accept such answers for full marks. Accepting the paired t-test as the better test will cap the answer at half marks, even if the paired test is properly interpreted.

F.3.2.2 Short answer question

The concepts of Type I and Type II errors, and how they are intrinsically related with statistical power, were returned to throughout the course. Bonferroni methods were mentioned in one lecture, and have a dedicated section in the handout. Answers should be able to articulate that (a) testing multiple times inflates the probability of getting a significant result by chance, and that Bonferroni-type corrections can reduce this likelihood. They should then point out that (b) multiple model tests means it is actually less likely that we will detect significant associations because we have a greater chance of making a mistake, and that Bonferroni-type corrections would actually increase the rate of Type II errors by reducing statistical power. Answers may also define Bonferroni corrections (or other corrections), and go into further details about Type I or Type II errors, or statistical power (alpha vs. beta). Answers do not need to address all the aspects above to get full marks; mentioning the key points (underlined above) and one or more of the other points would be sufficient for full marks.

F.3.2.3 Long answer question

A. An ANOVA of lm(spark ~ cullen + medical + socioeconomic) vs. lm(spark ~ medical + socioeconomic) would be the best choice here, because it is a simple, single test that contrasts an alternative model with a good null model. I would also accept model-averaging and/or information theoretic approaches, better answers to which would incorporate a description of standard effect sizes or some other way of assessing relative model support.

B. Mixed effects models (frequentist) would be acceptable here, with random effects for hall, but I would expect to a see a discussion of the difficulty of testing the significance of their terms. Information theoretic criteria would be acceptable, and would fit well with an answer given to A along those lines. The perfect answer would be a Bayesian hierarchical model, and I will accept the term ‘random’ as being applied to hierarchical terms for hall in such a model, and would want to see discussion of posterior distributions used to estimate the probability of an impact.

C. One of the first lecture’s exercises was to do calculations such as this, and I walked through the calculations on the board in three lectures. 0.1 x 0.3 x 0.1 = 0.003. If the answer inovlves summing them, erroneously thinking they are dealing with log-likelihoods, half marks will be given.

D. I would accept any well-reasoned answer here, since I have left this quite vague, but answers of the kind I expect (and would award full marks to) include: (1) No, because Michael has ‘p-hacked’ and increased his sampling until he found a correct answer (this is the most correct answer, but given the scope of the course I will accept others). (2) Yes/no, he’s correct in a sense, but this is clearly a small effect and so is not likely to be biologically meaningful. The answer should attempt some numerical argument with reference to the stated reference values, or requesting additional information (e.g., what the null expectation is, or the variances associated with the groups). (3) This was an unfair test – we don’t know if there are some other factors that drive cullen secretion that vary across these two groups. They seem random with respect to biology, but we can’t be certain of this until we check.

# Mock exams What follows are two 'mock exams'. Each has two short answer questions, and a longer-format question, of the kind you might expect to find in an exam. I expect you to focus on these during your own time if you want extra practice, hence answers are given at the end of this section because I won't be there to help you with the answers. I have tried to make the page formatting amenable to you treating these as if they were real exam questions, and I encourage you to try taking these under exam conditions. ## Mock Exam 1 ### Short answer question You have been given a dataset from an experiment on the impact of calcium and sodium on neural firing rates. Below is the output from a model of firing rate as a function of calcium and sodium. Describe what the experiment has found, using the information below to support your answer. ``` > summary(lm(firing.rate ~ calcium + sodium)) Call: lm(formula = firing.rate ~ calcium + sodium) Residuals: Min 1Q Median 3Q Max -2.09008 -0.55427 -0.04097 0.61000 2.16015 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -0.14895 0.09403 -1.584 0.116 calcium 1.10577 0.10005 11.052 < 2e-16 *** sodium -0.66157 0.09004 -7.348 6.35e-11 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.9358 on 97 degrees of freedom Multiple R-squared: 0.6325, Adjusted R-squared: 0.6249 F-statistic: 83.48 on 2 and 97 DF, p-value: < 2.2e-16 ``` ### Short answer question Your colleague Michael approaches you with a problem. He has developed a new COVID test with a false-positive rate of 50%: if applied to a patient who does not have COVID, 50% of the time it will come back positive. If the test is applied to four patients who do not have COVID, what is the probability of one test coming back positive? Show your working, either by using the Binomial distribution (defined below and in your handout), or some other means. We can use the Binomial distribution to calculate the probability of k successes happening out of n trials given a probability of success p which was defined in your handout as: $$ p(x=k) = \binom{n}{k} p^k (1-p)^{n-k} $$ $$ \binom{n}{k} = \frac{n!}{k!(n-k)!} $$ ### Long-answer question You are conducting a study of the impact of various chemical treatments on the expression of a gene – TheMC – in cell cultures. (A – 25%). You conduct a study of TheMC expression in 10 separate cultures before the addition of a calcium treatment and after calcium addition. You conduct two different analyses of the expression levels (measured in dimensionless units), and on the basis of these tests your supervisor feels there is no evidence of an impact of calcium. Do you agree with your supervisor? Support your answer with reference to the output below. ``` > t.test(before, after) Welch Two Sample t-test data: before and after t = -1.5054, df = 15.596, p-value = 0.1522 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -2.9807504 0.5082676 sample estimates: mean of x mean of y 4.968161 6.204402 > t.test(before, after, paired=TRUE) Paired t-test data: before and after t = -2.6183, df = 9, p-value = 0.02789 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -2.3043251 -0.1681576 sample estimates: mean of the differences -1.236241 ``` (B – 25%). You conduct an additional experiment where you apply various multipliers of normal human blood-calcium levels (0 = no calcium, 1 = normal calcium, 2=twice normal, 3=three times normal, 4=four times normal, 5=five times normal) either with or without a new potential therapeutical drug. On the basis of the model output below, describe the impact of the new drug and calcium on TheMC expression in this system. Make sure to reference the model output below explicitly, and to be quantitative (i.e., use numbers) in describing the impact of the drug and calcium. ``` > summary(lm(themc ~ drug * calcium, data=data)) Call: lm(formula = themc ~ drug * calcium, data = data) Residuals: Min 1Q Median 3Q Max -1.2476 -1.0820 0.2191 1.0199 1.0906 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 2.571 2.329e-01 11.038 < 2e-16 *** drugTRUE 0.001 3.294e-01 0.000 1.00000 calcium 0.250 7.692e-02 3.250 0.00151 ** drugTRUE:calcium 0.249 1.088e-01 2.298 0.02335 * --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 1.018 on 116 degrees of freedom Multiple R-squared: 0.356, Adjusted R-squared: 0.3394 F-statistic: 21.38 on 3 and 116 DF, p-value: 4.307e-11 ``` (C – 40%). Your supervisor is concerned that there is too much variation in the base magnitude of cell lines' TheMC expression, and that this is affecting your results. They ask you to repeat your experiment across a number of different cell lines. Outline how you would go about statistically analysing such data, taking care to describe both the kind of model(s) you would fit (e.g., t-tests), how you would assess support for your hypothesis (which should be the results you find in part B), and your reasoning. (D – 10%). A colleague suggests that you should re-fit the model in part B in a Bayesian setting because "Bayesian statistics are better". They also insist that you fit your model with a conservative prior on the intercept [ Normal(0, 0.1) ] and a prior that is strong and at odds with the data [ Normal(100, 0.1) ]. The output from the conservative prior is essentially the same as the output from part B, but the strong prior at odds with the data gives a radically different answer (given below). Your colleague insists that this means your model in part B cannot be trusted, and your results are wrong. What do you think, and why? (Note that the MCMC sampling statements are not given below to save space; assume they contain no indications of model fitting issues). ``` Model Info: function: stan_glm family: gaussian [identity] formula: themc ~ drug * calcium algorithm: sampling sample: 4000 (posterior sample size) priors: see help('prior_summary') observations: 120 predictors: 4 Estimates: mean sd 10% 50% 90% (Intercept) -421.5 3.1 -425.4 -421.5 -417.5 drugTRUE 100.0 0.1 99.9 100.0 100.1 calcium 100.0 0.1 99.9 100.0 100.1 drugTRUE:calcium 100.0 0.1 99.9 100.0 100.1 sigma 208.4 7.6 198.9 208.2 218.4 Fit Diagnostics: mean sd 10% 50% 90% mean_PPD 2.7 19.3 -21.6 2.7 27.0 The mean_ppd is the sample average posterior predictive distribution of the outcome variable (for details see help('summary.stanreg')). MCMC diagnostics mcse Rhat n_eff (Intercept) 0.0 1.0 5268 drugTRUE 0.0 1.0 5580 calcium 0.0 1.0 5983 drugTRUE:calcium 0.0 1.0 6296 sigma 0.1 1.0 5528 mean_PPD 0.3 1.0 4035 log-posterior 0.0 1.0 1862 ``` ## Mock Exam 2 ### Short answer question You are given the marks of two cohorts of Bioinformatics MSc students, each capped at exactly 30 students, and asked to see if the second cohort performed better than the first. To begin with, you perform a t-test between the two groups and get the following output: ``` > t.test(first, second) Welch Two Sample t-test data: first and second t = -1.1623, df = 57.889, p-value = 0.2499 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -2.595297 0.688538 sample estimates: mean of x mean of y 64.50909 65.46247 ``` ...but then your colleague Michael appears and tells you that a paired t-test would be most appropriate because the first cohort was in 2021 and the second in 2022. You get the following output: ``` > t.test(first, second, paired=TRUE) Paired t-test data: first and second t = -5.5523, df = 29, p-value = 5.473e-06 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -1.3045604 -0.6021984 sample estimates: mean of the differences -0.9533794 ``` Do you think there is evidence to suggest that the second cohort performed better than the first? Give your reasoning. ### Short Question Your supervisor asks you to quantify how Calcium injections affect neural firing rates differently in ten cell-lines. But instead of fitting a model to all ten lines at the same time like this: ```r model <- lm(rate ~ calcium * cellline, data=all.lines) ``` ...or this… ```r hierarchical.model <- stan_glmer(rate ~ (calcium|celline), data=all.lines) ``` ...your supervisor ask you to fit ten separate models like this: ```r linea.model <- lm(rate ~ calcium, data=linea) lineb.model <- lm(rate ~ calcium, data=linen) linec.model <- lm(rate ~ calcium, data=linec) lined.model <- lm(rate ~ calcium, data=lined) linee.model <- lm(rate ~ calcium, data=linee) linef.model <- lm(rate ~ calcium, data=linef) lineg.model <- lm(rate ~ calcium, data=lineg) lineh.model <- lm(rate ~ calcium, data=lineh) linei.model <- lm(rate ~ calcium, data=linei) linej.model <- lm(rate ~ calcium, data=linej) ``` Explain to your supervisor the impact of fitting this many models on (a) Type I errors (‘false positives’) and (b) Type II errors (‘false negatives’). Explain the impact of a Bonferroni correction (or any kind of multiple-testing correction) on Type I and Type II error rates in this case. ### Long Question It is found that all adults have, in varying concentrations, amounts of a chemical called ‘cullen’, which affects the degree to which adults sparkle. Your PhD supervisor has been given cullen concentrations, full sparkle estimates (a continuous variable measured in lumens), quantitative medical history information (e.g., blood tests of whatever kind you think appropriate), and socio-demographic backgrounds from 1000 undergraduates at various London universities. (A – 35%) Outline how you would test whether there is an association between sparkling and cullen concentrations, or whether the apparent concentration is driven by associations with medical history or socio-demographic background. Use classical frequentist or Bayesian techniques, not supervised machine learning approaches (e.g., regression trees, support vector machines, least-angle regression, or artificial neural networks). Ignore possible issues with sampling and/or pseudo-replication. Note that there are several appropriate ways to address this question; you need only give one, but you must justify and explain it. (B – 35%) Upon inspecting the data, you become concerned there are issues with its sampling: all of the data are from first-year undergraduates at Imperial College London who live in halls around the city. How would you fit a model to these data that accounted for this pseudoreplication (unequal sampling) where some students are taken from the same hall, and how would you judge whether cullen concentration was a factor in sparkle? Use classical frequentist or Bayesian techniques, not supervised machine learning approaches (e.g., regression trees, support vector machines, least-angle regression, or artificial neural networks). Note that there are several appropriate ways to address this question; you need only give one, but you must justify and explain it. (C – 20%) You have recorded the rate of cullen secretion from three epidermal cells in a dish, and have found them to have rates of exactly 1, 3, and 5 (ml/min). Your supervisor has a statistical model that gives a probability density function of secretion rates (below; note that likelihood, not log-likelihood, is plotted on the vertical axis). Using this information, what is the probability of observing these three epidermal cells releasing chemicals at these rates? (D – 10%) Your colleague Michael compares cullen secretion rate between 500,000 people whose surnames begin with the letter ‘A’ and 500,000 people whose surnames begin with the letter ‘B’. He finds that people in the ‘A’ group have a mean secretion rate of 3.1 ml/min and those in the ‘B’ group have a mean secretion rate of 3.1001. A t-test of the two groups is statistically significant (i.e., p < 0.05). Do you think these results are biologically meaningful? For reference, the highest cullen secretion rate ever recorded was 4 ml/min, and the lowest is 0 ml/min. Justify your answer. ![Figure to accompany long question of the second exam.](Figs/cullen-secretion.png) ## Answers ### Exam 1 #### Short answer question This question tests whether you can interpret model output. The model shows a positive correlation between calcium and firing rate and a negative correlation between sodium and firing rate. Identification of this from the model would be sufficient for full marks. #### Short answer question The students have practised calculations like this in class. The correct solution is: $$ \binom{n}{k} p^k (1-p)^{(n-k)} = \frac{4!}{(1!3!)} \times \frac{1}{2} \times \left(\frac{1}{2}\right)^3 = 4 \times \left(\frac{1}{2}\right)^4 = \frac{4}{(4\times4)} = \frac{1}{4} $$ Answers would also be marked as correct if they recognise that there are four 'chances' to get a positive result, and thus the correct answer is $p(\text{one success}) \times 4 = \left(\frac{1}{2}\right)^4 \times 4 = \frac{1}{4}$. #### Long answer question (A) You have had practice interpreting paired t-tests in the course and in your handout, and you are explicitly told that they are the best way to test treatment before/after data. Distinction answers will identify that a paired t-test is the best way to test these data, and will reference the test-statistic value and how it suggests these data are unlikely to be seen under a null hypothesis of no difference among the groups. Very high distinction answers might be particularly well-written, or identify the low sample size as giving poor statistical power (and potentially even identify the high variance in the samples from the output). Merits will get the correct answer, but show limited critical understanding of why. Pass answers may either select the wrong answer, or select the correct answer but struggle to explain why or to interpret the output. (B) You have practised how to interpret linear model output as contrasts, not absolute values. Distinction answers will correctly identify that the impact of calcium is twice as large in the presence of the drug; the coefficient of the effect of calcium is 0.25 without the drug and 0.5 in the presence of it. The overall intercept is 2.5; there is no effect of the drug on TheMC production on its own, its only effect is mediated through calcium. Merit answers will address most of the above, perhaps missing the fact that it has an effect only on its own, or being overly precise with effect sizes (e.g., claiming the effect in the presence of the drug is exactly 0.499 and not having the common sense to round to 0.5 even given the standard error of the interaction term). Pass answers will correctly identify that there is an interaction but fail to quantitatively identify it. (C) Essentially any valid answer that accounts for the effect of variation in cell line would be acceptable here. The canonical answer would be to fit a hierarchical model, either a mixed effects model or a Bayesian hierarchical model, that accounts for differences across lines while also incorporating the model structure in part (B). Assessing support for the hypothesis if a mixed effets model is fit would be tricky, because the students are taught in class that such approaches essentially cannot be done outside of information theoretic criteria, but answers that address the challenge of assessing fixed-effect structures and provide a potential way forward that is broadly acceptable can get a Distinction grade. Bayesian methods may also allow for direct posterior distribution tests, and all approaches could involve scaled coefficients as necessary. Distinction answers would be well-reasoned, coherent, and contain no obvious statistical deficiencies. Merit answers would be broadly correct but either unclear in reasoning or precise methodology. Pass answers will either clearly outline approaches that would not be appropriate (e.g., paired t-tests, PCAs, etc.) but are well-described, or correct approaches but in such a way that it is clear they are regurgitating and do not understand the reasoning for their choices. (D) Specifying an inappropriate prior that is at odds with the data means that the model’s output cannot be trusted. Distinction class answers will recognise this, and outline that the choice of prior is inappropriate and that a prior that is at odds with the data is rarely appropriate and no reasoning has been given for such an absurdly large coefficient value. Merit answers will address some of these concepts but will be confused. Pass answers may struggle to answer the question, or will agree with the colleague and regurgitate valid concerns about the definitions of priors that are irrelevant to the question at hand. ### Exam 2 #### Short answer question Marks in the second cohort are slightly higher than in the first, but the difference is slight (<1 mark) and not statistically significant. The use of a paired t-test is inappropriate because no evidence is given that the students are shared between the groups and the justification given by Michael is inappropriate. A perfect answer would make reference to the above, and also reference some additional aspect of the output from the first t-test model that supports their reasoning (e.g., the p-value). Since, in the lectures, students would have noticed that I am not a fan of using point estimates and p-values unless critically considered, I would accept answers that engage in a reasonable way with the output but seem reticent to report p-values and/or talk about rejecting H0/accepting H1, although I would also accept such answers for full marks. Accepting the paired t-test as the better test will cap the answer at half marks, even if the paired test is properly interpreted. #### Short answer question The concepts of Type I and Type II errors, and how they are intrinsically related with statistical power, were returned to throughout the course. Bonferroni methods were mentioned in one lecture, and have a dedicated section in the handout. Answers should be able to articulate that (a) testing multiple times inflates the probability of getting a significant result by chance, and that Bonferroni-type corrections can reduce this likelihood. They should then point out that (b) multiple model tests means it is actually less likely that we will detect significant associations because we have a greater chance of making a mistake, and that Bonferroni-type corrections would actually increase the rate of Type II errors by reducing statistical power. Answers may also define Bonferroni corrections (or other corrections), and go into further details about Type I or Type II errors, or statistical power (alpha vs. beta). Answers do not need to address all the aspects above to get full marks; mentioning the key points (underlined above) and one or more of the other points would be sufficient for full marks. #### Long answer question A. An ANOVA of `lm(spark ~ cullen + medical + socioeconomic)` vs. `lm(spark ~ medical + socioeconomic)` would be the best choice here, because it is a simple, single test that contrasts an alternative model with a good null model. I would also accept model-averaging and/or information theoretic approaches, better answers to which would incorporate a description of standard effect sizes or some other way of assessing relative model support. B. Mixed effects models (frequentist) would be acceptable here, with random effects for hall, but I would expect to a see a discussion of the difficulty of testing the significance of their terms. Information theoretic criteria would be acceptable, and would fit well with an answer given to A along those lines. The perfect answer would be a Bayesian hierarchical model, and I will accept the term ‘random’ as being applied to hierarchical terms for hall in such a model, and would want to see discussion of posterior distributions used to estimate the probability of an impact. C. One of the first lecture’s exercises was to do calculations such as this, and I walked through the calculations on the board in three lectures. 0.1 x 0.3 x 0.1 = 0.003. If the answer inovlves summing them, erroneously thinking they are dealing with log-likelihoods, half marks will be given. D. I would accept any well-reasoned answer here, since I have left this quite vague, but answers of the kind I expect (and would award full marks to) include: (1) No, because Michael has ‘p-hacked’ and increased his sampling until he found a correct answer (this is the most correct answer, but given the scope of the course I will accept others). (2) Yes/no, he’s correct in a sense, but this is clearly a small effect and so is not likely to be biologically meaningful. The answer should attempt some numerical argument with reference to the stated reference values, or requesting additional information (e.g., what the null expectation is, or the variances associated with the groups). (3) This was an unfair test – we don’t know if there are some other factors that drive cullen secretion that vary across these two groups. They seem random with respect to biology, but we can’t be certain of this until we check.