Week 8: Confidence Intervals and Hypothesis Testing

Where we’ve been

Where we’ve been

• Under i.i.d random sampling from the target population, the central limit theorem gives us…

\[\sqrt{n}\left(\hat{\mu} - \mu\right) \xrightarrow{d} \mathcal{N}\left(0, \sigma^2\right)\]

• Intuition: Averaging washes out every feature of the distribution beyond the mean and variance:
  • And the normal is the distribution fully defined by mean and variance alone
    • All odd central moments \(\ge 3\) are zero
    • All even central moments \(\ge 4\) are functions of \(\Var(X)\)

This week

• Confidence intervals
  • Interval estimator - An upper and lower bound constructed to cover the true parameter in some percentage of repeated samples.
  • Constructing confidence intervals under asymptotic normality
  • Small sample coverage rates and some typical corrections.
• Hypothesis testing
  • Statistical decision making through thought experiments.
  • Given some assumed value of the population parameter, how likely would I be to observe what I find in my sample (or more extreme)?
  • This is a p-value. If it’s sufficiently low, I may be willing to believe that the population parameter is not the hypothesized value.
• Connections between confidence intervals and hypothesis testing

Point estimation

Interval estimation

Motivation: ANES (2024) - Attitudes towards the parties

Motivation: ANES (2024) - Attitudes towards the parties

• Do Americans feel more warmly towards the Democratic party or the Republican party?
  • The 2024 American National Election Study (ANES) asks respondents to rate each party on a 0-100 feeling thermometer.

library(tidyverse)
# 2024 ANES pre-election wave
anes <- read_csv("data/anes24_therm.csv") |>
  filter(!is.na(ft_dem), !is.na(ft_rep)) |>
  mutate(ft_diff = ft_dem - ft_rep)   # Dem minus Rep for each respondent

• We’ll use a sample mean estimator for the within-respondent difference
  • As before, we’ll ignore the sample weights for now…

anes |> summarize(mean_ft_dem = mean(ft_dem),
                    mean_ft_rep = mean(ft_rep),
                    mean_ft_diff = mean(ft_diff))

# A tibble: 1 × 3
  mean_ft_dem mean_ft_rep mean_ft_diff
        <dbl>       <dbl>        <dbl>
1        45.7        43.4         2.28

• On average, respondents feel 2.28 points more warmly towards the Democrats?
  • But how confident are we that the truth isn’t zero?

Attitudes towards the parties

Attitudes towards the parties

• We can calculate an estimate of the standard error of the mean using our formula from before

  \[\widehat{SE}(\bar{Y}) = \frac{\hat{\sigma}}{\sqrt{n}} = \frac{\sqrt{\frac{1}{n-1}\sum_{i=1}^n \left(Y_i - \bar{Y}\right)^2}}{\sqrt{n}} \]

anes |> summarize(mean_ft_diff = mean(ft_diff),
                  n=n(),
                  sd_ft_diff = sd(ft_diff),
                  se_ft_diff = sd_ft_diff/sqrt(n))

# A tibble: 1 × 4
  mean_ft_diff     n sd_ft_diff se_ft_diff
         <dbl> <int>      <dbl>      <dbl>
1         2.28  5477       59.7      0.807

• In terms of RMSE, our estimator is 0.807 units from the true mean…which is about a third of our estimate - not too big but still sizeable.
  • It’s entirely plausible though that we could have obtained our estimate in a world where the true difference is zero.
  • Can we quantify our uncertainty in a more transparent and rigorous way?

Confidence interval

• A confidence interval is an estimator that consists of an upper and a lower bound.

  • These bounds are constructed to have a particular coverage rate

• A \(100(1-\alpha)\%\) confidence interval for population parameter \(\theta\) is an interval \([l, u]\) where \(l = l(Y_1, Y_2, \dotsc, Y_n)\) and \(u = u(Y_1, Y_2, \dotsc, Y_n)\) are functions of the data such that

  \[P(l \le \theta \le u) = 1 - \alpha\]

• The bounds are random variables

  • They’re functions of the data - just like our sample mean estimator

Confidence intervals under asymptotic normality

• How can we derive a confidence interval for the population mean \(\mu\)

  • We’ll need to calculate the probability of drawing an interval that either does or does not cover the target.
  • This means we need the distribution of our estimator - mean and variance alone aren’t enough!
  • That’s why we spent so much time on the central limit theorem - in large samples we can approximate the sampling distribution with a normal.

• We’ll focus on an interval that is symmetric and centered on the sample mean

  \[P(\underbrace{\bar{Y} - c}_{\text{lower}} \le \mu \le \underbrace{\bar{Y} + c}_{\text{upper}}) = 1 - \alpha\]

Confidence intervals under asymptotic normality

• Rearrange the probability so that the estimator is in the middle \[\begin{aligned} P(\bar{Y} - c \le \mu \le \bar{Y} + c) &= P(-c \le \mu - \bar{Y} \le c) && \small\color{#777777}{\text{subtract } \bar{Y} \text{ throughout}} \\ &= P(-c \le \bar{Y} - \mu \le c) && \small\color{#777777}{\text{multiply by } -1} \\ &= P\left(-\frac{c}{SE(\bar{Y})} \le \frac{\bar{Y} - \mu}{SE(\bar{Y})} \le \frac{c}{SE(\bar{Y})}\right) && \small\color{#777777}{\text{divide by } SE(\bar{Y}) = \sigma/\sqrt{n} > 0} \end{aligned}\]

• The term in the middle is the standardized sample mean \[\frac{\bar{Y} - \mu}{SE(\bar{Y})} = \frac{\bar{Y} - \mu}{\sigma/\sqrt{n}} = \frac{\sqrt{n}(\bar{Y} - \mu)}{\sigma}\]

• If we knew its distribution, we could choose \(c\) to make the probability exactly \(1 - \alpha\)

Confidence intervals under asymptotic normality

• Last week we used the central limit theorem to show that

  \[ \sqrt{n}(\bar{Y} - \mu) \xrightarrow{d} \mathcal{N}(0, \sigma^2) \]

• And, by extension (continuous mapping theorem), the standardized mean is asymptotically standard normal

  \[ \frac{\sqrt{n}(\bar{Y} - \mu)}{\sigma} \xrightarrow{d} \mathcal{N}(0, 1) \]

• We now have a pivotal quantity - a function of the data and parameter whose (limiting) distribution does not depend on any unknowns.

Confidence intervals under asymptotic normality

• Now we just need to find the value of \(c\) that satisfies

  \[ P\left(-\frac{c}{SE(\bar{Y})} \le \frac{\bar{Y} - \mu}{SE(\bar{Y})} \le \frac{c}{SE(\bar{Y})}\right) = 1 - \alpha\]

• The probability of landing in an interval is just difference of two CDFs. Let \(\Phi()\) be the standard normal CDF

  \[\begin{aligned} &= P\left(\frac{\bar{Y} - \mu}{SE(\bar{Y})} \le \frac{c}{SE(\bar{Y})}\right) - P\left(\frac{\bar{Y} - \mu}{SE(\bar{Y})} < -\frac{c}{SE(\bar{Y})}\right) && \small\color{#777777}{\text{upper minus lower}} \\ &\approx \Phi\left(\frac{c}{SE(\bar{Y})}\right) - \Phi\left(-\frac{c}{SE(\bar{Y})}\right) && \small\color{#777777}{\text{asymptotic normality}} \\ &= \Phi\left(\frac{c}{SE(\bar{Y})}\right) - \left[1 - \Phi\left(\frac{c}{SE(\bar{Y})}\right)\right] && \small\color{#777777}{\text{symmetry: } \Phi(-x) = 1 - \Phi(x)} \\ &= 2\Phi\left(\frac{c}{SE(\bar{Y})}\right) - 1 \end{aligned}\]

Confidence intervals under asymptotic normality

\[P\left(-\frac{c}{SE(\bar{Y})} \le \frac{\bar{Y} - \mu}{SE(\bar{Y})} \le \frac{c}{SE(\bar{Y})}\right)\]

Confidence intervals under asymptotic normality

• Set the probability equal to \(1 - \alpha\) and solve for \(c\) using the inverse CDF \(\Phi^{-1}()\)

  \[\begin{aligned} 2\Phi\left(\frac{c}{SE(\bar{Y})}\right) - 1 &= 1 - \alpha \\ \Phi\left(\frac{c}{SE(\bar{Y})}\right) &= 1 - \frac{\alpha}{2} && \small\color{#777777}{\text{add 1, divide by 2}} \\ \frac{c}{SE(\bar{Y})} &= \Phi^{-1}\left(1 - \frac{\alpha}{2}\right) && \small\color{#777777}{\text{apply the inverse CDF}} \\ c &= \Phi^{-1}\left(1 - \frac{\alpha}{2}\right) \times SE(\bar{Y}) \end{aligned}\]

• The critical value \(z_{1-\alpha/2} = \Phi^{-1}(1 - \alpha/2)\) is the \(1 - \alpha/2\) quantile of the standard normal

  • By symmetry, we allocate \(\alpha/2\) probability to each tail.

• For a 95% interval, \(\alpha = 0.05\) and \(z_{0.975} \approx 1.96\) (qnorm(0.975) in R)

Confidence intervals under asymptotic normality

\[\Phi(z_{0.975}) = \int_{-\infty}^{z_{0.975}} \frac{1}{\sqrt{2\pi}} e^{-z^2/2}\,dz = 0.975 \quad \Longleftrightarrow \quad z_{0.975} = \Phi^{-1}(0.975)\]

Critical values and coverage

\[P\left(-z \le \frac{\bar{Y} - \mu}{SE(\bar{Y})} \le z\right) \approx 2\Phi(z) - 1\]

Plugging in the estimated standard error

• Our two-sided \(100(1-\alpha)\%\) confidence interval is

  \[ \left[\bar{Y} - z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}},\ \ \bar{Y} + z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}}\right]\]

• Unfortunately, we don’t know \(\sigma\)! So let’s plug in our estimated \(\hat{\sigma}\) instead.

  \[ \left[\bar{Y} - z_{1-\alpha/2}\widehat{SE}(\bar{Y}),\ \ \bar{Y} + z_{1-\alpha/2}\widehat{SE}(\bar{Y})\right], \qquad \widehat{SE}(\bar{Y}) = \frac{\hat{\sigma}}{\sqrt{n}}\]

• Is this valid? Yes, by Slutsky’s theorem, if \(\hat{\sigma} \xrightarrow{p} \sigma\), then

  \[ \frac{\bar{Y} - \mu}{\widehat{SE}(\bar{Y})} = \frac{\overbrace{\sqrt{n}(\bar{Y} - \mu)}^{\xrightarrow{d} \mathcal{N}(0, \sigma^2)}}{\underbrace{\hat{\sigma}}_{\xrightarrow{p} \sigma}} \xrightarrow{d} \frac{1}{\sigma}\mathcal{N}(0, \sigma^2) = \mathcal{N}(0, 1)\]

• Our normalized mean with the estimated standard error is still asymptotically standard normal.

Attitudes towards the parties

anes |> summarize(est = mean(ft_diff),
                  se = sd(ft_diff) / sqrt(n()),
                  lower = est - qnorm(0.975) * se,
                  upper = est + qnorm(0.975) * se)

# A tibble: 1 × 4
    est    se lower upper
  <dbl> <dbl> <dbl> <dbl>
1  2.28 0.807 0.699  3.86

• Our 95% confidence interval for the average Dem \(-\) Rep difference is \([0.70, 3.86]\)

• Interpretation: We are 95% confident that the true population difference in Dem vs. Rep thermometers is between \([0.70, 3.86]\)

• This does not mean that there is a 95% probability that the true difference lies between \([0.70, 3.86]\)

  • \(\mu\) is fixed - it either is or isn’t in this particular interval.
  • The bounds are the estimator - they have randomness under repeated sampling.
  • The specific bounds we calculate are the estimate

Interpreting Confidence Intervals

Small sample confidence intervals

Small sample confidence intervals

Small sample confidence intervals

• Our 95% confidence interval from above is a large sample approximation

  • Relies on \(\frac{\sqrt{n}(\bar{Y} - \mu)}{\hat{\sigma}} \xrightarrow{d} \mathcal{N}(0, 1)\)

• Even if our data were normally distributed, \(Y_i \overset{\text{i.i.d.}}{\sim} \mathcal{N}(\mu, \sigma^2)\), the asymptotic confidence interval is not quite exact

  • Because we’re estimating the standard error.

• Getting exact coverage guarantees for our confidence intervals in a finite sample requires some additional assumptions.

  • Our sampling distribution may also not be normal in finite samples.

• Student (1908) showed that for normally distributed data, \(Y_i \overset{\text{i.i.d.}}{\sim} \mathcal{N}(\mu, \sigma^2)\) where \(\hat{\sigma^2}\) is the unbiased sample variance estimator:

  \[T = \frac{\sqrt{n}(\bar{Y} - \mu)}{\hat{\sigma}} \sim t_{(n-1)} \]

• \(t_{(n-1)}\) is the t-distribution with \(n-1\) “degrees of freedom”

Small sample confidence intervals

• The t distribution is the ratio of a standard normal and the square root of an independent chi-squared divided by its degrees of freedom.

  \[T = \frac{\sqrt{n}(\bar{Y} - \mu)}{\hat{\sigma}} = \underbrace{\frac{\sqrt{n}(\bar{Y} - \mu)}{\sigma}}_{\text{Normal}(0, 1)} \times \left(\underbrace{\frac{(n-1)\hat{\sigma}^2}{\sigma^2}}_{\chi^2_{(n-1)}} \bigg/ (n-1)\right)^{-1/2} \]

• The chi-squared distribution is the distribution of the sum of squared i.i.d. standard normals.

  • The parameter is the number of independent normals being summed - typically referred to as the “degrees of freedom”
  • Under normality, \(\frac{(n-1)\hat{\sigma}^2}{\sigma^2} = \sum_{i=1}^n \left(\frac{Y_i - \bar{Y}}{\sigma}\right)^2\) is the sum of \(n\) squared normal deviations.
  • But \(\sum_{i=1}^n (Y_i - \bar{Y}) = 0\). Once we know \(n-1\) of the outcomes, the last is fixed. So it behaves like a sum of only \(n-1\) independent squared standard normals: \(\chi^2_{(n-1)}\)

The t-distribution

One-sided confidence intervals

• Most of the time, researchers present two-sided and symmetric confidence intervals.

  • But there’s nothing saying that we have to be symmetric or two-sided!
  • It just depends on where we want to allocate our “coverage failure” \(\alpha\).

• We could instead stack it all on one side and construct a one-sided confidence interval!

  \[P( \theta \le u) = 1 - \alpha\]

• For an asymptotically normal CI of the sample mean, this is just

  \[u = \bar{Y} + z_{1-\alpha}\frac{\hat{\sigma}}{\sqrt{n}}\]

• For a 95% interval, \(\alpha = 0.05\) and \(z_{0.95} \approx 1.645\) (qnorm(0.95) in R)

  • We’ve narrowed the upper bound a bit in exchange for an arbitrary lower bound \(-\infty\).
  • Equivalent idea for a one-sided lower bound

One-sided confidence intervals

Hypothesis Testing

• Another approach to thinking about uncertainty in our estimator is from the perspective of evaluating hypotheses
• Assume a particular “truth” about the parameter we want to learn about.
  • Calculate the probability of observing our sample given this assumption.
  • A sufficiently unlikely sample may persuade us that our assumption should be rejected.
• The “null hypothesis statistical testing” (NHST) paradigm is ubiquitious in science.
  • Combines ideas from both Fisher (1925) and Neyman-Pearson (1933) approaches to hypothesis tests.

General Framework

1. Choose a null \(H_0\) and an alternative \(H_a\) hypothesis about some feature of the population
2. Choose a test statistic \(T\) that is a function of the data we observe \(T(Y_1, Y_2, \dotsc, Y_n)\)
3. Choose a desired Type I (false positive) error rate
4. Find the distribution of the test statistic under the null hypothesis
5. Define the rejection region for the chosen error rate in terms of values of the test statistic
6. Calculate whether the observed test statistic falls in the rejection region
7. If it does, reject the null. Otherwise, fail to reject the null.

Null and Alternative Hypotheses

• The null hypothesis is the statement we wish to find evidence against.
  • Usually a “baseline” assumption: “no effect/difference/deviation from what’s already known”
  • We denote this as \(H_0: \mu = \mu_0\)
• The alternative hypothesis is the research claim we wish to make
  • “There is an effect/difference/deviation from baseline”
  • We denote this as \(H_a: \mu \neq \mu_0\)
• Ex: The average population difference between Democratic and Republican feeling thermometer ratings
  • Null Hypothesis: \(H_0: \mu = 0\) - The population average is zero
  • Alternative Hypothesis: \(H_a: \mu \neq 0\) - The population average is not zero
• Most of the time we see two-sided hypothesis tests (\(\neq\))
  • We are willing to reject in favor of a difference in either direction.
  • Sometimes (though rarely), we might only want to reject the null if the result is in a particular direction.
  • \(H_0: \mu = 0, H_a: \mu > 0\)

Step 1: Null and alternative hypotheses

Choosing a test statistic

• Given a particular null and alternative, we want to choose a test statistic that will be large if the null is false (and not too large if it is true).

  • We also want to know (or be able to calculate) the distribution of the test statistic under the null

• Our most common test statistic is the t-statistic (esp. for tests based on asymptotic normality):

  \[ T = \frac{\hat{\mu} - \mu_0}{\widehat{SE}(\hat{\mu})} \]

• Ex. For testing the null of \(\mu = 0\) difference in Democratic v. Republican feeling thermometers using the sample mean estimator

  \[ T = \frac{\bar{Y} - 0}{\hat{\sigma}/\sqrt{n}} \]

• We’ve shown that under the null \(T\) is asymptotically standard normal

  \[T \overset{a}{\sim} \mathcal{N}(0, 1) \]

The test statistic

False Positives and False Negatives

• At the end of the test we will either reject the null \(H_0\) in favor of \(H_a\) or fail to reject
  • Each of these decisions carries with it a risk of error
• Type I error or false positives
  • We reject the null even though it is true
• Type II error or a false negative
  • We fail to reject the null when it is false.

Size and Power of a Test

• In NHST, we directly control the Type I or False positive rate
  • This is because we only have probabilities given \(H_0\) is true
• The level or size of our test, denoted \(\alpha\) is our chosen probability of a Type I error.
• Our choice of level depends on how much we’d like to avoid a false discovery
  • In social sciences, typical convention is \(\alpha = .05\) - but this is arbitrary!
  • To detect the Higgs Boson, CERN physicists used \(\alpha = 1 - \Phi(5) = 2.87 \times 10^{-7}\)
• Choosing a low Type I error trades off a potentially higher Type II error (more false negatives)
  • The power of a test is defined as \(1 - Pr(\text{False negative})\)
  • This is in part a function of our chosen Type I error, but also a function of an unknown: the true parameter value (relative to the sampling variance of our estimator)
  • We’ll often calculate power before running our study under hypothesized values of the parameter to figure out how large of an effect our study can plausibly detect.
    • You’ll be doing this in section this week!

The Type I error rate

The null distribution

The rejection region

• Once we know the distribution of the test statistic under the null, we need to decide what realizations of the test statistic will lead to a decision of rejecting \(H_0\)

  • This is known as the rejection region for the test.
  • How large of a \(T\) will lead us to conclude in favor of the alternative?

• To do this, we need to find the value \(c\) such that

  \[Pr(-c \le T \le c) = 1 - \alpha\]

• For a two-sided test, \(H_a: \mu \neq \mu_0\), we’ll allocate our “\(\alpha\) budget” symmetrically to both tails of the test statistic distribution

  • Equivalent to \(Pr(|T| \le c) = 1 - \alpha\)

The rejection region

• For an asymptotically standard normal test statistic, our critical values should look familiar

  \[\begin{aligned} Pr(-c \le T \le c) &\approx \Phi(c) - \Phi(-c) && \small\color{#777777}{\text{difference of two CDFs}} \\ &= \Phi(c) - \left[1 - \Phi(c)\right] && \small\color{#777777}{\text{symmetry: } \Phi(-x) = 1 - \Phi(x)} \\ &= 2\Phi(c) - 1 \end{aligned}\]

• Setting this equal to \(1 - \alpha\) and solving for \(c\)

  \[\begin{aligned} 2\Phi(c) - 1 &= 1 - \alpha \\ \Phi(c) &= 1 - \frac{\alpha}{2} && \small\color{#777777}{\text{add 1, divide by 2}} \\ c &= \Phi^{-1}\left(1 - \frac{\alpha}{2}\right) = z_{1 - \alpha/2} && \small\color{#777777}{\text{apply the inverse CDF}} \end{aligned}\]

• Reject \(H_0\) if \(|T| > z_{1-\alpha/2}\). For \(\alpha = 0.05\), reject if \(|T| > 1.96\) - the same critical value as our 95% confidence interval!

The rejection region

The decision

p-values

• The p-value for a particular hypothesis test is the probability of observing a test statistic at least as extreme as the one we observe in the sample given that the null is true
  • If the p-value falls below our chosen \(\alpha\), we reject the null
• The smallest \(\alpha\) for which we would still reject our null hypothesis.
  • Calculate using 2*pnorm(-abs(T)) for a two-sided asymptotically normal test.
• Be careful! P-values are not…
  • …evidence that the effect is substantively significant
  • …the probability that the null hypothesis is true \(P(H_0 \text{ true}|\text{observed data}) \neq P(\text{observed data} | H_0 \text{ true})\)
  • …the probability that the alternative is true (all probabilities here are conditional on the null)

Connection to confidence intervals

• Confidence intervals and hypothesis tests have a dual relationship
  • A \(100(1-\alpha)\%\) confidence interval contains all of the null hypotheses that we’d fail to reject with an \(\alpha\) level test.
• Two-sided CIs correspond to two-sided hypothesis tests.
  • One-sided CIs correspond to one-sided hypothesis tests.
• This relationship also gives us a way of generating a confidence interval if all we have is a hypothesis test (e.g. randomization tests in ps813).
  • Vary \(\mu_0\) across some range of values
  • Conduct the hypothesis test at each value
  • Collect all the values where \(p\) is greater than \(\alpha\).
  • That’s our \(100(1-\alpha)\%\) interval!

Multiple testing

Bonferroni correction

• When testing multiple hypotheses, we want to control the global error rate of the full “family” of tests

• One error rate we could target is family-wise error rate

  \[\text{FWER} = P(\text{at least one rejection} | \text{All nulls are true}) \]

• We can write this using the complement

  \[\text{FWER} = 1 - P(\text{fail to reject all tests} | \text{All nulls are true})\]

• If I have \(K\) independent tests where I reject each at the rate \(\alpha\), my FWER is

  \[\begin{aligned} \text{FWER} &= 1 - P(\text{fail to reject all } K | \text{All nulls are true}) \\ &= 1 - \textstyle\prod_{k=1}^K P(\text{fail to reject } H_{0k} | H_{0k} \text{ true}) && \small\color{#777777}{\text{independence}} \\ &= 1 - \textstyle\prod_{k=1}^K (1 - \alpha) && \small\color{#777777}{\text{Type I error rate } \alpha} \\ &= 1 - (1 - \alpha)^K \end{aligned}\]

Bonferroni correction

• Uncorrected, our testing procedure fails to control the FWER at the target \(\alpha\)

  • Since \(1-\alpha < 1\), \((1 - \alpha)^{K} \to 0\) as \(K \to \infty\), so the FWER \(\to 1\).

• One simple correction is to alter the rate that we reject each individual hypothesis so that the FWER remains at most \(\alpha\) even as we increase the number of tests we conduct.

  • The Bonferroni correction: reject each test at \(\alpha/K\) rather than \(\alpha\)
  • Equivalently, reject if \(|T_k| > z_{1 - \alpha/(2K)}\) - a wider rejection region for every test

• Why does this work? Remember that the probability of the union is bounded by the sum of the component probabilities.

  \[\begin{aligned} \text{FWER} &= P\left(\textstyle\bigcup_{k=1}^K \{\text{reject } H_{0k}\} \,\middle|\, \text{All nulls are true}\right) \\ &\le \textstyle\sum_{k=1}^K P(\text{reject } H_{0k} | H_{0k} \text{ true}) && \small\color{#777777}{\text{union bound}} \\ &\le K \times \frac{\alpha}{K} = \alpha && \small\color{#777777}{\text{each test at level } \alpha / K} \end{aligned}\]

• Ex: \(K = 20\) tests at \(\alpha = 0.05\). We’d reject each null only if \(p < 0.0025\) - \(|T| > 3.02\) instead of \(1.96\)

  • The consequence is a much higher bar for each test: lower power and more false negatives.

Holm correction

• An alternative approach is the Holm step-down procedure (Holm 1979)

  • Also controls the FWER at \(\alpha\) (for any dependence)
  • Rejects at least as often as Bonferroni (so more power)

• The Holm testing approach:

  1. Sort the p-values from smallest to largest: \(p_{(1)} \le p_{(2)} \le \dotsc \le p_{(K)}\)
  2. Start at \(k = 1\): if \(p_{(k)} \le \frac{\alpha}{K - k + 1}\), reject \(H_{0(k)}\) and move on to \(k + 1\)
  3. Stop at the first \(p_{(k)} > \frac{\alpha}{K - k + 1}\): fail to reject \(H_{0(k)}\) and every hypothesis after it

• The bar relaxes as we go - first step is just Bonferroni:

  \[\underbrace{\frac{\alpha}{K}}_{k = 1}, \quad \underbrace{\frac{\alpha}{K - 1}}_{k = 2}, \quad \underbrace{\frac{\alpha}{K - 2}}_{k = 3}, \quad \dotsc, \quad \underbrace{\frac{\alpha}{1}}_{k = K}\]

• Two equivalent ways to correct: shrink the threshold \((p_k < \alpha/K)\) or inflate the p-value \((K p_k < \alpha)\)

  • In R, p.adjust(p, method = "holm") (or "bonferroni") returns adjusted p-values we can compare to \(\alpha\) as if it were a single test

Multiple testing

Bonferroni and Holm

Attitudes towards the parties by age

Caution with NHST

• The American Statistical Association’s statement on p-values (Wasserstein and Lazar 2016)
  1. P-values can indicate how incompatible the data are with a specified statistical model
  2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
  3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
  4. Proper inference requires full reporting and transparency.
  5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
  6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
• A p-value is \(P(\text{data at least this extreme} | H_0)\), not \(P(H_0 | \text{data})\)

Caution with NHST

• “Significant” vs. “not significant” is not itself significant (Gelman and Stern 2006)
  • 65+ is “significant” and 18–29 is not - but the difference between them is 0.90 (SE 2.71), nowhere near significant!
• Researcher degrees of freedom - “p-hacking” and the “garden of forking paths” (Simmons et al. 2011; Gelman and Loken 2014)
  • Many choices of sample, outcome, controls and subgroups - an implicit multiple testing problem even if only one test is reported
• Publication bias - journals (and authors) favor results that clear \(p < 0.05\) (Gerber and Malhotra 2008)
  • Published estimates often bunch just past the critical value.

Caution with NHST

• The significance filter (Gelman and Carlin 2014)
  • In low-powered studies, the estimates that pass the threshold exaggerate the effect (Type M)
  • And very often have the wrong sign (Type S)
• Meaning of a null
  • Some failures to reject are genuinely informative zeroes (Rainey 2014; Hartman and Hidalgo 2018)
  • Failures for underpowered studies are often uninformative. Those studies would likely fail to detect most substantively interesting effects (Arel-Bundock et al. 2026).
• Proposed fixes
  • Stricter default threshold (\(p < 0.005\)) (Benjamin et al. 2018)
  • Abandon “statistical significance” altogether in favor of reporting estimates + ranges of values “compatible” with the data (Amrhein et al. 2019)

Next week

• Regression
  • How do we model the conditional expectation of \(Y\) as \(X\) changes
  • Flexible but noisy methods: binned scatterplots, kernel estimators
  • More precise but need assumptions: ordinary least squares
• Plug-in estimator for the “best linear predictor”
  • Later: Interpreting the BLP as the CEF.