Week 6: Estimation

Where we’ve been

Where we’re going

Defining the population

• Statistical inference is the task of learning about some feature of a target population
  • e.g. All U.S. voters
  • All voters in Wisconsin
  • All municipalities in the United States
  • All online reviews of restaurants
• The population is not (fully) observed
  • Otherwise we wouldn’t need statistics!
• In the case of a finite population, we could imagine listing the total number of units in the population \(N\)
  • This is our sampling frame (e.g. the voter file)

Running Example: 2024 General Social Survey

• Our running example for today is the General Social Survey - a nationally representative survey of U.S. adults that has been conducting regular interviews since 1972.
• Target population: U.S. adults (18+) living in noninstitutional housing, April–December 2024
• Sampling frame: the NORC 2020 National Sampling Frame
  • A list of housing units built mostly from the USPS delivery file
  • Supplemented by NORC directly identifying addresses where USPS coverage is below 90%

Defining the population

Defining the estimand

• Our goal is to learn about some feature of the target population.
• We might be interested in the distribution of some variable \(Y\) in that population
  • Turnout
  • Vote choice
  • Restaurant review score
• Often we want a summary of that distribution
  • An expected value: \(\mathbb{E}[Y]\)
  • Or, if we have two variables, a regression function: \(\mathbb{E}[Y | X]\)
• This is our quantity of interest or estimand
  • Often denoted with a greek letter: \(\mu\), \(\theta\), \(\tau\)

Defining the estimand

Running Example: 2024 General Social Survey

• We’ll focus on self-reported turnout in the 2020 U.S. presidential election

“In 2020, you remember that Joe Biden ran for President on the Democratic ticket against Donald Trump for the Republicans. Do you remember for sure whether or not you voted in that election?”

• Let \(\mathbb{E}[Y]\) denote the expected turnout in the target population
  • With a finite population, this is just the average of the response \(Y_i\) of every member \(i\)
  • \(Y_i = 1\) if voted, \(Y_i = 0\) otherwise

\[ \mu = \mathbb{E}[Y] = \frac{1}{N} \sum_{i=1}^N Y_i \]

Sampling design

• We do not observe the population directly. Instead, we observe a sample from that population

  • Knowing the sampling design - how that sample was obtained - is essential to figuring out how what we calculate in sample relates to the target population.

• In statistics, we focus mainly on probabilistic samples

  • Let \(Z_i = 1\) denote that unit \(i\) is sampled
  • Let \(n\) denote the size of the sample

• A simple random sample is a design where each sample of size \(n\) has equal probability of occurring.

  • Let \(\mathbf{Z} = (Z_1, \dotsc, Z_N)\) be the collection of sampling indicators

  \[P(\mathbf{Z}) = \begin{cases} 1/\binom{N}{n} & \text{if } \sum_{i=1}^N Z_i = n \\ 0 & \text{otherwise} \end{cases}\]

• \(P(Z_i = 1) = \frac{n}{N}\) for all units.

Sampling design

• A Bernoulli sampling design selects units by independent coin flips, each with some probability \(p\)

  \[ P(Z_i = 1) = p \quad \forall i \in \{1, 2, \dotsc, N\} \]

• What’s the difference?

  • Under a Bernoulli design, sample size \(n\) is random. Under an SRS, it’s fixed
  • Under a Bernoulli design, selection indicators are independent. Under an SRS, they’re (slightly) dependent.

• What’s the difference in practice?

  • None. If \(N\) is very large relative to \(n\), the two designs are indistinguishable
  • Under an SRS, learning that \(i\) was sampled barely changes the chance that \(j\) was: \(P(Z_j = 1 \mid Z_i = 1) = \frac{n-1}{N-1} \approx \frac{n}{N} = P(Z_j = 1)\)
  • Under a Bernoulli design, \(n\) concentrates around \(Np\). Conditional on \(n\), a Bernoulli sample is an SRS.
  • e.g. the GSS: \(n \approx 3{,}300\) out of \(N \approx 260\) million adults

Sampling design

• Alternatively, we’ll often describe our sampling design by characterizing the distribution from which the \(Y_i\) observed in our sample are drawn.
  • (e.g.) \(Y_i \sim ?(\mu, \sigma^2)\)
  • Each \(Y_i\) is a draw from some unknown distribution with finite mean \(\mu\) and finite variance \(\sigma^2\)
  • From here, we’ll use \(i\) to index observations in the sample of size \(n\).
• A key assumption that we’ll make is that observations in sample are independently and identically distributed (i.i.d.)
  • Identically distributed: Our units are each drawn from the population distribution of interest.
  • Independently distributed: Knowing who shows up in one draw tells us nothing about the next.
• If we sampled with replacement, we’d have exactly independent observations.
• Under SRS/Bernoulli sampling, there’s a slight dependence (drawing \(j\) rules them out in the next draw).
  • But this vanishes as the population size becomes arbitrarily large relative to sample.

Running Example: 2024 General Social Survey

• Real surveys are more involved. The 2024 GSS uses a multi-stage cluster design:
  1. Sample 74 areas (metros or groups of counties): 20 large areas with certainty, the rest w/ probability proportional to their number of housing units
  2. Within those, sample 400 segments (census tracts or block groups) (proportional to number of housing units)
  3. Within segments, sample 13,900 addresses so that every housing unit has the same chance of selection
  4. Within each responding household, select one adult at random
• This is not an SRS of adults
  • How do they correct this? Weights!

Sampling design

Estimators

• Once we have an estimand in mind, we want to figure out the algorithm to transform our sample of observations into a best guess.

  • Consider a sample of size \(n\) with observations indexed by \(i \in \{1,2, 3, \dotsc n\}\)

• An estimator is a function of our sample.

  • It’s a random variable too!

• Since we’re targeting the population mean, a natural choice for an estimator would be the sample mean

  \[ \hat{\mu} = \bar{Y}_n = \frac{1}{n} \sum_{i=1}^n Y_i \]

• We’ll often denote estimators with a hat put on top of the greek letter for the estimand: \(\hat{\mu}\), \(\hat{\theta}\), \(\hat{\tau}\)

  • Means are also often denoted with a bar on top of the variable

Estimates

• An estimate is a single realization of an estimator

  • It’s the number that you calculate for your dataset.

• In the GSS, \(n = 3{,}373\) respondents say whether they voted in 2020. The sample mean is

  \[\hat{\mu} = \bar{Y}_n = \frac{1}{n} \sum_{i=1}^n Y_i = 0.778\]

• An alternative estimator is the weighted mean using the survey weight \(w_i\)

  \[\hat{\mu}_w = \frac{\sum_{i=1}^n w_i Y_i}{\sum_{i=1}^n w_i} = 0.770\]

Estimates in R

library(tidyverse)

gss <- read_csv("data/gss2024.csv") |>
  filter(!is.na(voted20))   # said whether they voted in 2020

est <- gss |>
  summarize(n = n(),
            ybar = mean(voted20),
            ybar_w = weighted.mean(voted20, weight))
est

# A tibble: 1 × 3
      n  ybar ybar_w
  <int> <dbl>  <dbl>
1  3373 0.778  0.770

Estimators are random variables

Estimation

• How do we know if we’ve picked a good estimator?
  • Will it be close to the truth?
  • Will it be systematically higher or lower than the target?
• We cannot answer this question from our sample alone - we need theory.
• In statistics, we study the properties of estimators
  • Bias: \(\mathbb{E}[\hat{\mu}] - \mu\)
  • Variance: \(\Var(\hat{\mu})\)
  • Consistency: Does \(\hat{\mu}\) converge in probability to \(\mu\) as \(n\) goes to infinity?
  • Asymptotic distribution: Is the sampling distribution of \(\hat{\mu}\) well approximated by a known distribution?

Estimation

Unbiasedness of the sample mean

• Assume that each of our observations in the sample \(Y_i \sim ?(\mu, \sigma^2)\) come from the same population with a finite mean.

• Can we show that this estimator has zero bias?

  • In other words, is its expected value equal to the true parameter value?

• Start by writing down our estimator and taking the expectation

  \[\mathbb{E}[\hat{\mu}] = \mathbb{E}\left[\frac{1}{n}\sum_{i=1}^n Y_i\right]\]

• Next we use linearity of expectations

  \[\mathbb{E}[\hat{\mu}] = \mathbb{E}\left[\frac{1}{n}\sum_{i=1}^n Y_i\right] = \frac{1}{n}\mathbb{E}\left[\sum_{i=1}^n Y_i\right] = \frac{1}{n}\sum_{i=1}^n \mathbb{E}\left[Y_i\right]\]

Unbiasedness of the sample mean

• Finally, under our assumption that all \(Y_i\) come from the population of interest with expectation \(\mu\)

  \[\frac{1}{n}\sum_{i=1}^n \mathbb{E}\left[Y_i\right] = \frac{1}{n}\sum_{i=1}^n \mu = \frac{n \mu}{n} = \mu\]

• Therefore, the bias, \(\text{Bias}(\hat{\mu}) = \mathbb{E}[\hat{\mu}] - \mu = 0\)

Unbiasedness of the sample mean

Sampling variance of the sample mean

• What else can we learn about the distribution of an estimator?
  • Remember, an estimator is a random variable.
  • We’ll refer to the distributions of estimators as sampling distributions
• We’ve established \(\mathbb{E}[\hat{\mu}]\)…now what about \(\Var(\hat{\mu})\)?
  • How far do draws from our estimator tend to land from its expectation?
• Why do we want to know this?
  • So we can characterize the degree of uncertainty in our particular realization of the estimator.
  • Confidence intervals, hypothesis tests, etc… (next few weeks!)

Sampling variance of the sample mean

• Let’s take the variance of \(\hat{\mu}\)

  \[\Var(\hat{\mu}) = \Var\left(\frac{1}{n}\sum_{i=1}^n Y_i\right)\]

• Constants come out of a variance squared

  \[\Var(\hat{\mu}) = \Var\left(\frac{1}{n}\sum_{i=1}^n Y_i\right) = \frac{1}{n^2}\Var\left(\sum_{i=1}^n Y_i\right)\]

• The variance of a sum is the sum of the variances plus all of the covariances

  \[\frac{1}{n^2}\Var\left(\sum_{i=1}^n Y_i\right) = \frac{1}{n^2}\left[\sum_{i=1}^n \Var(Y_i) + \sum_{i=1}^n \sum_{j \neq i} \Cov(Y_i, Y_j)\right]\]

Sampling variance of the sample mean

• Under independence, all of the covariance terms are zero: \(\Cov(Y_i, Y_j) = 0\) for \(i \neq j\)

  \[\frac{1}{n^2}\left[\sum_{i=1}^n \Var(Y_i) + \sum_{i=1}^n \sum_{j \neq i} \Cov(Y_i, Y_j)\right] = \frac{1}{n^2}\sum_{i=1}^n \Var(Y_i)\]

• Under the assumption that each observation comes from the same target distribution, every observation has the same population variance \(\Var(Y_i) = \sigma^2\)

  \[\frac{1}{n^2}\sum_{i=1}^n \Var(Y_i) = \frac{1}{n^2}\sum_{i=1}^n \sigma^2 = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}\]

Sampling variance of the sample mean

• So under i.i.d. sampling, our sampling variance is

  \[\Var(\hat{\mu}) = \frac{\sigma^2}{n}\]

• We’ll also often talk about the standard deviation (square root of the variance) of a sampling distribution

  • For estimators we refer to this particular standard deviation as a standard error

  \[ \text{SE}(\hat{\mu}) = \sqrt{\Var(\hat{\mu})} = \frac{\sigma}{\sqrt{n}} \]

• Note what we needed for unbiasedness: identical sampling

  • But this variance expression requires: identical and independent draws.
  • What happens with dependence? We’ll have a different expression for the standard error.

• What happens as \(n\) gets large?

  • The standard error gets smaller! Our estimator becomes more precise.

Sampling variance of the sample mean

The sampling distribution of an estimator

• Under i.i.d. \(Y_i\), the sampling distribution of \(\hat{\mu}\) has mean \(\mu\) and variance \(\sigma^2/n\)
• Is that enough to pin down its full distribution? Not quite yet!

• A detour into large-sample approximations gets us an answer without more assumptions on \(Y_i \sim ?(\mu, \sigma^2)\)

Asymptotic properties

• Another way of talking about properties of estimators is to consider how they behave as the sample size gets large.
  • Sometimes this is just easier than working with finite sample expectations.
  • Additionally, we might have a biased estimator. Does that bias “go away” in large samples?
• This requires us to talk about sequences of estimators indexed (typically) by sample size.
  • We’ll typically denote this sequence with a subscript - for example, for the sample mean… \[\bar{Y}_n = \{\bar{Y}_1, \bar{Y}_2, \bar{Y}_3, \dotsc\}\]
• Can we characterize the limit of this sequence of estimators?
  • It’s a random variable…so we need a slightly different language compared to deterministic series.
  • But a lot of intuitions from convergence of series carry over!

Convergence in probability

• First, we can talk about the convergence of a sequence to a single value
  • There are lots of different forms of convergence here, but the most common that we use is convergence in probability
• Convergence in probability
  • A sequence of random variables \(\hat{\mu}_n\) is said to converge in probability to some value \(\mu\) if for every \(\epsilon > 0\), as \(n \to \infty\) \[\Pr(|\hat{\mu}_n - \mu| \ge \epsilon) \to 0\]
• We’ll denote this as \(\hat{\mu}_n \xrightarrow{p} \mu\). You’ll see other texts use the term “plim”
• An estimator that converges in probability to the target estimand is called consistent.

Weak Law of Large Numbers

• The weak law of large numbers states that the sample means of independent and identically distributed random variables…

  • … converge in probability to their population expected value
  • Assuming that mean is finite.

• Let’s prove this! We’ll also assume finite variance to make it easier. \[P(|\bar{Y}_n - \mu | \ge \epsilon) \to 0 \]

• The trick will rely on a known bound on how far a random variable can fall from its mean: Chebyshev’s Inequality

  • This bound depends on the variance of that random variable.
  • And we’ve already shown that the variance of \(\hat{\mu} = \frac{1}{n}\sum_{i=1}^n Y_i\) goes to zero as \(n\) gets large.

Chebyshev’s Inequality

• For any random variable \(X\) with a finite mean and variance, and any \(\epsilon > 0\), Chebyshev’s Inequality states:

  \[P(|X - \mathbb{E}[X]| \ge \epsilon) \le \frac{\Var(X)}{\epsilon^2}\]

• The chance of landing far from the mean is bounded by the variance

  • Holds for any distribution with a finite variance

Weak Law of Large Numbers

• Apply Chebyshev’s inequality to the sample mean, which has \(\mathbb{E}[\bar{Y}_n] = \mu\) and \(\Var(\bar{Y}_n) = \frac{\sigma^2}{n}\)

  \[P(|\bar{Y}_n - \mu| \ge \epsilon) \le \frac{\Var(\bar{Y}_n)}{\epsilon^2} = \frac{\sigma^2}{n\epsilon^2}\]

• For any fixed \(\epsilon > 0\), the right-hand side goes to zero as \(n\) gets large

  \[\lim_{n \to \infty} \frac{\sigma^2}{n\epsilon^2} = 0\]

• A probability can’t be negative, so it’s squeezed between \(0\) and something going to \(0\)

  \[P(|\bar{Y}_n - \mu| \ge \epsilon) \to 0\]

• Therefore, \(\bar{Y}_n \xrightarrow{p} \mu\): the sample mean is consistent for the population mean

Weak Law of Large Numbers

Properties of convergence in probability

• Consider two sequences \(X_n \xrightarrow{p} x\) and \(Y_n \xrightarrow{p} y\) and continuous function \(g()\).
• By the continuous mapping theorem

1. \(X_n + Y_n \xrightarrow{p} x + y\)
2. \(X_nY_n \xrightarrow{p} xy\)
3. \(X_n/Y_n \xrightarrow{p} x/y\) if \(y \neq 0\)
4. \(g(X_n) \xrightarrow{p} g(x)\)

Running Example: 2024 General Social Survey

• In our particular sample, the sample mean estimator gives an estimated turnout of \(\hat{\mu} = 0.778\)

• We can estimate the population variance in a similar manner (we’ll spend more time on this next week): plug in the sample variance

  \[\hat{\sigma}^2 = \frac{1}{n-1}\sum_{i=1}^n (Y_i - \bar{Y}_n)^2 = 0.173\]

• So we computed a guess of \(0.778\)

  • And we think it came from a distribution centered on the truth with a standard error of \(\sqrt{\hat{\sigma}^2/n} = 0.0072\)

• The Census reports that the true turnout rate in 2020 was \(0.668\) (Fabina and Scherer 2022)

• What drives the error?

  • Sampling error: Noise due to the fact that the sample isn’t the same as the population - what we can talk about.
  • Non-sampling error: Our sample isn’t drawn from a distribution with expected turnout \(0.668\) - it’s coming from a different distribution.

Running Example: 2024 GSS in R

truth <- 0.668   # Census CPS: share of citizens 18+ who voted in 2020

gss_se <- gss |>
  summarize(n = n(),
            ybar = mean(voted20),
            s2 = var(voted20),         
            se = sqrt(s2 / n),
            ybar_w = weighted.mean(voted20, weight))
gss_se

# A tibble: 1 × 5
      n  ybar    s2      se ybar_w
  <int> <dbl> <dbl>   <dbl>  <dbl>
1  3373 0.778 0.173 0.00716  0.770

Next week

• Estimation, continued
  • Convergence in distribution and the central limit theorem
  • Plug-in estimators
  • Density estimation