Week 9: Regression

Where we’ve been

Where we’ve been

Where we’ve been

Where we’ve been

Where we’re going

Regression

• In the social sciences, we often want to understand how our prediction for an outcome variable \(Y_i\) changes depending on the values of some set of covariates \(\mathbf{X}_i\)
  • Description: A parsimonious summary of the data
  • Forecasting: Given a new value of \(\mathbf{X}_i\), what’s our best prediction for \(Y_i\)?
  • Adjustment for causal inference: Can we explain away \(Y_i\) with confounders \(\mathbf{X}_i\) to make our causal assumptions more credible?
• In all cases, our estimand changes from a single parameter to a function
  • The conditional expectation function
  \[\mu(\mathbf{x}) = \mathbb{E}[Y_i \mid \mathbf{X}_i = \mathbf{x}]\]

Running Example: Hostile sexism and support for Trump

• Do gender attitudes predict support for right-wing political candidates even after we adjust for partisanship?

  • One particular attitude measured by political psychologists is hostile sexism
    • Antagonism towards women seen as seeking power or challenging traditional gender roles (Glick and Fiske 1996)
  • In 2016, hostile sexism predicted support for Trump over Clinton even after adjusting for partisanship and other predictors (Winter 2023).
    • Additionally, it appears to contribute to the education gap in political attitudes among white voters (Schaffner et al. 2018)
  • Does the relationship hold up in 2024?

• Outcome \(Y_i\): Trump feeling thermometer (0-100) from the 2024 ANES (American National Election Studies 2026)

• Regressor \(X_i\): responses to four statements, each on a 5-point scale, rescaled to \([0, 1]\) and averaged

  1. “Many women interpret innocent remarks or acts as being sexist.” (agree-disagree)
  2. “Women seek to gain power by getting control over men.” (agree-disagree)
  3. “When women demand equality these days, how often are they actually seeking special favors?” (always-never)
  4. “When women complain about discrimination, how often do they cause more problems than they solve?” (always-never)

Marginal distributions

Hostile sexism and support for Trump

Why models?

• It might seem like we can just use the same sample mean estimator as before but on subsets of the data
  • With discrete covariates, we can!
  • Regression is just taking a conditional mean.
• Suppose we separated our respondents into two “bins”: high sexism and low sexism

  • Call respondents who agree with the statements on average (\(X_i > 0.5\)) high and those below low

  • The CEF is just two distinct quantities.

  • We can estimate each by a sample mean within its group

    \[\hat{\mu}(\text{high}) = \frac{1}{n_{\text{high}}}\sum_{i : X_i > 0.5} Y_i \qquad \widehat{SE}\left(\hat{\mu}(\text{high})\right) = \frac{\hat{\sigma}_{\text{high}}}{\sqrt{n_{\text{high}}}}\]

  • Everything from the last three weeks carries through.

    • Under i.i.d. sampling from the target population, it’s unbiased, consistent, asymptotically normal

Two groups

• High-sexism respondents rate Trump 34.4 points more warmly.

Binning on every value of \(X\)

• Our hostile sexism measure only has 17 distinct values. We can calculate a conditional mean at every one \[\hat{\mu}(x) = \frac{1}{n_x}\sum_{i : X_i = x} Y_i \qquad n_x = \sum_{i=1}^n I(X_i = x)\]

• No assumptions about the shape of \(\mu(x)\). Our estimator is fully nonparametric

• But the precision at each \(x\) depends only on \(n_x\): \(\widehat{SE}\left(\hat{\mu}(x)\right) = \hat{\sigma}_x / \sqrt{n_x}\)

  • Some bins are very small!

Binning on every value of \(X\)

• Support rises until about \(0.6\) and then appears to level off
• We pay for flexibility with variance: the top two bins only hold 4 and 13 respondents!

The curse of dimensionality

• Binning worked because \(X_i\) is one variable with 17 values, and we have thousands of respondents

• What if we want the CEF given hostile sexism and gender, education, race, partisanship, ideology and age? \[\mu(\mathbf{x}) = \mathbb{E}[Y_i \mid \mathbf{X}_i = \mathbf{x}] \qquad \mathbf{X}_i = (X_{i1}, X_{i2}, \dotsc, X_{ik})\]

  • Binning needs a separate mean for every combination of covariate values
  • And if we have a continuous covariate, nearly every respondent has a unique value of that covariate

• This is the curse of dimensionality

  • As the number of covariates grows, the covariate space grows exponentially

The curse of dimensionality

• 3,810 respondents with every covariate observed. Once we condition on party ID and ideology, the typical respondent shares their covariate bin with only 2 others
• Add age, and 90% of respondents have unique covariate profiles.

Borrowing strength

• To estimate \(\mu(\mathbf{x})\) where we have little data, we have to use observations at other values of \(\mathbf{X}_i\)

• That requires assumptions about how \(\mu(\cdot)\) behaves between the points we observe.

• For example:

  • Smoothness: \(\mu(\cdot)\) doesn’t change much between nearby \(\mathbf{x}\)
  • Functional form: \(\mu(\cdot)\) has a particular shape everywhere (e.g. a line)

• This is why we need models. More covariates, more need to impose structure ex-ante

• Some alternative estimation strategies:

  • Kernel estimators: Average the observations near \(\mathbf{x}\) when estimating \(\mu(\mathbf{x})\)
  • Linear regression: Assume a shape for \(\mu(\cdot)\) and use all of the data to estimate the function.

Local averaging

• Instead of averaging only the exact matches, average every observation within a window \(h/2\) of \(x\) \[\hat{\mu}(x) = \frac{\sum_{i=1}^n I\left(|X_i - x| \le \tfrac{h}{2}\right) Y_i}{\sum_{i=1}^n I\left(|X_i - x| \le \tfrac{h}{2}\right)}\]

• This is the same idea as our kernel density estimator but applied to regression

  • Calculate the mean at each \(x\) using a window centered on \(x\)

• The width \(h\) is a tuning parameter

  • Too small: few observations in each window \(\leadsto\) noisy estimator
  • Too large: we average over a region where \(\mu(x)\) changes \(\leadsto\) bias

• Here, we’re assuming a uniform kernel

  • But as before, we can instead let the weight on each observation fall with its distance from \(x\)

Nadaraya-Watson kernel regression

• The Nadaraya-Watson estimator (Nadaraya 1964; Watson 1964) \[\hat{\mu}_h(\mathbf{x}) = \frac{\sum_{i=1}^n K_h(\mathbf{X}_i - \mathbf{x})\, Y_i}{\sum_{i=1}^n K_h(\mathbf{X}_i - \mathbf{x})} = \sum_{i=1}^n w_i(\mathbf{x})\, Y_i \qquad w_i(\mathbf{x}) = \frac{K_h(\mathbf{X}_i - \mathbf{x})}{\sum_{j=1}^n K_h(\mathbf{X}_j - \mathbf{x})}\]

• A weighted average of the outcomes: the weights are non-negative, sum to \(1\), and are larger for the observations closer to \(\mathbf{x}\)

• The same kernels as in density estimation, \(K_h(\mathbf{u}) = K(\mathbf{u}/h)\)

  • With \(k\) dimensions, often use the product kernel: \(K(\mathbf{u}) = \prod_{j=1}^k K(u_j)\) (with covariates on a common scale)
    • The product of covariate-specific kernels (which can be uniform/triangular/etc…)
  • Another approach is the “nearest neighbors” method: at each \(x\), average the outcomes of the \(m\) closest observations (based on some distance metric)
    • Larger windows in sparser parts of the covariate space.

The kernel weights

• Gaussian kernel, \(h = 0.05\)

Nadaraya-Watson in R

# Nadaraya-Watson with a Gaussian kernel: a weighted mean of Y around x0
nw <- function(x0, x, y, h) {
  w <- dnorm((x - x0) / h)
  sum(w * y) / sum(w)
}

grid <- (0:100) / 100
nw_fit <- tibble(hostile_sexism = grid,
                 fit = map_dbl(grid, nw, x = anes$hostile_sexism,
                               y = anes$ft_trump, h = 0.05))
nw_fit |> filter(hostile_sexism %in% c(0, 0.25, 0.5, 0.75, 1))

# A tibble: 5 × 2
  hostile_sexism   fit
           <dbl> <dbl>
1           0     6.63
2           0.25 29.2 
3           0.5  55.8 
4           0.75 69.8 
5           1    70.1 

Choosing the bandwidth

• The bias depends on how much \(\mu(\cdot)\) curves and whether \(\mathbf{X}_i\) are lopsided around \(\mathbf{x}\)
• The variance depends on \(\Var(Y_i \mid \mathbf{X}_i = \mathbf{x})\) and on how many observations are around \(\mathbf{x}\)

From a local to a global estimator

• What if we assume the CEF is a line, \(\mu(x) = \beta_0 + \beta_1 x\)?
  • Regardless of how many values \(X_i\) takes, we’re only ever estimating two things
• Every observation informs the fit at every \(x\), so the line is more precise
  • But at the cost of bias. Where \(\mu(\cdot)\) bends, the line is biased for the true CEF

Linear regression and ordinary least squares

Best Linear Predictor

• Let’s restrict our focus to functions of \(X\) that are linear predictors

  • That is, they’re linear combinations of the covariates and some set of parameters

• With a single covariate

  \[m(x) = \beta_0 + \beta_1 x\]

• With many covariates, we write

  \[m(\mathbf{x}) = \beta_0 + \beta_1 x_1 + \dotsc + \beta_{k-1} x_{k-1} = \mathbf{x}^\prime\boldsymbol{\beta}\]

  \[\mathbf{x} = (1, x_1, \dotsc, x_{k-1})^\prime \qquad \boldsymbol{\beta} = (\beta_0, \beta_1, \dotsc, \beta_{k-1})^\prime\]

• Review: Inner products

  \[\underbrace{\mathbf{x}^\prime}_{1 \times k} \quad \underbrace{\boldsymbol{\beta}}_{k \times 1} = \begin{pmatrix} 1 & x_1 & x_2 & \cdots & x_{k-1} \end{pmatrix} \begin{pmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \vdots \\ \beta_{k-1} \end{pmatrix} = 1 \cdot \beta_0 + x_1 \beta_1 + x_2 \beta_2 + \dotsc + x_{k-1} \beta_{k-1}\]

Best Linear Predictor

• The best linear predictor is the linear predictor that minimizes the expected squared prediction error

  \[\boldsymbol{\beta} = \arg\min_{\mathbf{b} \in \mathbb{R}^k} \ \mathbb{E}\big[(Y_i - \mathbf{X}_i^\prime\mathbf{b})^2\big]\]

• We found a closed form solution for the two-parameter case in a previous lecture

  \[\beta_1 = \frac{\Cov(X_i, Y_i)}{\Var(X_i)} \qquad \beta_0 = \mathbb{E}[Y_i] - \beta_1\,\mathbb{E}[X_i]\]

• Let’s derive the general form for many covariates

Best Linear Predictor

• We’ll start by taking the partial derivatives of the squared errors inside the expectation with respect to the parameters

  \[ \begin{aligned} \frac{\partial}{\partial \mathbf{b}} (Y_i - \mathbf{X}_i^\prime\mathbf{b})^2 &= \begin{pmatrix} \frac{\partial}{\partial b_0} (Y_i - \mathbf{X}_i^\prime\mathbf{b})^2 \\ \vdots \\ \frac{\partial}{\partial b_{k-1}} (Y_i - \mathbf{X}_i^\prime\mathbf{b})^2 \end{pmatrix} = \begin{pmatrix} -2 \cdot 1 \cdot (Y_i - \mathbf{X}_i^\prime\mathbf{b}) \\ \vdots \\ -2 X_{i,k-1} (Y_i - \mathbf{X}_i^\prime\mathbf{b}) \end{pmatrix} \\[0.4em] &= -2\,\mathbf{X}_i (Y_i - \mathbf{X}_i^\prime\mathbf{b}) = -2\,(\mathbf{X}_i Y_i - \mathbf{X}_i\mathbf{X}_i^\prime\mathbf{b}) \end{aligned} \]

• Review: outer products

  \[\underbrace{\mathbf{X}_i}_{k \times 1} \quad \underbrace{\mathbf{X}_i^\prime}_{1 \times k} = \begin{pmatrix} 1 \\ X_{i1} \\ \vdots \\ X_{i,k-1} \end{pmatrix} \begin{pmatrix} 1 & X_{i1} & \cdots & X_{i,k-1} \end{pmatrix} = \begin{pmatrix} 1 & X_{i1} & \cdots & X_{i,k-1} \\ X_{i1} & X_{i1}^2 & \cdots & X_{i1} X_{i,k-1} \\ \vdots & \vdots & \ddots & \vdots \\ X_{i,k-1} & X_{i,k-1} X_{i1} & \cdots & X_{i,k-1}^2 \end{pmatrix}\]

Best Linear Predictor

• Take the expected value and set it equal to \(\mathbf{0}\) to solve for \(\boldsymbol{\beta}\)

  \[ \begin{aligned} \mathbb{E}\big[{-2}\,(\mathbf{X}_i Y_i - \mathbf{X}_i\mathbf{X}_i^\prime\boldsymbol{\beta})\big] &= \mathbf{0} \\ \mathbb{E}[\mathbf{X}_i Y_i] - \mathbb{E}[\mathbf{X}_i\mathbf{X}_i^\prime]\,\boldsymbol{\beta} &= \mathbf{0} \\ \mathbb{E}[\mathbf{X}_i\mathbf{X}_i^\prime]\,\boldsymbol{\beta} &= \mathbb{E}[\mathbf{X}_i Y_i] \\ \boldsymbol{\beta} &= \big(\mathbb{E}[\mathbf{X}_i\mathbf{X}_i^\prime]\big)^{-1}\,\mathbb{E}[\mathbf{X}_i Y_i] \end{aligned} \]

• What have we assumed?

• Essentially just regularity conditions that ensure the BLP exists

  • Finite mean/variance for \(Y\)
  • Finite means/variances/covariances for \(\mathbf{X}\)
  • \((\mathbb{E}[\mathbf{X}_i\mathbf{X}_i^\prime])^{-1}\) exists (no perfect linear dependence between covariates)

Projection error

• What are the properties of the error between the true \(Y_i\) and the best linear predictor?

  \[e_i = Y_i - m(\mathbf{X}_i) = Y_i - \mathbf{X}_i^\prime\boldsymbol{\beta}\]

• One very important property is that the projection errors are uncorrelated with the covariates

  \[ \begin{aligned} \mathbb{E}[\mathbf{X}_i e_i] &= \mathbb{E}\big[\mathbf{X}_i (Y_i - \mathbf{X}_i^\prime\boldsymbol{\beta})\big] \\ &= \mathbb{E}[\mathbf{X}_i Y_i] - \mathbb{E}[\mathbf{X}_i\mathbf{X}_i^\prime]\,\boldsymbol{\beta} \\ &= \mathbb{E}[\mathbf{X}_i Y_i] - \mathbb{E}[\mathbf{X}_i\mathbf{X}_i^\prime]\big(\mathbb{E}[\mathbf{X}_i\mathbf{X}_i^\prime]\big)^{-1}\mathbb{E}[\mathbf{X}_i Y_i] \\ &= \mathbb{E}[\mathbf{X}_i Y_i] - \mathbb{E}[\mathbf{X}_i Y_i] = \mathbf{0} \end{aligned} \]

  \[\text{First entry of } \mathbf{X}_i \text{ is } 1 \;\Rightarrow\; \mathbb{E}[e_i] = 0 \;\Rightarrow\; \Cov(X_{ij}, e_i) = \mathbb{E}[X_{ij}\, e_i] - \mathbb{E}[X_{ij}]\,\mathbb{E}[e_i] = 0\]

• This holds mechanically with respect to the BLP - no assumptions beyond the regularity conditions above!

Ordinary Least Squares

• We have defined the population best linear predictor from a minimization problem

  \[m(\mathbf{x}) = \mathbf{x}^\prime \boldsymbol{\beta} \qquad \boldsymbol{\beta} = \arg\min_{\mathbf{b} \in \mathbb{R}^k} \ \mathbb{E}\big[(Y_i - \mathbf{X}_i^\prime\mathbf{b})^2\big]\]

• And we’ve written \(\boldsymbol{\beta}\) as a function of population expectations, \(\mathbb{E}[\mathbf{X}_i\mathbf{X}_i^\prime]\) and \(\mathbb{E}[\mathbf{X}_iY_i]\)

  \[\boldsymbol{\beta} = \big(\mathbb{E}[\mathbf{X}_i\mathbf{X}_i^\prime]\big)^{-1}\,\mathbb{E}[\mathbf{X}_i Y_i]\]

• How do we estimate \(\boldsymbol{\beta}\) from an i.i.d. sample from the target population?

• Plug-in principle again: replace each population expectation with a sample mean

  \[\hat{\boldsymbol{\beta}} = \left(\frac{1}{n}\sum_{i=1}^n \mathbf{X}_i\mathbf{X}_i^\prime\right)^{-1} \left(\frac{1}{n}\sum_{i=1}^n \mathbf{X}_iY_i\right)\]

Why “least squares”?

• We get the same estimator by applying the plug-in principle to the minimization problem

  • Minimize the sample mean of the squared errors

  \[\hat{\boldsymbol{\beta}} = \arg\min_{\mathbf{b} \in \mathbb{R}^k} \ \frac{1}{n}\sum_{i=1}^n (Y_i - \mathbf{X}_i^\prime\mathbf{b})^2\]

• Ordinary least squares (OLS): the coefficients with the smallest sum of squared residuals (SSR)

• The same derivative as before but with a sample mean in place of the expectation gives the normal equations

  \[\frac{1}{n}\sum_{i=1}^n \mathbf{X}_i (Y_i - \mathbf{X}_i^\prime\hat{\boldsymbol{\beta}}) = \mathbf{0}\]

• So the residuals \(\hat{e}_i = Y_i - \mathbf{X}_i^\prime\hat{\boldsymbol{\beta}}\) have the sample version of the projection error property

  • They average to \(0\) and are uncorrelated with every covariate in the sample, by construction

OLS in matrix form

• We’ll typically talk about the data in terms of an \(n\)-length outcome vector and an \(n \times k\) matrix of regressors

  • One row per respondent and one column per variable.

  \[\underbrace{\mathbf{Y}}_{n \times 1} = \begin{pmatrix} Y_1 \\ Y_2 \\ \vdots \\ Y_n \end{pmatrix} \qquad \underbrace{\mathbf{X}}_{n \times k} = \begin{pmatrix} \mathbf{X}_1^\prime \\ \mathbf{X}_2^\prime \\ \vdots \\ \mathbf{X}_n^\prime \end{pmatrix} = \begin{pmatrix} 1 & X_{11} & \cdots & X_{1,k-1} \\ 1 & X_{21} & \cdots & X_{2,k-1} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & X_{n1} & \cdots & X_{n,k-1} \end{pmatrix}\]

• The sums in \(\hat{\boldsymbol{\beta}}\) are matrix products

  \[\sum_{i=1}^n \mathbf{X}_i\mathbf{X}_i^\prime = \mathbf{X}^\prime\mathbf{X} \qquad \sum_{i=1}^n \mathbf{X}_iY_i = \mathbf{X}^\prime\mathbf{Y}\]

• The \(\frac{1}{n}\)’s cancel, which gives the common formula for OLS that you will often see:

  \[\hat{\boldsymbol{\beta}} = (\mathbf{X}^\prime\mathbf{X})^{-1}\mathbf{X}^\prime\mathbf{Y}\]

Review: matrix multiplication

• A \(p \times n\) matrix times an \(n \times q\) matrix is \(p \times q\).

  • Entry \((j, l)\) is the inner product of row \(j\) of the first and column \(l\) of the second

• Column \(i\) of \(\mathbf{X}^\prime\) is \(\mathbf{X}_i\) and row \(i\) of \(\mathbf{X}\) is \(\mathbf{X}_i^\prime\), so \(\mathbf{X}^\prime\mathbf{X}\) is a sum of outer products

  \[\underbrace{\mathbf{X}^\prime}_{k \times n} \quad \underbrace{\mathbf{X}}_{n \times k} = \begin{pmatrix} \mathbf{X}_1 & \mathbf{X}_2 & \cdots & \mathbf{X}_n \end{pmatrix} \begin{pmatrix} \mathbf{X}_1^\prime \\ \mathbf{X}_2^\prime \\ \vdots \\ \mathbf{X}_n^\prime \end{pmatrix} = \mathbf{X}_1\mathbf{X}_1^\prime + \mathbf{X}_2\mathbf{X}_2^\prime + \dotsc + \mathbf{X}_n\mathbf{X}_n^\prime\]

• Written out, \(\mathbf{X}^\prime\mathbf{X}\) holds the sums of squares and cross-products of the columns of \(\mathbf{X}\)

  \[\mathbf{X}^\prime\mathbf{X} = \begin{pmatrix} n & \sum_i X_{i1} & \cdots & \sum_i X_{i,k-1} \\ \sum_i X_{i1} & \sum_i X_{i1}^2 & \cdots & \sum_i X_{i1}X_{i,k-1} \\ \vdots & \vdots & \ddots & \vdots \\ \sum_i X_{i,k-1} & \sum_i X_{i,k-1}X_{i1} & \cdots & \sum_i X_{i,k-1}^2 \end{pmatrix} \qquad \mathbf{X}^\prime\mathbf{Y} = \begin{pmatrix} \sum_i Y_i \\ \sum_i X_{i1}Y_i \\ \vdots \\ \sum_i X_{i,k-1}Y_i \end{pmatrix}\]

When does the OLS estimator exist?

• For the BLP, we assumed \(\mathbb{E}[\mathbf{X}_i\mathbf{X}_i^\prime]\) is invertible

  • The sample version of this is that \(\mathbf{X}^\prime\mathbf{X}\) must be invertible

• \(\mathbf{X}^\prime\mathbf{X}\) is invertible if and only if \(\mathbf{X}\) has full column rank

  \[\text{rank}(\mathbf{X}) = k\]

  • No column of \(\mathbf{X}\) is a linear combination of the others
    • We have no perfect collinearity
  • And we need at least as many observations as coefficients, \(n \geq k\)

• Without full rank there is no unique \(\hat{\boldsymbol{\beta}}\) since there are multiple solutions which achieve the same minimum sum of squares.

Review: linear combinations and rank

• A linear combination of the vectors \(\mathbf{a}_1, \dotsc, \mathbf{a}_k\) is a weighted sum. A matrix times a vector is a linear combination of the matrix’s columns

  \[\mathbf{A}\mathbf{c} = c_1\mathbf{a}_1 + c_2\mathbf{a}_2 + \dotsc + c_k\mathbf{a}_k\]

• The columns are linearly independent if the only combination equal to \(\mathbf{0}\) is the trivial one: \(\mathbf{A}\mathbf{c} = \mathbf{0} \implies \mathbf{c} = \mathbf{0}\)

• The rank of a matrix is the largest number of linearly independent columns it contains

  \[\mathbf{A} = \begin{pmatrix} 1 & 2 & 3 \\ 1 & 0 & 1 \\ 1 & 5 & 6 \\ 1 & 1 & 2 \end{pmatrix} \qquad \mathbf{a}_1 + \mathbf{a}_2 - \mathbf{a}_3 = \mathbf{0} \qquad \text{rank}(\mathbf{A}) = 2 < 3\]

Example: dummy variables

• Consider three party categories: Democrat, Independent and Republican

  • Code each category as a dummy (indicator) variable, e.g. \(Z_i^{\text{Rep}} = I(\text{party}_i = \text{Republican})\)

• What if we include an intercept and all three dummies?

  \[\begin{array}{l|cccc} & 1 & Z^{\text{Dem}} & Z^{\text{Ind}} & Z^{\text{Rep}} \\ \hline \text{Democrat} & 1 & 1 & 0 & 0 \\ \text{Republican} & 1 & 0 & 0 & 1 \\ \text{Independent} & 1 & 0 & 1 & 0 \\ \text{Democrat} & 1 & 1 & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \vdots \end{array}\]

  • Everyone is in exactly one category, so \(Z^{\text{Dem}} + Z^{\text{Ind}} + Z^{\text{Rep}} = 1\): the intercept column. The rank is \(3 < 4\)

• If I include an intercept and all of these indicators, the OLS solution is undefined.

  • The fix is to drop one category which is the baseline
  • Each remaining dummy’s coefficient compares its category to the baseline

Dummy variables in R

anes <- anes |>
  mutate(party = case_when(pid7 <= 3 ~ "Democrat", pid7 == 4 ~ "Independent",
                           pid7 >= 5 ~ "Republican"),
         dem = as.numeric(party == "Democrat"),
         ind = as.numeric(party == "Independent"),
         rep = as.numeric(party == "Republican"))

# An intercept and all three dummies: R chooses a redundant column and drops it.
round(coef(lm(ft_trump ~ hostile_sexism + dem + ind + rep, data = anes)), 2)

   (Intercept) hostile_sexism            dem            ind            rep 
         58.88          32.67         -58.82         -36.79             NA 

# With factor variables, R automatically drops the first - by default the sort is alphabetical
round(coef(lm(ft_trump ~ hostile_sexism + party, data = anes)), 2)

     (Intercept)   hostile_sexism partyIndependent  partyRepublican 
            0.06            32.67            22.03            58.82 

OLS with a single predictor

• With one covariate, \(\mathbf{X}_i = (1, X_i)^\prime\) and \(k = 2\).
  • Solving \(\mathbf{X}^\prime\mathbf{X}\,\hat{\boldsymbol{\beta}} = \mathbf{X}^\prime\mathbf{Y}\) gives the plug-in version of the BLP we found before
  \[\hat{\beta}_1 = \frac{\sum_{i=1}^n (X_i - \bar{X})(Y_i - \bar{Y})}{\sum_{i=1}^n (X_i - \bar{X})^2} = \frac{\widehat{\Cov}(X_i, Y_i)}{\widehat{\Var}(X_i)} \qquad \hat{\beta}_0 = \bar{Y} - \hat{\beta}_1\bar{X}\]

OLS with a single predictor

• In R, lm() fits OLS
  • Be careful with SEs here - they make the classical OLS assumptions - we’ll talk about these next week

summary(lm(ft_trump ~ hostile_sexism, data = anes))

Call:
lm(formula = ft_trump ~ hostile_sexism, data = anes)

Residuals:
     Min       1Q   Median       3Q      Max 
-100.214  -26.475   -6.231   27.895   93.769 

Coefficients:
               Estimate Std. Error t value Pr(>|t|)    
(Intercept)      6.2306     0.9621   6.476 1.04e-10 ***
hostile_sexism  93.9838     2.3906  39.313  < 2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 33.9 on 4582 degrees of freedom
Multiple R-squared:  0.2522,    Adjusted R-squared:  0.2521 
F-statistic:  1546 on 1 and 4582 DF,  p-value: < 2.2e-16

OLS with a single predictor

• \(\hat{\beta}_1 = 94.0\): respondents a quarter-point apart on the scale differ by 23.5 points on the thermometer, on average
• \(\hat{\beta}_0 = 6.2\): the predicted thermometer rating at the lowest level of hostile sexism

Adding partisanship

• Partisanship predicts both hostile sexism and feelings about Trump.

  • Add the party dummies, with Democrats as the baseline

  \[\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_i + \hat{\beta}_2 Z_i^{\text{Ind}} + \hat{\beta}_3 Z_i^{\text{Rep}}\]

• Within each party, the fit is a line in \(X_i\). The lines share a slope and have different intercepts

  \[\begin{aligned} \text{Democrats:} \quad & \hat{\beta}_0 + \hat{\beta}_1 X_i \\ \text{Independents:} \quad & (\hat{\beta}_0 + \hat{\beta}_2) + \hat{\beta}_1 X_i \\ \text{Republicans:} \quad & (\hat{\beta}_0 + \hat{\beta}_3) + \hat{\beta}_1 X_i \end{aligned}\]

• \(\hat{\beta}_2\) and \(\hat{\beta}_3\) are the gaps between the parallel lines:

  • How much warmer Independents and Republicans are towards Trump than Democrats with the same level of hostile sexism
  • Assuming a common slope on hostile sexism across the three types of partisans.

Adding partisanship

• The slope falls from 94.0 to 32.7. After accounting for the explanatory power of partisanship, the relationship is smaller but still there

Interpreting multiple regression coefficients

• With a bivariate regression, interpreting the coefficient on a given \(X_i\) is straightforward

  • \(\hat{\beta}_1 = \widehat{\Cov}(X_i, Y_i)/\widehat{\Var}(X_i)\) is the difference in the expected \(Y_i\) between respondents whose \(X_i\) differs by one unit

• However, with a multivariate regression, the coefficient captures a different relationship.

  • The difference in the expected \(Y_i\) between respondents whose \(X_{i1}\) differs by one unit holding the other covariates fixed.
  • We’ll often talk about relationships between \(X\) and \(Y\) after controlling for or conditioning on the other covariates

• What is OLS actually doing? We’re not physically controlling anything!

  • The Frisch-Waugh-Lovell theorem gives us a way of interpreting multivariate regression coefficients in terms of a residuals-on-residuals regression.

Frisch-Waugh-Lovell Theorem

• Split the regressors into the one we care about, \(X_{i1}\), and everything else, \(\mathbf{W}_i\) (which includes the intercept)

  \[Y_i = \hat{\beta}_1 X_{i1} + \mathbf{W}_i^\prime\hat{\boldsymbol{\gamma}} + \hat{e}_i\]

• Frisch-Waugh-Lovell (FWL): we can get the same \(\hat{\beta}_1\) from three regressions

  1. Regress \(Y_i\) on \(\mathbf{W}_i\) and keep the residuals, \(\tilde{Y}_i\)
  2. Regress \(X_{i1}\) on \(\mathbf{W}_i\) and keep the residuals, \(\tilde{X}_{i1}\)
  3. Regress \(\tilde{Y}_i\) on \(\tilde{X}_{i1}\). The slope is exactly \(\hat{\beta}_1\)

  \[\hat{\beta}_1 = \frac{\widehat{\Cov}(\tilde{X}_{i1}, \tilde{Y}_i)}{\widehat{\Var}(\tilde{X}_{i1})}\]

What FWL tells us

• \(\hat{\beta}_1\) only uses the variation in \(X_{i1}\) that is not linearly predicted by the other covariates to explain \(Y_i\)

• “Controlling for” \(\mathbf{W}_i\) means partialling out its linear relationship with both \(X_{i1}\) and \(Y_i\)

• If \(X_{i1}\) is uncorrelated with the other covariates in the sample, then adding those covariates doesn’t change \(\hat{\beta}_1\)

• Remember that the “partialling out” is linear

  • If \(\mathbf{W}_i\) is related to \(X_{i1}\) or \(Y_i\) in a way that the linear regression does not capture, that relationship is still in the residuals and might be attributed to \(X_{i1}\).

Example: controlling for party

• In our example, \(\mathbf{W}_i\) is the intercept plus the Independent and Republican dummies

  • A regression on a full set of indicators predicts each category’s mean

• So partialling out party just subtracts each respondent’s party mean

  \[\tilde{X}_{i1} = X_{i1} - \bar{X}_{\text{party}(i)} \qquad \tilde{Y}_i = Y_i - \bar{Y}_{\text{party}(i)}\]

• \(\hat{\beta}_1\) is the slope using only the within-party variation

  • Differences in hostile sexism between the parties contribute nothing to \(\hat{\beta}_1\)

FWL in R

anes_p <- anes |> filter(!is.na(party))   # the same respondents in every regression

# The full regression
round(coef(lm(ft_trump ~ hostile_sexism + party, data = anes_p)), 2)

     (Intercept)   hostile_sexism partyIndependent  partyRepublican 
            0.06            32.67            22.03            58.82 

# Steps 1 and 2: partial party out of the outcome and out of hostile sexism
anes_p$y_tilde <- resid(lm(ft_trump ~ party, data = anes_p))
anes_p$x_tilde <- resid(lm(hostile_sexism ~ party, data = anes_p))

# Step 3: residuals on residuals
round(coef(lm(y_tilde ~ x_tilde, data = anes_p)), 2)

(Intercept)     x_tilde 
       0.00       32.67 

Interacting with partisanship

• To let the slope on hostile sexism differ between parties too we add interactions

  • The products of \(X_i\) with each dummy

  \[\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_i + \hat{\beta}_2 Z_i^{\text{Ind}} + \hat{\beta}_3 Z_i^{\text{Rep}} + \hat{\beta}_4 X_i Z_i^{\text{Ind}} + \hat{\beta}_5 X_i Z_i^{\text{Rep}}\]

• Within each party, the fit is still a line in \(X_i\), but now each has its own intercept and slope

  \[\begin{aligned} \text{Democrats:} \quad & \hat{\beta}_0 + \hat{\beta}_1 X_i \\ \text{Independents:} \quad & (\hat{\beta}_0 + \hat{\beta}_2) + (\hat{\beta}_1 + \hat{\beta}_4) X_i \\ \text{Republicans:} \quad & (\hat{\beta}_0 + \hat{\beta}_3) + (\hat{\beta}_1 + \hat{\beta}_5) X_i \end{aligned}\]

• \(\hat{\beta}_1\) is now the slope among Democrats (the baseline), and \(\hat{\beta}_4\), \(\hat{\beta}_5\) are differences in slopes relative to Democrats

• With every interaction in, this is the same as fitting a separate OLS line within each party

Interacting with partisanship

# x * z in a formula adds x, z, and their product x:z
round(coef(lm(ft_trump ~ hostile_sexism * party, data = anes)), 2)

                    (Intercept)                  hostile_sexism 
                           0.95                           29.06 
               partyIndependent                 partyRepublican 
                          20.93                           56.08 
hostile_sexism:partyIndependent  hostile_sexism:partyRepublican 
                           4.12                            7.76 

• Slopes actually aren’t that different!

Interacting with education

• Does the slope on hostile sexism differ by education?

  • Let \(Z_i^{\text{Col}} = I(\text{college degree}_i)\) and add its product with \(X_i\)

  \[\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_i + \hat{\beta}_2 Z_i^{\text{Col}} + \hat{\beta}_3 X_i Z_i^{\text{Col}}\]

• Within each education group, the fit is a line in \(X_i\) with its own intercept and slope

  \[\begin{aligned} \text{No college degree:} \quad & \hat{\beta}_0 + \hat{\beta}_1 X_i \\ \text{College degree:} \quad & (\hat{\beta}_0 + \hat{\beta}_2) + (\hat{\beta}_1 + \hat{\beta}_3) X_i \end{aligned}\]

• \(\hat{\beta}_1\) is the slope among respondents without a degree (the baseline), and \(\hat{\beta}_3\) is the difference in slopes for college graduates

Interacting with education

round(coef(lm(ft_trump ~ hostile_sexism * college, data = anes)), 2)

           (Intercept)         hostile_sexism                college 
                 15.42                  79.18                 -14.47 
hostile_sexism:college 
                 20.43 

A second continuous covariate

• Racial resentment: a three question measure from Kinder and Sanders (1996), also averaged and re-scaled to \([0, 1]\)

  • Correlated with hostile sexism: \(r = 0.50\)

• With two continuous covariates, the fit is a plane over \((X_{i1}, X_{i2})\)

  \[\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_{i1} + \hat{\beta}_2 X_{i2}\]

round(coef(lm(ft_trump ~ hostile_sexism + racial_resentment, data = anes)), 2)

      (Intercept)    hostile_sexism racial_resentment 
            -9.29             50.18             60.49 

• After partialling racial resentment out of both the outcome and hostile sexism, the slope on hostile sexism is 50.2, also down from the unconditional 94.0

Interacting two continuous covariates

• A linear interaction with two continuous covariates adds the product of hostile sexism and racial resentment to the equation.

  \[\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_{i1} + \hat{\beta}_2 X_{i2} + \hat{\beta}_3 X_{i1} X_{i2}\]

• Holding \(X_{i2}\) at a value \(x_2\), we’re still estimating a line for \(X_{i1}\), but its slope now depends on \(x_2\)

  \[\big[\hat{\beta}_0 + \hat{\beta}_1 (x_1 + 1) + \hat{\beta}_2 x_2 + \hat{\beta}_3 (x_1 + 1) x_2\big] - \big[\hat{\beta}_0 + \hat{\beta}_1 x_1 + \hat{\beta}_2 x_2 + \hat{\beta}_3 x_1 x_2\big] = \hat{\beta}_1 + \hat{\beta}_3 x_2\]

  • The marginal ‘effect’ of \(X_{i1}\) at \(X_{i2} = x_2\). (be careful though, this is just a slope, not yet a causal claim).
  • \(\hat{\beta}_1\) alone is the slope at \(X_{i2} = 0\) - sometimes this is meaningful, sometimes it isn’t!

round(coef(lm(ft_trump ~ hostile_sexism * racial_resentment, data = anes)), 2)

                     (Intercept)                   hostile_sexism 
                           -7.00                            41.42 
               racial_resentment hostile_sexism:racial_resentment 
                           54.87                            17.33 

Marginal effects plot

• The slope on hostile sexism rises from 41.4 at the lowest racial resentment to 58.8 at the highest, a straight line with slope \(\hat{\beta}_3 = 17.3\)
  • The band is a 95% confidence interval (using next week’s robust standard errors)
• The gray bars show where the data are. Always plot the marginal effect against the distribution of the moderator

Careful with linear interactions!

• \(\hat{\beta}_1 + \hat{\beta}_3 x_2\) assumes the slope on \(X_{i1}\) changes at a constant rate in \(X_{i2}\)
  • Where few respondents have a given \(X_{i2}\) (weak common support), the predicted marginal effect relies heavily on the functional form assumption!
• Hainmueller et al. (2019) recommend a check that just uses bins of the effect modifier
  • Estimate separate slopes and intercepts within each bin

Binned marginal effects

• The bins differ from the linear interaction results!
  • The slope is steepest in the middle tercile (64.9) and flattest in the top one (39.8), where the linear interaction puts its largest slope (57.3)
  • The linear interaction forced a monotone pattern onto a hump
• Likely because the thermometer is bounded:
  • Little room to move at either end, so the marginal “effects” get compressed.

Next week

• Inference for regression
  • The sampling distribution of the OLS estimator: consistency and asymptotic normality
  • Confidence intervals and hypothesis tests for regression coefficients
• Standard errors for OLS
  • Heteroskedasticity-consistent (“robust”) standard errors
  • Classical OLS standard errors