Week 1: Course Introduction and Review

Week overview

• This week
  • Introductions + course logistics
  • What is a statistical model and what is it good for?
  • Review of regression
• Next week
  • Parametric models and the likelihood function
  • Maximum likelihood estimation
• Long-run
  • likelihood \(\to\) Bayes \(\to\) semi-/non-parametrics

Course Overview

• Instructor: Anton Strezhnev
• Logistics:
  • Lectures Mon/Wed - 9:30am-10:45am, Sterling Hall 1339
  • 4 Problem Sets (~ 2 weeks each)
  • 2 in-class midterms (October 7, November 4)
  • Replication project poster (December 4th)

Course Overview

• My office hours: Tuesdays 9am-11am (North Hall 322D)
  • Or just drop in – I’m usually around and my door is open.
• Course Website: https://www.antonstrezhnev.com/ps818
• Announcements and discussion happen on Slack - you should have received an invite.
• Problem sets are submitted on Gradescope; solutions get posted to Canvas.

Course objectives

• What is this course about?
  • Defining statistical models via their data-generating process
  • Estimating model parameters and conducting inference
  • Interpreting model output and evaluating model quality
• Goals for the course
  • Give you the tools to understand descriptive and predictive inference via statistical models – and to comment on other researchers’ work.
  • Equip you with the fundamentals of likelihood and Bayesian inference so that you can learn new models that build on these principles.
  • Connect these principles to the particular research questions you want to answer.
  • Teach you how to program and implement estimators by yourself!

Course workflow

• Lectures (Mondays/Wednesdays)
  • Topics organized by week: Monday introduces the core theory, Wednesday goes deeper and works through applications.
  • Lectures are the “course notes” – readings are the reference manuals.
  • I want lectures to be interactive - you should ask questions and interrupt!
• Readings
  • Mix of textbooks and papers – see the Schedule page.
  • Main texts: Regression and Other Stories (Gelman, Hill, and Vehtari 2020), Bayesian Data Analysis (Gelman et al. 2013), Statistical Rethinking (McElreath 2020), and An Introduction to Statistical Learning (James et al. 2013)
  • Everything is available digitally, either free online or through the UW-Madison library.
  • Do the readings before that week’s lectures – definitely before the Wednesday.

Course workflow

• Problem sets (20% of your grade)
  • Four problem sets, each covering roughly a two-week block of material.
  • Meant as a check on your understanding of the material and a way of communicating with me about the course.
  • Collaboration is strongly encouraged – ask and answer questions on Slack.
    • But write up your own answers and your own code.
  • Distributed as .qmd + rendered .html; submit both on Gradescope.
  • Graded holistically on a plus/check/minus system.

Course workflow

• Two in-class midterms (20% of your grade each)
  • October 7 – material through Week 4 (likelihood inference and GLMs)
  • November 4 – material from Weeks 5-8 (Bayesian inference, multilevel models, mixtures)
  • Closed-book, pen-and-paper, in class. Both theory and interpretation of sample code/output.
  • This is a change from previous years – no more take-home exams.

Course workflow

• Replication project (30% of your grade)
  • Work in pairs to replicate and extend a published paper.
  • Start by reading King (1995) on why we do this.
  • Steps: find a paper \(\to\) obtain the data \(\to\) reproduce the result \(\to\) extend it \(\to\) write it up.
    • The extension is where this course comes in: flexible regression adjustment, different likelihood model, adding multilevel structure, additional confounders…
  • Ungraded check-in memo due October 21
  • Presented as a poster at the MEAD graduate poster session, December 4th.

Course workflow

• Participation (10% of your grade)
  • It is important that you actively engage with lecture – ask and answer questions.
  • Participating on Slack counts towards this as well, as does attending MEAD talks.
• Models, Experiments and Data (MEAD) workshop
  • Check out the Fall 2026 schedule: https://sites.google.com/view/meadwisc/home

Assignment timeline

• Problem Set 1: Assigned September 9, due September 23
• Problem Set 2: Assigned September 23, due October 7
• Midterm 1: In class, Wednesday, October 7
• Problem Set 3: Assigned October 7, due October 21
• Replication memo: Due October 21
• Problem Set 4: Assigned October 21, due November 4
• Midterm 2: In class, Wednesday, November 4
• Replication poster: MEAD poster session, Friday, December 4

Class Requirements

• Overall: An interest in learning and willingness to ask questions.
• I assume a background in intro probability and statistics (the 1st year sequence)
  • You should be comfortable thinking about basic estimands/estimators + their properties.
  • You should be able to interpret a confidence interval for (e.g.) a difference-in-means.
• Some prior knowledge of causal inference is helpful but not critical
  • We’ll be connecting predictive models to causal estimands.
  • Ideally you’re familiar with the potential outcomes framework.
• You should also be familiar with linear regression
  • \(\hat{\beta} = (\mathbf{X}^{\prime}\mathbf{X})^{-1}\mathbf{X}^{\prime}Y\) should be a familiar expression.
  • You should know when it’s unbiased for \(\mathbb{E}[Y|\mathbf{X}]\), and when it’s efficient.
• If you want some review, check out chapters 1-7 of Regression and Other Stories

Statistical computing

• All of the work in this course is done in
  • We will implement some estimators from scratch – optimizers, samplers, EM.
  • Packaged implementations are great, but you should know what’s inside the box.
• We’ll also use Stan for Bayesian models (via rstan/brms)
  • Stan needs a working C++ toolchain – get this installed early
• Assignments are written and submitted in Quarto
• More broadly, I want you to learn how your computer works for research
  • Kieran Healy’s Modern Plain Text Computing (Healy 2025) is a great resource.

LLM Policy

• See the syllabus for the long version.
• In short…
  • You should have LLMs do things for you
  • You should not have LLMs think for you
  • You should definitely not have LLMs speak for you
• Sometimes you may wish to not have the LLM do things in order for you to learn how to think
  • I’d strongly encourage limiting your LLM use on the problem sets

Acknowledgements

• This course is an iteration in the broader project of methods education in political methodology and is indebted to many of those who have taught versions of this course in the past and at other institutions.
  • Particularly Gov 2001/Gov 2003 at Harvard and Quant III at MIT.
  • Thanks to Matthew Blackwell, Brandon Stewart, Erin Hartman, Molly Roberts, Kosuke Imai, Teppei Yamamoto, Jens Hainmueller, Adam Glynn, Gary King, Justin Grimmer, and In Song Kim, whose lecture notes and syllabi have been immensely valuable.
  • And special thanks to Andrew Heiss for the modern website/Quarto approach to teaching materials.

A brief overview

• Weeks 2-4: Likelihood inference and generalized linear models
  • The likelihood function, MLE and its asymptotic properties
  • Binary outcome models, count models, duration models
  • First midterm: Wednesday, October 7
• Weeks 5-7: Bayesian inference and multilevel models
  • Posteriors, priors, and the data; posterior means and credible intervals
  • Estimation via MCMC in Stan
  • Multilevel regression and post-stratification (MRP)
• Week 8: Mixture models and the EM algorithm
• Week 9: Item response theory and ideal point models
  • Second midterm: Wednesday, November 4

A brief overview

• Week 10: Regularization and model selection (ridge, LASSO, cross-fitting)
• Week 11: Flexible regression – trees, forests and BART
• Week 12: Flexible regression – kernels and Gaussian processes
• Week 13: Causal inference with flexible regression
  • Influence functions and “doubly-robust”/Neyman-orthogonal estimators
  • MEAD poster session: Friday, December 4
• Week 14: Working with “big” datasets
  • Out-of-memory estimation, stochastic gradient descent, lossless compression

What is a statistical model?

• A statistical model is a set of assumptions about the process that generated the data.
  • A family of probability distributions \(\{P_\theta : \theta \in \Theta\}\) indexed by a parameter \(\theta\).
  • We assume the observed data are a draw from one member of that family, and we want to figure out which one.
• Parametric models assume \(\Theta \subseteq \mathbb{R}^k\) for finite \(k\)
  • A few parameters define the entire distribution – most of the first half of the course.
• Nonparametric models let \(\Theta\) be infinite-dimensional
  • Still involves assumptions, just weaker ones.

What is a statistical model?

All models are wrong, but some are useful.

— George Box (1976)

• The relevant question is never “is the model true?” (it isn’t)
• It’s: is the model wrong in a way that matters for the question I’m asking?
  • A misspecified likelihood can still give a consistent estimator of a well-defined estimand.
  • Or it can give you a confidently wrong answer.
• Much of this course is about developing the judgment to tell those two cases apart.

What is a model good for?

• Description: summarize a joint distribution
  • “How does turnout vary with age, once we account for education?”
• Prediction: guess \(Y\) for a unit where we observe only \(X\)
  • “What share of this state’s voters support the referendum?”
• Measurement: recover a latent quantity we never observe directly
  • “How liberal is this Supreme Court justice?”
• Causal inference: a design identifies the estimand; a model does the adjustment
  • “What would turnout have been under a different policy?”

Two cultures

• Breiman (2001) draws a distinction between two ways of using models:
  • Data modeling: assume a stochastic model generated the data; estimate and interpret its parameters.
  • Algorithmic modeling: treat the data-generating process as unknown; find a function that predicts well.
• This course starts with the first and ends with the second.
  • Weeks 2-9 are “data modeling” - we write down a DGP.
  • Weeks 10-12 are “algorithmic” - we care about fit, not parameters.
  • Week 13 shows how to use “algorithmic” tools for nuisances when we want to learn about a low-dimensional quantity of interest (e.g. treatment effect).

Regression review

• Our most common quantity of interest (or annoying nuisance function) in social science is a population conditional expectation \(\mathbb{E}[Y|\mathbf{X}]\)
  • \(Y_i\): outcome/response/dependent variable
  • \(X_i\): vector of regressors/independent variables
• “How does the expected value of \(Y\) differ across different values of \(X\)?”
• Suppose we observe \(n\) paired observations of \(\{Y_i, X_i\}\).
  • How do we construct a “good” estimator of \(\mathbb{E}[Y|\mathbf{X}]\)?
  • What assumptions do we have to make to get…consistency…unbiasedness…efficiency?

Regression review

• Consider the ordinary least squares estimator \(\hat{\beta}\), which solves the minimization problem

  \[\hat{\beta} = \argmin_b \ \sum_{i=1}^n (Y_i - X_i b)^2\]

• We can do some algebra and find a closed form solution for this optimization problem

  \[\hat{\beta} = (\mathbf{X}^{\prime}\mathbf{X})^{-1}(\mathbf{X}^{\prime}Y)\]

Regression review

• Assumption 1: Linearity

  \[Y = \mathbf{X}\beta + \epsilon\]

• Assumption 2: Strict exogeneity of the errors

  \[\mathbb{E}[\epsilon | \mathbf{X}] = 0\]

• These two imply a linear CEF

  \[\mathbb{E}[Y|\mathbf{X}] = \mathbf{X}\beta = \beta_0 + \beta_1X_{1} + \beta_2X_{2} + \dotsc + \beta_kX_{k}\]

• Best case: our CEF is truly linear (by luck, or because we have a saturated model)

• Usual case: Assumption 2 fails and the CEF is not linear

  • Even then, OLS is consistent for the best linear approximation to the CEF

Regression review

• Assumption 3: No perfect collinearity
  • \(\mathbf{X}^{\prime}\mathbf{X}\) is invertible
  • \(\mathbf{X}\) has full column rank
• This assumption is needed for identifiability – otherwise no unique solution to the least squares minimization problem exists!
• Fails when one column can be written as a linear combination of the others
  • Or when there are more regressors than observations, \(k > n\)
  • Keep this in mind – Week 10 is all about what to do when \(k > n\).

Regression review

• Under assumptions 1-3, our OLS estimator \(\hat{\beta}\) is unbiased for \(\beta\)
• Let’s do a quick proof for unbiasedness

\[\begin{align*}\hat{\beta} &= (\mathbf{X}^{\prime}\mathbf{X})^{-1}(\mathbf{X}^{\prime}Y)\\ &= (\mathbf{X}^{\prime}\mathbf{X})^{-1}(\mathbf{X}^{\prime}(\mathbf{X}\beta + \epsilon))\\ &= (\mathbf{X}^{\prime}\mathbf{X})^{-1}(\mathbf{X}^{\prime}\mathbf{X})\beta + (\mathbf{X}^{\prime}\mathbf{X})^{-1}(\mathbf{X}^{\prime}\epsilon)\\ &= \beta + (\mathbf{X}^{\prime}\mathbf{X})^{-1}(\mathbf{X}^{\prime}\epsilon) \end{align*}\]

• Then we can obtain the conditional expectation \(\mathbb{E}[\hat{\beta} | \mathbf{X}]\)

\[\begin{align*} \mathbb{E}[\hat{\beta} | \mathbf{X}] &= \mathbb{E}\bigg[\beta + (\mathbf{X}^{\prime}\mathbf{X})^{-1}(\mathbf{X}^{\prime}\epsilon) \bigg| \mathbf{X} \bigg]\\ &= \mathbb{E}[\beta | \mathbf{X}] + \mathbb{E}[(\mathbf{X}^{\prime}\mathbf{X})^{-1}(\mathbf{X}^{\prime}\epsilon) | \mathbf{X}]\\ &= \beta + (\mathbf{X}^{\prime}\mathbf{X})^{-1}\mathbf{X}^{\prime} \mathbb{E}[\epsilon | \mathbf{X}]\\ &= \beta + (\mathbf{X}^{\prime}\mathbf{X})^{-1}\mathbf{X}^{\prime}0\\ &= \beta \end{align*}\]

Regression review

• Lastly, by the law of total expectation

  \[\mathbb{E}[\hat{\beta}] = \mathbb{E}[\mathbb{E}[\hat{\beta}|\mathbf{X}]]\]

• Therefore

  \[\mathbb{E}[\hat{\beta}] = \mathbb{E}[\beta] = \beta\]

Regression review

• For consistency, we just need to stabilize by \(\frac{1}{n}\), since \(\mathbf{X}^{\prime}\mathbf{X}\) diverges as \(n\) grows

  \[\hat{\beta} - \beta = \left(\frac{1}{n}\mathbf{X}^{\prime}\mathbf{X}\right)^{-1}\left(\frac{1}{n}\mathbf{X}^{\prime}\epsilon\right) \overset{p}{\to} \mathbf{Q}^{-1} \cdot 0 = 0\]

• We need \(\mathbf{Q} = \mathbb{E}[X_iX_i^{\prime}]\) to be invertible – the population version of no perfect collinearity

• And we need \(\frac{1}{n}\mathbf{X}^{\prime}\epsilon \overset{p}{\to} 0\) – which requires only \(\mathbb{E}[X_i\epsilon_i] = 0\), weaker than the full \(\mathbb{E}[\epsilon | \mathbf{X}] = 0\).

• Notice what we still have not assumed: anything about the shape of the error distribution.

Regression review

• Assumption 4 – Spherical errors

  \[Var(\epsilon | \mathbf{X}) = \begin{bmatrix} \sigma^2 & 0 & \cdots & 0\\ 0 & \sigma^2 & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \sigma^2 \end{bmatrix} = \sigma^2 \mathbf{I}\]

• Benefits

  • Simple, unbiased estimator for the variance of \(\hat{\beta}\)
  • Completes the Gauss-Markov assumptions \(\leadsto\) OLS is BLUE (Best Linear Unbiased Estimator)

• Drawbacks

  • Basically never true

Regression review

• We can relax homoskedasticity (but still keep no correlation) and do inference on the variance of \(\hat{\beta}\)

  \[Var(\epsilon | \mathbf{X}) = \begin{bmatrix} \sigma^2_1 & 0 & \cdots & 0\\ 0 & \sigma^2_2 & \cdots & 0\\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \sigma^2_n \end{bmatrix}\]

• “Robust” standard errors using the Eicker-Huber-White “sandwich” estimator – consistent but not unbiased for the true sampling variance of \(\hat{\beta}\)

  \[\widehat{Var(\hat{\beta})} = (\mathbf{X}^{\prime}\mathbf{X})^{-1} \mathbf{X}^{\prime}\hat{\Sigma}\mathbf{X}(\mathbf{X}^{\prime}\mathbf{X})^{-1}\]

• \(\hat{\Sigma}\) is our estimate of the variance-covariance matrix, using the squared residuals on the diagonal

• Extensions to “clustered” standard errors allow arbitrary correlation within groups.

  • “Block-diagonal” structure on \(Var(\epsilon | \mathbf{X})\)

Regression review

• Assumption 5 – Normality of the errors

  \[\epsilon | \mathbf{X} \sim \mathcal{N}(0, \sigma^2\mathbf{I})\]

• Not necessary even for the Gauss-Markov result

• Not needed to do asymptotic inference on \(\hat{\beta}\)

  • Why? Central Limit Theorem!

• Benefits?

  • Finite-sample inference.

• And: this is the assumption that turns OLS into a likelihood model.

  • Which is exactly where we pick up next week.

Regression review

• What do we need for OLS to be consistent for the “best linear approximation” to the CEF?
  • Very little!
• What do we need for \(\mathbf{X}\hat{\beta}\) to be consistent and unbiased for the conditional expectation function itself?
  • A truly linear CEF
  • But still no assumptions about the outcome distribution!
• What do we need to do inference on \(\hat{\beta}\)?
  • We almost never assume homoskedasticity, because “robust” SE estimators are ubiquitous.
  • Even some forms of error correlation are permitted (“cluster”-robust SEs).
  • Sample sizes are usually large enough that the Central Limit Theorem makes a normal sampling distribution a reasonable approximation.

Next week

• Parametric models
  • What happens when we assume a distribution for \(Y\)?
• The likelihood function
  • Given the data, how relatively plausible is each candidate value of \(\theta\)?
• Maximum likelihood estimation
  • Finding the \(\hat{\theta}\) that maximizes the likelihood – analytically and numerically.
• Readings: Chapter 5 of Aronow and Miller (2019) and Chapter 6 of Evans and Rosenthal (2010)
• Problem Set 1 is assigned today and due Wednesday, September 23.