Week 9: Item Response Theory

PS 818 - Data Analysis with Statistical Models

Anton Strezhnev

strezhnev@wisc.edu

University of Wisconsin-Madison

November 2, 2026

Item Response Theory

Previously

Mixture models
- “Unsupervised” approach to clustering observations.
- A discrete latent variable $z_i$: cluster membership
- MLE/MAP estimation via Expectation-Maximization
Policy attitudes in the 2023 CES
- Two clusters $\approx$ party
- Five clusters show different types of “moderates”

This week

Item Response Theory
- Model discrete responses as a function of a continuous latent trait
- Logit/probit regression with unobserved regressors
Ideal point models
- Item response theory as a model of voting
- Identification, estimation, and interpreting the latent dimension
Running example: Last week’s 24 CES policy questions
Dynamic IRT
- Allow ideal points to drift over time

Item Response Theory

In many settings, we observe binary or ordinal responses to a set of questions among a sample of units.
- Students’ responses to test questions (correct/incorrect)
- Survey respondents’ answers to policy questions (support/oppose)
- Legislators voting on bills (yea/nay)
In these settings, we’re interested in learning about an (interval-scaled) latent trait of the units based on their discrete responses
- Measuring student ability
- Measuring voter ideology
- Measuring legislator ideology
Simple aggregation of the binary responses may give misleading results
- Some test questions are harder than others and some are more informative about student ability than others.
- Counting “liberal” answers treats a question that splits the parties as equivalent to one that almost everyone supports.

Item Response Theory

Item Response Theory (IRT) developed out of research on testing and evaluation.
Observed responses to test questions are a function of:
- Latent traits that are common features of respondents across all questions
- Item parameters that are common features of questions across all respondents.
Setup:
- Observe $N$ respondents indexed by $i \in \{1, 2, \dotsc, N\}$
- Observe $J$ questions indexed by $j \in \{1, 2, \dotsc, J\}$
- Observe $Y_{ij}$, the response to question $j$ by respondent $i$
  - We’ll work with binary $Y_{ij}$ to start, but can generalize to other outcome distributions (typically ordinal)

One Parameter Logit (1PL)

The baseline classic IRT model (sometimes called the “Rasch” model after Georg Rasch) assumes the following item response function:

\[Pr(Y_{ij} = 1) = F(\theta_i - \alpha_j)\]

where $F()$ is the logistic CDF.
In other words,

\[Pr(Y_{ij} = 1) = \frac{1}{1 + \exp[-(\theta_i - \alpha_j)]}\]
Or in “latent variable form”

\[Y_{ij} = \mathbf{1}(Y_{ij}^* > 0)\] \[Y_{ij}^* = \theta_i - \alpha_j + \epsilon_{ij}\]

where $\epsilon_{ij}$ are distributed i.i.d. standard logistic

One Parameter Logit (1PL)

In the 1PL model, we have one parameter describing the item and one parameter describing the individual’s latent trait

\[Pr(Y_{ij} = 1) = F(\theta_i - \alpha_j)\]
- $\theta_i$ is the latent trait for unit $i$
- $\alpha_j$ is the item difficulty of question $j$
Analogy to the GLM
- $\alpha_j$ is the intercept for item $j$
- $\theta_i$ is the regressor, common to all of unit $i$’s responses
- What are we implicitly assuming?
Every question counts the same w.r.t. $\theta_i$
- Doesn’t work well for our case where “Support” is the liberal answer on some questions and the conservative answer on others

Two Parameter Logit (2PL)

One drawback of the 1PL model is that it only allows questions to vary in their difficulty (intercept) and not in the extent to which variation in the responses captures variation in the latent trait.
- Essentially assuming a constant “slope” of $1$ on the $\theta$ parameter
The two-parameter logit adds an additional item parameter $\beta_j$ and assumes

\[Pr(Y_{ij} = 1) = F\bigg(\beta_j(\theta_i - \alpha_j)\bigg)\]
Now each item has:
- $\alpha_j$ - item difficulty
- $\beta_j$ - item discrimination
$\beta_j$ captures the extent to which the question reflects the latent trait
- $\beta_j$ close to $0$ means that the probability of $Y_{ij} = 1$ is essentially uncorrelated with the latent trait
- Negative $\beta_j$ implies that low latent trait values are more likely to answer $Y_{ij} = 1$ (this creates some identifiability issues!)

Two Parameter Logit (2PL)

Typically also see the 2PL written as:

\[Pr(Y_{ij} = 1) = F\bigg(\beta_j\theta_i - \tau_j\bigg)\]

with $\tau_j = \beta_j \alpha_j$
Think back again to the logistic regression
- $\tau_j$ is the “intercept”
- $\beta_j$ is the “slope”
- $\theta_i$ is the “regressor”
Compare to last week’s mixture model: $Pr(Y_{ij} = 1 | z_i = k) = \beta_{kj}$
- Both assume local independence given the latent variable
- (single-dimensional) IRT: $2 \times J$ item parameters
- Bernoulli mixture model: $K \times J$ item parameters

Interpretation as a voting model

In political science, the 2PL is the standard model for analyzing voting in legislatures (Clinton, Jackman, and Rivers 2004)
Often described in terms of utility maximization
- Consider a legislator (or survey respondent) choosing “Yea” $Y_{ij} = 1$ or “Nay” $Y_{ij} = 0$.
- These positions are located in some space (we’ll work in $\mathbb{R}^1$ for now). The “Yea” position is $\zeta_j$ and the “Nay” position is $\psi_j$.
Define a utility function for legislator $i$: $U_i()$
- $U_i(\zeta_j) = -\frac{1}{\sigma_j}(\theta_i - \zeta_j)^2 + \eta_{ij}$
- $U_i(\psi_j) = -\frac{1}{\sigma_j}(\theta_i - \psi_j)^2 + \nu_{ij}$
In other words, they get decreasing utility (in terms of quadratic distance) from policies that are further from their ideal point
- They will vote $Y_{ij} = 1$ if $U_i(\zeta_j) > U_i(\psi_j)$ and $Y_{ij} = 0$ otherwise.
$\eta_{ij}$ and $\nu_{ij}$ are the “vote-specific error terms”

Interpretation as a voting model

Returning to the “latent variable” formulation of the logit, we can write: $Y_{ij}^* = U_i(\zeta_j) - U_i(\psi_j)$
Then, some algebra \[\begin{align*} Y_{ij}^* &= -\frac{1}{\sigma_j}(\theta_i - \zeta_j)^2 + \eta_{ij} + \frac{1}{\sigma_j}(\theta_i - \psi_j)^2 - \nu_{ij}\\ Y_{ij}^* &= \frac{1}{\sigma_j}(-\theta_i^2 + 2\theta_i\zeta_j - \zeta_j^2) + \frac{1}{\sigma_j}(\theta_i^2 - 2\theta_i\psi_j + \psi_j^2) + (\eta_{ij} - \nu_{ij})\\ Y_{ij}^* &= \frac{2(\zeta_j - \psi_j)}{\sigma_j}\theta_i + \frac{(\psi_j^2 - \zeta_j^2)}{\sigma_{j}} + (\eta_{ij} - \nu_{ij}) \end{align*}\]
And with some assumptions on the error distribution of $\epsilon_{ij} = (\eta_{ij} - \nu_{ij})$ we have our 2-parameter logit!
- $\tau_j = -\frac{(\psi_j^2 - \zeta_j^2)}{\sigma_{j}}$
- $\beta_j = \frac{2(\zeta_j - \psi_j)}{\sigma_j}$
The sign of $\beta_j$ says which side of the space the “Yea” position is on. $|\beta_j|$ grows with the distance between the two positions relative to the noise.

Interpretation as a voting model

IRT also assumes local independence of responses given the latent ideology.

\[Pr(Y_{ij} =1 , Y_{ij^{\prime}} = 1 | \theta_i, \beta, \tau) = Pr(Y_{ij} = 1 | \theta_i, \beta, \tau) \times Pr(Y_{ij^{\prime}} = 1 | \theta_i, \beta, \tau)\]
In other words, knowing how a legislator voted on bill $j$ doesn’t tell you anything about how they will vote on bill $j^{\prime}$ if we already know the latent positions of the legislators and bills
- Could be violated under common legislative behavior (e.g. log-rolling, horse-trading, etc…)
- In surveys: two questions about the same issue (two abortion questions) may be correlated beyond what one ideological dimension explains
- You’ll still get something but interpreting ideal points in this setting is a bit harder - essentially some of the “closeness” in ideal points could be driven by non-ideological factors.
The “utility maximization” interpretation of the 2PL also implies single peaked preferences
- Legislators strictly prefer policies closer to their ideal point and disprefer ones that are further away
- But this assumption can be violated if some extreme legislators vote against their party and with the opposition but for differing reasons
  - “Ends against the middle”

Identification

While historically, IRT models were estimated via maximum-likelihood, there are many reasons why modern methods use Bayes.
1. Inconsistency - The number of parameters grows as we add more questions and as we add more respondents so our ML estimators are not consistent
2. Non-identifiability - The 2PL likelihood only depends on $\beta_j\theta_i - \tau_j$
  - Location: add $c$ to every $\theta_i$ and $c\beta_j$ to every $\tau_j$
  - Scale: multiply every $\theta_i$ by $c$ and divide every $\beta_j$ by $c$
  - Reflection: multiply every $\theta_i$ and every $\beta_j$ by $-1$. This flips what high $\theta$ means but leaves the likelihood unchanged
Putting a prior on $\theta$ pins down the location and scale
- Typically assume $\theta_i \sim \text{Normal}(0, 1)$
But the prior is symmetric, so we need an additional constraint to fix the reflection:
- Constraints on the sign of some $\beta_j$…
- …or setting some of the $\theta_i$ parameters to a known value (or a tight prior around it)

Estimation

Estimation typically relies on either MCMC or a (variational) EM algorithm
- The underlying trick is to use existing theory for Bayesian probit (and now logit) regression to derive the conditional distributions of $\theta_i$, $\beta_j$ and $\tau_j$
Consider the “latent variable” form of the logit/probit regression:

\[Y_{ij}^* = \beta_j\theta_i - \tau_j + \epsilon_{ij}\]
Conditional on $\theta$, we have $J$ regressions of $Y^*_{ij}$ on $\theta_i$, each with intercept $-\tau_j$ and slope $\beta_j$
Conditional on $\beta$ and $\tau$, we have $N$ regressions of $Y^*_{ij} + \tau_j$ on $\beta_j$ with slope $\theta_i$ and no intercept.
- With a probit link and $Y^*_{ij}$ treated as missing data, this gives a Gibbs sampler (Clinton, Jackman, and Rivers 2004)…
- …or an EM algorithm for the posterior mode that scales to millions of votes (Imai, Lo, and Olmsted 2016).
Stan doesn’t need any of this: HMC samples $\theta$, $\beta$, and $\tau$ jointly.

Application: Policy attitudes

Data: CES 2023

Same data as last week: the 24 support/oppose policy questions from the 2023 CES (Schaffner et al. 2024), 1 = Support, 0 = Oppose

library(haven)
ces_raw <- read_dta("../week8/data/CCES23_Common_OUTPUT.dta.gz") # the file from last week
items <- c(ban_assault_rifles = "CC23_321a", concealed_carry = "CC23_321b", background_checks = "CC23_321c",
           more_police = "CC23_321d", fewer_police = "CC23_321e", legal_status = "CC23_323a",
           border_patrols = "CC23_323b", border_wall = "CC23_323c", dreamers = "CC23_323d",
           abortion_choice = "CC23_324a", abortion_exceptions = "CC23_324b", abortion_ban = "CC23_324c",
           abortion_expand = "CC23_324d", epa_co2 = "CC23_326a", renewable_mandate = "CC23_326b",
           epa_enforcement = "CC23_326c", more_fossil_fuels = "CC23_326d", halt_leases = "CC23_326e",
           relax_zoning = "CC23_328a", housing_credits = "CC23_328b", medicaid_work_req = "CC23_328c",
           repeal_aca = "CC23_328d", expand_medicaid = "CC23_328e", forgive_loans = "CC23_328f")
ces <- ces_raw %>%
  transmute(caseid, commonweight, pid3 = as.character(as_factor(pid3)), ideo5 = as.character(as_factor(ideo5)),
            across(all_of(items), ~ case_when(.x == 1 ~ 1, .x == 2 ~ 0))) %>% # skipped/not asked become NA
  drop_na(all_of(names(items)))
Y <- as.matrix(select(ces, all_of(names(items)))) # 1 = Support, 0 = Oppose
dim(Y)

[1] 24453    24

A 2PL in Stan

irt_code <- "
data {
  int N; int J;
  array[N, J] int<lower=0, upper=1> Y; // 1 = Support, 0 = Oppose
}
parameters {
  vector[N] theta; // respondent ideal points
  vector[J] beta;  // item discrimination
  vector[J] tau;   // item intercept (difficulty)
}
model {
  theta ~ normal(0, 1); // fixes the location and scale
  beta ~ normal(0, 3);
  tau ~ normal(0, 3);
  for (j in 1:J) Y[, j] ~ bernoulli_logit(beta[j] * theta - tau[j]);
}
"

24,453 respondents means 24,453 $\theta_i$ parameters. For a quick illustration, we’ll fit a random subsample of 1,500

set.seed(53703)
sub <- sample(nrow(Y), 1500)
irt_data <- list(N = length(sub), J = ncol(Y), Y = Y[sub, ])

Fitting the model

irt_fit <- stan(
  model_code = irt_code,  # Stan code
  data = irt_data,        # named list of data
  chains = 4,             # number of Markov chains
  warmup = 500,           # number of warmup iterations per chain
  iter = 1000,            # total number of iterations per chain
  cores = 4,              # number of cores
  refresh = 0,            # no progress shown
  seed = 60637            # random seed
  )

Stan complains: the largest $\hat{R}$ is 16.85. Each chain’s posterior mean of $\beta_j$ for two of the items:

anchor <- which(colnames(Y) == "border_wall")
epa <- which(colnames(Y) == "epa_enforcement")
round(get_posterior_mean(irt_fit, pars = str_c("beta[", c(anchor, epa), "]"))[, 1:4], 2)

         mean-chain:1 mean-chain:2 mean-chain:3 mean-chain:4
beta[8]          2.49        -2.48         2.49        -2.48
beta[16]        -3.20         3.19        -3.20         3.20

What’s happening here?

traceplot(irt_fit, pars = str_c("beta[", c(anchor, epa), "]"))

What’s happening here?

Because the IRT model is identified only up to reflection, the chains end up going to one of two posterior modes
- $\beta_{\text{wall}} > 0$: high $\theta$ means conservative
- $\beta_{\text{wall}} < 0$: high $\theta$ means liberal
In educational testing, a common fix is to make all of the discrimination parameters positive
- A correct answer should never indicate lower ability
Ideology doesn’t work this way, but we can still fix the sign of some of the $\beta_j$

Constraining one item

Supporting a border wall is a conservative answer, so require $\beta_{\text{wall}} > 0$
Pass a lower and upper bound for each $\beta_j$ as data ($0$ or $\pm\infty$)

irt_sign_code <- "
data {
  int N; int J;
  array[N, J] int<lower=0, upper=1> Y;
  vector[J] lb; // lower bound on each beta_j: 0 or -infinity
  vector[J] ub; // upper bound on each beta_j: 0 or +infinity
}
parameters {
  vector[N] theta;
  vector<lower=lb, upper=ub>[J] beta;
  vector[J] tau;
}
model {
  theta ~ normal(0, 1);
  beta ~ normal(0, 3);
  tau ~ normal(0, 3);
  for (j in 1:J) Y[, j] ~ bernoulli_logit(beta[j] * theta - tau[j]);
}
"
lb_one <- if_else(colnames(Y) == "border_wall", 0, -Inf) # only the border wall is constrained
ub_one <- rep(Inf, ncol(Y))

irt_anchor_fit <- stan(model_code = irt_sign_code, data = c(irt_data, list(lb = lb_one, ub = ub_one)),
                       chains = 4, warmup = 500, iter = 1000, cores = 4, refresh = 0, seed = 60637)

Constraining one item

Not fixed! The chains that started in the “wrong” mode are stuck at the $\beta_{\text{wall}} = 0$ boundary.
- The other $\beta_j$ and all of the $\theta_i$ are still reflected, so HMC can’t get to the other mode.

Constraining every item

We know which answer is the conservative one on every question (last week’s coding), so constrain all of the signs
- $\beta_j > 0$ for the 9 conservative items, $\beta_j < 0$ for the other 15

lb_all <- if_else(colnames(Y) %in% conservative_items, 0, -Inf)
ub_all <- if_else(colnames(Y) %in% conservative_items, Inf, 0)

irt_sign_fit <- stan(model_code = irt_sign_code, data = c(irt_data, list(lb = lb_all, ub = ub_all)),
                     chains = 4, warmup = 500, iter = 1000, cores = 4, refresh = 0, seed = 60637)

draws <- as_draws_df(irt_sign_fit)
irt_summary <- summarise_draws(draws, "mean", ~ quantile2(.x, probs = c(.025, .975)), "rhat", "ess_bulk")
summary(irt_summary$rhat)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  0.999   1.001   1.002   1.003   1.004   1.014

Constraining every item

The four chains converge to the same mode: every $\hat{R}$ is close to 1
Other fixes: start the chains near the right mode…
- …or flip the draws after sampling.

Item discrimination

Positive $\beta_j$: conservatives (high $\theta$) more likely to support. Negative $\beta_j$: liberals more likely to support.
The most discriminating items: EPA enforcement and CO2 regulation, the border wall
The least: relaxing zoning (-0.42) and making abortion illegal (0.48)
- Few respondents of any ideology support a total abortion ban: 19% of the sample
- The ban question discriminates only among the most conservative respondents

Item characteristic curves

Each item’s $Pr(Y_{ij} = 1)$ as a function of $\theta_i$, at the posterior means of $\beta_j$ and $\tau_j$

Steep curves (large $|\beta_j|$) separate respondents near the point where $Pr = 50\%$, at $\theta = \tau_j / \beta_j$. Flat curves carry little information about $\theta$.

Respondent ideal points

How different is this from just counting?

Posterior means of $\theta_i$ against the number of liberal answers (0-24)

Strongly related (correlation -0.98), but not one-to-one.
Conditional on the same total, a conservative answer on a low-$|\beta_j|$ question (relaxing zoning) moves $\theta_i$ much less than one on a high-$|\beta_j|$ question (the border wall)

Scaling up: EM

MCMC on 1,500 respondents took about 1.1 minutes. All 24,453 would mean 16 times as much data and 16 times as many parameters.
- Roll-call datasets can be even larger: every legislator in every chamber, or millions of social media users “voting” by following accounts (Barberá 2015).
Imai, Lo, and Olmsted (2016): an EM algorithm for the posterior mode
- Probit link: treat the latent utilities $Y^*_{ij}$ as the missing data
- E-step: expectation of each $Y^*_{ij}$ given the current parameters (a truncated normal mean)
- M-step: the $N + J$ conditional regressions from before have closed-form solutions
- Variational EM for extensions (ordinal, dynamic, text)
Implemented in the emIRT package.

Scaling up: EM

binIRT() takes a roll call object from the pscl package: votes are recoded to $+1$/$-1$, with $0$ meaning missing

library(emIRT)
rc <- convertRC(rollcall(Y, yea = 1, nay = 0, missing = NA))

set.seed(53703)
em_time <- system.time(
  em_fit <- binIRT(.rc = rc, .starts = getStarts(rc$n, rc$m, 1), .priors = makePriors(rc$n, rc$m, 1),
                   .control = list(threads = 1, verbose = FALSE, thresh = 1e-6))
)

All 24,453 respondents in 3.4 seconds (80 iterations)
makePriors() sets $\theta_i \sim \text{Normal}(0, 1)$ and $\text{Normal}(0, 5^2)$ priors on the item parameters.

x_mode <- em_fit$means$x[, 1]
c(sd_theta = sd(x_mode), mean_abs_beta = mean(abs(em_fit$means$beta[, "d1"])))

     sd_theta mean_abs_beta 
       0.0446       27.7328

Scale at the posterior mode

The $\theta_i$ estimates are tiny and the $\beta_j$ are huge. Only the products $\beta_j\theta_i$ are identified by the likelihood.
- The prior on the 24,453 values of $\theta_i$ prefers small values far more than the prior on the 24 values of $\beta_j$ penalizes large ones
- This ultimately doesn’t really matter since the scale of the ideal points is arbitrary.
We can standardize the estimates after the fact.

EM vs. MCMC

Standardize and align direction.

em_sign <- sign(em_fit$means$beta[anchor, "d1"])
theta_em <- em_sign * (x_mode - mean(x_mode)) / sd(x_mode)

For the 1,500 respondents in both fits:

Correlation 0.979. The Stan estimates are shrunk toward zero at both extremes: with few answers on the other side to go on, the posterior is pulled back towards the $\text{Normal}(0, 1)$ prior.

Ideal points and party

With all 24,453 respondents (EM estimates)

Partisans are well separated, but there is a long tail of Democrats and Republicans in the middle. Independents are everywhere.

Ideal points and ideology

Distribution of $\theta_i$ by self-reported ideology (dashed line: group mean)

Self-described ideology lines up with $\theta_i$ on average, with plenty of spread within each category
“Not sure”: same average as moderates, but more tightly bunched in the middle

Where do last week’s clusters land?

Last week’s $K = 5$ mixture, with each respondent assigned to their most likely cluster

Where do last week’s clusters land?

The consistent liberal and conservative clusters are at the ends.
Clusters 2, 3, and 4 all land in the middle of the scale
- Cluster 2 is near the center because it is liberal on most issues and conservative on police and the border
- Cluster 3 is near the center because saying “support” to everything means half its answers are liberal and half are conservative
- Cluster 4 is near the center because its answers have little ideological structure
A single dimension maps all three patterns to a “moderate” ideal point
- This is the Fowler et al. (2023) vs. Broockman and Lauderdale (2025) debate: are moderates centrists, or are they unconstrained voters that a one-dimensional model assigns to the middle?
- The answer depends on which model we’re willing to assume.

Checking local independence

Like last week: does the model reproduce the observed correlations between pairs of questions?
- Posterior predictive check: simulate replicated datasets $Y^{\text{rep}}$ from 200 posterior draws and compute their correlation matrices

theta_draws <- unclass(as_draws_matrix(subset_draws(draws, variable = "theta"))) # draws x respondents
beta_draws <- unclass(as_draws_matrix(subset_draws(draws, variable = "beta")))
tau_draws <- unclass(as_draws_matrix(subset_draws(draws, variable = "tau")))
pairs <- upper.tri(diag(ncol(Y)))
set.seed(53703)
rep_cor <- sapply(sample(ndraws(draws), 200), function(s){
  eta <- outer(theta_draws[s, ], beta_draws[s, ]) - matrix(tau_draws[s, ], length(sub), ncol(Y), byrow = T)
  Y_rep <- matrix(rbinom(length(eta), 1, plogis(eta)), nrow(eta))
  cor(Y_rep)[pairs]
})
obs_cor <- cor(Y[sub, ])[pairs]

Checking local independence

40% of the 276 pairs fall outside the 95% predictive interval: 31 of 47 pairs on the same issue, vs. 34% of pairs across issue areas

Checking local independence

The worst-fitting pairs:

Question 1	Question 2	Observed	Replicated
Always allow abortion	Expand abortion access	0.67	0.36
Legal status for undocumented workers	Citizenship path for Dreamers	0.61	0.36
10% more police	10% fewer police	-0.41	-0.21
More border patrols	Build a border wall	0.49	0.30
Easier concealed-carry permits	Repeal the ACA	0.42	0.26
Tax credits for low-income housing	Expand Medicaid	0.43	0.27

Answers on the same issue are more strongly related than one ideological dimension can explain
- Someone who always allows abortion is much more likely to want to expand access than their $\theta_i$ alone predicts

Extensions

Ordinal responses (e.g. a 5-point agree/disagree scale): an ordered logit with item-specific cutpoints $\tau_{j1} < \tau_{j2} < \dotsc < \tau_{j,K_j-1}$

\[Pr(Y_{ij} \le k) = F(\tau_{jk} - \beta_j\theta_i)\]
Multiple dimensions: $\theta_i \in \mathbb{R}^D$, with more identification constraints (rotation, not just reflection)
Hierarchical priors on $\theta_i$: let the prior mean depend on covariates (party, region, …) and borrow strength like last week’s multilevel models
Dynamic models: let $\theta_i$ change over time

Dynamic IRT

Dynamic IRT model

So far, we’ve focused on settings with just one time period.
- But many datasets (e.g. legislative voting, survey panels) observe the same units over time.
One option: estimate each period’s ideal points entirely separately.
- But this seems inefficient. The previous year’s latent score is probably a good predictor of this year’s score.
- How do we incorporate this in the model?
Put a random walk prior on $\theta_i$
- In political science, first used by Martin and Quinn (2002) to study the ideology of Supreme Court justices.
- Many other applications to different settings (e.g. UN General Assembly voting by Bailey, Strezhnev, and Voeten (2017))

Dynamic IRT model

We observe roll call votes for $N$ legislators on $J$ bills across $T$ time periods.
- $Y_{ijt}$ denotes the choice ($1$ = Yea, $0$ = Nay)
- Note that not all legislators will vote in every time period, not every bill appears in every time period, etc…
Same two-parameter set-up as before with a time-specific ideal point $\theta_{it}$. Martin and Quinn (2002) use the probit link here.

\[Pr(Y_{ijt} = 1) = \Phi\bigg(\beta_j\theta_{it} - \tau_j\bigg)\]

Dynamic IRT model

Same normal priors on the bill parameters $\beta_j$ and $\tau_j$
For the first ideal point for legislator $i$, $\theta_{i0}$, we use a standard normal prior.
But each successive ideal point is assumed to come from a random walk centered on the previous ideal point

\[\theta_{it} \sim \mathcal{N}(\theta_{i(t-1)}, \omega^2)\]
$\omega^2$ is a user-specified “smoothing parameter” – what is our prior belief about how “jumpy” the ideal points are from year-to-year.
- $\omega^2 \to 0$ is equivalent to a pooled IRT model where every legislator has the same ideal point in each session.
- $\omega^2 \to \infty$ is equivalent to the setting where each time period is analyzed completely separately
- Same partial pooling logic as multilevel models, but pooling across adjacent periods

Illustration: Supreme Court Ideology

Martin and Quinn (2002) implement an MCMC algorithm for this model.
- Imai, Lo, and Olmsted (2016) develop a faster variational EM algorithm for obtaining the posterior modes.
- Implemented in emIRT as dynIRT(), which also ships with the Martin and Quinn data

data(mq_data)

Votes of every justice in every case, October Term 1937 through 2013. What does the “data matrix” look like? (One case, with $+1$/$-1$ for the two sides and $0$ for justices not on the Court)

mq_data$data.mq$rc[, 4]

     Harlan       Black     Douglas     Stewart    Marshall     Brennan 
          0           1           0           0           0           0 
      White      Warren       Clark Frankfurter   Whittaker      Burton 
          0           0           0           0           0           0 
       Reed      Fortas    Goldberg      Minton     Jackson      Burger 
          0           0           0           0           0           0 
   Blackmun      Powell   Rehnquist     Stevens    O.Connor      Scalia 
          0           0           0           0           0           0 
    Kennedy      Souter      Thomas    Ginsburg      Breyer    Rutledge 
          0           0           0           0           0           0 
     Murphy      Vinson      Byrnes  Sutherland     Cardozo    Brandeis 
          0           0           0          -1           1           1 
     Butler  McReynolds      Hughes   OJRoberts       Stone     Roberts 
         -1          -1           1          -1           1           0 
      Alito   Sotomayor       Kagan 
          0           0           0

Illustration: Supreme Court Ideology

Priors on the case parameters (emIRT writes the model as $\alpha_j + \beta_j\theta_{it}$, so its alpha is our $-\tau_j$)

mq_data$priors.mq$beta.mu

     [,1]
[1,]    0
[2,]    0

mq_data$priors.mq$beta.sigma

     [,1] [,2]
[1,]    1    0
[2,]    0    1

We use informative priors on some of the $\theta$ values to get identification

mq_data$priors.mq$x.mu0[mq_data$priors.mq$x.mu0 != 0]

[1] -2  1  3

mq_data$priors.mq$x.sigma0[mq_data$priors.mq$x.mu0 != 0]

[1] 0.1 0.1 0.1

Which justices do these correspond to?

rownames(mq_data$data.mq$rc)[mq_data$priors.mq$x.mu0 != 0]

[1] "Black"     "Stewart"   "Rehnquist"

Illustration: Supreme Court Ideology

How much smoothing are we doing?

head(mq_data$priors.mq$omega2)

     [,1]
[1,]  0.1
[2,]  0.1
[3,]  0.1
[4,]  0.1
[5,]  0.1
[6,]  0.1

Let’s estimate the model

mq_estimates <- dynIRT(.data = mq_data$data.mq,
                       .starts = mq_data$cur.mq,
                       .priors = mq_data$priors.mq,
                       .control = list(threads = 1, verbose = FALSE, thresh = 1e-6, maxit = 500))

mq_estimates$means$x is a justices $\times$ terms matrix of ideal points

Illustration: Supreme Court Ideology

Reshape to one row per justice per term they served (term $t$ is the October Term $1937 + t$)

mq_long <- function(x){ # justices x terms matrix -> data frame of served terms
  tibble(justice = rownames(mq_data$data.mq$rc), i = seq_along(justice),
         start = mq_data$data.mq$startlegis[, 1], end = mq_data$data.mq$endlegis[, 1]) %>%
    mutate(t = map2(start, end, seq)) %>% unnest(t) %>%
    mutate(term = 1937 + t, ideal_point = x[cbind(i, t + 1)])
}
results_df <- mq_long(mq_estimates$means$x)
results_df %>% select(justice, term, ideal_point) %>% slice(1:5)

# A tibble: 5 × 3
  justice  term ideal_point
  <chr>   <dbl>       <dbl>
1 Harlan   1954       0.432
2 Harlan   1955       0.714
3 Harlan   1956       0.900
4 Harlan   1957       1.06 
5 Harlan   1958       1.07

Illustration: Supreme Court Ideology

What happens when we use a lower smoothing variance?

priors_strong <- mq_data$priors.mq
priors_strong$omega2[, 1] <- .001

mq_estimates_heavysmooth <- dynIRT(.data = mq_data$data.mq,
                                   .starts = mq_data$cur.mq,
                                   .priors = priors_strong,
                                   .control = list(threads = 1, verbose = FALSE, thresh = 1e-6, maxit = 500))
results_heavysmooth <- mq_long(mq_estimates_heavysmooth$means$x)

Illustration: Supreme Court Ideology

$\omega^2 = 0.001$ instead of $0.1$:

Summary

Like mixture models, Item Response Theory (IRT) models treat the outcome as a function of a latent variable
- However, in IRT, the latent variable is continuous rather than discrete
As with clustering, imagine a regression model with unknown regressors
- Our goal is to draw inferences about these regressors under the model structure.
Inference algorithms leverage the fact that conditional on the latent regressors, the model has a familiar structure (e.g. logistic regression)
- But similar identifiability problems as clustering: invariance to reflection + rescaling.
Careful! - Results dependent on model structure!
- Interpretation of the model parameters is ultimately up to the researchers - no inherent “meaning” to the latent scale.
- In the CES, very different response patterns all end up in the “middle” of a single dimension.

Next week

Regularization and model selection
- Variable selection and penalized regression (Ridge, LASSO)
- Cross-fitting and out-of-sample validation
Readings: Chapters 6-7 of James et al. (2013)

References

Bailey, Michael A., Anton Strezhnev, and Erik Voeten. 2017. “Estimating Dynamic State Preferences from United Nations Voting Data.” Journal of Conflict Resolution 61 (2): 430–56. https://doi.org/10.1177/0022002715595700.

Barberá, Pablo. 2015. “Birds of the Same Feather Tweet Together: Bayesian Ideal Point Estimation Using Twitter Data.” Political Analysis 23 (1): 76–91. https://doi.org/10.1093/pan/mpu011.

Broockman, David E., and Benjamin E. Lauderdale. 2025. “‘Moderates’.” American Political Science Review. https://doi.org/10.1017/S0003055424001333.

Clinton, Joshua, Simon Jackman, and Douglas Rivers. 2004. “The Statistical Analysis of Roll Call Data.” American Political Science Review 98 (2): 355–70. https://doi.org/10.1017/S0003055404001194.

Fowler, Anthony, Seth J. Hill, Jeffrey B. Lewis, Chris Tausanovitch, Lynn Vavreck, and Christopher Warshaw. 2023. “Moderates.” American Political Science Review 117 (2): 643–60. https://doi.org/10.1017/S0003055422000818.

Imai, Kosuke, James Lo, and Jonathan Olmsted. 2016. “Fast Estimation of Ideal Points with Massive Data.” American Political Science Review 110 (4): 631–56. https://doi.org/10.1017/S000305541600037X.

James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2013. An Introduction to Statistical Learning. New York: Springer. https://www.statlearning.com/.

Martin, Andrew D., and Kevin M. Quinn. 2002. “Dynamic Ideal Point Estimation via Markov Chain Monte Carlo for the U.S. Supreme Court, 1953–1999.” Political Analysis 10 (2): 134–53. https://doi.org/10.1093/pan/10.2.134.

Schaffner, Brian, Stephen Ansolabehere, Jeremy Pope, and Marissa Shih. 2024. “CES Common Content, 2023.” Harvard Dataverse. https://doi.org/10.7910/DVN/JQJTCC.