Research
Working Papers
- Omitted variable bias sensitivity analysis with clustered treatment assignment
Cinelli and Hazlett (2020) develops a sensitivity analysis method for the linear regression model that parameterizes omitted variable bias in terms of two partial R² parameters capturing the residual variation explained by an omitted confounder in the treatment and outcome respectively. This method is often applied to regressions fit to unit-level data when treatment is assigned at a higher level of aggregation – as in clustered observational designs. This paper shows that despite the numerical equivalence of the unit-level regression and an appropriately weighted cluster-aggregated regression for estimating the treatment effect, the sensitivity analysis procedure yields different conclusions depending on the chosen level of analysis. The outcome-confounder partial R² reflects both between- and within- group variation but the latter is irrelevant to omitted variable bias as it by construction cannot be explained by a group-level confounder. Straightforward corrections to the robustness value and the extreme scenario analysis from the unit-level regression using Pearson’s partial-η recover equivalence between these two approaches. The paper concludes with a point of caution when benchmarking against unit-level covariates and recommends always including cluster-level averages of these covariates as regressors (Mundlak, 1978).
- Benchmarking parallel trends violations in regression imputation difference-in-differences
Difference-in-differences (DiD) studies increasingly use regression imputation methods as an alternative to the conventional two-way fixed effects (TWFE) estimator, fitting a TWFE regression on the controls to impute treated counterfactuals. A common method for obtaining pre-trend placebo estimates uses the same model to impute outcomes for control units – the default in the popular fect R package. We decompose this “in-sample imputation” estimator into its component 2 x 2 differences-in-differences to show two biases: an attenuation bias driven by redundant differences-in-differences that are zero by construction and a contamination bias resulting from the use of “early adopters” as controls under staggered adoption. These distort both the magnitude and the apparent shape of the pre-treatment trends. We show that both biases are addressed by a simple correction computed from the in-sample imputations. To illustrate, we re-analyze a study of the political effects of the 2008 “shale shock” on Republican vote share in U.S. coal counties (Gazmararian, 2025). While the original analysis used in-sample imputation and concluded pre-trends were small, corrected placebo estimates of pre-trends are comparable in magnitude to the estimated treatment effects. The observed post-2008 Republican gains in coal counties are likely artifacts of longer-running regional trends.
- Aggregation, Interpretation, and Estimation of Preferences in Conjoint Experiments
Conjoint experiments are widely used to study preferences in multidimensional choice settings. Commonly reported estimands in this literature summarize a feature’s average performance against the full field of alternatives induced by the design, aggregating direct comparisons between the two feature levels of interest together with indirect comparisons involving other levels of the same attribute. Yet researchers’ substantive questions often concern binary preference relations — whether respondents prefer feature A to feature B. These are distinct quantities, and we show that they can diverge when there are preference cycles: a feature may perform better against the field while losing in the direct comparison. We introduce a new estimand, the average feature choice probability (AFCP), that directly targets the binary preference relation; decompose widely used estimands into weighted averages of such pairwise comparisons; and develop statistical tools that diagnose divergence between binary preference relations and “against-the-field” estimands.
- A Guide to Dynamic Difference-in-Differences Regressions for Political Scientists
Difference-in-differences (DiD) designs for estimating causal effects have grown in popularity throughout political science. It is common for DiD studies to report their main results using a “dynamic” or “event study” two-way fixed effects (TWFE) regression. This regression combines estimates of average treatment effects for multiple post-treatment time periods alongside placebo tests of the main identifying assumption: parallel trends. Despite their ubiquity, there is little clear and consistent guidance in the discipline for how researchers should estimate dynamic treatment effects. This paper develops a novel decomposition of the dynamic TWFE regression coefficients in terms of their component 2x2 difference-in-differences comparisons in the style of Goodman-Bacon (2021). We use this decomposition to illustrate how bias can result from the incorrect specification of baseline time periods, the inclusion of units and time periods where all observations are treated, and heterogeneity in the dynamic treatment effects across different treatment timing groups. Our results provide additional intuition for the source of bias due to effect heterogeneity—what Sun and Abraham (2021) term “contamination bias”—by directly characterizing the contaminated 2x2 comparisons. We then provide a common framework for connecting the many proposed “heterogeneity-robust” estimators in the literature, noting that they vary primarily in which 2x2 comparisons they choose to include. Through a replication of three studies published in prominent political science journals, we conclude by showing how attentiveness to baseline selection and specification can alter findings.
- Group-specific linear trends and the triple-differences in time design
Differences-in-differences designs for estimating causal effects rely on an assumption of “parallel trends” – that in the absence of the intervention, treated units would have followed the same outcome trajectory as observed in control units. When parallel trends fails, researchers often turn to alternative strategies that relax this identifying assumption. One popular approach is the inclusion of a group-specific linear time trend in the commonly used two-way fixed effects (TWFE) estimator. In a setting with a single post-treatment and two pre-treatment periods it is well known that this is equivalent to a non-parametric “triple-differences” estimator which is valid under a “parallel trends-in-trends” assumption (Egami and Yamauchi, 2023). This paper analyzes the TWFE estimator with group-specific linear time trends in the more general setting with many pre- and post-treatment periods. It shows that this estimator can be interpreted as an average over triple-differences terms involving both pre-treatment and post-treatment observations. As a consequence, this estimator does not identify a convex average of post-treatment ATTs without additional effect homogeneity assumptions even when there is no staggering in treatment adoption. A straightforward solution is to make the TWFE specification fully dynamic with a separate parameter for each relative treatment time. However, identification requires that researchers omit at least two pre-treatment relative treatment time indicators to estimate a group-specific linear trend. The paper shows how to properly extend this estimator to the staggered adoption setting using the approach of Sun and Abraham (2021), correcting a perfect collinearity error in recent implementations of this method in Hassell and Holbein (2024). It concludes with a note of caution for researchers, showing through a replication of Kogan (2021) how inferences from group-specific time trend specifications can be extremely sensitive to arbitrary specification choices when parallel trends violations are present but do not follow an easily observed functional form.
- Decomposing Triple-Differences Regression with Staggered Adoption
The triple-differences (TD) design is a popular identification strategy for causal effects in settings where researchers do not believe the parallel trends assumption of conventional difference-in-differences (DiD) is satisfied. TD designs augment the conventional 2x2 DiD with a “placebo” stratum – observations that are nested in the same units and time periods but are known to be entirely unaffected by the treatment. However, many TD applications go beyond this simple 2x2x2 and use observations on many units in many “placebo” strata across multiple time periods. A popular estimator for this setting is the triple-differences regression (TDR) fixed-effects estimator – an extension of the common “two-way fixed effects” estimator for DiD. This paper decomposes the TDR estimator into its component two-group/two-period/two-strata triple-differences and illustrates how interpreting this parameter causally in settings with arbitrary staggered adoption requires strong effect homogeneity assumptions as many placebo DiDs incorporate observations under treatment. The decomposition clarifies the implied identifying variation behind the triple-differences regression estimator and suggests researchers should be cautious when implementing these estimators in settings more complex than the 2x2x2 case. Alternative approaches that only incorporate “clean placebos” such as direct imputation of the counterfactual may be more appropriate. The paper concludes by demonstrating the utility of this imputation estimator in an application of the “gravity model” to the estimation of the effect of the WTO/GATT on international trade.
Journal Articles
- The Political Benefits of the Monoculture: Estimating the Electoral Effect of the Market Facilitation ProgramJournal of Public Economics, 2026
Does the distribution of government transfers affect elections? We analyze a natural experiment in the 2019 wave of the US Department of Agriculture’s Market Facilitation Program (MFP). The 2019 MFP allocated $14.5 billion via a formula combining historical production data and commodity-specific trade damages. We show how a methodological quirk resulted in arbitrary variation in these damages that propagated through the formula into excessive county-level compensation rates. We estimate the effects of this payment shock using a novel, design-based, randomization inference approach to account for complex dependencies across US counties. We find that counties receiving greater compensation rates, on average, have higher two-party Republican presidential vote shares in the 2020 election. Instrumenting for actual 2019 MFP disbursements, we find an additional $1 million in payments to a county increased that county’s 2020 two-party Trump vote share by about .18 percentage points on average. Had the 2019 MFP maintained the spending levels of the 2018 wave, we estimate that Candidate Trump’s two-party vote share would have been .117 percentage points lower nationally, with particularly pronounced effects in swing states like Arizona (.41 percentage points) and Georgia (.159 percentage points).
- An improved method of automated nonparametric content analysis for social sciencePolitical Analysis, 2023
Some scholars build models to classify documents into chosen categories. Others, especially social scientists who tend to focus on population characteristics, instead usually estimate the proportion of documents in each category—using either parametric “classify-and-count” methods or “direct” nonparametric estimation of proportions without individual classification. Unfortunately, classify-and-count methods can be highly model-dependent or generate more bias in the proportions even as the percent of documents correctly classified increases. Direct estimation avoids these problems, but can suffer when the meaning of language changes between training and test sets or is too similar across categories. We develop an improved direct estimation approach without these issues by including and optimizing continuous text features, along with a form of matching adapted from the causal inference literature. Our approach substantially improves performance in a diverse collection of 73 datasets. We also offer easy-to-use software that implements all ideas discussed herein.
- Longer trips to court cause evictionsProceedings of the National Academy of Sciences, 2023
Studying ∼200,000 evictions filed against ∼300,000 Philadelphians from 2005 to 2021, we focus on the role of transit to court in preventing tenants from asserting their rights. In this period, nearly 40% of tenants facing eviction were ordered to leave their residences because they did not show up to contest cases against them and received a default judgment. Controlling for a variety of potential confounds at the tenant and landlord level, we find that residents of private tenancies with longer transit travel time to the courthouse were more likely to default. A 1-h increase in estimated travel time increases the probability of default by between 3.8% and 8.6% points across different model specifications. The effect holds after adjusting for direct distance to the court, unobserved landlord characteristics, and even baseline weekend travel time. However, it is absent in public housing evictions, where timing rules are significantly laxer, and during the COVID-19 pandemic, when tenants had the opportunity to be present virtually. We estimate that had all tenants been equally able to get to the court in 10 min, there would have been 4,000 to 9,000 fewer default evictions over the sample period. We replicate this commuting effect in another dataset of over 800,000 evictions from Harris County, Texas. These results open up a new way to study the physical determinants of access to justice, illustrating that the location and accessibility of a courthouse can affect individual case outcomes. We suggest that increased use of video technology in court may reduce barriers to justice.
- Telescope matching for reducing model dependence in the estimation of the effects of time‐varying treatmentsJournal of the Royal Statistical Society: Series A (Statistics in Society), 2022
Time-varying treatments are prevalent in the social sciences. For example, a political campaign might decide to air attack ads against an opponent, but this decision to go negative will impact polling and, thus, future campaign strategy. If an analyst naively applies methods for point exposures to estimate the effect of earlier treatments, this would lead to post-treatment bias. Several existing methods can adjust for this type of time-varying confounding, but they typically rely on strong modelling assumptions. In this paper, we propose a novel two-step matching procedure for estimating the effect of two-period treatments. This method, telescope matching, reduces model dependence without inducing post-treatment bias by using matching with replacement to impute missing counterfactual outcomes. It then employs flexible regression models to correct for bias induced by imperfect matches. We derive the asymptotic properties of the telescope matching estimator and provide a consistent estimator for its variance. We illustrate telescope matching by investigating the effect of negative campaigning in US Senate and gubernatorial elections. Using the method, we uncover a positive effect on turnout of negative ads early in a campaign and a negative effect of early negativity on vote shares.
- International investment disputes, media coverage, and backlash against international lawJournal of Conflict Resolution, 2022
This paper puts forth a theory explaining domestic backlash against international investment law by connecting media coverage—specifically the bias in the news media’s selection of international disputes—to public opinion formation towards international agreements. To test our theory, we examine both the content and effects of the media’s reporting on international disputes, focusing on the increasingly controversial form known as investor-state dispute settlement (ISDS). We find that newspaper outlets in both the United States and Canada have a bias in favor of covering disputes filed against their home country as opposed to those filed by home country firms. Using two national survey experiments fielded in the United States and Canada, we further find that the bias in news story selection has a strong negative effect on attitudes towards ISDS and related agreements, especially among highly nationalistic individuals.
- Leases as formsJournal of Empirical Legal Studies, 2022
We offer the first large scale descriptive study of residential leases, based on a novel dataset of ~170,000 residential leases filed in support of over ~200,000 Philadelphia eviction proceedings from 2005 through 2019. These leases are highly likely to contain unenforceable terms, and their pro-landlord tilt has increased sharply over time. Matching leases with individual tenant characteristics, and to 16,261 unique owner-landlords, we show that unenforceable terms are likely to be associated with more expensive leaseholds in richer, whiter parts of the city. This result is linked to particular landlords growing adoption of shared forms, originally created by non-profit landlord associations, and more recently available online for a nominal fee. Generally, such shared form leases contain worse rules for tenants than the proprietary leases they replace. Over time, it has become easier and cheaper for landlords to adopt such common forms, meaning that access to justice for landlords strips tenants of rights.
- Sovereignty, Substance, and Public Support for European Courts' Human Rights RulingsAmerican Political Science Review, 2022
Is the public backlash against human rights rulings from European courts driven by substantive concerns over case outcomes, procedural concerns over sovereignty, or combinations thereof? We conducted preregistered survey experiments in Denmark, France, Poland, Spain, and the United Kingdom using three vignettes: a foreigner who faces extradition, a person fighting a fine for burning Qurans, and a home owner contesting eviction. Each vignette varies with respect to whether a European court disagrees with a national court (deference treatment) and whether an applicant wins a case (outcome treatment). We find little evidence that deference moves willingness to implement judgments or acceptance of court authority but ample evidence that case outcomes matter. Even nationalists and authoritarians are unmoved by European court decisions as long as they agree with the case outcome. These findings imply that nationalist opposition to European courts is more about content than the location of authority and that backlash to domestic and international courts may be driven by similar forces.
- Testing for Negative Spillovers: Is Promoting Human Rights Really Part of the 'Problem'?International Organization, 2021
The international community often seeks to promote political reforms in recalcitrant states. Recently, some scholars have argued that, rather than helping, international law and advocacy create new problems because they have negative spillovers that increase rights violations. We review three mechanisms for such spillovers: backlash, trade-offs, and counteraction and concentrate on the last of these. Some researchers assert that governments sometimes “counteract” international human rights pressures by strategically substituting violations in adjacent areas that are either not targeted or are harder to monitor. However, most such research shows only that both outcomes correlate with an intervention—the targeted positively and the spillover negatively. The burden of proof, however, should be as rigorous as those for studies of first-order policy consequences. We show that these correlations by themselves are insufficient to demonstrate counteraction outside of the narrow case where the intervention is assumed to have no direct effect on the spillover, a situation akin to having a valid instrumental variable design. We revisit two prominent findings and show that the evidence for the counteraction claim is weak in both cases. The article contributes methodologically to the study of negative spillovers in general by proposing mediation and sensitivity analysis within an instrumental variables framework for assessing such arguments. It revisits important prior findings that claim negative consequences to human rights law and/or advocacy, and raises critical normative questions regarding how we empirically evaluate hypotheses about causal mechanisms.
- Rulers or Rules? International Law, Elite Cues and Public OpinionEuropean Journal of International Law, 2019
One of the mechanisms by which international law can shape domestic politics is through its effects on public opinion. However, a growing number of national leaders have begun to advocate policies that ignore or even deny international law constraints. This article investigates whether international law messages can still shift public opinion even in the face of countervailing elite cues. It reports results from survey experiments conducted in three countries – the USA, Australia and India – which examined attitudes on a highly salient domestic political issue: restrictions on refugee admissions. In each experimental vignette, respondents were asked about their opinion on a proposed or ongoing restrictive refugee policy that was endorsed by the government but also likely contravened international refugee law. Respondents were randomly exposed to messages highlighting the policy’s illegality and/or elite endorsement. The results show that, on average, the international law messages had a small but significant persuasive effect in reducing support for the restrictive policy, at most 10 percentage points. Surprisingly, there was no evidence that the countervailing elite endorsement was a significant moderator of this effect. However, in the case of the USA and among Republican co-partisans of the president, the elite endorsement independently increased respondents’ beliefs that the restriction was legal under international law while having no effect on support for the policy. The results suggest that cues from domestic elites do not strictly trump those from international sources and that, despite cues about national leaders’ policy advocacy, international law can affect the attitudes of some voters even on an issue as heavily politicized as refugee policy.
- Investigator Characteristics and Respondent Behavior in Online SurveysJournal of Experimental Political Science, 2018
Prior research demonstrates that responses to surveys can vary depending on the race, gender, or ethnicity of the investigator asking the question. We build upon this research by empirically testing how information about researcher identity in online surveys affects subject responses. We do so by conducting an experiment on Amazon’s Mechanical Turk in which we vary the name of the researcher in the advertisement for the experiment and on the informed consent page in order to cue different racial and gender identities. We fail to reject the null hypothesis that there is no difference in how respondents answer questions when assigned to a putatively black/white or male/female researcher.
- The David Effect and ISDSEuropean Journal of International Law, 2017
The legitimacy of international investment law is fiercely contested. Chiefly, scholars argue that investor–state dispute settlement empowers corporations from rich nations over governments of poor ones. Some also assert that poor nations facing investment claims have limited ability to improve their standing in this setting of adjudication. Based on a first-of-its-kind experiment conducted on 257 international arbitrators, this article argues that one avenue to improve standing is for developing countries to exploit their ‘underdog’ status. We presented arbitrators with a vignette describing an investor–state dispute and randomly assigned different features to test their effect. Our results suggest arbitrators are prone to a particular type of bias – surveyed professionals were more likely to grant poor respondent states reimbursement of their legal costs compared to wealthy states when the respondent won the dispute. Based on this ‘David effect’, we argue for re-conceptualizing investor–state arbitration as a tool for partially mitigating power imbalances.
- Affiliation Bias in Arbitration: An Experimental ApproachThe Journal of Legal Studies, 2017
A characteristic feature of arbitration, a growing form of legal adjudication, is that each disputing party appoints an arbitrator. Commentators, however, suggest that party-appointed arbitrators tend to be biased in favor of their appointers. Evaluating this claim from data on historical disputes is problematic because of nonrandom selection of arbitrators. Here we use a novel experimental approach to estimate the causal effect of the appointing party. Using survey experiments with arbitration experts around the world, we show that professional arbitrators suffer from affiliation effects — a cognitive predisposition to favor the appointing party. At a methodological level, we offer a solution to the problem of measuring this effect when credible observational designs are lacking.
- Estimating Dynamic State Preferences from United Nations Voting DataJournal of Conflict Resolution, 2017
United Nations (UN) General Assembly votes have become the standard data source for measures of states preferences over foreign policy. Most papers use dyadic indicators of voting similarity between states. We propose a dynamic ordinal spatial model to estimate state ideal points from 1946 to 2012 on a single dimension that reflects state positions toward the US-led liberal order. We use information about the content of the UN’s agenda to make estimates comparable across time. Compared to existing measures, our estimates better separate signal from noise in identifying foreign policy shifts, have greater face validity, allow for better intertemporal comparisons, are less sensitive to shifts in the UN’ agenda, and are strongly correlated with measures of liberalism. We show that the choice of preference measures affects conclusions about the democratic peace.
Edited Volumes and Conference Proceedings
- Court Performance within the Multilateral Trade RegimeIn The Performance of International Courts and Tribunals, 2018
- boocio: An Education System with Hierarchical Concept Maps and Dynamic Nonlinear Learning PlansIEEE Transactions on Visualization and Computer Graphics, 2017
Information hierarchies are difficult to express when real-world space or time constraints force traversing the hierarchy in linear presentations, such as in educational books and classroom courses. We present booc.io, which allows linear and non-linear presentation and navigation of educational concepts and material. To support a breadth of material for each concept, booc.io is Web based, which allows adding material such as lecture slides, book chapters, videos, and LTIs. A visual interface assists the creation of the needed hierarchical structures. The goals of our system were formed in expert interviews, and we explain how our design meets these goals. We adapt a real-world course into booc.io, and perform introductory qualitative evaluation with students.
- Human Rights and Human Welfare: Looking for a Dark Side to International Human Rights LawIn Human Rights Futures, 2017