Testing Attrition Bias in Field Experiments (with Sarojini Hirshleifer and Karen Ortiz-Becerra)

• Development Workshop

Date and Location

Tuesday, November 6, 2018, 4:10 PM - 5:30 PM
ARE Library Conference Room, 4101 Social Sciences and Humanities

Abstract

Attrition is a common threat to the internal validity of field experiments in economics. This paper brings insights from the nonparametric identification literature in panel models to address the question of how to formally test for attrition bias in the presence of baseline outcome data. We first conduct a systematic review of the field experiment literature in economics and find that there is no consensus on how to test for attrition bias. Of the field experiments we reviewed: 81% test for differential attrition rates, but only 59% conduct some type of test for internal validity conditional on response status. We then formally demonstrate that: (1) the differential attrition rate test does not control size in general, and (2) internal validity has a sharp testable restriction on the baseline outcome distribution. We show that this restriction implies joint (as opposed to simple) tests of distributional equality, which are not widely used in the literature. We also demonstrate that the restriction differs depending on whether the authors’ object of interest is the treatment effect for respondents or for the population selected for the evaluation. We propose randomization procedures to obtain p-values for Kolmogorov-Smirnov- and Cramer-von-Mises-type statistics of the joint hypothesis. We further extend our framework to testing internal validity in the case of stratified randomization. Finally, we include simulation experiments to illustrate the implications of these results for empirical practice.