Seminar with Gary King - Who’s to Blame for Survey Instability: Respondents with Nonexistent Preferences or Researchers with Flawed Measures? 

Seminars - Theory and Experiments Seminar Series
(joint with Department of Decision Sciences)
Speakers
Gary King, Harvard University
ROOM 3-E4-SR03 - Via Roentgen 1
srf-019

Neither. For at least 75 years, survey researchers have found that about a quarter of respondents give different answers when asked the same question twice (even if no material changes occur and respondents do not remember being asked the first time). This “survey instability” problem casts doubt on a vast research enterprise spanning large areas of academia and industry, is core to ongoing substantive debates in numerous scholarly fields, and requires a resolution for proper survey design and analysis methods. To tackle this open question, we collect a wide variety of observational and experimental evidence, including 53 unique surveys. We first show that instability barely drops after accounting for both existing explanations, i.e., when respondents have fixed preferences and researchers use high quality, unbiased survey instruments. We trace most survey instability to a different source recognized only in fields with non-survey measurement instruments -- intrinsic human stochasticity. We then trace precursors of this stochasticity and reveal their wide ranging implications for understanding respondents, designing surveys, and building statistical analysis methods.