@Who? Investigating Possible Errors in Studies Linking Survey and Twitter Data

Abstract

Expanding global usage of social media and growing questions about its societal impact have led scholars to investigate the relationship between individuals' offline and online behaviors and characteristics. Such inquiries, which compare individuals' survey responses to their social media behavior, typically do not address whether the elicitation of survey respondents' social media information introduces any systematic errors. However, making inferences from a survey-linked sample to a social media platform, and finally to a survey sample or broader target population, can be imperiled when systematic differences exist between those who provide and those who deny researchers access to their social media accounts. In this paper, we ask: Do survey respondents who say they use Twitter differ from the subset providing validated Twitter handles, as well as from the overall survey sample? Pooling across five datasets and over 31,000 respondents, we show first that samples of stated Twitter users differ from the initial survey samples from which they are drawn on several socio-demographic characteristics. Second and reassuringly as concerns possible errors due to survey-linkage, we report few systematic differences between those who say they use Twitter and those who provide validated Twitter handles. Nevertheless, we do document differences on some demographics, and we illustrate how errors could carry potential consequences for sample composition of which researchers should be aware. Finally, we conclude with a discussion of our results, their possible generalizability, and areas for future research.