Disentangling Selection into Mode from Mode Effects

Colm O'Muircheartaigh; L Philip Schumm; Ned English; Becki Curtis

doi:10.1093/geronb/gbae140

Disentangling Selection into Mode from Mode Effects

J Gerontol B Psychol Sci Soc Sci. 2024 Aug 21:gbae140. doi: 10.1093/geronb/gbae140. Online ahead of print.

Authors

Colm O'Muircheartaigh¹, L Philip Schumm², Ned English³, Becki Curtis³

Affiliations

¹ Harris School of Public Policy, University of Chicago, Chicago, Illinois, USA.
² Center for Translational Data Science, University of Chicago, Chicago, Illinois, USA.
³ NORC at the University of Chicago, Methodology and Quantitative Social Sciences, Chicago, Illinois, USA.

PMID: 39164215
DOI: 10.1093/geronb/gbae140

Abstract

Objectives: We investigate the impact of data collection mode on responses to variables in NSHAP Round 4 and discuss how potential mode differences should (and should not) be addressed in substantive analyses.

Methods: Among the set of respondents who were eligible to be contacted remotely in Round 4, we randomly selected 398 to be contacted instead for an in-person interview. We compare response rates and the distributions of selected key outcomes among those 398 respondents to those among the control group who were initially approached remotely. As a contrast, we compare all R4 respondents according to the mode in which they completed the interview, including those not part of the randomized experiment.

Results: Among those included in the experiment, there was no evidence of systematic differences in responses to physical and mental health questions between remote and in-person modes, nor in responses to number recall measures. In-person respondents scored moderately lower on cognitive function measures requiring careful attention to a figure and/or task, though this difference became less with each similar item. Remote respondents named fewer social network members. Comparing all respondents according to their final mode yielded substantially different results in all cases.

Discussion: Mode did not appear to affect reports of physical and mental health based on a randomized comparison, though it did moderately affect other items in predictable ways. Naïve estimates of mode effects based on comparing all respondents according to mode yielded misleading results, and should not be used to adjust for mode differences in analyses.

Keywords: Experimental design; Mode effects; Multimode; Surveys.

© The Author(s) 2024. Published by Oxford University Press on behalf of The Gerontological Society of America. All rights reserved. For commercial re-use, please contact reprints@oup.com for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site—for further information please contact journals.permissions@oup.com.