Skip to main content
SpringerLink
Log in
Book cover

International Conference on Information

iConference 2021: Diversity, Divergence, Dialogue pp 384–393Cite as

Conducting Quantitative Research with Hard-To-Reach-Online Populations: Using Prime Panels to Rapidly Survey Older Adults During a Pandemic

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12646))

Abstract

Vulnerable populations (e.g., older adults) can be hard to reach online. During a pandemic like COVID-19 when much research data collection must be conducted online only, these populations risk being further underrepresented. This paper explores methodological strategies for rigorous, efficient survey research with a large number of older adults online, focusing on (1) the design of a survey instrument both comprehensible and usable by older adults, (2) rapid collection (within hours) of data from a large number of older adults, and (3) validation of data using attention checks, independent validation of age, and detection of careless responses to ensure data quality. These methodological strategies have important implications for the inclusion of older adults in online research.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    Formerly known as TurkPrime, and also offering MTurk Toolkit, a platform designed to integrate MTurk into the social science workflow [24].

  2. 2.

    https://www.w3.org/TR/WCAG20/.

  3. 3.

    We recruited older adult participants from our research team’s established relationship with local community partners. We chose to partner with local participants instead of crowdsourced participants due to our established relationship, rapport, level of engagement, and length of the task.

References

  1. Chandler, J., Rosenzweig, C., Moss, A.J., Robinson, J., Litman, L.: Online panels in social science research: expanding sampling methods beyond Mechanical Turk. Behav. Res. Methods 51(5), 2022–2038 (2019). https://doi.org/10.3758/s13428-019-01273-7

    Article  Google Scholar 

  2. Levay, K.E., Freese, J., Druckman, J.N.: The demographic and political composition of mechanical Turk samples. Sage Open. (2016). https://doi.org/10.1177/2158244016636433

    Article  Google Scholar 

  3. He, W., Goodkind, D., Kowal, P.: An aging world: 2015. International Population Reports P95/16–1, United States Census Bureau (2016). https://www.census.gov/content/dam/Census/library/publications/2016/demo/p95-16-1.pdf

  4. Weil, J., Mendoza, A.N., McGavin, E.: Recruiting older adults as participants in applied social research: applying and evaluating approaches from clinical studies. Educ. Gerontol. 43(12), 662–673 (2017)

    Google Scholar 

  5. Zickuhr, K., Madden, M.: Older adults and internet use. Pew Research Center (2012). https://www.pewresearch.org/internet/2012/06/06/older-adults-and-internet-use

  6. Kwak, J., Xie, B., Champion, J.D., Fleischmann, K.R.: Rural dementia caregivers in Southwest Texas: an exploratory study of advance directives and end-of-life proxy decision making. J. Gerontol. Nurs. 45(9), 11–17 (2019). https://doi.org/10.3928/00989134-20190530-01

    Article  Google Scholar 

  7. Xie, B.: Effects of an e-health literacy intervention for older adults. J. Med. Internet Res. 13(4), e90 (2011). https://doi.org/10.2196/jmir.1880

    Article  Google Scholar 

  8. Xie, B.: Improving older adults’ e-health literacy through computer training using NIH online resources. Libr. Inf. Sci. Res. 34(1), 63–71 (2012). https://doi.org/10.1016/j.lisr.2011.07.006

    Article  Google Scholar 

  9. Xie, B., Bugg, J.M.: Public library computer training for older adults to access high-quality Internet health information. Libr. Inf. Sci. Res. 31(3), 155–162 (2009). https://doi.org/10.1016/j.lisr.2009.03.004

    Article  Google Scholar 

  10. Clay, R.A.: Conducting research during the COVID-19 pandemic: advice from psychological researchers on protecting participants, animals and research plans. APA News, American Psychological Association, 19 March 2020. https://www.apa.org/news/apa/2020/03/conducting-research-covid-19

  11. Behrend, T.S., Sharek, D.J., Meade, A.W., Wiebe, E.N.: The viability of crowdsourcing for survey research. Behav. Res. Methods 43(3), 800–813 (2011). https://doi.org/10.3758/s13428-011-0081-0

    Article  Google Scholar 

  12. Geldsetzer, P.: Use of rapid online surveys to assess people’s perceptions during infectious disease outbreaks: a cross-sectional survey on COVID-19. J. Med. Internet Res. 22(4), e18790 (2020). https://doi.org/10.2196/18790

    Article  Google Scholar 

  13. Lin, S.-Y., Thompson, H.J., Hart, L.A., Fu, M.C., Demiris, G.: Evaluation of pharmaceutical pictograms by older “turkers”: a cross-sectional crowdsourced study. Res. Soc. Admin. Pharm. (2020). https://doi.org/10.1016/j.sapharm.2020.08.006

  14. Brewer, R., Morris, M.R., Piper, A.M.: Why would anybody do this? Older adults’ understanding of and experiences with crowd work. In: CHI 2016: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 2246–2257. ACM Digital Library (2016). https://doi.org/10.1145/2858036.2858198

  15. McDuffie, D.: Using Amazon’s mechanical Turk: benefits, drawbacks, and suggestions. APS Observer. 32(2), 34–35 (2019). https://www.psychologicalscience.org/observer/using-amazons-mechanical-turk-benefits-drawbacks-and-suggestions

  16. Skorupska, K., Núñez, M., Kopec, W., Nielek, R.: Older adults and crowdsourcing: Android TV app for evaluating TEDx subtitle quality. In: Proceedings of the ACM on Human–Computer Interaction, Article 159. ACM Digital Library (2018). https://doi.org/10.1145/3274428

  17. Cheung, J.H., Burns, D.K., Sinclair, R.R., Sliter, M.: Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations. J. Bus. Psychol. 32(4), 347–361 (2016). https://doi.org/10.1007/s10869-016-9458-5

    Article  Google Scholar 

  18. Ogletree, A.M., Katz, B.: How do older adults recruited using MTurk differ from those in a national probability sample? Int. J. Aging Hum. Devel. (2020). https://doi.org/10.1177/0091415020940197

    Article  Google Scholar 

  19. Difallah, D., Filatova, E., Ipeirotis, P.: Demographics and dynamics of mechanical Turk workers. In: WSDM 2018: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pp. 135–143. ACM Digital Library (2018). https://doi.org/10.1145/3159652.3159661

  20. Robinson, J., Rosenzweig, C., Moss, A.J., Litman, L.: Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the mechanical Turk participant pool. PLoS ONE 14(12), e0226394 (2019). https://doi.org/10.1371/journal.pone.0226394

    Article  Google Scholar 

  21. Stewart, N., Ungemach, C., Harris, A.J.L., Bartels, D.M., Newell, B.R., Paolacci, G., Chandler, J.: The average laboratory samples a population of 7,300 Amazon mechanical Turk workers. Judgm. Decis. Making. 10(5), 479–491 (2015). https://journal.sjdm.org/14/14725/jdm14725.pdf

  22. Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P.D., Jewell, R., Winter, N.J.G.: The shape of and solutions to the MTurk quality crisis. Pol. Sci. Res. Methods (2020). https://doi.org/10.1017/psrm.2020.6

    Article  Google Scholar 

  23. Kapelner, A., Chandler, D.: Preventing satisficing in online surveys: a “kaptcha” to ensure higher quality data. In: CrowdConf 2010, San Francisco (2010)

    Google Scholar 

  24. Litman, L., Robinson, J., Abberbock, T.: TurkPrime.com: a versatile crowdsourcing data acquisition platform for the behavioral sciences. Behav. Res. Methods 49(2), 433–442 (2016). https://doi.org/10.3758/s13428-016-0727-z

    Article  Google Scholar 

  25. Kane, J.V., Barabas, J.: No harm in checking: Using factual manipulation checks to assess attentiveness in experiments. Am. J. Pol. Sci. 63(1), 234–249 (2019). https://doi.org/10.1111/ajps.12396

    Article  Google Scholar 

  26. Oppenheimer, D.M., Meyvis, T., Davidenko, N.: Instructional manipulation checks: detecting satisficing to increase statistical power. J. Exp. Soc. Psychol. 45(4), 867–872 (2009). https://doi.org/10.1016/j.jesp.2009.03.009

    Article  Google Scholar 

  27. Bishop, G.F.: Experiments with the middle response alternative in survey questions. Public Opin. Q. 51(2), 220–232 (1987). https://doi.org/10.1086/269030

    Article  Google Scholar 

  28. Weijters, B., Cabooter, E., Schillewaert, N.: The effect of rating scale format on response styles: the number of response categories and response category labels. Int. J. Res. Mark. 27(3), 236–247 (2010). https://doi.org/10.1016/j.ijresmar.2010.02.004

    Article  Google Scholar 

  29. Aday, L.A., Cornelius, L.J.: Formulating questions about knowledge and attitudes. In: Designing and Conducting Health Surveys: A Comprehensive Guide, 3rd ed., pp. 268–287. Wiley, Jossey-Bass (2006)

    Google Scholar 

  30. Beatty, P.C., Willis, G.B.: Research synthesis: The practice of cognitive interviewing. Public Opin. Q. 71(2), 287–311 (2007). https://doi.org/10.1093/poq/nfm006

    Article  Google Scholar 

  31. Drennan, J.: Cognitive interviewing: verbal data in the design and pretesting of questionnaires. J. Adv. Nurs. 42(1), 57–63 (2003). https://doi.org/10.1046/j.1365-2648.2003.02579.x

    Article  Google Scholar 

  32. Jobe, J.B., Mingay, D.J.: Cognitive laboratory approach to designing questionnaires for surveys of the elderly. Public Health Rep. 105(5), 518–524 (1990). PMCID: PMC1580104

    Google Scholar 

  33. Meade, A.W., Craig, S.B.: Identifying careless responses in survey data. Psychol. Methods. 17, 437 (2012). 10/f399k2

    Google Scholar 

  34. Huang, J.L., Curran, P.G., Keeney, J., Poposki, E.M., DeShon, R.P.: Detecting and deterring insufficient effort responding to surveys. J. Bus. Psychol. 27, 99–114 (2012). 10/dppm46

    Google Scholar 

  35. Curran, P.G.: Methods for the detection of carelessly invalid responses in survey data. J. Exp. Soc. Psychol. 66, 4–19 (2016). 10/f8zmvf

    Google Scholar 

  36. Seifert, A., Cotten, S. R., Xie, B.: A double burden of exclusion? Digital and social exclusion of older adults in times of COVID-19. J. Gerontol. B Psychol. Sci. Soc. Sci. (2020). https://doi.org/10.1093/geronb/gbaa098

  37. Xie, B., Charness, N., Fingerman, K., Kaye, J., Kim, M.T., Khurshid, A.: When going digital becomes a necessity: ensuring older adults’ needs for information, services, and social inclusion during COVID-19. J. Aging Soc. Policy. 32(4–5), 460–470 (2020). https://doi.org/10.1080/08959420.2020.1771237

    Article  Google Scholar 

  38. Anderson, M., Perrin, A.: Tech adoption climbs among older adults. Pew Res. Center (2017). https://www.pewinternet.org/2017/05/17/tech-adoption-climbs-among-older-adults/

  39. Semuels, A.: The internet is enabling a new kind of poorly paid hell. The Atlantic, 23 January 2018. https://www.theatlantic.com/business/archive/2018/01/amazon-mechanical-turk/551192/. Accessed 16 Oct 2020

  40. Silberman, M.S., Tomlinson, B., LaPlante, R., Ross, J., Irani, L., Zaldivar, A.: Responsible research with crowds: pay crowdworkers at least minimum wage. Commun. ACM 61(3), 39–41 (2018). https://doi.org/10.1145/3180492

    Article  Google Scholar 

  41. Templeton, T.C., Fleischmann, K.R.: The relationship between human values and attitudes toward the Park51 and nuclear power controversies. Proc. Am. Soc. Info. Sci. Tech. 48, 1–10 (2011). https://doi.org/10.1002/meet.2011.14504801172

    Article  Google Scholar 

  42. Verma, N., Fleischmann, K.R., Koltai, K.S.: Human values and trust in scientific journals, the mainstream media and fake news. Proc. Assoc. Inf. Sci. Technol. 54, 426–435 (2017). 10/ghkc6k

    Google Scholar 

  43. Lee, M.K.: Understanding perception of algorithmic decisions: fairness, trust, and emotion in response to algorithmic management. Big Data Soc. 5, 2053951718756684 (2018). 10/ggsfsp

    Google Scholar 

  44. CloudResearch: How are Participants on Prime Panels Compensated? (2019). https://go.cloudresearch.com/knowledge/how-are-participants-on-prime-panels-compensated. Accessed 17 Oct 2020

  45. SurveyMonkey: SurveyMonkey Raises $15M for Charitable Causes, One Survey at a Time. SurveyMonkey, 18 February 2020. https://www.surveymonkey.com/newsroom/surveymonkey-raises-15m-for-charitable-causes/

  46. Lamb, J.: Clear Outcomes. Americans’ COVID-19 preventative practices in April and May 2020. Inter-university Consortium for Political and Social Research (ICPSR). 10/ghfwtc

    Google Scholar 

Download references

Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant No. 2027426. We thank John Bellquist, Ph.D., Editor of the Cain Center in the School of Nursing at The University of Texas at Austin, for his professional proofreading of an earlier draft of this manuscript; Le (Betty) Zhou, Ph.D., at Carlson School of Management, the University of Minnesota for helping us with data validation; and the anonymous participants of this study.

Author information

Authors and Affiliations

  1. School of Information, The University of Texas at Austin, Austin, TX, 78701, USA

    Nitin Verma, Kristina Shiroma, Kate Rich, Kenneth R. Fleischmann, Bo Xie & Min Kyung Lee

  2. School of Nursing, The University of Texas at Austin, Austin, TX, 78712, USA

    Bo Xie

  3. Department of Communication, The University of Washington, Seattle, WA, 98195, USA

    Kate Rich

Authors
  1. Nitin Verma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kristina Shiroma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kate Rich
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kenneth R. Fleischmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bo Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Min Kyung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nitin Verma .

Editor information

Editors and Affiliations

  1. Humboldt University of Berlin, Berlin, Germany

    Katharina Toeppe

  2. Renmin University of China, Beijing, China

    Hui Yan

  3. The University of Hong Kong, Hong Kong, Hong Kong

    Samuel Kai Wah Chu

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Verma, N., Shiroma, K., Rich, K., Fleischmann, K.R., Xie, B., Lee, M.K. (2021). Conducting Quantitative Research with Hard-To-Reach-Online Populations: Using Prime Panels to Rapidly Survey Older Adults During a Pandemic. In: Toeppe, K., Yan, H., Chu, S.K.W. (eds) Diversity, Divergence, Dialogue. iConference 2021. Lecture Notes in Computer Science(), vol 12646. Springer, Cham. https://doi.org/10.1007/978-3-030-71305-8_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-71305-8_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-71304-1

  • Online ISBN: 978-3-030-71305-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation