Qisheng Li
Krzysztof Z. Gajos
ABSTRACT
There are few large-scale empirical studies with people with disabilities or older adults, mainly because recruiting participants with specific characteristics is even harder than recruiting young and/or non-disabled populations. Analyzing four online experiments on LabintheWild with a total of 355,656 participants, we show that volunteer-based online experiments that provide personalized feedback attract large numbers of participants with diverse disabilities and ages and allow robust studies with these populations that replicate and extend the findings of prior laboratory studies. To find out what motivates people with disabilities to take part, we additionally analyzed participants' feedback and forum entries that discuss LabintheWild experiments. The results show that participants use the studies to diagnose themselves, compare their abilities to others, quantify potential impairments, self-experiment, and share their own stories -- findings that we use to inform design guidelines for online experiment platforms that adequately support and engage people with disabilities.
- Simon Baron-Cohen, Sally Wheelwright, Jacqueline Hill, Yogini Raste, and Ian Plumb. 2001. The "Reading the Mind in the Eyes" Test revised version: a study with normal adults, and adults with Asperger syndrome or high-functioning autism. The Journal of Child Psychology and Psychiatry and Allied Disciplines 42, 2 (2001), 241--251.Google Scholar
Cross Ref
- Marialena Barouti, Konstantinos Papadopoulos, and Georgios Kouroupetroglou. 2013. Synthetic and natural speech intelligibility in individuals with visual impairments: Effects of experience and presentation rate. In European AAATE Conference, Portugal. 695--699.Google Scholar
- Heather Becker, Greg Roberts, Janet Morrison, and Julie Silver. 2004. Recruiting people with disabilities as research participants: Challenges and strategies to address them. Mental Retardation 42, 6 (2004), 471--475.Google ScholarCross Ref
- Adam J Berinsky, Gregory A Huber, and Gabriel S Lenz. 2012. Using Mechanical Turk as a subject recruitment tool for experimental research. Political Analysis 20 (2012), 351--68.Google ScholarCross Ref
- Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and others. 2010. VizWiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 333--342. Google ScholarDigital Library
- J. Birch. 2001. Diagnosis of Defective Colour Vision. Oxford: Butterworth-Heinemann. 92048986Google Scholar
- Danielle Bragg, Cynthia Bennett, Katharina Reinecke, and Richard Ladner. 2018. A Large Inclusive Study of Human Listening Rates. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '18). ACM. Google ScholarDigital Library
- Robin Brewer, Meredith Ringel Morris, and Anne Marie Piper. 2016. "Why Would Anybody Do This?": Understanding Older Adults' Motivations and Challenges in Crowd Work. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 2246--2257. Google ScholarDigital Library
- Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. 2011. Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science 6, 1 (2011), 3--5.Google ScholarCross Ref
- Rocıo Calvo, Shaun K. Kane, and Amy Hurst. 2014. Evaluating the Accessibility of Crowdsourcing Tasks on Amazon's Mechanical Turk. In Proceedings of the 16th international ACM SIGACCESS conference on Computers and Accessibility (ASSETS '14). ACM, New York, NY, USA, 257--258. Google ScholarDigital Library
- Alaina Carr. 2014. An exploration of Mechanical Turk as a feasible recruitment platform for cancer survivors. Undergraduate Honors Theses 59 (2014).Google Scholar
- Marianne Dee and Vicki L. Hanson. 2014. A Large User Pool for Accessibility Research with Representative Users. In Proceedings of the 16th international ACM SIGACCESS conference on Computers and Accessibility (ASSETS '14). ACM, New York, NY, USA, 35--42. Google ScholarDigital Library
- Marianne Dee and Vicki L. Hanson. 2016. A Pool of Representative Users for Accessibility Research: Seeing Through the Eyes of the Users. ACM Trans. Access. Comput. 8, 1, Article 4 (Jan. 2016), 31 pages. Google ScholarDigital Library
- Xianghua Ding, Patrick C. Shih, and Ning Gu. 2017. Socially Embedded Work: A Study of Wheelchair Users Performing Online Crowd Work in China. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 642--654. Google ScholarDigital Library
- Yafit Gabay, Eli Vakil, Rachel Schiff, and Lori L Holt. 2015. Probabilistic category learning in developmental dyslexia: Evidence from feedback and paired-associate weather prediction tasks. Neuropsychology 29, 6 (2015), 844.Google ScholarCross Ref
- Laura Germine, Ken Nakayama, Bradley C Duchaine, Christopher F Chabris, Garga Chatterjee, and Jeremy B Wilmer. 2012. Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic bulletin & review 19, 5 (2012), 847--857.Google Scholar
- Barney G Glaser and Anselm L Strauss. 2009. Discovery of grounded theory: Strategies for qualitative research. Transaction Publishers.Google Scholar
- Samuel D Gosling, Simine Vazire, Sanjay Srivastava, and Oliver P John. 2004. Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American psychologist 59, 2 (2004), 93.Google Scholar
- Joshua K Hartshorne and Laura T Germine. 2015. When does cognitive functioning peak? The asynchronous rise and fall of different cognitive abilities across the life span. Psychological science (2015), 0956797614567339.Google Scholar
- Jeffrey Heer and Michael Bostock. 2010. Crowdsourcing graphical perception: Using Mechanical Turk to assess visualization design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). 203--212. Google ScholarDigital Library
- J J Horton, D G Rand, and R J Zeckhauser. 2011. The online laboratory: Conducting experiments in a real labor market. Experimental Economics (2011).Google Scholar
- Bernd Huber, Katharina Reinecke, and Krzysztof Z. Gajos. 2017. The Effect of Performance Feedback on Social Media Sharing at Volunteer-Based Online Experiment Platforms. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1882--1886. Google ScholarDigital Library
- International Organization for Standardization. 2000. 9241--9 Ergonomic requirements for office work with visual display terminals (VDTs)-Part 9: Requirements for non-keyboard input devices. (2000).Google Scholar
- P. Ipeirotis. 2010. Demographics of mechanical turk. (March 2010). NYU Working Paper No. CEDER-10-01.Google Scholar
- Stefan Johansson, Jan Gulliksen, and Ann Lantz. 2015. User Participation When Users Have Mental and Cognitive Disabilities. In Proceedings of the 17th international ACM SIGACCESS conference on Computers and Accessibility (ASSETS '15). ACM, New York, NY, USA, 69--76. Google ScholarDigital Library
- Simeon Keates and Shari Trewin. 2005. Effect of age and Parkinson's disease on cursor positioning using a mouse. In Proceedings of the 7th international ACM SIGACCESS conference on Computers and Accessibility (ASSETS 05'). ACM, 68--75. Google ScholarDigital Library
- Szabolcs Kéri, O Kelemen, G Szekeres, N Bagoczky, R Erdelyi, A Antal, G Benedek, and Z Janka. 2000. Schizophrenics know more than they can tell: probabilistic classification learning in schizophrenia. Psychological medicine 30, 1 (2000), 149--155.Google Scholar
- Szabolcs Kéri, Csaba Szlobodnyik, György Benedek, Zoltán Janka, and Júlia Gádoros. 2002. Probabilistic classification learning in Tourette syndrome. Neuropsychologia 40, 8 (2002), 1356--1362.Google ScholarCross Ref
- Caroline J Ketcham, Rachael D Seidler, Arend W A Van Gemmert, and George E Stelmach. 2002. Age-related kinematic differences as influenced by task difficulty, target size, and movement amplitude. J Gerontol B Psychol Sci Soc Sci 57, 1 (Jan. 2002), P54--64.Google ScholarCross Ref
- Shinobu Kitayama, Sean Duffy, Tadashi Kawamura, and Jeff T Larsen. 2003. Perceiving an object and its context in different cultures: A cultural look at new look. Psychological science 14, 3 (2003), 201--206.Google ScholarCross Ref
- Barbara J Knowlton, Larry R Squire, and Mark A Gluck. 1994. Probabilistic classification learning in amnesia. Learning & Memory 1, 2 (1994), 106--120.Google ScholarCross Ref
- Steven Komarov, Katharina Reinecke, and Krzysztof Z. Gajos. 2013. Crowdsourcing performance evaluations of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 207--216. Google ScholarDigital Library
- Rachel Marsh, Gerianne M Alexander, Mark G Packard, Hongtu Zhu, Jeffrey C Wingard, Georgette Quackenbush, and Bradley S Peterson. 2004. Habit learning in Tourette syndrome: a translational neuroscience approach to a developmental psychopathology. Archives of general psychiatry 61, 12 (2004), 1259--1268.Google ScholarCross Ref
- Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods 44, 1 (2012), 1--23.Google Scholar
- Nigini Oliveira, Eunice Jun, and Katharina Reinecke. 2017. Citizen Science Opportunities in Volunteer-Based Online Experiments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '17). ACM, 6800--6812. Google ScholarDigital Library
- Timothy J Perfect and Elizabeth A Maylor. 2000. Rejecting the dull hypothesis: The relation between method and theory in cognitive aging research. Oxford University Press.Google Scholar
- Helen Petrie, Fraser Hamilton, Neil King, and Pete Pavan. 2006. Remote Usability Evaluations With Disabled People. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, New York, NY, USA, 1133--1141. Google ScholarDigital Library
- Katharina Reinecke, David R. Flatla, and Christopher Brooks. 2016. Enabling Designers to Foresee Which Colors Users Cannot See. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16). 2693--2704. Google ScholarDigital Library
- Katharina Reinecke and Krzysztof Z. Gajos. 2015. LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing - CSCW '15. 1364--1378. Google ScholarDigital Library
- Andrew Sears and Vicki L. Hanson. 2012. Representing Users in Accessibility Research. ACM Trans. Access. Comput. 4, 2, Article 7 (March 2012), 6 pages. Google ScholarDigital Library
- Nicholas A. Smith, Isaac E. Sabat, Larry R. Martinez, Kayla Weaver, and Shi Xu. 2015. A Convenient Solution: Using MTurk To Sample From Hard-To-Reach Populations. Industrial and Organizational Psychology 8, 2 (2015), 220--228.Google ScholarCross Ref
- Saul Sternberg. 1966. High-speed scanning in human memory. Science 153, 3736 (1966), 652--654.Google Scholar
- Saiganesh Swaminathan, Kotaro Hara, and Jeffrey P. Bigham. 2017. The Crowd Work Accessibility Problem. In Proceedings of the 14th Web for All Conference on The Future of Accessible Work (W4A '17). ACM, New York, NY, USA, Article 6, 4 pages. Google ScholarDigital Library
- Rachel Z Tenenbaum, Conor J Byrne, and Jason J Dahling. 2014. Interactive effects of physical disability severity and age of disability onset on RIASEC self-efficacies. Journal of Career Assessment 22, 2 (2014), 274--289.Google ScholarCross Ref
- International Telecommunication Union. 2015. ICT Facts & Figures: The World in 2015. (2015).Google Scholar
- Rajan Vaish, Snehalkumar Neil S Gaikwad, Geza Kovacs, Andreas Veit, Ranjay Krishna, Imanol Arrieta Ibarra, Camelia Simoiu, Michael Wilber, Serge Belongie, Sharad Goel, and others. 2017. Crowd Research: Open and Scalable University Laboratories. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, 829--843. Google ScholarDigital Library
- Pascal WM Van Gerven, Fred Paas, Jeroen JG Van Merriënboer, and Henk G Schmidt. 2004. Memory load and the cognitive pupillary response in aging. Psychophysiology 41, 2 (2004), 167--174.Google ScholarCross Ref
- N. Walker, D. A. Philbin, and A. D. Fisk. 1997. Age-related differences in movement control: adjusting submovement structure to optimize performance. J Gerontol B Psychol Sci Soc Sci 52, 1 (January 1997).Google Scholar
- Kathryn Zyskowski, Meredith Ringel Morris, Jeffrey P. Bigham, Mary L. Gray, and Shaun K. Kane. 2015. Accessible Crowdwork?: Understanding the Value in and Challenge of Microtask Employment for People with Disabilities. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, New York, NY, USA, 1682--1693. Google ScholarDigital Library
Index Terms
- Volunteer-Based Online Studies With Older Adults and People with Disabilities
Recommendations
Citizen Science Opportunities in Volunteer-Based Online Experiments
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsOnline experimentation with volunteers could be described as a form of citizen science in which participants take part in behavioral studies without financial compensation. However, while citizen science projects aim to improve scientific understanding, ...
Sustaining Engagement in Volunteer Activities for Older Adults
CSCW '20 Companion: Companion Publication of the 2020 Conference on Computer Supported Cooperative Work and Social ComputingSustaining engagement in volunteering opportunities is important for older adults to maintain an active lifestyle. We conducted an observation and interview study with older adults who volunteer in a water monitoring group to investigate the role of ...
Older adults' perceptions and experiences of online social support
This paper reports an investigation of older adults' needs and preferences concerning online social support. We focused our analysis on seven different aspects of online support: Self disclosure, Deep support, Light support, Community building, ...
Comments