Abstract
Thinking aloud (TA) is a widely employed usability testing method and allows identifying usability problems based on testers’ verbalization. Conducting TA studies is laborious and expensive due to the recruiting of participants, the required workforce and facilities, and the resulting monetary costs. Also, it is usually carried out in very controlled settings like laboratories, thereby limiting participants’ diversity, test realism, and the quality of the results. Crowdsourcing, in contrast, provides access to a wide and diverse workforce at moderate costs. Hence, it might help to overcome the limitations of traditional TA. Although usability testing has been successfully crowdsourced with other methods before, there is still no evidence on the feasibility of conducting TA involving unsupervised workers as testers. Thus, we conducted a between-subjects user study using the example of an online web page, both via a traditional remote TA setup and a crowdsourcing setup with a self-developed platform. The results do not show a significant difference between both setups regarding participants’ performance, and the quality and quantity of the identified usability problems. Therefore, we conclude that crowdsourcing is a feasible and cost-effective solution to conduct usability testing with the TA method, at least for selected use cases.
Keywords
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
https://www.webex.com/ Accessed June 2021.
- 2.
https://www.edx.org/ Accessed June 2021.
- 3.
- 4.
https://www.microworkers.com/ Accessed June 2021.
References
Alhadreti, O., Mayhew, P.: To intervene or not to intervene: an investigation of three think-aloud protocols in usability testing. J. Usability Stud. 12(3), 111–132 (2017)
Behrend, T.S., Sharek, D.J., Meade, A.W., Wiebe, E.N.: The viability of crowdsourcing for survey research. Behav. Res. Methods 43(3), 800–813 (2011). https://doi.org/10.3758/s13428-011-0081-0
Bruggemann, J., Lander, G.C., Su, A.I.: Exploring applications of crowdsourcing to cryo-EM. J. Struct. Biol. 203(1), 37–45 (2018). https://doi.org/10.1016/j.jsb.2018.02.006
Denning, S., Hoiem, D., Simpson, M., Sullivan, K.: The value of thinking-aloud protocols in industry: a case study at microsoft corporation. Proc. Hum. Factors Soc. Ann. Meet. 34(17), 1285–1289 (1990). https://doi.org/10.1177/154193129003401723
Egger-Lampl, S., et al.: Crowdsourcing quality of experience experiments. In: Archambault, D., Purchase, H., Hoßfeld, T. (eds.) Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments, LNCS, vol. 10264, pp. 154–190. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66435-4_7
Ericsson, K.A., Simon, H.A.: Protocol Analysis: Verbal Reports as Data. The MIT Press, Cambridge (1984)
Estellés-Arolas, E.L., Guevara, F.G., Towards an integrated crowdsourcing definition: Towards an integrated crowdsourcing definition. J. Inf. Sci. 38, 189–200 (2012). https://doi.org/10.1177/0165551512437638
Fan, M., Shi, S., Truong, K.N.: Practices and challenges of using think-aloud protocols in industry: an international survey. J. Usability Stud. 15(2) (2020)
Haak, M.J., Jong, M.D., Schellens, P.J.: Evaluating municipal websites: a methodological comparison of three think-aloud variants. Gov. Inf. Qual. 26(1) (2009). https://doi.org/10.1016/j.giq.2007.11.003
Hirth, M., Hoßfeld, T., Tran-Gia, P.: Anatomy of a crowdsourcing platform - using the example of Microworkers.com. In: International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (2011). https://doi.org/10.1109/IMIS.2011.89
Hossfeld, T., et al.: Best practices for QOE crowdtesting: QOE assessment with crowdsourcing. IEEE Trans. Multimedia 16(2), 541–558 (2014). https://doi.org/10.1109/TMM.2013.2291663
Kittur, A., Chi, E.H., Suh, B.: Crowdsourcing user studies with mechanical Turk. In: Conference on Human Factors in Computing Systems (2008). https://doi.org/10.1145/1357054.1357127
Krug, S.: Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems. New Riders (2009)
Krug, S.: Don’t Make Me Think. Revisited - A Common Sense Approach to Web Usability, New Riders (2014)
Lewis, C.: Using the “Thinking Aloud’’ Method in Cognitive Interface Design. IBM TJ Watson Research Center Yorktown Heights, NY (1982)
Liu, D., Bias, R.G., Lease, M., Kuipers, R.: Crowdsourcing for usability testing. In: Proceedings of the American Society for Information Science and Technology 49(1) (2012). https://doi.org/10.1002/meet.14504901100
Maier-Hein, L., et al.: Can masses of non-experts train highly accurate image classifiers? a crowdsourcing approach to instrument segmentation in laparoscopic images. Med. Image Comput. Comput.-Assist. Interv. - MICCAI 2014(17), 438–445 (2014). https://doi.org/10.1007/978-3-319-10470-6_55
Meier, F.: Crowdsourcing als Rekrutierungsstrategie im Asynchronen Remote-Usability-Test. Information-Wissenschaft und Praxis, 63(5) (2012). https://doi.org/10.1515/iwp-2012-0063
Nebeling, M., Speicher, M., Norrie, M.C.: CrowdStudy: general toolkit for Crowdsourced evaluation of web interfaces. In: SIGCHI Symposium on Engineering Interactive Computing Systems (2013). https://doi.org/10.1145/2494603.2480303
Nielsen, J.: Usability Engineering. Elsevier Science (1994)
Nielsen, J.: Estimating the number of subjects needed for a thinking aloud test. Int. J. Hum. - Comput. Stud. 41(3) (1994)
Nielsen, L., Chavan, S.: Differences in task descriptions in the think aloud test. In: Aykin, N. (ed.) Usability and Internationalization. HCI and Culture, UI-HCII 2007. LNCS, vol. 4559, pp. 174–180. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73287-7_22
Rosenthal, R.: Meta-Analytic Procedures for Social Research. SAGE Publications, Inc. (1991)
Rubin, J., Chisnell, D., Spool, J.: Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley, Hoboken (2011)
Schneider, H., Frison, K., Wagner, J., Butz, A.: CrowdUX: a case for using widespread and lightweight tools in the quest for UX. In: Conference on Designing Interactive Systems (2016). https://doi.org/10.1145/2901790.2901814
Skov, M.B., Stage, J.: Supporting problem identification in usability evaluations. In: Australia Conference on Computer-Human Interaction (2005)
Sova, D.H., Nielsen, J.: 234 Tips and Tricks for Recruiting Users as Participants in Usability Studies. Nielsen N Group (2003)
Thompson, K.E., Rozanski, E.P., Haake, A.R.: Here, there, anywhere: remote usability testing that works. In: Conference on Information Technology Education (2004). https://doi.org/10.1145/1029533.1029567
Vermeeren, A.P.O.S., den Bouwmeester, K., Aasman, J., Ridder, H., de Ridder, H.: DEVAN: a tool for detailed video analysis of user test aata. Behav. IT 21(6) (2002)
Yuhui, W., Tian, L., Xinxiong, L.: Reliability of perceived usability assessment via crowdsourcing platform: retrospective analysis and novel feedback quality inspection method. Int. J. Hum.-Comput. Int. 36(11) (2020). https://doi.org/10.1080/10447318.2019.1709339
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Gamboa, E., Galda, R., Mayas, C., Hirth, M. (2021). The Crowd Thinks Aloud: Crowdsourcing Usability Testing with the Thinking Aloud Method. In: Stephanidis, C., et al. HCI International 2021 - Late Breaking Papers: Design and User Experience. HCII 2021. Lecture Notes in Computer Science(), vol 13094. Springer, Cham. https://doi.org/10.1007/978-3-030-90238-4_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-90238-4_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-90237-7
Online ISBN: 978-3-030-90238-4
eBook Packages: Computer ScienceComputer Science (R0)