skip to main content
research-article

Concurrent Think-Aloud Verbalizations and Usability Problems

Published: 19 July 2019 Publication History

Abstract

The concurrent think-aloud protocol—in which participants verbalize their thoughts when performing tasks—is a widely employed approach in usability testing. Despite its value, analyzing think-aloud sessions can be onerous because it often entails assessing all of a user's verbalizations. This has motivated previous research on developing categories to segment verbalizations into manageable units of analysis. However, the way in which a category might relate to usability problems is currently unclear. In this research, we sought to address this gap in our understanding. We also studied how speech features might relate to usability problems. Through two studies, this research demonstrates that certain patterns of verbalizations are more telling of usability problems than others and that these patterns are robust to different types of test products (i.e., physical devices and digital systems), access to different types of information (i.e., video and audio modality), and the presence or absence of a visualization of verbalizations. The implication is that the verbalization and speech patterns can potentially reduce the time and effort required for analysis by enabling evaluators to focus more on the important aspects of a user's verbalizations. The patterns could also potentially be used to inform the design of systems to automatically detect when in the recorded think-aloud sessions users experience problems.

References

[1]
Obead Alhadreti and Pam Mayhew. 2018. Rethinking thinking aloud. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI’18). 1--12.
[2]
Obead Alhadreti, Pam Mayhew, and Senior Lecturer. 2017. To intervene or not to intervene: An investigation of three think-aloud protocols in usability testing. J. Usability Stud. 12, 3 (2017), 111--132. Retrieved from http://www.upassoc.org.
[3]
Pamela M. Auble, Jeffery J. Franks, Salvatore A. Soraci, Salvatore A. Soraci, and Salvatore A. Soraci. 1979. Effort toward comprehension: Elaboration or “aha”? Mem. Cognit. 7, 6 (1979), 426--434.
[4]
Victoria A. Bowers and Harry L. Snyder. 1990. Concurrent versus retrospective verbal protocol for comparing window usability. Proc. Hum. Factors Soc. 34, 17 (1990), 1270--1274.
[5]
Elizabeth Charters. 2003. The use of think-aloud methods in qualitative research an introduction to think-aloud methods. Brock Educ. J. 12, 2 (2003), 68--82.
[6]
Michelene T. H. Chi, Nicholas De Leeuw, Mei Hung Chiu, and Christian Lavancher. 1994. Eliciting self-explanations improves understanding. Cogn. Sci. 18, 3 (1994), 439--477.
[7]
Jason M. Chin and Jonathan W. Schooler. 2008. Why do words hurt? Content, process, and criterion shift accounts of verbal overshadowing. Eur. J. Cogn. Psychol. 20, 3 (2008), 396--413.
[8]
H. H. Clark and J. E. Fox Tree. 2002. Using “uh” and “um” in spontaneous speaking. Cognition 84, 1 (2002), 73--111.
[9]
Lynne Cooke. 2010. Assessing concurrent think-aloud protocol as a usability test method: A technical communication approach. IEEE Trans. Prof. Commun. 53, 3 (2010), 202--215.
[10]
Elling Sanne, Lentz Leo, and Menno De Jong. 2012. Combining concurrent think-aloud protocols and eye-tracking observations : An analysis of verbalizations. IEEE Trans. Prof. Commun. 55, 3 (2012), 206--220.
[11]
K. Anders Ericsson and Herbert A. Simon. 1984. Protocol Analisys: Verbal Reports as Data. MIT Press.
[12]
Asbjørn Følstad. 2007. Work-domain experts as evaluators: Usability inspection of domain-specific work-support systems. Int. J. Hum. Comput. Interact. 22, 3 (2007), 217--245.
[13]
Mark C. Fox, K. Anders Ericsson, and Ryan Best. 2011. Do procedures for verbal reporting of thinking have to be reactive? Psychol. Bull. 137, 2 (2011), 316.
[14]
Amy M. Gill and Blair Nonnecke. 2012. Think aloud. In Proceedings of the 30th ACM International Conference on Design of Communication (SIGDOC’12). 31--36.
[15]
F. Goldman-Eisler. 1986. Cycle Linguistics: Experiments in Spontaneous Speech. Double Day, New York, NJ.
[16]
Morten Hertzum, Pia Borlund, and Kristina B. Kristoffersen. 2015. What do thinking-aloud participants say? A comparison of moderated and unmoderated usability sessions. Int. J. Hum. Comput. Interact. 31, 9 (2015), 557--570.
[17]
Morten Hertzum, Kristin D. Hansen, and Hans H. K. Andersen. 2009. Scrutinising usability evaluation: Does thinking aloud affect behaviour and mental workload? Behav. Inf. Technol. 28, 2 (2009), 165--181.
[18]
Morten Hertzum and Niels Ebbe Jacobsen. 2009. International journal of human-computer interaction the evaluator effect: A chilling fact about usability evaluation methods the evaluator effect: A chilling fact about usability evaluation methods. Int. J. Hum. Comput. Interact. 13, 4 (2009), 421--443.
[19]
Morten Hertzum, Rolf Molich, and Niels Ebbe Jacobsen. 2014. What you get is what you see: Revisiting the evaluator effect in usability tests. Behav. Inf. Technol. 33, 2 (2014), 143--161.
[20]
Masahiro Hori, Yasunori Kihara, and Takashi Kato. 2011. Investigation of indirect oral operation method for think aloud usability testing. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer, 38--46.
[21]
Clayton J. Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the 8th International AAAI Conference on Weblogs and Social Media. 216--225.
[22]
M. D. T. de Jong, P. J. Schellens, and M. J. Van den Haak. 2004. Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: A methodological comparison. Interact. Comput. 16, 6 (2004), 1153--1170. Retrieved from https://academic.oup.com/iwc/article-abstract/16/6/1153/769631.
[23]
Claire-Marie Karat, Robert Campbell, and Tarra Fiegel. 1992. Comparison of empirical testing and walkthrough methods in user interface evaluation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’92). 397--404.
[24]
Emiel Krahmer and Nicole Ummelen. 2004. Thinking about thinking aloud: A comparison of two verbal protocols for usability testing. IEEE Trans. Prof. Commun. 47, 2 (2004), 105--117.
[25]
Petri Laukka, Clas Linnman, Fredrik Åhs, Anna Pissiota, Örjan Frans, Vanda Faria, Åsa Michelgård, Lieuwe Appel, Mats Fredrikson, and Tomas Furmark. 2008. In a nervous voice: Acoustic analysis and perception of anxiety in social phobics’ speech. J. Nonverbal Behav. 32, 4 (2008), 195--214.
[26]
Darryn Lavery, Gilbert Cockton, and Malcolm P. Atkinson. 1997. Comparison of evaluation methods using structured usability problem reports. Behav. Inf. Technol. 16, 4--5 (1997), 246--266.
[27]
S. Mcdonald, T. Zhao, and H. M. Edwards. 2016. Look who's talking: Evaluating the utility of interventions during an interactive think-aloud. Interact. Comput. 28, 3 (2016), 387--403.
[28]
Sharon McDonald and Helen Petrie. 2013. The effect of global instructions on think-aloud testing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’13). 2941--2944.
[29]
Sharon McDonald, Tingting Zhao, and Helen M. Edwards. 2013. Dual verbal elicitation: The complementary use of concurrent and retrospective reporting within a usability test. Int. J. Hum. Comput. Interact. 29, 10 (2013), 647--660.
[30]
Jakob Nielsen. 1993. Usability Engineering. Elsevier.
[31]
Jakob Nielson. 2014. Demonstrate thinking aloud by showing users a video. In Evidence-Based User Experience Research, Training, and Consulting. Nielson Norman Group. Retrieved October 2, 2018 from https://www.nngroup.com/articles/thinking-aloud-demo-video/.
[32]
Mie Nørgaard and Kasper Hornbæk. 2006. What do usability evaluators do in practice? In Proceedings of the 6th ACM conference on Designing Interactive Systems (DIS’06). 209--218.
[33]
K. R. Ohnemus and D. W Biers. 1993. Retrospective versus concurrent thinking-out-loud in usability testing. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 37, 17 (1993), 1127--1131.
[34]
Erica Olmsted-Hawala and Jennifer Romano Bergstrom. 2012. Think-aloud protocols: Does age make a difference? In Proceedings of the STC Technical Communication Summit.
[35]
Erica L. Olmsted-Hawala, Elizabeth D. Murphy, Sam Hawala, and Kathleen T. Ashenfelter. 2010. Think-aloud protocols analyzing three different think-aloud protocols with counts of verbalized frustrations in a usability study of an information-rich web site think-aloud protocols alterna.pdf. In Proceedings of the 2010 IEEE International Professional Communication Conference (IPCC’10). 60--66.
[36]
Alex Pentland. 2009. Honest Signals: How They Shape Our World. MIT Press.
[37]
Lloyd R. Peterson. 1969. Concurrent verbal activity. Psychol. Rev. 76, 4 (1969), 376--386.
[38]
Matthew F. Pike, Horia A. Maior, Martin Porcheron, Sarah C. Sharples, and Max L. Wilson. 2014. Measuring the effect of think aloud protocols on workload using fNIRS. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI’14). 3807--3816.
[39]
J. Ramey and T. Boren. 2000. Thinking aloud: Reconciling theory and practice. IEEE Trans. Prof. Commun. 43, 3 (2000), 261.
[40]
Detlef Rhenius and Gerhard Deffner. 1990. Evaluation of concurrent thinking aloud using eye-tracking data. Proc. Hum. Factors Soc. Annu. Meet. 34, 17 (1990), 1265--1269.
[41]
Sharon McDonald, Helen M. Edwards, and Zhao Tingting. 2012. Exploring think-alouds in usability testing: An international survey. IEEE Trans. Prof. Commun. 55, 1 (2012), 2--19.
[42]
Qingxin Shi. 2008. A field study of the relationship and communication between Chinese evaluators and users in thinking aloud usability tests. In Proceedings of the 5th Nordic Conference on Human-Computer Interaction Building Bridges (NordiCHI’08). 344--352.
[43]
Glen Shires and Hans Wennborg. 2012. Web Speech API Specification. Speech API Community Group, W3C.
[44]
Andreas Sonderegger, Sven Schmutz, and Juergen Sauer. 2016. The influence of age in usability testing. Appl. Ergon. 52 (2016), 291--300.
[45]
Siegfried L. Sporer and Barbara Schwandt. 2006. Paraverbal indicators of deception: A meta-analytic synthesis. J. Appl. Cogn. Psychol. 20, 4 (2006), 421--446.
[46]
Howard Tamler. 1998. How (much) to intervene in a usability testing session. Common Gr. 8, 3 (1998), 11--15.
[47]
S. Tirkkonen-Condit. 2006. Think-aloud protocols. In Encyclopedia of Language and Linguistics. Elsevier, 678--686.
[48]
R. B. Wright and S. A. Converse. 1992. Method bias and concurrent verbal protocol in software usability testing. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 36, 16 (1992), 1220--1224.
[49]
Zhao Tingting and McDonald Sharon. 2010. Keep talking. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction Extending Boundaries (NordiCHI’10). 581--590.
[50]
Zhao Tingting, McDonald Sharon, and Helen M. Edwards. 2014. The impact of two different think-aloud instructions in a usability test: A case of just following orders? Behav. Inf. Technol. 33, 2 (2014), 162--182.
[51]
Praat: Doing Phonetics by Computer. Retrieved from http://www.fon.hum.uva.nl/praat/.

Cited By

View all
  • (2024)Metaphors in Educational VideosEducation Sciences10.3390/educsci1402017714:2(177)Online publication date: 8-Feb-2024
  • (2024)Collaborative System Usability in Spaceflight Analog Environments through Remote ObservationsApplied Sciences10.3390/app1405200514:5(2005)Online publication date: 28-Feb-2024
  • (2024)Usability Testing of Mobile Applications: A Methodological FrameworkApplied Sciences10.3390/app1405179214:5(1792)Online publication date: 22-Feb-2024
  • Show More Cited By

Index Terms

  1. Concurrent Think-Aloud Verbalizations and Usability Problems

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Computer-Human Interaction
    ACM Transactions on Computer-Human Interaction  Volume 26, Issue 5
    October 2019
    249 pages
    ISSN:1073-0516
    EISSN:1557-7325
    DOI:10.1145/3349608
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 19 July 2019
    Accepted: 01 April 2019
    Revised: 01 March 2019
    Received: 01 May 2018
    Published in TOCHI Volume 26, Issue 5

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Concurrent think-aloud
    2. loudness
    3. pitch
    4. sentiment
    5. silence
    6. speech features
    7. speech rate
    8. usability problems
    9. usability testing
    10. verbal fillers
    11. verbalization
    12. verbalization categories

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)264
    • Downloads (Last 6 weeks)17
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Metaphors in Educational VideosEducation Sciences10.3390/educsci1402017714:2(177)Online publication date: 8-Feb-2024
    • (2024)Collaborative System Usability in Spaceflight Analog Environments through Remote ObservationsApplied Sciences10.3390/app1405200514:5(2005)Online publication date: 28-Feb-2024
    • (2024)Usability Testing of Mobile Applications: A Methodological FrameworkApplied Sciences10.3390/app1405179214:5(1792)Online publication date: 22-Feb-2024
    • (2024)Usability Evaluation Methods Used in Electronic Discharge Summaries: Literature ReviewJournal of Medical Internet Research10.2196/5524726(e55247)Online publication date: 12-Sep-2024
    • (2024)Evaluating the usability of Iran’s national comprehensive health information system: a think-aloud study to uncover usability problems in the recording of childcare dataBMC Medical Informatics and Decision Making10.1186/s12911-024-02746-224:1Online publication date: 16-Nov-2024
    • (2024)Internet-delivered emotional self-management program for the general population during the COVID-19 pandemic: Usability testingDIGITAL HEALTH10.1177/2055207624125841910Online publication date: 11-Sep-2024
    • (2024)Concurrent or Retrospective Thinking Aloud in Usability Tests: A Meta-Analytic ReviewACM Transactions on Computer-Human Interaction10.1145/366532731:3(1-29)Online publication date: 17-May-2024
    • (2024)Design Principles for a Study Planning Assistant in Higher EducationProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638327(243-253)Online publication date: 10-Mar-2024
    • (2024)Enhancing UX Evaluation Through Collaboration with Conversational AI Assistants: Effects of Proactive Dialogue and TimingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642168(1-16)Online publication date: 11-May-2024
    • (2024)Exploring accessibility, user experience and engagement of digital media among older patients with depression: a pilot and observational screening study protocol of the DiGA4Aged studyBMJ Open10.1136/bmjopen-2024-08677914:11(e086779)Online publication date: 7-Nov-2024
    • Show More Cited By

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media