skip to main content
10.1145/3411764.3445066acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Think-Aloud Computing: Supporting Rich and Low-Effort Knowledge Capture

Published: 07 May 2021 Publication History

Abstract

When users complete tasks on the computer, the knowledge they leverage and their intent is often lost because it is tedious or challenging to capture. This makes it harder to understand why a colleague designed a component a certain way or to remember requirements for software you wrote a year ago. We introduce think-aloud computing, a novel application of the think-aloud protocol where computer users are encouraged to speak while working to capture rich knowledge with relatively low effort. Through a formative study we find people shared information about design intent, work processes, problems encountered, to-do items, and other useful information. We developed a prototype that supports think-aloud computing by prompting users to speak and contextualizing speech with labels and application context. Our evaluation shows more subtle design decisions and process explanations were captured in think-aloud than via traditional documentation. Participants reported that think-aloud required similar effort as traditional documentation.

Supplementary Material

VTT File (3411764.3445066_videofigurecaptions.vtt)
Supplementary Materials (3411764.3445066_supplementalmaterials.zip)
MP4 File (3411764.3445066_videofigure.mp4)
Supplemental video

References

[1]
Abdulaziz Alaboudi and Thomas D. LaToza. 2019. An Exploratory Study of Live-Streamed Programming. In 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), IEEE, Memphis, TN, USA, 5–13.
[2]
John M. Carroll, Sherman R. Alpert, John Karat, Mary Van Deusen, and Mary Beth Rosson. 1994. Raison d'etre: capturing design history and rationale in multimedia narratives. In Proceedings of the CHI’94 Conference on Human Factors in Computing systems, Publ by ACM, 192–197.
[3]
Senthil Chandrasegaran, Chris Bryan, Hidekazu Shidara, Tung-Yen Chuang, and Kwan-Liu Ma. 2019. TalkTraces: Real-Time Capture and Visualization of Verbal Content in Meetings. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, 577.
[4]
Yan Chen, Walter S. Lasecki, and Tao Dong. 2021. Towards Supporting Programming Education at Scale via Live Streaming. Proc. ACM Hum.-Comput. Interact. 4, CSCW3 (January 2021), 1–19.
[5]
Pei-Yu Chi, Sally Ahn, Amanda Ren, Mira Dontcheva, Wilmot Li, and Björn Hartmann. 2012. MixT: automatic generation of step-by-step mixed media tutorials. In Proceedings of the 25th annual ACM symposium on User interface software and technology, ACM, 93–102.
[6]
Ionut Damian, Chiew Seng (Sean) Tan, Tobias Baur, Johannes Schöning, Kris Luyten, and Elisabeth André. 2015. Augmenting Social Interactions: Realtime Behavioural Feedback using Social Signal Processing Techniques. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI ’15, ACM Press, Seoul, Republic of Korea, 565–574.
[7]
Sidney D'Mello, Tanner Jackson, Scotty Craig, Brent Morgan, P. Chipman, Holly White, Natalie Person, Barry Kort, R. El Kaliouby, and Rosalind Picard. 2008. AutoTutor detects and responds to learners affective and cognitive states. In Workshop on emotional and cognitive issues at the international conference on intelligent tutoring systems, 306–308.
[8]
Anton N. Dragunov, Thomas G. Dietterich, Kevin Johnsrude, Matthew McLaughlin, Lida Li, and Jonathan L. Herlocker. 2005. TaskTracer: a desktop environment to support multi-tasking knowledge workers. In Proceedings of the 10th international conference on Intelligent user interfaces, ACM, 75–82.
[9]
K. Anders Ericsson and Herbert A. Simon. 1980. Verbal reports as data. Psychological review 87, 3 (1980), 215.
[10]
C. Ailie Fraser, Joy O. Kim, Hijung Valentina Shin, Joel Brandt, and Mira Dontcheva. 2020. Temporal Segmentation of Creative Live Streams. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, Honolulu HI USA, 1–12.
[11]
C. Ailie Fraser, Joy O. Kim, Alison Thornsberry, Scott Klemmer, and Mira Dontcheva. 2019. Sharing the Studio: How Creative Livestreaming can Inspire, Educate, and Engage. In Proceedings of the 2019 on Creativity and Cognition, ACM, San Diego CA USA, 144–155.
[12]
Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2010. Chronicle: capture, exploration, and playback of document workflow histories. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, ACM, 143–152.
[13]
William A. Hamilton, Oliver Garretson, and Andruid Kerne. 2014. Streaming on twitch: fostering participatory communities of play within live mixed media. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI ’14, ACM Press, Toronto, Ontario, Canada, 1315–1324.
[14]
William A. Hamilton, Nic Lupfer, Nicolas Botello, Tyler Tesch, Alex Stacy, Jeremy Merrill, Blake Williford, Frank R. Bentley, and Andruid Kerne. 2018. Collaborative Live Media Curation: Shared Context for Participation in Online Learning. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18, ACM Press, Montreal QC, Canada, 1–14.
[15]
Yiyang Hao, Ge Li, Lili Mou, Lu Zhang, and Zhi Jin. 2013. Mct: A tool for commenting programs by multimedia comments. In Proceedings of the 2013 International Conference on Software Engineering, IEEE Press, 1339–1342.
[16]
Morten Hertzum, Kristin D. Hansen, and Hans HK Andersen. 2009. Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload? Behaviour & Information Technology 28, 2 (2009), 165–181.
[17]
Donghan Hu and Sang Won Lee. 2020. ScreenTrack: Using a Visual History of a Computer Screen to Retrieve Documents and Web Pages. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, Honolulu HI USA, 1–13.
[18]
Hyeongcheol Kim, Shengdong Zhao, Can Liu, and Kotaro Hara. 2020. LiveSnippets: Voice-based Live Authoring of Multimedia Articles about Experiences. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services, ACM, Oldenburg Germany, 1–11.
[19]
Taemie Kim, Agnes Chang, Lindsey Holland, and Alex Sandy Pentland. 2008. Meeting mediator: enhancing group collaborationusing sociometric feedback. In Proceedings of the ACM 2008 conference on Computer supported cooperative work - CSCW ’08, ACM Press, San Diego, CA, USA, 457.
[20]
Hannu Kuusela and Paul Pallab. 2000. A comparison of concurrent and retrospective verbal protocol analysis. The American journal of psychology 113, 3 (2000), 387.
[21]
Matthew L. Lee and Anind K. Dey. 2008. Using lifelogging to support recollection for people with episodic memory impairment and their caregivers. In Proceedings of the 2nd International Workshop on Systems and Networking Support for Health Care and Assisted Living Environments, ACM, 14.
[22]
Matthew L. Lee and Anind K. Dey. 2008. Lifelogging memory appliance for people with episodic memory impairment. In Proceedings of the 10th international conference on Ubiquitous computing, ACM, 44–53.
[23]
Pascal Lessel, Alexander Vielhauer, and Antonio Krüger. 2017. Expanding Video Game Live-Streams with Enhanced Communication Channels: A Case Study. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, ACM, Denver Colorado USA, 1571–1576.
[24]
Clayton Lewis. 1982. Using the'thinking-aloud'method in cognitive interface design. Research Report RC9265, IBM TJ Watson Research Center (1982).
[25]
Ian Li, Jon Froehlich, Jakob E. Larsen, Catherine Grevet, and Ernesto Ramirez. 2013. Personal informatics in the wild: hacking habits for health & happiness. In CHI’13 Extended Abstracts on Human Factors in Computing Systems, ACM, 3179–3182.
[26]
Zhicong Lu, Seongkook Heo, and Daniel J. Wigdor. 2018. StreamWiki: Enabling Viewers of Knowledge Sharing Live Streams to Collaboratively Generate Archival Documentation for Effective In-Stream and Post Hoc Learning. Proc. ACM Hum.-Comput. Interact. 2, CSCW (November 2018), 1–26.
[27]
Gloria Mark, Daniela Gudith, and Ulrich Klocke. 2008. The cost of interrupted work: more speed and stress. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, ACM, 107–110.
[28]
Richard E. Mayer and Cheryl I. Johnson. 2008. Revising the redundancy principle in multimedia learning. Journal of Educational Psychology 100, 2 (2008), 380–386.
[29]
Moira McGregor and John C. Tang. 2017. More to meetings: challenges in using speech-based technology to support meetings. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, ACM, 2208–2220.
[30]
Alok Mysore and Philip J. Guo. 2017. Torta: Generating mixed-media gui and command-line app tutorials using operating-system-wide activity tracing. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, ACM, 703–714.
[31]
Soya Park, Amy X. Zhang, and David R. Karger. 2018. Post-literate Programming: Linking Discussion and Code in Software Development Teams. In The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings, ACM, 51–53.
[32]
Rupa Patel, Andrea Hartzler, Mary Czerwinski, Wanda Pratt, Anthony Back, and Asta Roseway. 2013. Visual Feedback on Nonverbal Communication: A Design Exploration with Healthcare Professionals. In Proceedings of the ICTs for improving Patients Rehabilitation Research Techniques, IEEE, Venice, Italy.
[33]
Zachary Pousman and John Stasko. 2006. A taxonomy of ambient information systems: four patterns of design. In Proceedings of the working conference on Advanced visual interfaces, ACM, 67–74.
[34]
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf (2018).
[35]
Søren Rasmussen, Jeanette Falk Olesen, and Kim Halskov. 2019. Co-notate: Exploring Real-time Annotations to Capture Situational Design Knowledge. In Proceedings of the 2019 on Designing Interactive Systems Conference, ACM, 161–172.
[36]
Raquel Robinson, Jessica Hammer, and Katherine Isbister. 2019. All the World (Wide Web)’s a Stage: A Workshop on Live Streaming. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland Uk, 1–8.
[37]
Daniela K. Rosner and Kimiko Ryokai. 2008. Spyn: augmenting knitting to support storytelling and reflection. In Proceedings of the 10th international conference on Ubiquitous computing, ACM, 340–349.
[38]
Donald A. Schon. 1984. The reflective practitioner: How professionals think in action. Basic books.
[39]
Abigail Sellen and Steve Whittaker. 2010. Beyond total capture: a constructive critique of lifelogging. Communications of the ACM (2010).
[40]
Mohit Shah, Brian Mears, Chaitali Chakrabarti, and Andreas Spanias. 2012. Lifelogging: Archival and retrieval of continuously recorded audio using wearable devices. In 2012 IEEE International Conference on Emerging Signal Processing Applications, IEEE, 99–102.
[41]
Alice Thudt, Dominikus Baur, Samuel Huron, and Sheelagh Carpendale. 2015. Visual mementos: Reflecting memories with personal data. IEEE transactions on visualization and computer graphics 22, 1 (2015), 369–378.
[42]
Tiffany Tseng. 2016. Build in progress: Building process-oriented documentation. Makeology: Makerspaces as learning environments. New York: Routledge (2016).
[43]
Maaike Van Den Haak, Menno De Jong, and Peter Jan Schellens. 2003. Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behaviour & information technology 22, 5 (2003), 339–351.
[44]
Maaike J. Van den Haak and Menno DT De Jong. 2003. Exploring two methods of usability testing: concurrent versus retrospective think-aloud protocols. In IEEE International Professional Communication Conference, 2003. IPCC 2003. Proceedings., IEEE, 3 pp.
[45]
April Yi Wang, Zihan Wu, Christopher Brooks, and Steve Oney. 2020. Callisto: Capturing the “Why” by Connecting Conversations with Computational Narratives. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, Honolulu HI USA, 1–13.
[46]
Saelyne Yang, Changyoon Lee, Hijung Valentina Shin, and Juho Kim. 2020. Snapstream: Snapshot-based Interaction in Live Streaming for Visual Art. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, Honolulu HI USA, 1–12.
[47]
Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018. Recent trends in deep learning based natural language processing. ieee Computational intelligenCe magazine 13, 3 (2018), 55–75.
[48]
Thinking Aloud: The #1 Usability Tool. Nielsen Norman Group. Retrieved September 18, 2019 from https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/

Cited By

View all
  • (2024)Estimation, Impact and Visualization of Schema Evolution in Graph DatabasesProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3652620.3688196(123-129)Online publication date: 22-Sep-2024
  • (2024)Demonstrating PANDALens: Enhancing Daily Activity Documentation with AI-assisted In-Context Writing on OHMDExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3648644(1-7)Online publication date: 11-May-2024
  • (2024)PANDALens: Towards AI-Assisted In-Context Writing on OHMD During TravelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642320(1-24)Online publication date: 11-May-2024
  • Show More Cited By

Index Terms

  1. Think-Aloud Computing: Supporting Rich and Low-Effort Knowledge Capture
    Index terms have been assigned to the content through auto-classification.

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
    May 2021
    10862 pages
    ISBN:9781450380966
    DOI:10.1145/3411764
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 May 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Think-aloud
    2. documentation
    3. knowledge capture
    4. process

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    CHI '21
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)150
    • Downloads (Last 6 weeks)17
    Reflects downloads up to 08 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Estimation, Impact and Visualization of Schema Evolution in Graph DatabasesProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3652620.3688196(123-129)Online publication date: 22-Sep-2024
    • (2024)Demonstrating PANDALens: Enhancing Daily Activity Documentation with AI-assisted In-Context Writing on OHMDExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3648644(1-7)Online publication date: 11-May-2024
    • (2024)PANDALens: Towards AI-Assisted In-Context Writing on OHMD During TravelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642320(1-24)Online publication date: 11-May-2024
    • (2024)GazePuffer: Hands-Free Input Method Leveraging Puff Cheeks for VR2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00055(331-341)Online publication date: 16-Mar-2024
    • (2024)Exploring the impacts of semi-automated storytelling on programmers’ comprehension of software histories2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC60511.2024.00025(148-162)Online publication date: 2-Sep-2024
    • (2023)Retrospector: Rapid collaborative reflection to improve collaborative practicesProceedings of the ACM on Human-Computer Interaction10.1145/36100847:CSCW2(1-20)Online publication date: 4-Oct-2023
    • (2023)Making From Home: Reflections on Crafting Tangible Interfaces for Stay-at-home LivingProceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3569009.3572744(1-16)Online publication date: 26-Feb-2023

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media