Skip to main content
SpringerLink
Log in
Book cover

Ways of Knowing in HCI pp 373–393Cite as

Looking Back: Retrospective Study Methods for HCI

  • Chapter
  • First Online:

Abstract

Many people infer reasons for behavior without actually knowing. Two methods that can be used to actually gain insight into thought processes are the think-aloud protocol and retrospective cued recall. These two methods gather natural user behavior fairly unobtrusively over a period of time while allowing for some insight into what people are thinking as they are doing. These methods can be used to understand the reasons for tasks that require focused attention. In this chapter we illustrate these ideas in the domain of “searching on the Internet,” but these methods are broadly applicable.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   99.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  • Akers, D., Simpson, M., Jeffries, R., & Winograd, T. (2009, April). Undo and erase events as indicators of usability problems. In Proceedings of the 27th International Conference on Human Factors in Computing Systems (pp. 659–668). New York, NY: ACM.

    Google Scholar 

  • Al-Qaimari, G., & McRostie, D. (1999). KALDI: A computer-aided usability engineering tool for supporting testing and analysis of human computer interaction. In J. Vanderdonckt & A. Puerta (Eds.), Proceedings of the 3rd International Conference on Computer-Aided Design of User Interfaces (CADUI‘99), Dordrecht, October, 1999. Louvain-la-Neuve: Kluwer.

    Google Scholar 

  • Baauw, E., & Markopoulous, P. (2004, June). A comparison of think-aloud and post-task interview for usability testing with children. In Proceedings of the 2004 Conference on Interaction Design and Children: Building a Community (pp. 115–116). New York, NY: ACM.

    Google Scholar 

  • Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287–311.

    Article  Google Scholar 

  • Blackwell, A., Jones, R., Milic-Frayling, N., & Rodden, K. (2005, April). Combining logging with interviews to investigate web browser usage in the workplace. Position paper for Workshop Usage Analysis: Combining Logging and Qualitative Methods, ACM Conference on Human Factors in Computing Systems (ACM CHI 2005).

    Google Scholar 

  • Brandt, J., Weiss, H., & Klemmer, S. (2007). txt 4l8r: Lowering the burden for diary studies under mobile conditions. In CHI ‘07, April 23–May 3, 2007, San Jose, California.

    Google Scholar 

  • Brewer, W. F. (1986). What is autobiographical memory? In D. Rubin (Ed.), Autobiographical memory (pp. 25–49). Cambridge, UK: Cambridge University Press.

    Chapter  Google Scholar 

  • Brewer, W. F. (1988). Qualitative analysis of the recalls of randomly sampled autobiographical events. In M. M. Gruneberg, P. E. Morris, & R. N. Sykes (Eds.), Practical aspects of memory: Current research and issues (Vol. 1, pp. 263–268). Chichester, UK: Wiley.

    Google Scholar 

  • Brush, A. J., Ames, M., & Davis, J. (2004, April). A comparison of synchronous remote and local usability studies for an expert interface. In CHI‘04 Extended Abstracts on Human Factors in Computing Systems (pp. 1179–1182). New York, NY: ACM.

    Google Scholar 

  • Capra, M. (2002). Contemporaneous versus retrospective user-reported critical incidents in usability evaluation. In Proc. Human Factors 2002, HFES, 1973–1977.

    Google Scholar 

  • Chi, E. H., Pirolli, P., & Pitkow, J. (2000). The scent of a site: A system for analyzing and predicting information scent, usage, and usability of a web site. In Proc. CHI 2000 (pp. 161–167). New York, NY: ACM Press.

    Google Scholar 

  • Collier, J. (1967). Visual anthropology: Photography as a research method. New York, NY: Holt, Rinehart and Winston.

    Google Scholar 

  • Czerwinski, M., Horvitz, E., & Wilhite, S. (2004). A diary study of task switching and interruptions. In CHI (pp. 175–182). New York, NY: ACM Press.

    Google Scholar 

  • Dickson, J., McLennan, J., & Omodei, M. M. (2000). Effects of concurrent verbalization on a time-critical, dynamic decision-making task. The Journal of General Psychology, 127(2), 217–228.

    Article  Google Scholar 

  • Ericsson, K. A. (2006). Protocol analysis and expert thought: Concurrent verbalizations of thinking during experts’ performance on representative tasks. In K. A. Ericsson, N. Charness, P. Feltovich, & R. R. Hoffman (Eds.), Cambridge handbook of expertise and expert performance (pp. 223–242). Cambridge, UK: Cambridge University Press.

    Chapter  Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1985). Protocol analysis. Cambridge, MA: MIT Press.

    Google Scholar 

  • Guan, Z., Lee, S., Cuddihy, E., & Ramey, J. (2006). The validity of the stimulated retrospective think-aloud method as measured by eye tracking. In Proc. CHI 2006 (pp. 1253–1262). New York, NY: ACM Press.

    Google Scholar 

  • Hektner, J. M., Schmidt, J. A., & Csikszentmihalyi, M. (2007). Experience sampling method: Measuring the quality of everyday life. Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Holbrook, A. L., Green, M. C., & Krosnick, J. A. (2003). Telephone versus face-to-face interviewing of national probability samples with long questionnaires: Comparisons of respondent satisficing and social desirability response bias. Public Opinion Quarterly, 67(1), 79–125.

    Article  Google Scholar 

  • Höysniemi, J., Hämäläinen, P., & Turkki, L. (2003). Using peer tutoring in evaluating the usability of a physically interactive computer game with children. Interacting with Computers, 15(2), 203–225.

    Article  Google Scholar 

  • Hyrskykari, A., Ovaska, S., Räihä, K., Majaranta, P., & Lehtinen, M. (2008). Gaze path stimulation in retrospective think-aloud. Journal of Eye Movement Research, 2(4), 1–18.

    Google Scholar 

  • Intille, S. S., Kukla, C., & Ma, X. (2002). Eliciting user preferences using image-based experience sampling and reflection. In Proc. CHI ‘02 Extended Abstracts on Human Factors in Computing Systems (pp. 738–739). New York, NY: ACM Press.

    Google Scholar 

  • Ivory, M. Y., & Hearst, M. A. (2001). The state of the art in automated usability evaluation of user interfaces. ACM Computing Surveys, 33(4), 470–516.

    Article  Google Scholar 

  • Jain, J., & Boyce, S. (2012). Case study: Longitudinal comparative analysis for analyzing user behavior. In Proceedings of the 2012 ACM Annual Conference Extended Abstracts on Human Factors in Computing Systems Extended Abstracts. New York, NY: ACM.

    Google Scholar 

  • Jones, R., Milic-Frayling, N., Rodden, K., & Blackwell, A. (2007). Contextual method for the redesign of existing software products. International Journal of Human-Computer Interaction, 22(1–2), 81–101.

    Article  Google Scholar 

  • Kahneman, D., Krueger, A. B., Schkade, D. A., Schwarz, N., & Stone, A. A. (2004). A survey method for characterizing daily life experience: The day reconstruction method. Science, 306(5702), 1776.

    Article  Google Scholar 

  • Karapanos, E., Martens, J.-B., & Hassenzahl, M. (2009). Reconstructing experiences through sketching. Arxiv preprint, arXiv:0912.5343.

    Google Scholar 

  • Karapanos, E., Martens, J., & Hassenzahl, M. (2010). On the retrospective assessment of users’ experiences over time: Memory or actuality?. In Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing systems. New York, NY: ACM.

    Google Scholar 

  • Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. B. (2009). User experience over time: An initial framework. In Proceedings of the 27th International Conference on Human Factors in Computing Systems (pp. 729–738). New York, NY: ACM.

    Google Scholar 

  • Kellar, M., Watters, C., & Shepherd, M. (2006). A goal-based classification of web information tasks. In Proceedings of the Annual Meeting of the American Society for Information Science and Technology, Austin, TX (ASIS&T).

    Google Scholar 

  • Kuniavsky, M. (2003). Observing the user experience: A practioner’s guide to user research. New York, NY: Morgan Kaufman.

    Google Scholar 

  • Kuusela, H., & Paul, P. (2000). A comparison of concurrent and retrospective verbal protocol analysis. American Journal of Psychology, 113(3), 387–404.

    Article  Google Scholar 

  • Lamming, M., Brown, P., Carter, K., Eldridge, M., Flynn, M., Louie, G., et al. (1994). The design of a human memory prosthesis. Computer Journal, 37(3), 153–163.

    Article  Google Scholar 

  • Larson, R., & Csikszentmihalyi, M. (1983). The experience sampling method. In H. T. Reis (Ed.), Naturalistic approaches to studying social interaction: New directions for methodology of social and behavioral science. San Francisco, CA: Jossey-Bass.

    Google Scholar 

  • Loftus, E. F. (1996). Memory distortion and false memory creation. Journal of the American Academy of Psychiatry and the Law Online, 24(3), 281–295.

    Google Scholar 

  • Loftus, E. F. (2005). Planting misinformation in the human mind: A 30-year investigation of the malleability of memory. Learning & Memory, 12(4), 361–366.

    Article  Google Scholar 

  • McCarney, R., Warner, J., Iliffe, S., van Haselen, R., Griffin, M., & Fisher, P. (2007). The Hawthorne effect: A randomised, controlled trial. BMC Medical Research Methodology, 7, 30.

    Article  Google Scholar 

  • Memon, A., Wark, L., Holley, A., Bull, R., & Koehnken, G. (1997). Eyewitness performance in cognitive and structured interviews. Memory, 5(5), 639.

    Article  Google Scholar 

  • Muralidharan, A., Gyongyi, Z., & Chi, E. (2012). Social annotations in web search. In Proceedings of the 2012 ACM Annual Conference on Human Factors in Computing Systems. New York, NY: ACM.

    Google Scholar 

  • Norman, D. (2009). Memory is more important that actuality. Interactions, 16(2), 24–26.

    Article  MathSciNet  Google Scholar 

  • Novick, D. G., Santaella, B., Cervantes, A., & Andrade, C. (2012, October). Short-term methodology for long-term usability. In Proceedings of the 30th ACM International Conference on Design of Communication (pp. 205–212). New York, NY: ACM.

    Google Scholar 

  • Rieman, J. (1993). The diary study: A workplace-oriented research tool to guide laboratory efforts. In Proceedings of CHI: ACM Conference on Human Factors in Computing Systems (pp. 321–326).

    Google Scholar 

  • Roediger, H. L., & McDermott, K. B. (1995). Creating false memories: Remembering words that were not presented in lists. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 803–814.

    Google Scholar 

  • Russell, D. M., & Grimes, C. (2007). Assigned tasks are not the same as self-chosen Web search tasks. In System Sciences, 2007. HICSS 2007. 40th Annual Hawaii International Conference on (pp. 83-83). IEEE.

    Google Scholar 

  • Russell, D. M., & Oren, M. (2009, January). Retrospective cued recall: A method for accurately recalling previous user behaviors. In System Sciences, 2009, HICSS‘09. 42nd Hawaii International Conference on (pp. 1–9). IEEE.

    Google Scholar 

  • Schacter, D. L. (1999). The seven sins of memory: Insights from psychology and cognitive neuroscience. American Psychologist, 54(3), 182–203.

    Article  Google Scholar 

  • Schacter, D. L. (2001). The seven sins of memory: How the mind forgets and remembers. Boston, MA: Houghton Mifflin.

    Google Scholar 

  • Schwarz, N., Kahneman, D., Xu, J., Belli, R., Stafford, F., & Alwin, D. (2009). Global and episodic reports of hedonic experience. In R. Belli, D. Alwin, & F. Stafford (Eds.), Using calendar and diary methods in life events research (pp. 157–174). Newbury Park, CA: Sage Publishing.

    Google Scholar 

  • Shiffman, S., Hufford, M., Hickcox, M., Paty, J. A., Gnys, M., & Kassel, J. D. (1997). Remember that? A comparison of real-time versus retrospective recall of smoking lapses. Journal of Consulting and Clinical Psychology, 65, 292.

    Article  Google Scholar 

  • Siochi, A. C., & Hid, D. (1991). A study of computer-supported user interface evaluation using maximal repeating pattern analysis. In Proceedings of ACM CHI ‘91 (pp. 301–305).

    Google Scholar 

  • Steele-Johnson, D. (2000). Goal orientation and task demand effects on motivation, affect, and performance. The Journal of Applied Psychology, 85(5), 724–738.

    Article  Google Scholar 

  • Teevan, J., & Karger, D. (2005). The research engine: Helping people return to information on the Web. Paper presented at the Proceedings of the ACM Symposium on User Interface Software and Technology (UIST‘05), Seattle, WA.

    Google Scholar 

  • Underwood, B. J. (1965). False recognition produced by implicit verbal responses. Journal of Experimental Psychology, 70, 122–129.

    Article  Google Scholar 

  • Van Boven, L., Kane, J., & McGraw, A. P. (2009). Temporally asymmetric constraints on mental simulation: Retrospection is more constrained than prospection. The handbook of imagination and mental simulation (131–147).

    Google Scholar 

  • van den Haak, M., De Jong, M., & Schellens, P. J. (2003). Retrospective versus concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology, 22, 339–351.

    Article  Google Scholar 

  • Van Gog, T., Paas, F., & Van Merriënboer, J. J. G. (2005). Uncovering the problem-solving process: Cued retrospective reporting versus concurrent and retrospective reporting. Journal of Experimental Psychology: Applied, 11(4), 237–244.

    Google Scholar 

  • Van House, N. (2006). Interview viz: Visualization-assisted photo elicitation. Ext. Abstracts CHI 2006 (pp. 1463–1468). New York, NY: ACM Press.

    Google Scholar 

  • van Kesteren, I. E., Bekker, M. M., Vermeeren, A. P., & Lloyd, P. A. (2003). Assessing usability evaluation methods on their effectiveness to elicit verbal comments from children subjects. In Proceedings of the 2003 Conference on Interaction Design and Children (pp. 41–49). New York, NY: ACM.

    Google Scholar 

  • Weisberg, H., Krosnick, J. A., & Bowen, B. D. (1996). An introduction to survey research, polling, and data analysis. Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Wilson, B. A., Evans, J. J., Emslie, H., & Malinek, V. (1997). Evaluation of NeuroPage: A new memory aid. Journal of Neurology, Neurosurgery and Psychiatry, 63, 113–115.

    Article  Google Scholar 

  • Wilson, T. D. (1994). The Proper Protocol: Validity and Completeness of Verbal Reports. Psychological Science, 5(5), 249–252.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Google, Inc., 1600 Amphitheatre Parkway, Mountain View, CA, 94043, USA

    Daniel M. Russell & Ed H. Chi

Authors
  1. Daniel M. Russell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ed H. Chi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel M. Russell .

Editor information

Editors and Affiliations

  1. University of California, Irvine, Irvine, California, USA

    Judith S. Olson

  2. Group Social Computing, IBM T.J. Watson Research Center, Yorktown Heights, New York, USA

    Wendy A. Kellogg

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer Science+Business Media New York

About this chapter

Cite this chapter

Russell, D.M., Chi, E.H. (2014). Looking Back: Retrospective Study Methods for HCI. In: Olson, J., Kellogg, W. (eds) Ways of Knowing in HCI. Springer, New York, NY. https://doi.org/10.1007/978-1-4939-0378-8_15

Download citation

  • DOI: https://doi.org/10.1007/978-1-4939-0378-8_15

  • Published:

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4939-0377-1

  • Online ISBN: 978-1-4939-0378-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation