Amelia Cole
ABSTRACT
Capturing authentic information behaviors, feelings, and attitudes in natural settings is a challenge in interactive information retrieval research (IIR). Quantitative data collection is useful for understanding IIR at scale. Yet common data collection techniques, such as surveys, lack participants’ reasoning behind their choices. Qualitative data collection, such as think aloud and after protocols, support understanding how people behave, think, and feel immediately following an experience, but may be subject to cognitive biases and are challenging to deliver longitudinally. IIR studies have an opportunity to enhance ecological validity by using mixed and multi-method studies. This paper applies learnings from a longitudinal mixed method study that combines the 'in-situ' benefits of collecting real-time data over time with the benefits of retrospectives. This approach has potential to advance SAL research by providing a contextualised approach to longitudinal data collection and can be used to gain deeper insights into subjective experiences.
- Obead Alhadreti and Pam Mayhew. 2018. Rethinking Thinking Aloud: A Comparison of Three Think-Aloud Protocols. ACM Press, 1–12. DOI:https://doi.org/10.1145/3173574.3173618Google Scholar
Digital Library
- Albert Bandura. 1977. Self-efficacy: Toward a unifying theory of behavioral change. Psychol. Rev. 84, 2 (1977), 191–215.Google ScholarCross Ref
- Niels van Berkel, Denzil Ferreira, and Vassilis Kostakos. 2018. The Experience Sampling Method on Mobile Devices. ACM Comput. Surv. 50, 6 (January 2018), 1–40. DOI:https://doi.org/10.1145/3123988Google ScholarDigital Library
- Pia Borlund and Toine Bogers. 2018. Injecting realism into simulated work tasks: A case study of the book domain. Proc. Assoc. Inf. Sci. Technol. 55, 1 (2018), 759–761. DOI:https://doi.org/10.1002/pra2.2018.14505501105Google ScholarCross Ref
- Saskia Brand-Gruwel, Iwan Wopereis, and Amber Walraven. 2009. A descriptive model of information problem solving while using internet. Comput. Educ. 53, 4 (December 2009), 1207–1217. DOI:https://doi.org/10.1016/j.compedu.2009.06.004Google ScholarDigital Library
- Joel Brandt, Philip J. Guo, Joel Lewenstein, Mira Dontcheva, and Scott R. Klemmer. 2009. Two studies of opportunistic programming: Interleaving web foraging, learning, and writing code. In CHI2009, ACM, Boston, MA, USA.Google Scholar
- Virgina Braun and Victoria Clark. 2021. Thematic Analysis: A Practical Guide. SAGE Publications. Retrieved December 6, 2021 from https://us.sagepub.com/en-us/nam/thematic-analysis/book248481Google Scholar
- Alan Bryman. 2012. Social research methods (4th ed ed.). Oxford University Press, Oxford ; New York.Google Scholar
- Yu Chi. 2021. Health consumer's knowledge learning in online health information seeking (Doctoral Dissertation). University of Pittsburgh, Pittsburgh, PA, USA.Google Scholar
- Amelia Cole and Kelsey Urgo. 2021. Multi-method experience sampling in information behaviour research. Proc. Annu. Conf. CAIS Actes Congrès Annu. ACSI (May 2021). DOI:https://doi.org/10.29173/cais1187Google ScholarCross Ref
- Amelia W. Cole. 2022. Understanding self-efficacy in search as self-determined learning. University of British Columbia. DOI:https://doi.org/10.14288/1.0416302Google ScholarCross Ref
- Kevyn Collins-Thompson, Soo Young Rieh, Carl C. Haynes, and Rohail Syed. 2016. Assessing learning outcomes in web search: A comparison of tasks and query strategies. In 2016 ACM on Conference on Human Information Interaction and Retrieval, ACM Press, 163–172. DOI:https://doi.org/10.1145/2854946.2854972Google ScholarDigital Library
- John W. Creswell and Vicki L. Plano Clark. 2017. Designing and Conducting Mixed Methods Research (3rd Edition ed.). SAGE Publications, Inc.Google Scholar
- Mihaly Csikszentmihalyi. 1997. Creativity: flow and the psychology of discovery and invention. HarperPerennial, New York.Google Scholar
- David W. Eccles and Güler Arsal. 2017. The think aloud method: what is it and how do I use it? Qual. Res. Sport Exerc. Health 9, 4 (August 2017), 514–531. DOI:https://doi.org/10.1080/2159676X.2017.1331501Google ScholarCross Ref
- Anastasia Efklides. 2008. Metacognition: Defining Its Facets and Levels of Functioning in Relation to Self-Regulation and Co-regulation. Eur. Psychol. 13, 4 (January 2008), 277–287. DOI:https://doi.org/10.1027/1016-9040.13.4.277Google ScholarCross Ref
- K. Anders Ericsson and Herbert A. Simon. 1998. How to Study Thinking in Everyday Life: Contrasting Think-Aloud Protocols With Descriptions and Explanations of Thinking. Mind Cult. Act. 5, 3 (July 1998), 178–186. DOI:https://doi.org/10.1207/s15327884mca0503_3Google ScholarCross Ref
- John H. Flavell. 1979. Metacognition and cognitive monitoring. Am. Psychol. 34, 10 (1979), 906–911.Google ScholarCross Ref
- Luanne Freund, Heather O'Brien, and Rick Kopak. 2014. Getting the big picture: supporting comprehension and learning in search. In SAL Workshop, ACM, Regensburg Germany, 5.Google Scholar
- Ujwal Gadiraju, Ran Yu, Stefan Dietze, and Peter Holtz. 2018. Analyzing Knowledge Gain of Users in Informational Search Sessions on the Web. In Proceedings of the 2018 Conference on Human Information Interaction&Retrieval - CHIIR ’18, ACM Press, New Brunswick, NJ, USA, 2–11. DOI:https://doi.org/10.1145/3176349.3176381Google ScholarDigital Library
- Souvick Ghosh, Manasa Rath, and Chirag Shah. 2018. Searching as Learning: Exploring Search Behavior and Learning Outcomes in Learning-related Tasks. In Proceedings of the 2018 Conference on Human Information Interaction&Retrieval - CHIIR ’18, ACM Press, New Brunswick, NJ, USA, 22–31. DOI:https://doi.org/10.1145/3176349.3176386Google ScholarDigital Library
- Hayley Guillou, Kevin Chow, Thomas Fritz, and Joanna McGrenere. 2020. Is Your Time Well Spent? Reflecting on Knowledge Work More Holistically. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, Honolulu HI USA, 1–9. DOI:https://doi.org/10.1145/3313831.3376586Google ScholarDigital Library
- Joel Hektner, Jennifer Schmidt, and Mihaly Csikszentmihalyi. 2007. Experience Sampling Method. SAGE Publications, Inc., 2455 Teller Road, Thousand Oaks California 91320 United States of America. DOI:https://doi.org/10.4135/9781412984201Google ScholarCross Ref
- Ken Hillis, Michael Petit, and Kylie Jarrett. 2013. Google and the culture of search. Routledge Taylor & Francis Group, New York ; London.Google Scholar
- Barbara K. Hofer. 2004. Epistemological Understanding as a Metacognitive Process: Thinking Aloud During Online Searching. Educ. Psychol. 39, 1 (March 2004), 43–55. DOI:https://doi.org/10.1207/s15326985ep3901_5Google ScholarCross Ref
- Stephen Intille, Charles Kukla, and Xiaoyi Ma. 2002. Eliciting user preferences using image-based experience sampling and reflection. In Proceedings of the CHI ’02 Extended Abstracts on Human Factors in Computing Systems, ACM, 2.Google ScholarDigital Library
- Diane Kelly. 2009. Methods for Evaluating Interactive Information Retrieval Systems with Users. Found. Trends® Inf. Retr. 3, 1—2 (2009), 1–224. DOI:https://doi.org/10.1561/1500000012Google ScholarDigital Library
- Weize Kong, Rui Li, Jie Luo, Aston Zhang, Yi Chang, and James Allan. 2015. Predicting Search Intent Based on Pre-Search Context. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, Santiago Chile, 503–512. DOI:https://doi.org/10.1145/2766462.2767757Google ScholarDigital Library
- Carol Kuhlthau. 1991. Insider the search process: Information seeking from the user's perspective. J. Am. Soc. Inf. Sci. (1991), 361–371.Google Scholar
- Dominika Kwasnicka, Stephan U. Dombrowski, Martin White, and Falko F. Sniehotta. 2015. Data-prompted interviews: Using individual ecological data to stimulate narratives and explore meanings. Health Psychol. 34, 12 (December 2015), 1191–1194. DOI:https://doi.org/10.1037/hea0000234Google ScholarCross Ref
- Dominika Kwasnicka, Stephan U. Dombrowski, Martin White, and Falko F. Sniehotta. 2019. ‘It's not a diet, it's a lifestyle’: a longitudinal, data-prompted interview study of weight loss maintenance. Psychol. Health 34, 8 (August 2019), 963–982. DOI:https://doi.org/10.1080/08870446.2019.1579913Google ScholarCross Ref
- J. C. R. Licklider. 1960. Man-Computer Symbiosis. IRE Trans. Hum. Factors Electron. HFE-1, 1 (March 1960), 4–11. DOI:https://doi.org/10.1109/THFE2.1960.4503259Google ScholarCross Ref
- Gary Marchionini. 2006. Exploratory search: from finding to understanding. Commun. ACM 49, 4 (April 2006), 41. DOI:https://doi.org/10.1145/1121949.1121979Google ScholarDigital Library
- Lucia Mason, Angela Boldrin, and Nicola Ariasi. 2010. Epistemic metacognition in context: evaluating and learning online information. Metacognition Learn. 5, 1 (April 2010), 67–90. DOI:https://doi.org/10.1007/s11409-009-9048-2Google ScholarCross Ref
- Julia McQuoid, Johannes Thrul, and Pamela Ling. 2018. A geographically explicit ecological momentary assessment (GEMA) mixed method for understanding substance use. Soc. Sci. Med. 202, (April 2018), 89–98. DOI:https://doi.org/10.1016/j.socscimed.2018.02.014Google ScholarCross Ref
- Andre N. Meyer, Gail C. Murphy, Thomas Zimmermann, and Thomas Fritz. 2017. Design Recommendations for Self-Monitoring in the Workplace: Studies in Software Development. Proc. ACM Hum.-Comput. Interact. 1, CSCW (December 2017), 79:1-79:24. DOI:https://doi.org/10.1145/3134714Google ScholarDigital Library
- Felipe Moraes, Sindunuraga Rikarno Putra, and Claudia Hauff. 2017. Contrasting search as a learning activity with instructor-designed learning. In Conference on Information and Knowledge Management, ACM, Torino, Italy, 10. DOI:https://doi.org/10.1145/3269206.3271676Google ScholarDigital Library
- Barbara Oakley, Terrence Sejnowski, and Alistair McConville. 2018. Learning How to Learn: How to Succeed in School Without Spending All Your Time Studying; A Guide for Kids and Teens. TarcherPerigee, New York.Google Scholar
- Heather O'Brien, Amelia Cole, Andrea Kampen, and Kathy Brennan. 2022. The Effects of Domain and Search Expertise on Learning Outcomes in Digital Library Use. In ACM SIGIR Conference on Human Information Interaction and Retrieval, ACM, Regensburg Germany, 202–210. DOI:https://doi.org/10.1145/3498366.3505761Google ScholarDigital Library
- Heather L. O'Brien, Andrea Kampen, Amelia W. Cole, and Kathleen Brennan. 2020. The Role of Domain Knowledge in Search as Learning. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, ACM, Vancouver BC Canada, 313–317. DOI:https://doi.org/10.1145/3343413.3377989Google ScholarDigital Library
- Heather Lynn O'Brien. 2008. Defining and measuring *engagement in user experiences with technology. Ph.D. Dalhousie University (Canada), Canada. Retrieved July 21, 2020 from http://search.proquest.com/pqdtglobal/docview/304405933/abstract/D4C6E6851AA2497BPQ/2Google Scholar
- R.N. Oddy. 1977. Information retrieval through man-machine dialogue. J. Doc. 33, 1 (January 1977), 1–14. DOI:https://doi.org/10.1108/eb026631Google ScholarCross Ref
- Gabriele Pätsch, Thomas Mandl, and Christa Womser-Hacker. 2014. Using sensor graphs to stimulate recall in retrospective think-aloud protocols. ACM Press, 303–307. DOI:https://doi.org/10.1145/2637002.2637048Google ScholarDigital Library
- Paul R Pintrich, Christopher A Wolters, and Gail P Baxter. 2000. Assessing Metacognition and Self-Regulated Learning. In Issues in the Measurement of Metacognition, Gregory Schraw and James C. Impara (eds.). Buros Institute of Mental Measurements, Lincoln, NE, 57.Google Scholar
- Soo Young Rieh. 2002. Judgment of information quality and cognitive authority in the Web. J. Am. Soc. Inf. Sci. Technol. 53, 2 (2002), 145–161. DOI:https://doi.org/10.1002/asi.10017Google ScholarCross Ref
- Soo Young Rieh. 2018. Searching as learning: Perspectives, assessment, and future directions. In SLAIS Colloquium. Univeresity of British Columbia.Google Scholar
- Soo Young Rieh, Kevyn Collins-Thompson, Preben Hansen, and Hye-Jung Lee. 2016. Towards searching as a learning process: A review of current perspectives and future directions. J. Inf. Sci. 42, 1 (2016), 19–34. DOI:https://doi.org/10.1177/0165551515615841Google ScholarDigital Library
- Nirmal Roy, Felipe Moraes, and Claudia Hauff. 2020. Exploring Users’ Learning Gains within Search Sessions. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, ACM, Vancouver BC Canada, 432–436. DOI:https://doi.org/10.1145/3343413.3378012Google ScholarDigital Library
- Alan H. Schoenfeld. 1987. What's all the fuss about metacognition? In Cognitive Science and Mathematics Education. Taylor & Francis Group, New York. Retrieved August 9, 2018 from https://www.taylorfrancis.com/Google Scholar
- Gregory Schraw and Rayne Sperling. 1994. Assessing metacognitive awareness. Contemp. Educ. Psychol. 19, (1994), 469–475.Google ScholarCross Ref
- Dale H. Schunk and Jeffrey Alan Greene (Eds.). 2018. Handbook of Self-Regulation of Learning and Performance (2nd ed.). Routledge, Taylor & Francis Group, New York, NY.Google Scholar
- Vikram Singh. 2022. Correlating pre-search and in-search context to predict search intent for exploratory search. Int. J. Bus. Intell. Data Min. 20, 3 (January 2022), 274–298. DOI:https://doi.org/10.1504/ijbidm.2022.122154Google ScholarDigital Library
- Penny Trieu, Joseph B. Bayer, Nicole B. Ellison, Sarita Schoenebeck, and Emily Falk. 2019. Who likes to be reachable? Availability preferences, weak ties, and bridging social capital. Inf. Commun. Soc. 22, 8 (July 2019), 1096–1111. DOI:https://doi.org/10.1080/1369118X.2017.1405060Google ScholarCross Ref
- Timothy J. Trull and Ulrich W. Ebner-Priemer. 2009. Using experience sampling methods/ecological momentary assessment (ESM/EMA) in clinical assessment and clinical research: Introduction to the special section. Psychol. Assess. 21, 4 (December 2009), 457–462. DOI:https://doi.org/10.1037/a0017653Google ScholarCross Ref
- Kelsey Urgo and Jaime Arguello. 2022. Learning assessments in search-as-learning: A survey of prior work and opportunities for future research. Inf. Process. Manag. 59, 2 (March 2022), 102821. DOI:https://doi.org/10.1016/j.ipm.2021.102821Google ScholarDigital Library
- Pertti Vakkari. 2016. Searching as learning: A systematization based on literature. J. Inf. Sci. 42, 1 (February 2016), 7–18. DOI:https://doi.org/10.1177/0165551515615833Google ScholarDigital Library
- Tamara Van Gog, Fred Paas, J. G. Van Merriëmboer, and Puk Witte. 2005. Uncovering the Problem-Solving Process: Cued Retrospective Reporting Versus Concurrent and Retrospective Reporting. J. Exp. Psychol. Appl. 11, 4 (December 2005), 237–244. DOI:https://doi.org/10.1037/1076-898X.11.4.237Google ScholarCross Ref
- Mathew J. Wilson and Max L. Wilson. 2013. A comparison of techniques for measuring sensemaking and learning within participant-generated summaries. J. Am. Soc. Inf. Sci. Technol. 64, 2 (February 2013), 291–306. DOI:https://doi.org/10.1002/asi.22758Google ScholarDigital Library
- Kathleen A Yearick. 2017. Experience sampling methods in organizations: A review. Master's Thesis. University of Illinois at Urbana-Champaign, Illinois. Retrieved from https://www.ideals.illinois.edu/bitstream/handle/2142/97635/YEARICK-THESIS-2017.pdf?sequence=1&isAllowed=yGoogle Scholar
- Ran Yu, Ujwal Gadiraju, Peter Holtz, Markus Rokicki, Philipp Kemkes, and Stefan Dietze. 2018. Predicting User Knowledge Gain in Informational Search Sessions. 41st Int. ACM SIGIR Conf. Res. Dev. Inf. Retr. (June 2018), 75–84. DOI:https://doi.org/10.1145/3209978.3210064Google ScholarDigital Library
- Yinglong Zhang, Rob Capra, and Yuan Li. 2020. An In-situ Study of Information Needs in Design-related Creative Projects. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, ACM, Vancouver BC Canada, 113–123. DOI:https://doi.org/10.1145/3343413.3377973Google ScholarDigital Library
- Don H. Zimmerman and D. Lawrence Wieder. 1977. The Diary: Diary-Interview Method. Urban Life 5, 4 (January 1977), 479–498. DOI:https://doi.org/10.1177/089124167700500406Google ScholarCross Ref
- Sabrina Zirkel, Julie A. Garcia, and Mary C. Murphy. 2015. Experience-Sampling Research Methods and Their Potential for Education Research. Educ. Res. 44, 1 (2015), 7–16.Google ScholarCross Ref
Index Terms
- Using Data-Prompted Interviews in Interactive Information Retrieval Research: A Reflection on The Study of Self-Efficacy When Learning Using Search
Index terms have been assigned to the content through auto-classification.
Recommendations
Designing interviews to generate rich data for information systems research
Information Systems (IS) publications that use interviews for data generation tend to provide very little insight into the research process and very few rely on a carefully chosen and well-articulated interviewing method. Given the wide variety of ...
Revealing sensitive information in personal interviews
Despite the many advantages of computer-assisted data collection, it is unclear if, when, or under what conditions embodied conversational agents (i.e., ECA, virtual humans) can replace human interviewers to collect personal information in interviews, ...
The Value of Answers without Question[s]: A Qualitative Approach to User Experience and Aging
HCD 09: Proceedings of the 1st International Conference on Human Centered Design: Held as Part of HCI International 2009This project investigates reasons for use and non-use of interactive products by two age groups. It was motivated by the assumption that older adults, when given the chance, report more than just usability-related aspects of interactive products. In ...
Comments