research-article

How Does Domain Expertise Affect Users’ Search Interaction and Outcome in Exploratory Search?

Authors:

Shaoping MaAuthors Info & Claims

ACM Transactions on Information Systems (TOIS), Volume 36, Issue 4

Article No.: 42, Pages 1 - 30

https://doi.org/10.1145/3223045

Published: 17 July 2018 Publication History

Abstract

People often conduct exploratory search to explore unfamiliar information space and learn new knowledge. While supporting the highly dynamic and interactive exploratory search is still challenging for the search system, we want to investigate which factors can make the exploratory search successful and satisfying from the user’s perspective. Previous research suggests that domain experts have different search strategies and are more successful in finding domain-specific information, but how the domain expertise level will influence users’ interaction and search outcomes in exploratory search, especially in different knowledge domains, is still unclear. In this work, via a carefully designed user study that involves 30 participants, we investigate the influence of domain expertise levels on the interaction and outcome of exploratory search in three different domains: environment, medicine, and politics. We record participants’ search behaviors, including their explicit feedback and eye fixation sequences, in a laboratory setting. With this dataset, we identify both domain-independent and domain-dependent effects on user behaviors and search outcomes. Our results extend existing research on the effect of domain expertise in search and suggest different strategies for exploiting domain expertise to support exploratory search in different knowledge domains.

References

[1]

John R. Anderson. 2013. The Architecture of Cognition. Psychology Press.

[2]

Kumaripaba Athukorala, Dorota Głowacka, Giulio Jacucci, Antti Oulasvirta, and Jilles Vreeken. 2016. Is exploratory search different? A comparison of information search behavior for exploratory and lookup tasks. Journal of the Association for Information Science and Technology 67, 11 (2016), 2635--2651.

Digital Library

[3]

Kumaripaba Athukorala, Antti Oulasvirta, Dorota Głowacka, Jilles Vreeken, and Giulio Jacucci. 2014. Narrow or broad?: Estimating subjective specificity in exploratory search. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM’14). ACM, 819--828.

Digital Library

[4]

Suresh K. Bhavnani. 2001. Important cognitive components of domain-specific search knowledge. In Proceedings of the Tenth Text Retrieval Conference (TREC’01). 571--578.

[5]

Suresh K. Bhavnani. 2002. Domain-specific search strategies for the effective retrieval of healthcare and shopping information. In CHI’02 Extended Abstracts on Human Factors in Computing Systems. ACM, 610--611.

Digital Library

[6]

Pia Borlund. 2000. Experimental components for the evaluation of interactive information retrieval systems. Journal of Documentation 56, 1 (2000), 71--90.

[7]

Georg Buscher, Andreas Dengel, and Ludger van Elst. 2008. Eye movements as implicit relevance feedback. In CHI’08 Extended Abstracts on Human Factors in Computing Systems. ACM, 2991--2996.

Digital Library

[8]

Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences. Hilsdale, NJ: Lawrence Earlbaum Associates.

[9]

Michael J. Cole, Jacek Gwizdka, Chang Liu, Nicholas J. Belkin, and Xiangmin Zhang. 2013. Inferring user knowledge level from eye movement patterns. Information Processing 8 Management 49, 5 (2013), 1075--1091.

Digital Library

[10]

Michael J. Cole, Xiangmin Zhang, Chang Liu, Nicholas J. Belkin, and Jacek Gwizdka. 2011. Knowledge effects on document selection in search results pages. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’11). ACM, 1219--1220.

Digital Library

[11]

Kevyn Collins-Thompson, Soo Young Rieh, Carl C. Haynes, and Rohail Syed. 2016. Assessing learning outcomes in web search: A comparison of tasks and query strategies. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (CHIIR’16). ACM, 163--172.

Digital Library

[12]

Geoffrey B. Duggan and Stephen J. Payne. 2008. Knowledge in the head and on the web: Using topic expertise to aid search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’08). ACM, 39--48.

Digital Library

[13]

Yuka Egusa, Hitomi Saito, Masao Takaku, Hitoshi Terai, Makiko Miwa, and Noriko Kando. 2010. Using a concept map to evaluate exploratory search. In Proceedings of the Third Symposium on Information Interaction in Context (IIiX’10). ACM, 175--184.

Digital Library

[14]

Carsten Eickhoff, Sebastian Dungs, and Vu Tran. 2015. An eye-tracking study of query reformulation. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’15). ACM, 13--22.

Digital Library

[15]

Carsten Eickhoff, Jaime Teevan, Ryen White, and Susan Dumais. 2014. Lessons from the journey: A query log analysis of within-session learning. In Proceedings of the 7th ACM International Conference on Web Search and Data Mining (WSDM’14). ACM, 223--232.

Digital Library

[16]

Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. 2007. G* power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods 39, 2 (2007), 175--191.

[17]

Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais, and Thomas White. 2005. Evaluating implicit measures to improve web search. ACM Transactions on Information Systems (TOIS) 23, 2 (2005), 147--168.

Digital Library

[18]

Ahmed Hassan Awadallah, Ryen W. White, Patrick Pantel, Susan T. Dumais, and Yi-Min Wang. 2014. Supporting complex search tasks. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM’14). ACM, 829--838.

Digital Library

[19]

Jiepu Jiang, Daqing He, and James Allan. 2014. Searching, browsing, and clicking in a search session: Changes in user behavior by task and over time. In Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’14). ACM, 607--616.

Digital Library

[20]

Ruogu Kang and Wai-Tat Fu. 2010. Exploratory information search by domain experts and novices. In Proceedings of the 15th International Conference on Intelligent User Interfaces (IUI’10). ACM, 329--332.

Digital Library

[21]

Evangelos Kanoulas, Ben Carterette, Mark Hall, Paul Clough, and Mark Sanderson. 2011. Overview of the trec 2011 session track. In Proceedings of the Twentieth Text REtrieval Conference (TREC’11).

[22]

Weize Kong and James Allan. 2014. Extending faceted search to the general web. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM’14). ACM, 839--848.

Digital Library

[23]

Tessa Lau and Eric Horvitz. 1999. Patterns of search: Analyzing and modeling web query refinement. In User Modeling (UM’99). Springer, 119--128.

Digital Library

[24]

Xin Li, Yiqun Liu, Rongjie Cai, and Shaoping Ma. 2017. Investigation of user search behavior while facing heterogeneous search services. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining (WSDM’17). ACM, 161--170.

Digital Library

[25]

Yuelin Li and Nicholas J. Belkin. 2008. A faceted approach to conceptualizing tasks in information seeking. Information Processing 8 Management 44, 6 (2008), 1822--1837.

Digital Library

[26]

Chang Liu, Xiangmin Zhang, and Wei Huang. 2016. The exploration of objective task difficulty and domain knowledge effects on users’ query formulation. In Proceedings of the Association for Information Science and Technology (ASIST’16). American Society for Information Science, Article 63, 9 pages.

[27]

Mengyang Liu, Yiqun Liu, Jiaxin Mao, Cheng Luo, Min Zhang, and Shaoping Ma. 2018. “Satisfaction with failure” or “unsatisfied success”: Investigating the relationship between search success and user satisfaction. In Proceedings of the 2018 World Wide Web Conference (WWW’18). ACM.

Digital Library

[28]

Yiqun Liu, Ye Chen, Jinhui Tang, Jiashen Sun, Min Zhang, Shaoping Ma, and Xuan Zhu. 2015. Different users, different opinions: Predicting search satisfaction with mouse movement information. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’15). ACM, 493--502.

Digital Library

[29]

Zeyang Liu, Yiqun Liu, Ke Zhou, Min Zhang, and Shaoping Ma. 2015. Influence of vertical result in web search examination. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’15). ACM, 193--202.

Digital Library

[30]

Henry B. Mann and Donald R. Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics 18, 1 (1947), 50--60.

[31]

Jiaxin Mao, Yiqun Liu, Ke Zhou, Jian-Yun Nie, Jingtao Song, Min Zhang, Shaoping Ma, Jiashen Sun, and Hengliang Luo. 2016. When does relevance mean usefulness and user satisfaction in web search? In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’16). ACM, 463--472.

Digital Library

[32]

Gary Marchionini. 2006. Exploratory search: From finding to understanding. Communications of the ACM 49, 4 (2006), 41--46.

Digital Library

[33]

Daan Odijk, Ryen W. White, Ahmed Hassan Awadallah, and Susan T. Dumais. 2015. Struggling and success in web search. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (CIKM’15). ACM, 1551--1560.

Digital Library

[34]

Eyal M. Reingold, Erik D. Reichle, Mackenzie G. Glaholt, and Heather Sheridan. 2012. Direct lexical control of eye movements in reading: Evidence from a survival analysis of fixation durations. Cognitive Psychology 65, 2 (2012), 177--206.

[35]

Lynda Tamine and Cecile Chouquet. 2017. On the impact of domain expertise on query formulation, relevance assessment and retrieval performance in clinical settings. Information Processing 8 Management 53, 2 (2017), 332--350.

Digital Library

[36]

Pertti Vakkari, Mikko Pennanen, and Sami Serola. 2003. Changes of search terms and tactics while writing a research proposal: A longitudinal case study. Information Processing 8 Management 39, 3 (2003), 445--463.

Digital Library

[37]

Ryen W. White, Susan T. Dumais, and Jaime Teevan. 2009. Characterizing the influence of domain expertise on web search behavior. In Proceedings of the Second ACM International Conference on Web Search and Data Mining (WSDM’09). ACM, 132--141.

Digital Library

[38]

Ryen W. White and Resa A. Roth. 2009. Exploratory search: Beyond the query-response paradigm. Synthesis Lectures on Information Concepts, Retrieval, and Services 1, 1 (2009), 1--98.

[39]

Barbara M. Wildemuth. 2004. The effects of domain knowledge on search tactic formulation. Journal of the American Society for Information Science and Technology 55, 3 (2004), 246--258.

Digital Library

[40]

Xiangmin Zhang, Hermina G. B. Anghelescu, and Xiaojun Yuan. 2005. Domain knowledge, search behaviour, and search effectiveness of engineering and science students: An exploratory study. Information Research: An International Electronic Journal 10, 2 (2005), n2.

[41]

Xiangmin Zhang, Jingjing Liu, Michael Cole, and Nicholas Belkin. 2015. Predicting users’ domain knowledge in information retrieval using multiple regression analysis of search behaviors. Journal of the Association for Information Science and Technology 66, 5 (2015), 980--1000.

Digital Library

Cited By

Chen NLiu JFang HLuo YSakai TWu X(2024)Decoy Effect in Search Interaction: Understanding User Behavior and Measuring System VulnerabilityACM Transactions on Information Systems10.1145/370888443:2(1-58)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3708884
Liu SHu YTian ZJin ZRuan SMao JSakai TIshita EOhshima HHasibi FMao JJose J(2024)Investigating Users' Search Behavior and Outcome with ChatGPT in Learning-oriented Search TasksProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698406(103-113)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1145/3673791.3698406
Teixeira Lopes CHenriques M(2024)Unveiling Health Literacy through Web Search Behavior: A Classification-Based Analysis of User InteractionsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638318(1-11)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638318
Show More Cited By

Index Terms

How Does Domain Expertise Affect Users’ Search Interaction and Outcome in Exploratory Search?
1. Information systems
  1. Information retrieval
    1. Users and interactive retrieval
      1. Personalization
  2. World Wide Web
    1. Web searching and information discovery
      1. Web search engines

Recommendations

Characterizing the influence of domain expertise on web search behavior
WSDM '09: Proceedings of the Second ACM International Conference on Web Search and Data Mining

Domain experts search differently than people with little or no domain knowledge. Previous research suggests that domain experts employ different search strategies and are more successful in finding what they are looking for than non-experts. In this ...
How medical expertise influences web search interaction
SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval

Domain expertise can have an important influence on how people search. In this poster we present findings from a log-based study into how medical domain experts search the Web for information related to their expertise, as compared with non-experts. We ...
Exploratory information search by domain experts and novices
IUI '10: Proceedings of the 15th international conference on Intelligent user interfaces

The arising popularity of social tagging system has the potential to transform traditional web search into a new era of social search. Based on the finding that domain expertise could influence search behavior in traditional search engines, we ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems

ACM Transactions on Information Systems Volume 36, Issue 4

October 2018

365 pages

ISSN:1046-8188

EISSN:1558-2868

DOI:10.1145/3211967

Editor:
Maarten de Rijke
University of Amsterdam, The Netherlands

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 July 2018

Accepted: 01 May 2018

Revised: 01 March 2018

Received: 01 July 2017

Published in TOIS Volume 36, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Natural Science Foundation of China
JSPS KAKENHI

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

34
Total Citations
View Citations
586
Total Downloads

Downloads (Last 12 months)53
Downloads (Last 6 weeks)14

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen NLiu JFang HLuo YSakai TWu X(2024)Decoy Effect in Search Interaction: Understanding User Behavior and Measuring System VulnerabilityACM Transactions on Information Systems10.1145/370888443:2(1-58)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3708884
Liu SHu YTian ZJin ZRuan SMao JSakai TIshita EOhshima HHasibi FMao JJose J(2024)Investigating Users' Search Behavior and Outcome with ChatGPT in Learning-oriented Search TasksProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698406(103-113)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1145/3673791.3698406
Teixeira Lopes CHenriques M(2024)Unveiling Health Literacy through Web Search Behavior: A Classification-Based Analysis of User InteractionsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638318(1-11)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638318
Nimmo RConstantinides MZhou KQuercia DStumpf S(2024)User Characteristics in Explainable AI: The Rabbit Hole of Personalization?Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642352(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642352
Liu YQin CWang YMa X(2024)Exploratory search in information systems: a systematic reviewThe Electronic Library10.1108/EL-11-2023-026442:2(308-339)Online publication date: 14-Feb-2024
https://doi.org/10.1108/EL-11-2023-0264
Liu YQin CMa XLi FWang Y(2024)Comparison of information search behavior for different exploratory tasksInformation Processing and Management: an International Journal10.1016/j.ipm.2024.10379461:5Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1016/j.ipm.2024.103794
Dosso CChevalier ATamine LPaubel PSalmerón L(2024)People also ask: How does this tool affect exploration-exploitation strategies with regard to prior domain knowledge and search context? An eye-tracking studyApplied Ergonomics10.1016/j.apergo.2024.104367121(104367)Online publication date: Nov-2024
https://doi.org/10.1016/j.apergo.2024.104367
Li PDong SZhang YZhang B(2023)Predicting Task Planning Ability for Learners Engaged in Searching as Learning Based on Tree-Structured Long Short-Term Memory NetworksApplied Sciences10.3390/app13231284013:23(12840)Online publication date: 30-Nov-2023
https://doi.org/10.3390/app132312840
Shao YWu YLiu YMao JMa S(2023)Understanding Relevance Judgments in Legal Case RetrievalACM Transactions on Information Systems10.1145/356992941:3(1-32)Online publication date: 7-Feb-2023
https://dl.acm.org/doi/10.1145/3569929
Liu QWu JHuang ZWang HNing YChen MChen EYi JZhou B(2023)Federated User Modeling from Hierarchical InformationACM Transactions on Information Systems10.1145/356048541:2(1-33)Online publication date: 3-Apr-2023
https://dl.acm.org/doi/10.1145/3560485
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents