skip to main content
10.1145/1718487.1718515acmconferencesArticle/Chapter ViewAbstractPublication PageswsdmConference Proceedingsconference-collections
research-article

Beyond DCG: user behavior as a predictor of a successful search

Published: 04 February 2010 Publication History

Abstract

Web search engines are traditionally evaluated in terms of the relevance of web pages to individual queries. However, relevance of web pages does not tell the complete picture, since an individual query may represent only a piece of the user's information need and users may have different information needs underlying the same queries. In this work, we address the problem of predicting user search goal success by modeling user behavior. We show empirically that user behavior alone can give an accurate picture of the success of the user's web search goals, without considering the relevance of the documents displayed. In fact, our experiments show that models using user behavior are more predictive of goal success than those using document relevance. We build novel sequence models incorporating time distributions for this task and our experiments show that the sequence and time distribution models are more accurate than static models based on user behavior, or predictions based on document relevance.

References

[1]
E. Agichtein, E. Brill, and S. T. Dumais. Improving web search ranking by incorporating user behavior information. In SIGIR 2006: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 19--26, New York, NY, USA, 2006. ACM.
[2]
P. Boldi, F. Bonchi, C. Castillo, D. Donato, A. Gionis, and S. Vigna. The query-flow graph: model and applications. In Proceeding of the 17th ACM conference on Information and knowledge management (CIKM 2008), pages 609--618, 2008.
[3]
J. Borges and M. Levene. Data mining of user navigation patterns. In WEBKDD, pages 92--111, 1999.
[4]
B. Carterette and R. Jones. Evaluating search engines by modeling the relationship between relevance and clicks. In Proceedings of Twenty-First Annual Conference on Neural Information Processing Systems (NIPS 2007), 2007.
[5]
O. Chapelle and Y. Zhang. A dynamic bayesian network click model for web search ranking. In J. Quemada, G. León, Y.S. Maarek, and W. Nejdl, editors, WWW, pages 1--10. ACM, 2009.
[6]
D. Downey, S. Dumais, and E. Horvitz. Models of searching and browsing: Languages, studies, and applications. Journal of the American Society for Information Science and Technology (JASIST), 58(6):862--871, 2007.
[7]
S. Fox, K. Karnawat, M. Mydland, S.T. Dumais, and T. White. Evaluating implicit measures to improve web search. ACM Trans. Inf. Syst., 23(2):147--168, 2005.
[8]
J.H. Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29:1189--1232, 2001.
[9]
R.V. Hogg and A.T. Craig. Introduction to Mathematical Statistics. Macmillan, New York, 4th edition edition, 1978.
[10]
S.B. Huffman and M. Hochster. How well does result relevance predict session satisfaction? In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 567--574, 2007.
[11]
B.J. Jansen, M. Zhang, and A. Spink. Patterns and transitions of query reformulation during web searching. IJWIS, 3(4):328--340, 2007.
[12]
K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422--446, 2002.
[13]
K. Järvelin, S.L. Price, L.M.L. Delcambre, and M.L. Nielsen. Discounted cumulated gain based evaluation of multiple-query IR sessions. In C. Macdonald, I. Ounis, V. Plachouras, I. Ruthven, and R.W. White, editors, ECIR, volume 4956 of Lecture Notes in Computer Science, pages 4--15. Springer, 2008.
[14]
R. Jones and K.L. Klinkner. Beyond the session timeout: Automatic hierarchical segmentation of search topics in query logs. In Proceedings of ACM 17th Conference on Information and Knowledge Management (CIKM 2008), 2008.
[15]
S. Jung, J.L. Herlocker, and J. Webster. Click data as implicit relevance feedback in web search. Information Processing and Management (IPM), 43(3):791--807, 2007.
[16]
J. Li, S. Huffman, and A. Tokuda. Good abandonment in mobile and pc internet search. In SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 43--50, New York, NY, USA, 2009. ACM.
[17]
S. Ozmutlu. Automatic new topic identification using multiple linear regression. Information Processing and Management, 42(4):934--950, 2006.
[18]
B. Piwowarski, G. Dupret, and R. Jones. Mining user web search activity with layered bayesian networks or how to capture a click in its context. In Proceedings of the Second ACM International Conference on Web Search and Data Mining, 2009.
[19]
F. Radlinski and T. Joachims. Query chains: learning to rank from implicit feedback. In R. Grossman, R. Bayardo, and K.P. Bennett, editors, KDD, pages 239--248. ACM, 2005.
[20]
F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In J.G. Shanahan, S. Amer-Yahia, I. Manolescu, Y. Zhang, D.A. Evans, A. Kolcz, K.-S. Choi, and A. Chowdhury, editors, CIKM, pages 43--52. ACM, 2008.
[21]
N. Sadagopan and J. Li. Characterizing typical and atypical user sessions in clickstreams. In Proceedings of the Seventeenth International Conference on the World-Wide Web (WWW08), 2008.
[22]
C. Silverstein, M.R. Henzinger, H. Marais, and M. Moricz. Analysis of a very large web search engine query log. SIGIR Forum, 33(1):6--12, 1999.
[23]
A. Turpin and F. Scholer. User performance versus precision measures for simple search tasks. In SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 11--18, New York, NY, USA, 2006. ACM.
[24]
Y. Xu and D. Mease. Evaluating web search using task completion time. In J. Allan, J.A. Aslam, M. Sanderson, C. Zhai, and J. Zobel, editors, SIGIR, pages 676--677. ACM, 2009.

Cited By

View all
  • (2024)Advancing Spatial Drought Forecasts by Integrating an Improved Outlier Robust Extreme Learning Machine with Gridded Data: A Case Study of the Lower Mainland Basin, British Columbia, CanadaSustainability10.3390/su1608346116:8(3461)Online publication date: 21-Apr-2024
  • (2023)Investigating Online Art Search through Quantitative Behavioral Data and Machine Learning TechniquesAnalytics10.3390/analytics20200212:2(359-392)Online publication date: 26-Apr-2023
  • (2023)Report on the Dagstuhl Seminar on Frontiers of Information Access Experimentation for Research and EducationACM SIGIR Forum10.1145/3636341.363635157:1(1-28)Online publication date: 1-Jun-2023
  • Show More Cited By

Index Terms

  1. Beyond DCG: user behavior as a predictor of a successful search

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    WSDM '10: Proceedings of the third ACM international conference on Web search and data mining
    February 2010
    468 pages
    ISBN:9781605588896
    DOI:10.1145/1718487
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 04 February 2010

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. query log analysis
    2. search engine evaluation
    3. search sessions
    4. user behavior models
    5. user satisfaction

    Qualifiers

    • Research-article

    Conference

    Acceptance Rates

    Overall Acceptance Rate 498 of 2,863 submissions, 17%

    Upcoming Conference

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)41
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 10 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Advancing Spatial Drought Forecasts by Integrating an Improved Outlier Robust Extreme Learning Machine with Gridded Data: A Case Study of the Lower Mainland Basin, British Columbia, CanadaSustainability10.3390/su1608346116:8(3461)Online publication date: 21-Apr-2024
    • (2023)Investigating Online Art Search through Quantitative Behavioral Data and Machine Learning TechniquesAnalytics10.3390/analytics20200212:2(359-392)Online publication date: 26-Apr-2023
    • (2023)Report on the Dagstuhl Seminar on Frontiers of Information Access Experimentation for Research and EducationACM SIGIR Forum10.1145/3636341.363635157:1(1-28)Online publication date: 1-Jun-2023
    • (2023)Taking Search to TaskProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578288(1-13)Online publication date: 19-Mar-2023
    • (2023)Priming and Actions: An Analysis in Conversational Search SystemsProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3592041(2277-2281)Online publication date: 19-Jul-2023
    • (2023)Metric-agnostic Ranking OptimizationProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591935(2669-2680)Online publication date: 19-Jul-2023
    • (2023)Practice and Challenges in Building a Business-oriented Search Engine Quality MetricProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591841(3295-3299)Online publication date: 19-Jul-2023
    • (2023)Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and BeyondProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591683(632-642)Online publication date: 19-Jul-2023
    • (2022)Understanding Query Combination Behavior in Exploratory SearchesApplied Sciences10.3390/app1202070612:2(706)Online publication date: 11-Jan-2022
    • (2022)Understanding and Evaluating Search ExperienceSynthesis Lectures on Information Concepts, Retrieval, and Services10.2200/S01166ED1V01Y202202ICR07714:1(1-105)Online publication date: 28-Mar-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media