Skip to main content
SpringerLink
Log in

A new algorithm for modeling online search behavior and studying ranking reliability variations

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

We design an information retrieval algorithm that mimics the stochastic behavior of decision-makers (DMs) when evaluating the alternatives displayed by an online search engine. The algorithm consists of a decision tree that incorporates all the 1024 decision nodes that may arise from the information retrieval process of DMs. We calibrate the behavior of the algorithm to the one observed from online users and run several sets of 1,000,000 queries. Each query lets DMs decide which subset of the ten alternatives composing the initial page of results to click, allowing us to evaluate their behavior as ranking reliability is assumed to decrease when DMs decide not to click on an alternative. We compare the click-through rates (CTRs) obtained when modifying the degree of ranking reliability derived from the alternatives displayed on the first page of search results. We illustrate how the stability of the CTR prevails among the top-ranked alternatives within relatively reliable scenarios while it drops when imposing large initial decrements in reliability. The resulting consequences regarding the importance of relative ranking positions are analyzed, the top three alternatives exhibiting a generally contained decrease in their CTRs that contrasts with the cumulative pattern arising from the fourth position onwards.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Epstein R, Robertson RE (2015) The Search Engine Manipulation Effect (SEME) and its possible impact on the outcomes of elections. Proc Natl Acad Sci USA 112:E4512–E4521

    Article  Google Scholar 

  2. Jansen MBJ, Spink A, Saracevic T (2000) Real life, real users, and real needs: a study and analysis of user queries on the Web. Inf Process Manage 36:207–227

    Article  Google Scholar 

  3. Luo W, Cook D, Karson EJ (2011) Search advertising placement strategy: exploring the efficacy of the conventional wisdom. Inf Manag 48:404–411

    Article  Google Scholar 

  4. Lewandowski D, Kammerer Y (2020) Factors influencing viewing behaviour on search engine results pages: a review of eye-tracking research. Behav Inf Technol. https://doi.org/10.1080/0144929X.2020.1761450

    Article  Google Scholar 

  5. Lorigo L, Haridasan M, Brynjarsdóttir H, Xia L, Joachims T, Gay G, Granka L, Pellacini F, Pan B (2008) Eye tracking and online search: lessons learned and challenges ahead. J Am Soc Inf Sci Technol 59:1041–1052

    Article  Google Scholar 

  6. Chitika (2013) The value of Google result positioning. Chitika Insights, June 7, 2013. Chitika, Westborough. perma.cc/7AGC-HTDH

  7. Dean B (2019) We analyzed 5 million Google search results. Here’s what we learned about organic click through rate. https://backlinko.com/google-ctr-stats. Accessed 23 Sep 2021

  8. Baeza-Yates R (2005) Applications of web query mining. In: Losada DE, Fernández-Luna JM (eds) Advances in information retrieval. information systems and applications, incl. Internet/Web, and HCI, vol 3408. Springer, Berlin, pp 7–22

  9. Jansen MBJ, Spink A, Bateman J, Saracevic T (1998) Real life information retrieval: a study of user queries on the web. ACM SIGIR Forum 32:5–17

    Article  Google Scholar 

  10. European Commission (2016) Online platforms. Special Eurobarometer No. 447, April 2016. European Union

  11. Hendahewa C, Shah C (2017) Evaluating user search trails in exploratory search tasks. Inf Process Manage 53:905–922

    Article  Google Scholar 

  12. Yu H, Jatowt A, Blanco R, Joho H, Jose JM (2017) Decoding multi-click search behavior based on marginal utility. Inf Retrieval J 20:25–52

    Article  Google Scholar 

  13. Simon HA (1997) Administrative behavior. The Free Press, New York

    Google Scholar 

  14. Schneider M, Deck C, Shor M, Besedeš T, Sarangi S (2019) Optimizing choice architectures. Decis Anal 16:2–30

    Article  Google Scholar 

  15. Speier-Pero C (2019) Using aggregated data under time pressure: a mechanism for coping with information overload. J Decis Syst 28:82–100

    Article  Google Scholar 

  16. Victorelli EZ, Dos Reis JC, Hornung H, Prado AB (2020) Understanding human-data interaction: literature review and recommendations for design. Int J Hum Comput Stud 134:13–32

    Article  Google Scholar 

  17. Preibusch S, Kübler D, Beresford AR (2013) Price versus privacy: an experiment into the competitive advantage of collecting less personal information. Electron Commer Res 13:423–455

    Article  Google Scholar 

  18. Sun H, Fan M, Tan Y (2020) An empirical analysis of seller advertising strategies in an online marketplace. Inf Syst Res 31:37–56

    Article  Google Scholar 

  19. Yoo B, Jeon S, Han T (2016) An analysis of popularity information effects: field experiments in an online marketplace. Electron Commer Res Appl 17:87–98

    Article  Google Scholar 

  20. Bell D, Mgbemena C (2018) Data-driven agent-based exploration of customer behavior. SIMULATION 94:195–212

    Article  Google Scholar 

  21. Dunke F, Nickel S (2020) Neural networks for the metamodeling of simulation models with online decision making. Simul Model Pract Theory 99:102016

    Article  Google Scholar 

  22. Mahony C, Sammon D, Heavin C (2016) Design guidelines for online resources: a longitudinal analysis of information processing. J Decis Syst 25:329–342

    Article  Google Scholar 

  23. Zhang J, Adomavicius G, Gupta A, Ketter W (2020) Consumption and performance: understanding longitudinal dynamics of recommender systems via an agent-based simulation framework. Inf Syst Res 31:76–101

    Article  Google Scholar 

  24. Unkel J, Haas A (2017) The effects of credibility cues on the selection of search engine results. J Am Soc Inf Sci 68:1850–1862

    Google Scholar 

  25. Wang J, Yang Z, Brocato ED (2018) An investigation into the antecedents of prepurchase online search. Inf Manag 55:285–293

    Article  Google Scholar 

  26. Power DJ, Cyphert D, Roth RM (2019) Analytics, bias, and evidence: the quest for rational decision making. J Decis Syst 28:120–137

    Article  Google Scholar 

  27. Yi C, Jiang Z, Li X, Lu X (2019) Leveraging user-generated content for product promotion: the effects of firm-highlighted reviews. Inf Syst Res 30:711–725

    Article  Google Scholar 

  28. Li H, Duan H, Zheng Y, Wang Q, Wang Y (2020) A CTR prediction model based on user interest via attention mechanism. Appl Intell 50:1192–1203

    Article  Google Scholar 

  29. Wang Q, Liu F, Xing S, Zhao X (2019) Research on CTR prediction based on stacked autoencoder. Appl Intell 49:2970–2981

    Article  Google Scholar 

  30. Qin J, Zhang W, Wu X, Jin J, Fang Y, Yu Y (2020) User behavior retrieval for Click-Through Rate prediction. In: Proceedings of the 43rd international ACM SIGIR Conference on research and development in information retrieval (SIGIR’ 20). Association for Computing Machinery, New York, pp 2347–2356

  31. Nelson P (1970) Information and consumer behavior. J Polit Econ 78:311–329

    Article  Google Scholar 

  32. Tavana M, Di Caprio D, Santos Arteaga FJ, O’Connor A (2015) A novel entropy-based decision support framework for uncertainty resolution in the initial subjective evaluations of experts: the NATO enlargement problem. Decis Support Syst 74:135–149

    Article  Google Scholar 

  33. Herrmann JW (2015) Engineering decision making and risk management. Wiley, Hoboken

    MATH  Google Scholar 

  34. Pei S, Hu Q (2018) Partially monotonic decision trees. Inf Sci 424:104–117

    Article  MathSciNet  Google Scholar 

  35. Sagi O, Rokach L (2020) Explainable decision forest: transforming a decision forest into an interpretable tree. Inf Fusion 61:124–138

    Article  Google Scholar 

  36. Herce-Zelaya J, Porcel C, Bernabé-Moreno J, Tejeda-Lorente A, Herrera-Viedma E (2020) New technique to alleviate the cold start problem in recommender systems using information from social media and random decision forests. Inf Sci 536:156–170

    Article  MathSciNet  Google Scholar 

  37. Rokach L (2016) Decision forest: twenty years of research. Inf Fusion 27:111–125

    Article  Google Scholar 

  38. Lim S (2013) College students’ credibility judgments and heuristics concerning Wikipedia. Inf Process Manage 49:405–419

    Article  Google Scholar 

  39. Tavana M, Santos-Arteaga FJ, Di Caprio D, Tierney K (2016) Modeling signal-based decisions in online search environments: a non-recursive forward-looking approach. Information & Management 53:207–226

    Article  Google Scholar 

  40. Schulz F (2008) Trees with exponentially growing costs. Inf Comput 206:569–578

    Article  MathSciNet  Google Scholar 

  41. Gao R, Shah C (2020) Toward creating a fairer ranking in search engine results. Inf Process Manag 57:102138

    Article  Google Scholar 

  42. Pan B, Hembrooke H, Joachims T, Lorigo L, Gay G, Granka L (2007) In Google we trust: users’ decisions on rank, position, and relevance. J Comput Mediated Commun 12:801–823

    Article  Google Scholar 

  43. Brihaye T, Geeraerts G, Hallet M, Le Roux S (2020) On the termination of dynamics in sequential games. Inf Comput 272:104505

    Article  MathSciNet  Google Scholar 

  44. van Benthem J (2018) Computation as social agency: what, how and who. Inf Comput 261:519–535

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

Dr. Madjid Tavana is grateful for the partial support he received from the Czech Science Foundation (GAČR19-13946S) for this research.

Author information

Authors and Affiliations

  1. Department of Economics and Management, University of Trento, Trento, Italy

    Debora Di Caprio

  2. Faculty of Economics and Management, Free University of Bolzano, Bolzano, Italy

    Francisco J. Santos-Arteaga

  3. Business Systems and Analytics Department, La Salle University, Philadelphia, PA, 19141, USA

    Madjid Tavana

  4. Business Information Systems Department, Faculty of Business Administration and Economics, University of Paderborn, Paderborn, Germany

    Madjid Tavana

Authors
  1. Debora Di Caprio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Francisco J. Santos-Arteaga
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Madjid Tavana
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Madjid Tavana.

Ethics declarations

Conflict of interest

The authors declared that they have no conflict of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Di Caprio, D., Santos-Arteaga, F.J. & Tavana, M. A new algorithm for modeling online search behavior and studying ranking reliability variations. Appl Intell 52, 7529–7549 (2022). https://doi.org/10.1007/s10489-021-02856-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02856-8

Keywords

Search

Navigation