Skip to main content
Springer Nature Link
Log in

A Qualitative Evaluation of User Preference for Link-Based vs. Text-Based Recommendations of Wikipedia Articles

  • Conference paper
  • First Online:
Towards Open and Trustworthy Digital Societies (ICADL 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 13133))

Included in the following conference series:

  • 1116 Accesses

Abstract

Literature recommendation systems (LRS) assist readers in the discovery of relevant content from the overwhelming amount of literature available. Despite the widespread adoption of LRS, there is a lack of research on the user-perceived recommendation characteristics for fundamentally different approaches to content-based literature recommendation. To complement existing quantitative studies on literature recommendation, we present qualitative study results that report on users’ perceptions for two contrasting recommendation classes: (1) link-based recommendation represented by the Co-Citation Proximity (CPA) approach, and (2) text-based recommendation represented by Lucene’s MoreLikeThis (MLT) algorithm. The empirical data analyzed in our study with twenty users and a diverse set of 40 Wikipedia articles indicate a noticeable difference between text- and link-based recommendation generation approaches along several key dimensions. The text-based MLT method receives higher satisfaction ratings in terms of user-perceived similarity of recommended articles. In contrast, the CPA approach receives higher satisfaction scores in terms of diversity and serendipity of recommendations. We conclude that users of literature recommendation systems can benefit most from hybrid approaches that combine both link- and text-based approaches, where the user’s information needs and preferences should control the weighting for the approaches used. The optimal weighting of multiple approaches used in a hybrid recommendation system is highly dependent on a user’s shifting needs.

M. Ostendorff and C. Breitinger—Both authors contributed equally to this research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    We judge the article quality using Wikipedia’s vital article policy [40].

  2. 2.

    https://github.com/malteos/wikipedia-article-recommendations.

References

  1. Apache Lucence - More Like This. https://lucene.apache.org/core/7_2_0/queries/org/apache/lucene/queries/mlt/MoreLikeThis.html. Accessed 16 June 2021

  2. Bär, D., Zesch, T., Gurevych, I.: A reflective view on text similarity. In: Proceedings of the International Conference Recent Advances in Natural Language Processing 2011, pp. 515–520. Association for Computational Linguistics, Hissar (2011). https://www.aclweb.org/anthology/R11-1071

  3. Beel, J., Breitinger, C., Langer, S., Lommatzsch, A., Gipp, B.: Towards reproducibility in recommender-systems research. User Model. User-Adap. Inter. 26(1), 69–101 (2016). https://doi.org/10.1007/s11257-016-9174-x

    Article  Google Scholar 

  4. Beel, J., Gipp, B., Langer, S., Breitinger, C.: Research-paper recommender systems: a literature survey. Int. J. Digit. Libr. 17(4), 305–338 (2015). https://doi.org/10.1007/s00799-015-0156-0

    Article  Google Scholar 

  5. Cremonesi, P., Garzotto, F., Negro, S., Papadopoulos, A.V., Turrin, R.: Looking for “Good’’ recommendations: a comparative evaluation of recommender systems. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011. LNCS, vol. 6948, pp. 152–168. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23765-2_11

    Chapter  Google Scholar 

  6. De Gemmis, M., Lops, P., Semeraro, G., Musto, C.: An investigation on the serendipity problem in recommender systems. Inf. Process. Manag. 51(5), 695–717 (2015). https://doi.org/10.1016/j.ipm.2015.06.008

    Article  Google Scholar 

  7. ElasticSearch - More like this query. https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html. Accessed 16 June 2021

  8. Färber, M., Sampath, A.: HybridCite: a hybrid model for context-aware citation recommendation. arXiv (2020). http://arxiv.org/abs/2002.06406

  9. Ge, M., Delgado-Battenfeld, C., Jannach, D.: Beyond accuracy: evaluating recommender systems by coverage and serendipity. In: Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys ’10, pp. 257–260. Association for Computing Machinery, New York (2010). https://doi.org/10.1145/1864708.1864761

  10. Gipp, B., Beel, J.: Citation Proximity Analysis (CPA) – a new approach for identifying related work based on co-citation analysis. In: Proceedings of the 12th International Conference on Scientometrics and Informetrics (ISSI ’09), 2 July, pp. 571–575 (2009). https://ag-gipp.github.io/bib/preprints/gipp09a.pdf

  11. Gravino, P., Monechi, B., Loreto, V.: Towards novelty-driven recommender systems. C R Phys. 20(4), 371–379 (2019). https://doi.org/10.1016/j.crhy.2019.05.014

    Article  Google Scholar 

  12. Jannach, D., Zanker, M., Felfernig, A., Friedrich, G.: Recommender Systems - An Introduction, 1st edn. Cambridge University Press, London (2010)

    Book  Google Scholar 

  13. Jinha, A.E.: Article 50 million: an estimate of the number of scholarly articles in existence. Learn. Publish. 23(3), 258–263 (2010). https://doi.org/10.1087/20100308

    Article  Google Scholar 

  14. Joachims, T., Granka, L., Pan, B.: Accurately interpreting clickthrough data as implicit feedback. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (2005). https://doi.org/10.1145/1076034.1076063

  15. Johnson, R., Watkinson, A., Mabe, M.: The STM report: an overview of scientific and scholarly publishing. Int. Assoc. Sci. Tech. Med. Publish. 5(October), 212 (2018)

    Google Scholar 

  16. Jones, K.S.: Index term weighting. Inf. Storage Retrieval 9(11), 619–633 (1973). https://doi.org/10.1016/0020-0271(73)90043-0

    Article  Google Scholar 

  17. Kaminskas, M., Bridge, D.: Measuring surprise in recommender systems. In: RecSys REDD 2014: International Workshop on Recommender Systems Evaluation: Dimensions and Design, vol. 69, pp. 2–7 (2014)

    Google Scholar 

  18. Kaminskas, M., Bridge, D.: Diversity, serendipity, novelty, and coverage: a survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Trans. Interact. Intell. Syst. 7(1), 1–42 (2017). https://doi.org/10.1145/2926720

    Article  Google Scholar 

  19. Kanakia, A., Shen, Z., Eide, D., Wang, K.: A scalable hybrid research paper recommender system for microsoft academic. In: The World Wide Web Conference (WWW 2019), pp. 2893–2899. ACM Press, New York (2019). https://doi.org/10.1145/3308558.3313700

  20. Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems. User Model. User-Adap. Inter. 22(4–5), 441–504 (2012). https://doi.org/10.1007/s11257-011-9118-4

    Article  Google Scholar 

  21. Knoth, P., Khadka, A.: Can we do better than co-citations? - Bringing citation proximity analysis from idea to practice in research article recommendation. In: 2nd Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries, Tokyo, Japan (2017)

    Google Scholar 

  22. Kunaver, M., Požrl, T.: Diversity in recommender systems - a survey. Knowl.-Based Syst. 123, 154–162 (2017). https://doi.org/10.1016/j.knosys.2017.02.009

    Article  Google Scholar 

  23. Kunkel, J., Donkers, T., Michael, L., Barbu, C.M., Ziegler, J.: Let me explain: impact of personal and impersonal explanations on trust in recommender systems. In: Conference on Human Factors in Computing Systems - Proceedings (2019). https://doi.org/10.1145/3290605.3300717

  24. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977). http://www.jstor.org/stable/2529310

  25. Lenhart, P., Herzog, D.: Combining content-based and collaborative filtering for personalized sports news recommendations. In: Proceedings of the 3rd Workshop on New Trends in Content-Based RecommenderSystems (CBRecSys ’16) at RecSys’16, vol. 1673, pp. 3–10 (2016)

    Google Scholar 

  26. Liu, S., Chen, C.: The effects of co-citation proximity on co-citation analysis. In: Proceedings of the 13th Conference of the International Society for Scientometrics and Informetrics (ISSI 2011) (2011)

    Google Scholar 

  27. Maksai, A., Garcin, F., Faltings, B.: Predicting online performance of news recommender systems through richer evaluation metrics. In: Proceedings of the 9th ACM Conference on Recommender Systems, pp. 179–186. ACM, New York (2015). https://doi.org/10.1145/2792838.2800184

  28. Media Wiki Extension - CirrusSearch. https://www.mediawiki.org/wiki/Extension:CirrusSearch. Accessed 16 June 2021

  29. Mendoza, M., Torres, N.: Evaluating content novelty in recommender systems. J. Intell. Inf. Syst. 54(2), 297–316 (2019). https://doi.org/10.1007/s10844-019-00548-x

    Article  Google Scholar 

  30. Molloy, P., Beel, J., Aizawa, A.: Virtual Citation Proximity (VCP): learning a hypothetical in-text citation-proximity metric for uncited documents. In: Proceedings of the 8th International Workshop on Mining Scientific Publications, pp. 1–8 (2020). https://doi.org/10.31219/osf.io/t5aqf

  31. Ostendorff, M., Ash, E., Ruas, T., Gipp, B., Moreno-Schneider, J., Rehm, G.: Evaluating document representations for content-based legal literature recommendations. In: Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, ICAIL ’21, pp. 109–118. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3462757.3466073

  32. Pu, P., Chen, L., Hu, R.: A user-centric evaluation framework for recommender systems. In: Proceedings of the Fifth ACM Conference on Recommender Systems - RecSys ’11, p. 157. ACM Press, New York (2011). https://doi.org/10.1145/2043932.2043962

  33. Salton, G., Wong, A., Yang, C.S.: Vector space model for automatic indexing. Information retrieval and language processing. Commun. ACM 18(11), 613–620 (1975)

    Google Scholar 

  34. Schwarzer, M., Breitinger, C., Schubotz, M., Meuschke, N., Gipp, B.: Citolytics: a link-based recommender system for wikipedia. In: Proceedings of the Eleventh ACM Conference on Recommender Systems - RecSys ’17, pp. 360–361. ACM, New York (2017). https://doi.org/10.1145/3109859.3109981

  35. Schwarzer, M., Schubotz, M., Meuschke, N., Breitinger, C., Markl, V., Gipp, B.: Evaluating Link-based recommendations for wikipedia. In: Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries - JCDL ’16, pp. 191–200. ACM Press, New York (2016). https://doi.org/10.1145/2910896.2910908

  36. Sinha, R., Swearingen, K.: The role of transparency in recommender systems. In: CHI ’02 Extended Abstracts on Human Factors in Computing Systems - CHI ’02, p. 830. ACM Press, New York (2002)

    Google Scholar 

  37. Small, H.: Co-citation in the scientific literature: a new measure of the relationship between two documents. J. Am. Soc. Inf. Sci. 24(4), 265–269 (1973). https://doi.org/10.1002/asi.4630240406

    Article  Google Scholar 

  38. Sparck Jones, K., Walker, S., Robertson, S.E.: A probabilistic model of information retrieval: development and comparative experiments, Part 2. Inf. Process. Manag. 36(6), 809–840 (2000). https://doi.org/10.1016/S0306-4573(00)00015-7

    Article  Google Scholar 

  39. Wikipedia - Size of Wikipedia. https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia. Accessed 16 June 2021

  40. Wikipedia - Vital articles. https://en.wikipedia.org/wiki/Wikipedia:Vital_articles. Accessed 16 June 2021

  41. Yu, C., Lakshmanan, L., Amer-Yahia, S.: It takes variety to make a world: diversification in recommender systems. In: Edbt, pp. 368–378 (2009). https://doi.org/10.1145/1516360.1516404

  42. Zhang, Y., Chen, X.: Explainable recommendation: a survey and new perspectives. Now Publish. (2020). https://doi.org/10.1561/9781680836592

    Article  Google Scholar 

  43. Zhao, Q., Harper, F.M., Adomavicius, G., Konstan, J.A.: Explicit or implicit feedback? Engagement or satisfaction? a field experiment on machine-learning-based recommender systems. In: Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC ’18, pp. 1331–1340. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3167132.3167275

Download references

Acknowledgements

We thank the anonymous reviewers, all participants, and especially Olha Yarikova for her assistance in the completion of the interviews.

Author information

Authors and Affiliations

  1. Open Legal Data, Berlin, Germany

    Malte Ostendorff

  2. University of Wuppertal, Wuppertal, Germany

    Corinna Breitinger & Bela Gipp

Authors
  1. Malte Ostendorff
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Corinna Breitinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bela Gipp
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Malte Ostendorff .

Editor information

Editors and Affiliations

  1. National Taiwan Normal University, Taipei, Taiwan

    Hao-Ren Ke

  2. Nanyang Technological University, Singapore, Singapore

    Chei Sian Lee

  3. Kyoto University, Kyoto, Japan

    Kazunari Sugiyama

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ostendorff, M., Breitinger, C., Gipp, B. (2021). A Qualitative Evaluation of User Preference for Link-Based vs. Text-Based Recommendations of Wikipedia Articles. In: Ke, HR., Lee, C.S., Sugiyama, K. (eds) Towards Open and Trustworthy Digital Societies. ICADL 2021. Lecture Notes in Computer Science(), vol 13133. Springer, Cham. https://doi.org/10.1007/978-3-030-91669-5_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-91669-5_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-91668-8

  • Online ISBN: 978-3-030-91669-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics