Skip to main content
SpringerLink
Log in

Meta-analysis of evaluation methods and metrics used in context-aware scholarly recommender systems

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

With the current growth of the proposed contextual recommending algorithms, evaluating them becomes more critical. Researchers of recommender systems have expressed concerns that the evaluation quality cannot be properly judged. We carried out meta-analyses of the evaluation methods and metrics of 67 studies related to context-aware scholarly recommender systems, from the years 2000 to 2014. The analysis of variance results shows that offline evaluation methods are more commonly used compared to online and user studies, with the maximum rate of success. It also reveals the popularity order of accuracy metrics (31%) including “Recall, Precision, F-Measure”, “Mean Absolute Error, and Questionnaire studies, Reliability, Accessibility, Feasibility, Usability, Applicability and Performance”. By using factor analysis, 28 different evaluation metrics were classified into eight groups. The results of analysis have shown the difference in evaluation methods in applying different groups of metrics. This study highlights the importance of how an evaluation method should be adequately designed and implemented. Additionally, a few recommendations for future investigations on recommending evaluation are proposed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Champiri ZD, Shahamiri SR, Salim SSB (2015) A systematic review of scholar context-aware recommender systems. Expert Syst Appl 42(3):1743–1758

    Article  Google Scholar 

  2. Riboni D, Bettini C (2012) Private context-aware recommendation of points of interest: an initial investigation. In: IEEE international conference on pervasive computing and communications workshops (PERCOM Workshops). IEEE

  3. Gediminas Adomavicius BM, Francesco R, Alex T (2011) Context-aware recommender systems. Association for the Advancement of Artificial Intelligence. All rights reserved. ISSN 0738-4602

  4. Dehghani Champiri Z et al (2011) A multi-layer contextual model for recommender systems in digital libraries. In: Aslib proceedings. Emerald Group Publishing Limited

  5. Panniello U, Tuzhilin A, Gorgoglione M (2014) Comparing context-aware recommender systems in terms of accuracy and diversity. User Model User Adapt Interact 24(1–2):35–65

    Article  Google Scholar 

  6. Lim BY, Dey AK, Avrahami D (2009) Why and why not explanations improve the intelligibility of context-aware intelligent systems. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM

  7. Gunawardana A, Shani G (2009) A survey of accuracy evaluation metrics of recommendation tasks. J Mach Learn Res 10:2935–2962

    MathSciNet  MATH  Google Scholar 

  8. Beel J, Langer S (2014) A comparison of offline evaluations, online evaluations, and user studies in the context of research paper recommender systems. Under Review. Pre-print available at http://www.docear.org/publications

  9. Said A, Bellogín A (2014) Comparative recommender system evaluation: benchmarking recommendation frameworks. In: Proceedings of the 8th ACM conference on recommender systems. ACM

  10. Ekstrand MD (2014) Towards recommender engineering tools and experiments for identifying recommender differences. University of Minnesota, Minneapolis

    Google Scholar 

  11. Kluver D, Konstan JA (2014) Evaluating recommender behavior for new users. In: Proceedings of the 8th ACM conference on recommender systems. ACM

  12. Manouselis N, Karagiannidis C, Sampson D (2014) Layered evaluation in recommender systems: a retrospective assessment. J e-Learn Knowl Soc 10(1)

  13. Champiri ZD, Salim SSB, Shahamiri SR (2015) The role of context for recommendations in digital libraries. Int J Soc Sci Humanity 5(11):948

    Article  Google Scholar 

  14. Baltrunas L et al (2012) Context relevance assessment and exploitation in mobile recommender systems. Pers Ubiquit Comput 16(5):507–526

    Article  Google Scholar 

  15. Adomavicius G, Tuzhilin A (2005) Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6):734–749

    Article  Google Scholar 

  16. Panniello U, Gorgoglione M (2011) Context-aware recommender systems: a comparison of three approaches. In DART@ AI* IA

  17. Adomavicius G, Tuzhilin A (2011) Context-aware recommender systems. In: Recommender systems handbook. Springer, pp 217–253

  18. Liu L (2013) The implication of context and criteria information in recommender systems as applied to the service domain. University of Manchester, Manchester

    Google Scholar 

  19. Adomavicius G et al (2005) Incorporating contextual information in recommender systems using a multidimensional approach. ACM Trans Inf Syst 23(1):103–145

    Article  Google Scholar 

  20. Adomavicius G, Tuzhilin A (2011) Context-aware recommender systems. In: Recommender systems handbook. Springer, Boston, MA, pp 217–253

  21. Kantor PB et al (2011) Recommender systems handbook. Springer, New York

    MATH  Google Scholar 

  22. Baltrunas L, Ricci F (2009) Context-based splitting of item ratings in collaborative filtering. In: Proceedings of the third ACM conference on recommender systems. ACM

  23. Panniello U et al (2009) Experimental comparison of pre-versus post-filtering approaches in context-aware recommender systems. In: Proceedings of the third ACM conference on recommender systems. ACM

  24. Yujie Z, Licai W (2010) Some challenges for context-aware recommender systems. In: 5th International conference on computer science and education (ICCSE). IEEE

  25. Shani G, Gunawardana A (2011) Evaluating recommendation systems. In: Recommender systems handbook. Springer, pp 257–297

  26. Herlocker JL et al (2004) Evaluating collaborative filtering recommender systems. ACM Trans Inf Syst 22(1):5–53

    Article  Google Scholar 

  27. Beel J et al (2013) A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation. In: Proceedings of the international workshop on reproducibility and replication in recommender systems evaluation. ACM

  28. Said A (2013) Evaluating the accuracy and utility of recommender systems. Doctoral dissertation, Universitätsbibliothek der Technischen Universität, Berlin

  29. de Wit J (2008) Evaluating recommender systems. In: An evaluation framework to predict user satisfaction for recommender systems in an electronic program guide context

  30. Jannach D et al (2013) What recommenders recommend—an analysis of accuracy, popularity, and sales diversity effects. In: User modeling, adaptation, and personalization. Springer, pp 25–37

  31. Zaier Z, Godin R, Faucher L (2008) Evaluating recommender systems. In: Automated solutions for cross media content and multi-channel distribution, AXMEDIS’08 international conference. IEEE

  32. Pu P, Chen L, Hu R (2012) Evaluating recommender systems from the user’s perspective: survey of the state of the art. User Model User Adapt Interact 22(4–5):317–355

    Article  Google Scholar 

  33. Murakami T, Mori K, Orihara R (2007) Metrics for evaluating the serendipity of recommendation lists. In: Annual conference of the Japanese society for artificial intelligence. Springer

  34. Adamopoulos P, Tuzhilin A (2011) On unexpectedness in recommender systems: or how to expect the unexpected. In: Workshop on novelty and diversity in recommender systems (DiveRS 2011), at the 5th ACM international conference on recommender systems (RecSys’11). ACM, Chicago

  35. Ge M, Delgado-Battenfeld C, Jannach D (2010) Beyond accuracy: evaluating recommender systems by coverage and serendipity. In: Proceedings of the fourth ACM conference on recommender systems. ACM

  36. Parra D, Sahebi S (2013) Recommender systems: sources of knowledge and evaluation metrics. In: Advanced techniques in web intelligence-2. Springer, pp 149–175

  37. Schröder G, Thiele M, Lehner W (2011) Setting goals and choosing metrics for recommender system evaluations. In: UCERSTI2 workshop at the 5th ACM conference on recommender systems, Chicago, USA

  38. Beel J et al (2013) Research paper recommender system evaluation: a quantitative literature survey. In: Proceedings of the international workshop on reproducibility and replication in recommender systems evaluation. ACM

  39. Erdt M, Fernández A, Rensing C (2015) Evaluating recommender systems for technology enhanced learning: a quantitative survey. IEEE Trans Learn Technol 8(4):326–344

    Article  Google Scholar 

  40. Bobadilla J et al (2013) Recommender systems survey. Knowl Based Syst 46:109–132

    Article  Google Scholar 

  41. Tintarev N, Masthoff J (2007) A survey of explanations in recommender systems. In: IEEE 23rd international conference on data engineering workshop. IEEE

  42. McNee SM, Riedl J, Konstan JA (2006) Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: CHI’06 extended abstracts on human factors in computing systems. ACM

  43. Konstan JA (2004) Introduction to recommender systems: algorithms and evaluation. ACM Trans Inf Syst 22(1):1–4

    Article  Google Scholar 

  44. Ferrier L et al (1995) Dysarthric speakers’ intelligibility and speech characteristics in relation to computer speech recognition. Augment Altern Commun 11(3):165–175

    Article  Google Scholar 

  45. Tenenhaus M, Amato S, Esposito Vinzi V (2004) A global goodness-of-fit index for PLS structural equation modelling. In: Proceedings of the XLII SIS scientific meeting

  46. Kitchenham BA, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. EBSE technical report EBSE, pp 1–57

  47. Knijnenburg BP et al (2012) Explaining the user experience of recommender systems. User Model User Adapt Interact 22(4–5):441–504

    Article  Google Scholar 

  48. Porcel C, Herrera-Viedma E (2010) Dealing with incomplete information in a fuzzy linguistic recommender system to disseminate information in university digital libraries. Knowl Based Syst 23(1):32–39

    Article  Google Scholar 

  49. Reuters T (2013) EndNote X7. Thomson Reuters, Philadelphia

    Google Scholar 

  50. Dey AK (2001) Understanding and using context. Pers Ubiquit Comput 5(1):4–7

    Article  Google Scholar 

  51. Harman HH (1976) Modern factor analysis. University of Chicago Press

  52. Polit DF, Beck CT (2008) Nursing research: generating and assessing evidence for nursing practice. Lippincott Williams & Wilkins, Philadelphia

    Google Scholar 

  53. Croxton FE, Cowden DJ, Klein S, Prentice-Hall Inc, Englewood Cliffs NJ (1967) Applied General Statistics. J Am Stat Assoc 63(322):738

    Google Scholar 

  54. Geisler G, McArthur D, Giersch S (2001) Developing recommendation services for a digital library with uncertain and changing data. In: Proceedings of the 1st ACM/IEEE-CS joint conference on digital libraries. ACM

  55. De Giusti MR et al (2010) An ontology-based context aware system for selective dissemination of information in a digital library. arXiv preprint arXiv:1005.4008

  56. Torres R et al (2004) Enhancing digital libraries with TechLens+. In: Proceedings of the 4th ACM/IEEE-CS joint conference on digital libraries. ACM

  57. Gantner Z et al (2011) MyMediaLite: a free recommender system library. In: Proceedings of the fifth ACM conference on recommender systems. ACM

  58. Hwang S-Y, Hsiung W-C, Yang W-S (2003) A prototype WWW literature recommendation system for digital libraries. Online Inf Rev 27(3):169–182

    Article  Google Scholar 

  59. Sugiyama K, Kan MY (2010) Scholarly paper recommendation via user’s recent research interests. In: Proceedings of the 10th annual joint conference on digital libraries. ACM

  60. Wang CY et al (2004) Extending e-books with contextual knowledge recommenders by analyzing personal portfolio and annotation to help learners solve problems in time. In: Proceedings of IEEE international conference on advanced learning technologies. IEEE

  61. Wang F-H, Shao H-M (2004) Effective personalized recommendation based on time-framed navigation clustering and association mining. Expert Syst Appl 27(3):365–377

    Article  Google Scholar 

  62. Konstan JA et al (2005) Techlens: exploring the use of recommenders to support users of digital libraries. In: CNI fall task force meeting project briefing. Coalition for networked information, Phoenix

  63. Liao I-E et al (2010) A library recommender system based on a personal ontology model and collaborative filtering technique for English collections. Electron Lib 28(3):386–400

    Article  Google Scholar 

  64. Wu D et al (2012) Temporal social tagging based collaborative filtering recommender for digital library. In: The outreach of digital libraries: a globalized resource network. Springer, pp 199–208

  65. Yuan Z, Yu T, Zhang J (2011) A social tagging based collaborative filtering recommendation algorithm for digital library. In: Digital libraries: for cultural heritage, knowledge dissemination, and future creation. Springer, pp 192–201

  66. Trujillo M, Millan M, Ortiz E (2007) A recommender system based on multi-features. In: Computational science and its applications—ICCSA 2007. Springer, pp 370–382

  67. He Q et al (2010) Context-aware citation recommendation. In: Proceedings of the 19th international conference on world wide web. ACM

  68. Porcel C, Moreno JM, Herrera-Viedma E (2009) A multi-disciplinar recommender system to advice research resources in university digital libraries. Expert Syst Appl 36(10):12520–12528

    Article  Google Scholar 

  69. Porcel C, Herrera-Viedma E (2010) Dealing with incomplete information in a fuzzy linguistic recommender system to disseminate information in university digital libraries. Knowl Based Syst 23(1):32–39

    Article  Google Scholar 

  70. Sun Y, Ni W, Men R (2009) A personalized paper recommendation approach based on web paper mining and reviewer’s interest modeling. In: ICRCCS’09 international conference on research challenges in computer science. IEEE

  71. Morales-del-Castillo JM, Peis E, Herrera-Viedma E (2009) A filtering and recommender system prototype for scholarly users of digital libraries. Springer, New York

    Book  Google Scholar 

  72. Hwang S-Y, Wei C-P, Liao Y-F (2010) Coauthorship networks and academic literature recommendation. Electron Commer Res Appl 9(4):323–334

    Article  Google Scholar 

  73. Rocha LM (2001) TalkMine: a soft computing approach to adaptive knowledge recommendation. In: Soft computing agents. Springer, pp 89–116

  74. Rao KN, Talwar VG (2011) Content-based document recommender system for aerospace grey literature: system design. DESIDOC J Lib Inf Technol. https://doi.org/10.14429/djlit.31.3.1046

  75. Pagonis J, Clark AF (2010) Engene: a genetic algorithm classifier for content-based recommender systems that does not require continuous user feedback. In: 2010 UK workshop on computational intelligence (UKCI)

  76. Serrano-Guerrero J et al (2011) A google wave-based fuzzy recommender system to disseminate information in University Digital Libraries 2.0. Inf Sci 181(9):1503–1516

    Article  Google Scholar 

  77. Will T et al (2009) Search personalization: Knowledge-based recommendation in digital libraries. In: AMCIS 2009 proceedings, p 728

  78. Tsai C-S, Chen M-Y (2008) Using adaptive resonance theory and data-mining techniques for materials recommendation based on the e-library environment. Electron Lib 26(3):287–302

    Article  Google Scholar 

  79. Rodriguez MA et al (2009) A recommender system to support the scholarly communication process. arXiv preprint arXiv:0905.1594

  80. Middleton SE, Shadbolt NR, De Roure DC (2004) Ontological user profiling in recommender systems. ACM Trans Inf Syst 22(1):54–88

    Article  Google Scholar 

  81. Amini B et al (2011) Incorporating scholar’s background knowledge into recommender system for digital libraries. In: 5th Malaysian conference in software engineering (MySEC). IEEE

  82. Lopes GR et al (2008) A personalized recommender system for digital libraries. In: Proceedings of the 14th Brazilian symposium on multimedia and the web. ACM

  83. Gipp B, Beel J, Hentschel C (2009) Scienstein: a research paper recommender system. In: Proceedings of the international conference on emerging trends in computing (ICETiC’09)

  84. Vellino A, Zeber D (2007) A hybrid, multi-dimensional recommender for journal articles in a scientific digital library. In: Proceedings of the 2007 IEEE/WIC/ACM international conference on web intelligence and international conference on intelligent agent technology

  85. McNee SM, Riedl J, Konstan JA (2006) Making recommendations better: an analytic model for human-recommender interaction. In: CHI’06 extended abstracts on human factors in computing systems. ACM

  86. Pham MC et al (2011) A clustering approach for collaborative filtering recommendation using social network analysis. J UCS 17(4):583–604

    Google Scholar 

  87. Yang Y, Yun L (2010) Literature recommendation based on reference graph. In: 3rd International conference on advanced computer theory and engineering (ICACTE). IEEE

  88. Yang W-S, Lin Y-R (2013) A task-focused literature recommender system for digital libraries. Online Inf Rev 37(4):581–601

    Article  MathSciNet  Google Scholar 

  89. Patton RM, Potok TE, Worley BA (2012) Discovery & refinement of scientific information via a recommender system. In: The second international conference on advanced communications and computation

  90. Herlocker J, Jung S, Webster JG (2012) Collaborative filtering for digital libraries

  91. Nakagawa A, Ito T (2002) An implementation of a knowledge recommendation system based on similarity among users’ profiles. In: Proceedings of the 41st SICE annual conference on SICE 2002. IEEE

  92. Renda ME, Straccia U (2005) A personalized collaborative digital library environment: a model and an application. Inf Process Manag 41(1):5–21

    Article  MATH  Google Scholar 

  93. Webster J, Jung S, Herlocker J (2004) Collaborative filtering: a new approach to searching digital libraries. N Rev Inf Netw 10(2):177–191

    Article  Google Scholar 

  94. Tejeda-Lorente Á et al (2014) A quality based recommender system to disseminate information in a university digital library. Inf Sci 261:52–69

    Article  Google Scholar 

  95. Mcnee SM (2006) Meeting user information needs in recommender systems. Doctoral degree. University of Minnesota

  96. Pennock DM et al (2000) Collaborative filtering by personality diagnosis: a hybrid memory-and model-based approach. In: Proceedings of the sixteenth conference on uncertainty in artificial intelligence. Morgan Kaufmann Publishers Inc

  97. Steinberg RM et al (2010) SmartSearch: automated recommendations using librarian expertise and the National Center for Biotechnology Information’s Entrez Programming Utilities. J Med Lib Assoc 98(2):171

    Article  Google Scholar 

  98. Franke M, Geyer-Schulz A, Neumann AW (2008) Recommender services in scientific digital libraries. In: Multimedia services in intelligent environments. Springer, pp 377–417

  99. Zhang M, Wang W, Li X (2008) A paper recommender for scientific literatures based on semantic concept similarity. In: Digital libraries: universal and ubiquitous access to information. Springer, pp 359–362

  100. Kang J, Choi J (2011) An ontology-based recommendation system using long-term and short-term preferences. In: International conference on information science and applications (ICISA). IEEE

  101. Wakeling S (2012) The user-centered design of a recommender system for a universal library catalogue. In: Proceedings of the sixth ACM conference on recommender systems. ACM

  102. De Nart D, Ferrara F, Tasso C (2013) Personalized access to scientific publications: from recommendation to explanation. In: User modeling, adaptation, and personalization. Springer, pp 296–301

  103. Li QC, Dong ZH, Li T (2008) Research of information recommendation system based on reading behavior. In: International conference on machine learning and cybernetics. IEEE

  104. Kuo JJ, Zhang YJ (2012) A library recommender system using interest change over time and matrix clustering. In: The outreach of digital libraries: a globalized resource network. Springer, pp 259–268

  105. Tsuji K et al (2012) Use of library loan records for book recommendation. In: IIAI international conference on advanced applied informatics (IIAIAAI). IEEE

  106. Herlocker JL, Konstan J (2001) Content-independent task-focused recommendation. Internet Comput IEEE 5(6):40–47

    Article  Google Scholar 

  107. Chandrasekaran K et al (2008) Concept-based document recommendations for citeseer authors. In: Adaptive hypermedia and adaptive web-based systems. Springer

  108. Whitney C, Schiff LR (2006) The Melvyl recommender project: developing library recommendation services. California Digital Library, Oakland

    Book  Google Scholar 

  109. Aittola M, Ryhänen T, Ojala T (2003) SmartLibrary—location-aware mobile library service, in human–computer interaction with mobile devices and services. Springer, pp 411–416

  110. Zarrinkalam F, Kahani M (2013) SemCiR: a citation recommendation system based on a novel semantic distance measure. Program 47(1):92–112

    Article  Google Scholar 

  111. Middleton SE, Alani H, De Roure DC (2002) Exploiting synergy between ontologies and recommender systems. arXiv preprint cs/0204012

  112. Middleton SE, Shadbolt NR, De Roure DC (2003) Capturing interest through inference and visualization: ontological user profiling in recommender systems. In: Proceedings of the 2nd international conference on knowledge capture. ACM

  113. Geyer-Schulz A, Neumann A, Thede A (2003) Others also use: a robust recommender system for scientific libraries. In: Research and advanced technology for digital libraries. Springer, pp 113–125

  114. Jung S et al (2004) SERF: integrating human recommendations with search. In: Proceedings of the thirteenth ACM international conference on information and knowledge management. ACM

  115. Amini B, Ibrahim R, Othman MS (2013) Data sets for offline evaluation of scholar’s recommender system. In: Intelligent information and database systems. Springer, pp 158–167

  116. McNee SM et al (2002) On the recommending of citations for research papers. In: Proceedings of the 2002 ACM conference on computer supported cooperative work. ACM

  117. Konstan JA, Riedl J (1999) Research resources for recommender systems. In: CHI’99 workshop interacting with recommender systems

  118. Beel J et al (2016) Towards reproducibility in recommender-systems research. User Model User Adapt Interact 26(1):69–101

    Article  Google Scholar 

  119. del Olmo FH, Gaudioso E (2008) Evaluation of recommender systems: a new approach. Expert Syst Appl 35(3):790–804

    Article  Google Scholar 

  120. Pu P, Chen L, Hu R (2011) A user-centric evaluation framework for recommender systems. In: Proceedings of the fifth ACM conference on recommender systems. ACM

  121. Hayes C, Cunningham P (2002) An on-line evaluation framework for recommender systems. Trinity College Dublin, Department of Computer Science, Dublin

    Google Scholar 

  122. Hanson EM (2014) A beginner’s guide to creating library linked data: lessons from NCSU’s organization name linked data project. Ser Rev 40(4):251–258

    Article  Google Scholar 

  123. Figueroa C, Vagliano I, Rocha OR, Morisio M (2015) A systematic literature review of linked data-based recommender systems. Concurr Comput Pract Exp 27(17):4659–4684

    Article  Google Scholar 

  124. Cremonesi P, Garzotto F, Turrin R (2013) User-centric versus system-centric evaluation of recommender systems. In: Human–Computer interaction–INTERACT 2013. Springer, pp 334–351

  125. Levy M (2013) Offline evaluation of recommender systems: all pain and no gain? In: Proceedings of the international workshop on reproducibility and replication in recommender systems evaluation. ACM

  126. Mobasher B et al (2001) Effective personalization based on association rule discovery from web usage data. In: Proceedings of the 3rd international workshop on web information and data management. ACM

  127. Sinha R, Swearingen K (2002) The role of transparency in recommender systems. In: CHI’02 extended abstracts on human factors in computing systems. ACM

  128. Wu W, He L, Yang J (2012) Evaluating recommender systems. In: Seventh international conference on digital information management (ICDIM). IEEE

  129. Powers D (2007) Evaluation: from precision, recall and F-factor to ROC, informedness, markedness & correlation (Technical Report). Adelaide, Australia

  130. Iaquinta L et al (2008) Introducing serendipity in a content-based recommender system. In: Hybrid intelligent systems. HIS’08 Eighth international conference. IEEE

  131. Kotkov D, Wang S, Veijalainen J (2016) A survey of serendipity in recommender systems. Knowl Based Syst 111:180–192

    Article  Google Scholar 

  132. Konstan JA, Riedl J (2012) Recommender systems: from algorithms to user experience. User Model User Adapt Interact 22(1):101–123

    Article  Google Scholar 

  133. Ozok AA, Fan Q, Norcio AF (2010) Design guidelines for effective recommender system interfaces based on a usability criteria conceptual model: results from a college student population. Behav Inf Technol 29(1):57–83

    Article  Google Scholar 

  134. Hiesel P et al (2016) A user interface concept for context-aware recommender systems. Mensch und Computer, Tagungsband

  135. di Sciascio C (2017) Advanced user interfaces and hybrid recommendations for exploratory search. In: Proceedings of the 22nd iaaanternational conference on intelligent user interfaces companion. ACM

  136. Calero Valdez A, Ziefle M, Verbert K (2016) HCI for recommender systems: the past, the present and the future. In: Proceedings of the 10th ACM conference on recommender systems. ACM

Download references

Author information

Authors and Affiliations

  1. Department of Software Engineering, Faculty of Computer Science and Information Technology, University of Malaya, 50603, Lembah Pantai, Kuala Lumpur, Malaysia

    Zohreh Dehghani Champiri & Salim Siti Salwah Binti

  2. Safahan Institute of Higher Education, Isfahan, Iran

    Adeleh Asemi

Authors
  1. Zohreh Dehghani Champiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adeleh Asemi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Salim Siti Salwah Binti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Zohreh Dehghani Champiri, Adeleh Asemi or Salim Siti Salwah Binti.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dehghani Champiri, Z., Asemi, A. & Siti Salwah Binti, S. Meta-analysis of evaluation methods and metrics used in context-aware scholarly recommender systems. Knowl Inf Syst 61, 1147–1178 (2019). https://doi.org/10.1007/s10115-018-1324-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-018-1324-5

Keywords

Search

Navigation