Skip to main content
Log in

A systematic literature review of multicriteria recommender systems

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Since the first years of the 90s, recommender systems have emerged as effective tools for automatically selecting items according to user preferences. Traditional recommenders rely on the relevance assessments that users express using a single rating for each item. However, some authors started to suggest that this approach could be limited, as we naturally tend to formulate different judgments according to multiple criteria. During the last decade, several studies introduced novel recommender systems capable of exploiting user preferences expressed over multiple criteria. This work proposes a systematic literature review in the field of multicriteria recommender systems. Following a replicable protocol, we selected a total number of 93 studies dealing with this topic. We subsequently analyzed them to provide an answer to five different research questions. We considered what are the most common research problems, recommendation approaches, data mining and machine learning algorithms mentioned in these studies. Furthermore, we investigated the domains of application, the exploited evaluation protocols, metrics and datasets, and the most promising suggestions for future works.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://www.qsrinternational.com/nvivo.

  2. Please note that the results for the year 2019 are not available, as the selection was performed in January 2019.

  3. https://webscope.sandbox.yahoo.com.

  4. https://www.cs.cmu.edu/~jiweil/html/hotel-review.html.

  5. http://www.cs.virginia.edu/~hw5x/Data/LARA/TripAdvisor.

  6. https://www.tripadvisor.com.

  7. https://www.bcnrestaurantes.com.

References

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Diego Monti.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix: Selected studies

Appendix: Selected studies

Code

Author

Title

Year

Publication

Source

P1

Liu, L.; Mehandjiev, N.; Xu, D.-L.

Multi-criteria service recommendation based on user criteria preferences

2011

Fifth ACM Conference on Recommender Systems

ACM Digital Library

P2

Shambour, Q.; Lu, J.

A hybrid multi-criteria semantic-enhanced collaborative filtering approach for personalized recommendations

2011

International Conferences on Web Intelligence and Intelligent Agent Technology

ACM Digital Library

P3

Jannach, D.; Karakaya, Z.; Gedikli, F.

Accuracy improvements for multi-criteria recommender systems

2012

13th ACM Conference on Electronic Commerce

ACM Digital Library

P4

Hdioud, F.; Frikh, B.; Ouhbi, B.

Multi-criteria recommender systems based on multi-attribute decision making

2013

International Conference on Information Integration and Web-based Applications & Services

ACM Digital Library

P5

Choudhary, P.; Kant, V.; Dwivedi, P.

A particle swarm optimization approach to multi criteria recommender system utilizing effective similarity measures

2017

9th International Conference on Machine Learning and Computing

ACM Digital Library

P6

Musto, C.; de Gemmis, M.; Semeraro, G.; Lops, P.

A multi-criteria recommender system exploiting aspect-based sentiment analysis of users’ reviews

2017

Eleventh ACM Conference on Recommender Systems

ACM Digital Library

P7

Park, Y.

Recommending personalized tips on new courses for guiding course selection

2017

South-East Conference

ACM Digital Library

P8

Sreepada, R. S.; Patra, B. K.; Hernando, A.

Multi-criteria recommendations through preference learning

2017

Fourth ACM IKDD Conferences on Data Sciences

ACM Digital Library

P9

Zheng, Y.

Criteria chains: A novel multi-criteria recommendation approach

2017

22nd International Conference on Intelligent User Interfaces

ACM Digital Library

P10

Tallapally, D.; Sreepada, R. S.; Patra, B. K.; Babu, K. S.

User preference learning in multi-criteria recommendations using stacked auto encoders

2018

12th ACM Conference on Recommender Systems

ACM Digital Library

P11

Zheng, Y.; Dave, T.; Mishra, N.; Kumar, H.

Fairness in reciprocal recommendations

2018

26th Conference on User Modeling, Adaptation and Personalization

ACM Digital Library

P12

Niknafs, A.; Charkari, N. M.; Niknafs, A. A.

PROMETHEE-based recommender system for multi-sort recommendations in on-line stores

2008

Third International Conference on Digital Information Management

IEEE Xplore

P13

Hwang, C.-S.; Kao, Y.-C.; Yu, P.

Integrating multiple linear regression and multicriteria collaborative filtering for better recommendation

2010

International Conference on Computational Aspects of Social Networks

IEEE Xplore

P14

Liu, L.; Lecue, F.; Mehandjiev, N.; Xu, L.

Using context similarity for service recommendation

2010

IEEE Fourth International Conference on Semantic Computing

IEEE Xplore

P15

Shambour, Q.; Lu, J.

A framework of hybrid recommendation system for government-to-business personalized e-services

2010

Seventh International Conference on Information Technology: New Generations

IEEE Xplore

P16

Zarrinkalam, F.; Kahani, M.

A multi-criteria hybrid citation recommendation system based on linked data

2012

2nd International eConference on Computer and Knowledge Engineering

IEEE Xplore

P17

Boulkrinat, S.; Hadjali, A.; Mokhtari, A.

Enhancing recommender systems prediction through qualitative preference relations

2013

11th International Symposium on Programming and Systems

IEEE Xplore

P18

Samatthiyadikun, P.; Takasu, A.; Maneeroj, S.

Bayesian model for a multicriteria recommender system with support vector regression

2013

IEEE 14th International Conference on Information Reuse & Integration

IEEE Xplore

P19

Bokde, D. K.; Girase, S.; Mukhopadhyay, D.

An approach to a university recommendation by multi-criteria collaborative filtering and dimensionality reduction techniques

2015

IEEE International Symposium on Nanoelectronic and Information Systems

IEEE Xplore

P20

Sharma, Y.; Bhatt, J.; Magon, R.

A multi-criteria review-based hotel recommendation system

2015

IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing

IEEE Xplore

P21

Asawarangsee, T.; Maneeroj, S.

A novel aggregation technique for multi-criteria recommendation

2016

13th International Joint Conference on Computer Science and Software Engineering

IEEE Xplore

P22

Ashley-Dejo, E.; Ngwira, S. M.; Zuva, T.

A context-aware proactive recommender system for tourist

2016

International Conference on Advances in Computing and Communication Engineering

IEEE Xplore

P23

Hassan, M.; Hamada, M.

Enhancing learning objects recommendation using multi-criteria recommender systems

2016

IEEE International Conference on Teaching, Assessment, and Learning for Engineering

IEEE Xplore

P24

Wijayanto, A.; Winarko, E.

Implementation of multi-criteria collaborative filtering on cluster using Apache Spark

2016

2nd International Conference on Science and Technology-Computer

IEEE Xplore

P25

Hamada, M.; Odu, N. B.; Hassan, M.

A fuzzy-based approach for modelling preferences of users in multi-criteria recommender systems

2018

IEEE 12th International Symposium on Embedded Multicore/Many-core Systems-on-Chip

IEEE Xplore

P26

Mohamed, H.; Abdulsalam, L.; Mohammed, H.

Adaptive genetic algorithm for improving prediction accuracy of a multi-criteria recommender system

2018

IEEE 12th International Symposium on Embedded Multicore/Many-core Systems-on-Chip

IEEE Xplore

P27

Turk, A. M.; Bilge, A.

A robust multi-criteria collaborative filtering algorithm

2018

Innovations in Intelligent Systems and Applications

IEEE Xplore

P28

Manouselis, N.; Costopoulou, C.

Experimental analysis of design choices in multiattribute utility collaborative filtering

2007

International Journal of Pattern Recognition and Artificial Intelligence

ISI Web of Knowledge

P29

Yin, Z.; Yueting, Z.; Jiangqin, W.; Liang, Z.

Applying probabilistic latent semantic analysis to multi-criteria recommender system

2009

AI Communications

ISI Web of Knowledge

P30

Palanivel, K.; Sivakumar, R.

A study on implicit feedback in multicriteria e-commerce recommender system

2010

Journal of Electronic Commerce Research

ISI Web of Knowledge

P31

Dixit, V. S.; Mehta, H.; Bedi, P.

A proposed framework for group-based multi-criteria recommendations

2014

Applied Artificial Intelligence

ISI Web of Knowledge

P32

Mikeli, A.; Apostolou, D.; Despotis, D.

A new recommendation technique for interval scaled multi-criteria rating systems incorporating intensity of preferences

2015

Intelligent Decision Technologies

ISI Web of Knowledge

P33

Hamada, M.; Hassan, M.

Artificial neural networks and particle swarm optimization algorithms for preference prediction in multi-criteria recommender systems

2018

Informatics

ISI Web of Knowledge

P34

Chen, D.-N.; Hu, P. J.-H.; Kuo, Y.-R.; Liang, T.-P.

A web-based personalized recommendation system for mobile phone selection: Design, implementation, and evaluation

2010

Expert Systems with Applications

ScienceDirect

P35

Huang, S.-l.

Designing utility-based recommender systems for e-commerce: Evaluation of preference-elicitation methods

2011

Electronic Commerce Research and Applications

ScienceDirect

P36

Liu, H.; He, J.; Wang, T.; Song, W.; Du, X.

Combining user preferences and user opinions for accurate recommendation

2013

Electronic Commerce Research and Applications

ScienceDirect

P37

Hu, Y.-C.

Nonadditive similarity-based single-layer perceptron for multi-criteria collaborative filtering

2014

Neurocomputing

ScienceDirect

P38

Li, Y.-M.; Chou, C.-L.; Lin, L.-F.

A social recommender mechanism for location-based group commerce

2014

Information Sciences

ScienceDirect

P39

Marin, L.; Moreno, A.; Isern, D.

Automatic preference learning on numeric and multi-valued categorical attributes

2014

Knowledge-Based Systems

ScienceDirect

P40

Nilashi, M.; bin Ibrahim, O.; Ithnin, N.

Hybrid recommendation approaches for multi-criteria collaborative filtering

2014

Expert Systems with Applications

ScienceDirect

P41

Son, L. H.; Thong, N. T.

Intuitionistic fuzzy recommender systems: An effective tool for medical diagnosis

2015

Knowledge-Based Systems

ScienceDirect

P42

Ali, M.; Son, L. H.; Thanh, N. D.; Minh, N. V.

A neutrosophic recommender system for medical diagnosis based on algebraic neutrosophic measures

2017

Applied Soft Computing

ScienceDirect

P43

Amoretti, M.; Belli, L.; Zanichelli, F.

UTravel: Smart mobility with a novel user profiling and recommendation approach

2017

Pervasive and Mobile Computing

ScienceDirect

P44

Choudhary, P.; Kant, V.; Dwivedi, P.

Handling natural noise in multi criteria recommender system utilizing effective similarity measure and particle swarm optimization

2017

Procedia Computer Science

ScienceDirect

P45

Kermany, N. R.; Alizadeh, S. H.

A hybrid multi-criteria recommender system using ontology and neuro-fuzzy techniques

2017

Electronic Commerce Research and Applications

ScienceDirect

P46

Yuen, K. K. F.

The fuzzy cognitive pairwise comparisons for ranking and grade clustering to build a recommender system: An application of smartphone recommendation

2017

Engineering Applications of Artificial Intelligence

ScienceDirect

P47

Akcayol, M. A.; Utku, A.; Aydoğan, E.; Mutlu, B.

A weighted multi-attribute-based recommender system using extended user behavior analysis

2018

Electronic Commerce Research and Applications

ScienceDirect

P48

Castillo, A.; Meer, D. V.; Castellanos, A.

ExUP recommendations: Inferring user’s product metadata preferences from single-criterion rating systems

2018

Decision Support Systems

ScienceDirect

P49

Nilashi, M.; Ibrahim, O.; Yadegaridehkordi, E.; Samad, S.; Akbari, E.; Alizadeh, A.

Travelers decision making using online review in social network sites: A case on TripAdvisor

2018

Journal of Computational Science

ScienceDirect

P50

Núñez-Valdez, E. R.; Quintana, D.; Crespo, R. G.; Isasi, P.; Herrera-Viedma, E.

A recommender system based on implicit feedback for selective dissemination of ebooks

2018

Information Sciences

ScienceDirect

P51

Song, W.; Sakao, T.

An environmentally conscious PSS recommendation method based on users’ vague ratings: A rough multi-criteria approach

2018

Journal of Cleaner Production

ScienceDirect

P52

Wasid, M.; Ali, R.

An improved recommender system based on multi-criteria clustering approach

2018

Procedia Computer Science

ScienceDirect

P53

Adomavicius, G.; Kwon, Y.

New recommendation techniques for multicriteria rating systems

2007

IEEE Intelligent Systems

Scopus

P54

Lee, H.-H.; Teng, W.-G.

Incorporating multi-criteria ratings in recommendation systems

2007

IEEE International Conference on Information Reuse and Integration

Scopus

P55

Hwang, C.-S.

Genetic algorithms for feature weighting in multi-criteria recommender systems

2010

Journal of Convergence Information Technology

Scopus

P56

Lousame, F. P.; Sanchez, E.

Multicriteria predictors using aggregation functions based on item views

2010

10th International Conference on Intelligent Systems Design and Applications

Scopus

P57

Tangphoklang, P.; Maneeroj, S.; Takasu, A.

Advanced representative and dynamic user profile based on MCDM for multi-criteria RS

2010

IADIS International Conference Information Systems

Scopus

P58

Akhtarzada, A.; Calude, C. S.; Hosking, J.

A multi-criteria metric algorithm for recommender systems

2011

Fundamenta Informaticae

Scopus

P59

Lakiotaki, K.; Matsatsinis, N. F.; Tsoukias, A.

Multicriteria user modeling in recommender systems

2011

IEEE Intelligent Systems

Scopus

P60

Palanivel, K.; Sivakumar, R.

A study on collaborative recommender system using fuzzy-multicriteria approaches

2011

International Journal of Business Information Systems

Scopus

P61

Shambour, Q.; Lu, J.

Integrating multi-criteria collaborative filtering and trust filtering for personalized recommender systems

2011

IEEE Symposium on Computational Intelligence in Multicriteria Decision-Making

Scopus

P62

Samatthiyadikun, P.; Takasu, A.; Maneeroj, S.

Multicriteria collaborative filtering by Bayesian model-based user profiling

2012

IEEE 13th International Conference on Information Reuse & Integration

Scopus

P63

Premchaiswadi, W.; Poompuang, P.

Hybrid profiling for hybrid multicriteria recommendation based on implicit multicriteria information

2013

Applied Artificial Intelligence

Scopus

P64

Agathokleous, M.; Tsapatsoulis, N.

Learning user models in multi-criteria recommender systems

2014

Engineering Applications of Neural Networks

Scopus

P65

Bilge, A.; Kaleli, C.

A multi-criteria item-based collaborative filtering framework

2014

11th International Joint Conference on Computer Science and Software Engineering

Scopus

P66

Hdioud, F.; Frikh, B.; Ouhbi, B.

Bootstrapping recommender systems based on a multi-criteria decision making approach

2014

International Conference on Next Generation Networks and Services

Scopus

P67

Hu, Y.-C.

A multicriteria collaborative filtering approach using the indifference relation and its application to initiator recommendation for group-buying

2014

Applied Artificial Intelligence

Scopus

P68

Manouselis, N.; Kyrgiazos, G.; Stoitsis, G.

Exploratory study of multi-criteria recommendation algorithms over technology enhanced learning datasets

2014

Journal of E-Learning and Knowledge Society

Scopus

P69

Pinandito, A.; Ananta, M. T.; Brata, K. C.; Fanani, L.

Alternatives weighting in analytic hierarchy process of mobile culinary recommendation system using fuzzy

2015

ARPN Journal of Engineering and Applied Sciences

Scopus

P70

Sneha, Y. S.; Mahadevan, G.

A novel approach to personalized recommender systems based on multi criteria ratings

2015

Research Journal of Applied Sciences, Engineering and Technology

Scopus

P71

Bankshinategh, B.; Spanakis, G.; Zaiane, O.; ElAtia, S.

A course recommender system based on graduating attributes

2017

9th International Conference on Computer Supported Education

Scopus

P72

Goswami, A.; Dwivedi, P.; Kant, V.

Trust-enhanced multi-criteria recommender system

2017

Advances in Intelligent Systems and Computing

Scopus

P73

Hassan, M.; Hamada, M.

A neural networks approach for improving the accuracy of multi-criteria recommender systems

2017

Applied Sciences

Scopus

P74

Leal, F.; González-Vélez, H.; Malheiro, B.; Burguillo, J. C.

Profiling and rating prediction from multi-criteria crowd-sourced hotel ratings

2017

31st European Conference on Modelling and Simulation

Scopus

P75

Majumder, G. S.; Dwivedi, P.; Kant, V.

Matrix factorization and regression-based approach for multi-criteria recommender system

2017

Information and Communication Technology for Intelligent Systems

Scopus

P76

Karacapilidis, N.; Hatzieleftheriou, L.

Exploiting similarity measures in multi-criteria based recommendations

2003

E-Commerce and Web Technologies

Springer Link

P77

Manouselis, N.; Costopoulou, C.

Preliminary study of the expected performance of MAUT collaborative filtering algorithms

2008

The Open Knowlege Society

Springer Link

P78

Matsatsinis, N. F.; Manarolis, E. A.

New hybrid recommender approaches: An application to equity funds selection

2009

Algorithmic Decision Theory

Springer Link

P79

Naak, A.; Hage, H.; Aïmeur, E.

A multi-criteria collaborative filtering approach for research paper recommendation in Papyres

2009

E-Technologies: Innovation in an Open World

Springer Link

P80

Bitonto, P. D.; Laterza, M.; Roselli, T.; Rossano, V.

Multi-criteria retrieval in cultural heritage recommendation systems

2010

Knowledge-Based and Intelligent Information and Engineering Systems

Springer Link

P81

Maneeroj, S.; Samatthiyadikun, P.; Chalermpornpong, W.; Panthuwadeethorn, S.; Takasu, A.

Ranked criteria profile for multi-criteria rating recommender

2012

Information Systems, Technology and Management

Springer Link

P82

Fan, J.; Xu, L.

A robust multi-criteria recommendation approach with preference-based similarity and support vector machine

2013

Advances in Neural Networks

Springer Link

P83

Jannach, D.; Zanker, M.; Fuchs, M.

Leveraging multi-criteria customer feedback for satisfaction analysis and improved recommendations

2014

Information Technology & Tourism

Springer Link

P84

Nilashi, M.; Ibrahim, O. B.; Ithnin, N.; Zakaria, R.

A multi-criteria recommendation system using dimensionality reduction and neuro-fuzzy techniques

2014

Soft Computing

Springer Link

P85

Chen, T.; Chuang, Y. H.

Fuzzy and nonlinear programming approach for optimizing the performance of ubiquitous hotel recommendation

2015

Journal of Ambient Intelligence and Humanized Computing

Springer Link

P86

Li, S. T.; Pham, T. T.; Chuang, H. C.; Wang, Z.-W.

Does reliable information matter? Towards a trustworthy co-created recommendation model by mining unboxing reviews

2015

Information Systems and e-Business Management

Springer Link

P87

Parveen, R.; Kant, V.; Dwivedi, P.; Jaiswal, A. K.

Enhancing recommendation quality of a multi criterion recommender system using genetic algorithm

2015

Mining Intelligence and Knowledge Exploration

Springer Link

P88

Jhalani, T.; Kant, V.; Dwivedi, P.

A linear regression approach to multi-criteria recommender system

2016

Data Mining and Big Data

Springer Link

P89

Kant, V.; Jhalani, T.; Dwivedi, P.

Enhanced multi-criteria recommender system based on fuzzy Bayesian approach

2017

Multimedia Tools and Applications

Springer Link

P90

Ko, H.-G.; Ko, I.-Y.; Lee, D.

Multi-criteria matrix localization and integration for personalized collaborative filtering in IoT environments

2017

Multimedia Tools and Applications

Springer Link

P91

Leal, F.; Malheiro, B.; González-Vélez, H.; Burguillo, J. C.

Trust-based modelling of multi-criteria crowdsourced data

2017

Data Science and Engineering

Springer Link

P92

Ding, Y.; Li, S.; Yu, W.

Multi-criteria recommendation schemes based on factorization machines

2018

Cluster Computing

Springer Link

P93

Ding, Y.; Li, S.; Yu, W.; Wang, J.; Liu, M.

A unified neural model for review-based rating prediction by leveraging multi-criteria ratings and review text

2018

Cluster Computing

Springer Link

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Monti, D., Rizzo, G. & Morisio, M. A systematic literature review of multicriteria recommender systems. Artif Intell Rev 54, 427–468 (2021). https://doi.org/10.1007/s10462-020-09851-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-020-09851-4

Keywords

Navigation