Abstract
In this paper we introduce a new ranking algorithm, called Collaborative Judgement (CJ), that takes into account peer opinions of agents and/or humans on objects (e.g. products, exams, papers) as well as peer judgements over those opinions. The combination of these two types of information has not been studied in previous work in order to produce object rankings. We apply CJ to the use case of scientific paper assessment and we validate it over simulated data. The results show that the rankings produced by our algorithm improve current scientific paper ranking practice based on averages of opinions weighted by their reviewers’ self-assessments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Charlin, L., Zemel, R.S., Boutilier, C.: A framework for optimizing paper matching. CoRR, abs/1202.3706 (2012)
Cormen, T.H., Stein, C., Rivest, R.L., Leiserson, C.E.: Introduction to Algorithms, 2nd edn. McGraw-Hill Higher Education (2001)
de Alfaro, L., Shavlovsky, M.: Crowdgrader: Crowdsourcing the evaluation of homework assignments. Thech. Report 1308.5273, arXiv.org (2013)
Fagin, R., Kumar, R., Mahdian, M., Sivakumar, D., Vee, E.: Comparing and aggregating rankings with ties. In: Proceedings of the Twenty-Third ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, PODS ’04, pp. 47–58. ACM, New York (2004)
Haque, M., Egerstedt, M., Rahmani, A.: Multilevel coalition formation strategy for suppression of enemy air defenses missions. Journal of Aerospace Information Systems 10(6), 287–296 (2013)
Kamvar, S.D., Schlosser, M.T., Garcia-Molina, H.: The eigentrust algorithm for reputation management in p2p networks. In: Proceedings of the 12th International Conference on World Wide Web, WWW ’03, pp. 640–651. ACM, New York (2003)
Lappas, T., Liu, K., Terzi, E.: Finding a team of experts in social networks. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’09, pp. 467–476. ACM, New York (2009)
Nair, R., Tambe, M., Marsella, S.C.: Team formation for reformation in multiagent domains like RoboCupRescue. In: Kaminka, G.A., Lima, P.U., Rojas, R. (eds.) RoboCup 2002. LNCS (LNAI), vol. 2752, pp. 150–161. Springer, Heidelberg (2003)
Osman, N., Gutierrez, P., Sierra, C.: Trustworthy advice. Knowl.-Based Syst. 82, 41–59 (2015)
Osman, N., Sierra, C., McNeill, F., Pane, J., Debenham, J.K.: Trust and matching algorithms for selecting suitable agents. ACM TIST 5(1), 16 (2013)
Osman, N., Sierra, C., Sabater-Mir, J.: Propagation of opinions in structural graphs. In: Coelho, H., Studer, R., Wooldridge, M. (eds.) ECAI 2010–19th European Conference on Artificial Intelligence, Lisbon, Portugal, 16–20 August 2010. Frontiers in Artificial Intelligence and Applications, vol. 215, pp. 595–600. IOS Press (2010)
Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., Koller, D.: Tuned models of peer assessment in moocs. In: Proc. of the 6th International Conference on Educational Data Mining (EDM 2013) (2013)
Ramchurn, S.D., Farinelli, A., Macarthur, K.S., Jennings, N.R.: Decentralized coordination in robocup rescue. Comput. J. 53(9), 1447–1461 (2010)
Sierra, C., Debenham, J.K.: Trust and honour in information-based agency. In: Nakashima, H., Wellman, M.P., Weiss, G., Stone, P. (eds.) 5th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2006), Hakodate, Japan, 8–12 May 2006, pp. 1225–1232. ACM (2006)
Walsh, T.: The peerrank method for peer assessment. In: Schaub, T., Friedrich, G., O’Sullivan, B. (eds.) ECAI 2014–21st European Conference on Artificial Intelligence, 18–22 August 2014, Prague, Czech Republic - Including Prestigious Applications of Intelligent Systems (PAIS 2014). Frontiers in Artificial Intelligence and Applications, vol. 263, pp. 909–914. IOS Press (2014)
Wu, J., Chiclana, F., Herrera-Viedma, E.: Trust based consensus model for social network in an incompletelinguistic information context. Applied Soft Computing (2015)
Zhang, J., Ghorbani, A.A., Cohen, R.: A familiarity-based trust model for effective selection of sellers in multiagent e-commerce systems. Int. J. Inf. Sec. 6(5), 333–344 (2007)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Andrejczuk, E., Rodriguez-Aguilar, J.A., Sierra, C. (2015). Collaborative Judgement. In: Chen, Q., Torroni, P., Villata, S., Hsu, J., Omicini, A. (eds) PRIMA 2015: Principles and Practice of Multi-Agent Systems. PRIMA 2015. Lecture Notes in Computer Science(), vol 9387. Springer, Cham. https://doi.org/10.1007/978-3-319-25524-8_46
Download citation
DOI: https://doi.org/10.1007/978-3-319-25524-8_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-25523-1
Online ISBN: 978-3-319-25524-8
eBook Packages: Computer ScienceComputer Science (R0)