ABSTRACT
Given a social issue that needs to be solved, decision-makers need to listen to the crowd opinions and preferences. However, existing online voting systems with limited capabilities cannot conduct such investigations. Our idea is that decision-makers can collect many human opinions from crowds on the web and then prioritize them for social decision-making. A solution of the prioritization entails collecting a large amount of pairwise preference comparisons from crowds and utilizing the aggregated preference labels as the collective preferences on the opinions. In practice, because there is a large number of combinations of all candidate opinion pairs, we can only collect a small number of labels for a small subset of pairs. How to utilize only a small number of pairwise crowd preferences on the opinions to estimate collective preferences is the problem. Existing works on preference aggregation methods for general scenarios utilize only the pairwise preference labels. In our scenario, additional contextual information, such as the text contents of the opinions, can potentially promote the aggregation performance. Therefore, we propose preference aggregation approaches that can effectively incorporate contextual information by externally or internally building the relations between the opinion contexts and preference scores. We propose approaches for both the homogeneous and heterogeneous settings of modeling the evaluators. The experiments conducted on real datasets collected from real-world crowdsourcing platform show that our approaches can generate better aggregation results than the baselines for estimating collective preferences, especially when there are only a small number of preference labels available.
- Sameer Agarwal, Josh Wills, Lawrence Cayton, Gert Lanckriet, David Kriegman, and Serge Belongie. 2007. Generalized non-metric multidimensional scaling. In Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics (AISTATS). 11–18.Google Scholar
- Yukino Baba, Jiyi Li, and Hisashi Kashima. 2020. CrowDEA: Multi-View Idea Prioritization with Crowds. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP) 8, 1 (Oct. 2020), 23–32. https://ojs.aaai.org/index.php/HCOMP/article/view/7460Google ScholarCross Ref
- Roy Bar-Haim, Lilach Eden, Roni Friedman, Yoav Kantor, Dan Lahav, and Noam Slonim. 2020. From Arguments to Key Points: Towards Automatic Argument Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, Online, 4029–4039. https://doi.org/10.18653/v1/2020.acl-main.371Google ScholarCross Ref
- Alexander Bondarenko, Maik Fröbe, Meriem Beloucif, Lukas Gienapp, Yamen Ajjour, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Potthast, and Matthias Hagen. 2020. Overview of Touché 2020: Argument Retrieval. In Working Notes Papers of the CLEF 2020 Evaluation Labs(CEUR Workshop Proceedings, Vol. 2696). 22 pages. http://ceur-ws.org/Vol-2696/Google ScholarDigital Library
- Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39, 3/4 (1952), 324–345.Google ScholarCross Ref
- Manuela Cattelan. 2012. Models for paired comparison data: A review with emphasis on dependent data. Statist. Sci. (2012), 412–433.Google Scholar
- David Causeur and François Husson. 2005. A 2-dimensional extension of the Bradley–Terry model for paired comparisons. Journal of Statistical Planning and Inference 135, 2(2005), 245–259.Google ScholarCross Ref
- Shuo Chen and Thorsten Joachims. 2016. Modeling Intransitivity in Matchup and Comparison Data. In Proceedings of the 9th ACM International Conference on Web Search and Data Mining (WSDM). 227–236. https://doi.org/10.1145/2835776.2835787Google ScholarDigital Library
- Xi Chen, Paul N. Bennett, Kevyn Collins-Thompson, and Eric Horvitz. 2013. Pairwise Ranking Aggregation in a Crowdsourced Setting. In Proceedings of the 6th ACM International Conference on Web Search and Data Mining (WSDM). 193–202. https://doi.org/10.1145/2433396.2433420Google ScholarDigital Library
- Roger R Davidson. 1970. On extending the Bradley-Terry model to accommodate ties in paired comparison experiments. J. Amer. Statist. Assoc. 65, 329 (1970), 317–328.Google ScholarCross Ref
- Jiuding Duan, Jiyi Li, Yukino Baba, and Hisashi Kashima. 2017. A Generalized Model for Multidimensional Intransitivity. In Prceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD). 840–852.Google ScholarCross Ref
- Cynthia Dwork, Ravi Kumar, Moni Naor, and D. Sivakumar. 2001. Rank Aggregation Methods for the Web. In Proceedings of the 10th International Conference on World Wide Web (WWW). Association for Computing Machinery, New York, NY, USA, 613–622. https://doi.org/10.1145/371920.372165Google ScholarDigital Library
- Lukas Gienapp, Benno Stein, Matthias Hagen, and Martin Potthast. 2020. Efficient Pairwise Annotation of Argument Quality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, Online, 5772–5781. https://doi.org/10.18653/v1/2020.acl-main.511Google ScholarCross Ref
- Shai Gretz, Roni Friedman, Edo Cohen-Karlik, Assaf Toledo, Dan Lahav, Ranit Aharonov, and Noam Slonim. 2020. A Large-Scale Dataset for Argument Quality Ranking: Construction and Analysis. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) 34, 05 (Apr. 2020), 7805–7813. https://doi.org/10.1609/aaai.v34i05.6285Google ScholarCross Ref
- T.H. Haveliwala. 2003. Topic-sensitive PageRank: a context-sensitive ranking algorithm for Web search. IEEE Transactions on Knowledge and Data Engineering 15, 4(2003), 784–796. https://doi.org/10.1109/TKDE.2003.1208999Google ScholarDigital Library
- David R Hunter. 2004. MM algorithms for generalized Bradley-Terry models. Annals of Statistics(2004), 384–406.Google Scholar
- Tao Jin, Pan Xu, Quanquan Gu, and Farzad Farnoud. 2020. Rank Aggregation via Heterogeneous Thurstone Preference Models. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) 34, 04 (Apr. 2020), 4353–4360. https://doi.org/10.1609/aaai.v34i04.5860Google ScholarCross Ref
- Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining Human and Machine Intelligence in Large-Scale Crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1 (AAMAS). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 467–474.Google ScholarDigital Library
- Caitlin Kuhlman and Elke Rundensteiner. 2020. Rank Aggregation Algorithms for Fair Consensus. Proc. VLDB Endow. 13, 12 (Jul 2020), 2706–2719. https://doi.org/10.14778/3407790.3407855Google ScholarDigital Library
- Jiyi Li. 2020. Crowdsourced Text Sequence Aggregation Based on Hybrid Reliability and Representation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). Association for Computing Machinery, New York, NY, USA, 1761–1764. https://doi.org/10.1145/3397271.3401239Google ScholarDigital Library
- Jiyi Li, Yukino Baba, and Hisashi Kashima. 2017. Hyper Questions: Unsupervised Targeting of a Few Experts in Crowdsourcing. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM). 1069–1078. https://doi.org/10.1145/3132847.3132971Google ScholarDigital Library
- Jiyi Li, Yukino Baba, and Hisashi Kashima. 2018. Incorporating Worker Similarity for Label Aggregation in Crowdsourcing. In Proceedings of the 27th International Conference on Artificial Neural Networks (ICANN). 596–606. https://doi.org/10.1007/978-3-030-01421-6_57Google ScholarCross Ref
- Jiyi Li, Yukino Baba, and Hisashi Kashima. 2018. Simultaneous Clustering and Ranking from Pairwise Comparisons. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI). 1554–1560. https://doi.org/10.24963/ijcai.2018/215Google ScholarCross Ref
- Jiyi Li, Lucas Ryo Endo, and Hisashi Kashima. 2021. In Proceedings of the 28th International Conference on Neural Information Processing (ICONIP). Springer International Publishing, Cham, 176–185. https://doi.org/10.1007/978-3-030-92310-5_21Google ScholarCross Ref
- Jiyi Li and Fumiyo Fukumoto. 2019. A Dataset of Crowdsourced Word Sequences: Collections and Answer Aggregation for Ground Truth Creation. In Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP. 24–28. https://aclanthology.org/D19-5904Google ScholarCross Ref
- Jiyi Li, Yasushi Kawase, Yukino Baba, and Hisashi Kashima. 2020. Performance as a Constraint: An Improved Wisdom of Crowds Using Performance Regularization. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI), Christian Bessiere (Ed.). 1534–1541. https://doi.org/10.24963/ijcai.2020/213 Main track.Google ScholarCross Ref
- R Duncan Luce. 2012. Individual choice behavior: A theoretical analysis. Courier Corporation.Google Scholar
- Sahand Negahban, Sewoong Oh, and Devavrat Shah. 2017. Rank Centrality: Ranking from Pairwise Comparisons. Oper. Res. 65, 1 (Feb. 2017), 266–287. https://doi.org/10.1287/opre.2016.1534Google ScholarDigital Library
- Sewoong Oh, Kiran K Thekumparampil, and Jiaming Xu. 2015. Collaboratively Learning Preferences from Ordinal Data. In Advances in Neural Information Processing Systems 28 (NIPS). 1909–1917.Google Scholar
- Peter Potash, Adam Ferguson, and Timothy J. Hazen. 2019. Ranking Passages for Argument Convincingness. In Proceedings of the 6th Workshop on Argument Mining. Association for Computational Linguistics, Florence, Italy, 146–155. https://doi.org/10.18653/v1/W19-4517Google ScholarCross Ref
- Karthik Raman and Thorsten Joachims. 2014. Methods for Ordinal Peer Grading. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 1037–1046. https://doi.org/10.1145/2623330.2623654Google ScholarDigital Library
- Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Hong Kong, China, 3982–3992. https://doi.org/10.18653/v1/D19-1410Google ScholarCross Ref
- Yoshihiko Suhara, Xiaolan Wang, Stefanos Angelidis, and Wang-Chiew Tan. 2020. OpinionDigest: A Simple Framework for Opinion Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, Online, 5789–5798. https://doi.org/10.18653/v1/2020.acl-main.513Google ScholarCross Ref
- Omer Tamuz, Ce Liu, Serge Belongie, Ohad Shamir, and Adam Tauman Kalai. 2011. Adaptively Learning the Crowd Kernel. In Proceedings of the 28th International Conference on International Conference on Machine Learning (ICML). 673–680.Google Scholar
- Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, and Feng Wu. 2020. SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, Online, 4067–4076. https://doi.org/10.18653/v1/2020.acl-main.374Google ScholarCross Ref
- Antti Ukkonen, Behrouz Derakhshan, and Hannes Heikinheimo. 2015. Crowdsourced nonparametric density estimation using relative distances. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing (HCOMP).Google ScholarCross Ref
- Laurens van Der Maaten and Kilian Weinberger. 2012. Stochastic triplet embedding. In Proceedings of 2012 IEEE International Workshop on Machine Learning for Signal Processing (MLSP). 1–6.Google ScholarCross Ref
- Catherine Wah, Grant Van Horn, Steve Branson, Subhransu Maji, Pietro Perona, and Serge Belongie. 2014. Similarity Comparisons for Interactive Fine-Grained Categorization. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 859–866.Google ScholarDigital Library
- Zhisong Zhang, Xiang Kong, Zhengzhong Liu, Xuezhe Ma, and Eduard Hovy. 2020. A Two-Step Approach for Implicit Event Argument Detection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). Online, 7479–7485. https://doi.org/10.18653/v1/2020.acl-main.667Google ScholarCross Ref
- He Zhao, Longtao Huang, Rong Zhang, Quan Lu, and Hui Xue. 2020. SpanMlt: A Span-based Multi-Task Learning Framework for Pair-wise Aspect and Opinion Terms Extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). Online, 3239–3248. https://doi.org/10.18653/v1/2020.acl-main.296Google ScholarCross Ref
Index Terms
- Context-based Collective Preference Aggregation for Prioritizing Crowd Opinions in Social Decision-making
Recommendations
Aggregation of fuzzy preference relations to multicriteria decision making
Weighted aggregation of fuzzy preference relations on the set of alternatives by several criteria in decision-making problems is considered. Pairwise comparisons with respect to importance of the criteria are given in fuzzy preference relation as well. ...
Multi-criteria decision making with incomplete linguistic preference relations
ACOS'07: Proceedings of the 6th Conference on WSEAS International Conference on Applied Computer Science - Volume 6This study proposes some issues to solve the Incomplete Linguistic Preference Relations under Multi-Criteria Decision Making. The proposed method has simple calculation and can speed up the process of comparison and selection of alternative. Experts ...
Integrating multiple types of incomplete linguistic preference relations in multi-person decision making
FSKD'06: Proceedings of the Third international conference on Fuzzy Systems and Knowledge DiscoveryIn this paper, the multi-person decision making problems with various different types of incomplete linguistic preference relations are studied. Some new concepts, including incomplete uncertain linguistic preference relation, incomplete triangular ...
Comments