Siva Likitha Valluru
Biplav Srivastava
ABSTRACT
This paper demonstrates ULTRA (University-Lead Team Builder from RFPs and Analysis), a novel AI-based recommendation system for team formation, where (1) candidate teams are formed with the goal to reach highest possible skill coverage, as demanded by an opportunity, and (2) the challenge of fair distribution of opportunities is balanced amongst all available members. Using this tool, users can explore the skills required in open data from proposal calls (demand) and adeptly assemble teams from candidate researcher profiles (supply). The efficiency of these teams is then evaluated using an innovative goodness metric and validated through both quantitative and qualitative experiments. Beyond teaming, the tool design and evaluation of this work could interest researchers exploring the potential of set recommendation in other applications, rather than the well-understood traditional single-item recommendations. We deploy this system in two major institutions from diverse geographical regions of the world (United States and India), and in doing so, we show that ULTRA can generate good candidate teams across differing teaming contexts, and support the notion that our system is widely expandable.
Supplemental Material
- Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2019. Managing popularity bias in recommender systems with personalized re-ranking. arXiv preprint arXiv:1901.07555 (2019).Google Scholar
- Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. 2020. The connection between popularity bias, calibration, and fairness in recommendation. In Proceedings of the 14th ACM Conference on Recommender Systems. 726–731.Google ScholarDigital Library
- ACM. 2012. ACM Classification Scheme. In https://www.acm.org/publications/computing-classification-system/how-to-use.Google Scholar
- Faez Ahmed, Kalyanmoy Deb, and Abhilash Jindal. 2013. Multi-objective optimization and decision making approaches to cricket team selection. Applied Soft Computing 13, 1 (2013), 402–414.Google ScholarDigital Library
- Aris Anagnostopoulos, Luca Becchetti, Carlos Castillo, Aristides Gionis, and Stefano Leonardi. 2012. Online team formation in social networks. In Proc. 21st international conference on WWW.Google ScholarDigital Library
- Ashwathy Ashokan and Christian Haas. 2021. Fairness metrics and bias mitigation strategies for rating predictions. Information Processing & Management 58, 5 (2021), 102646.Google ScholarDigital Library
- Coralio Ballester. 2004. NP-completeness in hedonic games. Games and Economic Behavior 49, 1 (2004), 1–30.Google ScholarCross Ref
- Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Allison Woodruff, Christine Luu, Pierre Kreitmann, Jonathan Bischof, and Ed H Chi. 2019. Putting fairness principles into practice: Challenges, metrics, and improvements. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 453–459.Google ScholarDigital Library
- Avradeep Bhowmik, Vivek Borkar, Dinesh Garg, and Madhavan Pallan. 2014. Submodularity in team formation problem. In Proceedings of the 2014 SIAM international conference on data mining. SIAM, 893–901.Google ScholarCross Ref
- Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. 2020. Fairlearn: A toolkit for assessing and improving fairness in AI. Microsoft, Tech. Rep. MSR-TR-2020-32 (2020).Google Scholar
- Jesús Bobadilla, Fernando Ortega, Antonio Hernando, and Abraham Gutiérrez. 2013. Recommender systems survey. Knowledge-based systems 46 (2013), 109–132.Google ScholarDigital Library
- Bart H Boon and Gerard Sierksma. 2003. Team formation: Matching quality supply and quality demand. European Journal of Operational Research 148, 2 (2003), 277–292.Google ScholarCross Ref
- Chad Crawford, Zenefa Rahaman, and Sandip Sen. 2016. Evaluating the efficiency of robust team formation algorithms. In International Conference on Autonomous Agents and Multiagent Systems. Springer, 14–29.Google ScholarCross Ref
- Wilmax Marreiro Cruz and Seiji Isotani. 2014. Group formation algorithms in collaborative learning contexts: A systematic mapping of the literature. In Collaboration and Technology: 20th International Conference, CRIWG 2014, Santiago, Chile, September 7-10, 2014. Proceedings 20. Springer, 199–214.Google ScholarCross Ref
- Stanislav Dadelo, Zenonas Turskis, Edmundas Kazimieras Zavadskas, and Ruta Dadeliene. 2014. Multi-criteria assessment and ranking system of sport team formation based on objective-measured values of criteria set. Expert Systems with Applns. 41, 14 (2014), 6106–6113.Google ScholarCross Ref
- DST. 2023. Department of Science & Technology. https://dst.gov.in/. Government of India | Ministry of Science and Technology (2023).Google Scholar
- Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference. 214–226.Google ScholarDigital Library
- Arik Friedman, Bart P Knijnenburg, Kris Vanhecke, Luc Martens, and Shlomo Berkovsky. 2015. Privacy aspects of recommender systems. Recommender systems handbook (2015), 649–688.Google Scholar
- Pratyush Garg, John Villasenor, and Virginia Foggo. 2020. Fairness metrics: A comparative analysis. In 2020 IEEE International Conference on Big Data (Big Data). IEEE, 3662–3666.Google ScholarCross Ref
- Matthew E. Gaston and Marie desJardins. 2005. Agent-Organized Networks for Dynamic Team Formation. In Proc. AAMAS (The Netherlands). ACM, 230–237. https://doi.org/10.1145/1082473.1082508Google ScholarDigital Library
- Matthew E Gaston, John Simmons, and Marie desJardins. 2004. Adapting Network Structure for Efficient Team Formation.. In AAAI Technical Report (2). 1–8.Google Scholar
- James Hardy, Mark A Eys, and Albert V Carron. 2005. Exploring the potential disadvantages of high cohesion in sports teams. Small group research 36, 2 (2005), 166–187.Google Scholar
- J Henry Hinnefeld, Peter Cooman, Nat Mammo, and Rupert Deese. 2018. Evaluating fairness metrics in the presence of dataset bias. arXiv preprint arXiv:1809.09245 (2018).Google Scholar
- Martin Hoefer, Daniel Vaz, and Lisa Wagner. 2015. Hedonic coalition formation in networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 29.Google ScholarCross Ref
- Jiuchuan Jiang, Bo An, Yichuan Jiang, Chenyan Zhang, Zhan Bu, and Jie Cao. 2019. Group-oriented task allocation for crowdsourcing in social networks. IEEE Transactions on Systems, Man, and Cybernetics: Systems 51, 7 (2019), 4417–4432.Google ScholarCross Ref
- Julio Juárez and Carlos A Brizuela. 2018. A multi-objective formulation of the team formation problem in social networks: preliminary results. In Proceedings of the genetic and evolutionary computation conference. 261–268.Google ScholarDigital Library
- Julio Juárez, Cipriano (Pano) Santos, and Carlos A. Brizuela. 2021. A Comprehensive Review and a Taxonomy Proposal of Team Formation Problems. ACM Computing Survey 54, 7, Article 153 (jul 2021), 33 pages. https://doi.org/10.1145/3465399Google ScholarDigital Library
- Ashutosh Kakadiya, Sriraam Natarajan, and Balaraman Ravindran. 2021. Relational boosted bandits. In Proceedings of the AAAI Conference on AI, Vol. 35. 12123–12130.Google ScholarCross Ref
- Mehdi Kargar, Aijun An, and Morteza Zihayat. 2012. Efficient bi-objective team formation in social networks. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23. Springer, 483–498.Google ScholarCross Ref
- Hyeyoung Ko, Suyeon Lee, Yoonseo Park, and Anna Choi. 2022. A survey of recommendation systems: recommendation models, techniques, and application fields. Electronics 11, 1 (2022), 141.Google ScholarCross Ref
- Madhusree Kuanr and Puspanjali Mohapatra. 2021. Recent challenges in recommender systems: a survey. In Progress in Advanced Computing and Intelligent Engineering: Proceedings of ICACIE 2019, Volume 2. Springer, 353–365.Google ScholarCross Ref
- Theodoros Lappas, Kun Liu, and Evimaria Terzi. 2009. Finding a team of experts in social networks. In Proc. 15th ACM SIGKDD. 467–476.Google ScholarDigital Library
- Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2021. User-oriented fairness in recommendation. In Proceedings of the Web Conference 2021. 624–632.Google ScholarDigital Library
- Yunqi Li, Hanxiong Chen, Shuyuan Xu, Yingqiang Ge, Juntao Tan, Shuchang Liu, and Yongfeng Zhang. 2023. Fairness in Recommendation: Foundations, Methods and Applications. ACM Transactions on Intelligent Systems and Technology (2023).Google Scholar
- Somchaya Liemhetcharat and Manuela Veloso. 2012. Modeling and learning synergy for team formation with heterogeneous agents. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1. 365–374.Google ScholarDigital Library
- Somchaya Liemhetcharat and Manuela Veloso. 2014. Weighted synergy graphs for effective team formation with heterogeneous ad hoc agents. Artificial Intelligence 208 (2014), 41–65.Google ScholarDigital Library
- Lucas Machado and Kostas Stefanidis. 2019. Fair team recommendations for multidisciplinary projects. In IEEE/WIC/ACM International Conference on Web Intelligence. 293–297.Google ScholarDigital Library
- Hannan Amoozad Mahdiraji, Elham Razghandi, and Adel Hatami-Marbini. 2021. Overlapping coalition formation in game theory: A state-of-the-art review. Expert Systems with Applications 174 (2021), 114752.Google ScholarDigital Library
- Anirban Majumder, Samik Datta, and KVM Naidu. 2012. Capacitated team formation problem on social networks. In Proceedings of the 18th ACM SIGKDD international conference on knowledge discovery and data mining. 1005–1013.Google ScholarDigital Library
- Naseebah Maqtary, Abdulqader Mohsen, and Kamal Bechkoum. 2019. Group formation techniques in computer-supported collaborative learning: A systematic literature review. Technology, Knowledge and Learning 24 (2019), 169–190.Google ScholarCross Ref
- Staša Milojević. 2014. Principles of scientific research team formation and evolution. Proceedings of the National Academy of Sciences 111, 11 (2014), 3984–3989.Google ScholarCross Ref
- E Molleman and J Slomp. 1999. Functional flexibility and team performance. International Journal of Production Research 37, 8 (1999), 1837–1858.Google ScholarCross Ref
- Chinasa Odo, Judith Masthoff, and Nigel Beacham. 2019. Group formation for collaborative learning: A systematic literature review. In Artificial Intelligence in Education: 20th International Conference, AIED 2019, Chicago, IL, USA, June 25-29, 2019, Proceedings, Part II 20. Springer, 206–212.Google ScholarDigital Library
- Tenda Okimoto, Nicolas Schwind, Maxime Clement, Tony Ribeiro, Katsumi Inoue, and Pierre Marquis. 2015. How to Form a Task-Oriented Robust Team.. In AAMAS. 395–403.Google Scholar
- Deepak Kumar Panda and Sanjog Ray. 2022. Approaches and algorithms to mitigate cold start problems in recommender systems: a systematic literature review. JIIS 59, 2 (2022), 341–366.Google ScholarDigital Library
- Evaggelia Pitoura, Kostas Stefanidis, and Georgia Koutrika. 2022. Fairness in rankings and recommendations: an overview. The VLDB Journal (2022), 1–28.Google Scholar
- Radin Hamidi Rad, Shirin Seyedsalehi, Mehdi Kargar, Morteza Zihayat, and Ebrahim Bagheri. 2022. A Neural Approach to Forming Coherent Teams in Collaboration Networks.. In EDBT. 2–440.Google Scholar
- Syama Sundar Rangapuram, Thomas Bühler, and Matthias Hein. 2013. Towards realistic team formation in social networks based on densest subgraphs. In Proceedings of the 22nd international conference on World Wide Web. 1077–1088.Google ScholarDigital Library
- Elham Semsar-Kazerooni and Khashayar Khorasani. 2009. Multi-agent team cooperation: A game theory approach. Automatica 45, 10 (2009), 2205–2213.Google ScholarDigital Library
- Nasim Sonboli, Masoud Mansoury, Ziyue Guo, Shreyas Kadekodi, Weiwen Liu, Zijun Liu, Andrew Schwartz, and Robin Burke. 2021. librec-auto: A Tool for Recommender Systems Experimentation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 4584–4593.Google ScholarDigital Library
- Matthew Spradling, Judy Goldsmith, Xudong Liu, Chandrima Dadi, and Zhiyu Li. 2013. Roles and teams hedonic game. In Algorithmic Decision Theory: Third International Conference, ADT 2013, Bruxelles, Belgium, November 12-14, 2013, Proceedings 3. Springer, 351–362.Google ScholarDigital Library
- Biplav Srivastava, Tarmo Koppel, Sai Teja Paladi, Siva Likitha Valluru, Rohit Sharma, and Owen Bond. 2022. ULTRA: A Data-driven Approach for Recommending Team Formation in Response to Proposal Calls. In IEEE ICDM Workshops 2022. https://doi.org/10.1109/ICDMW58026.2022.00130Google ScholarCross Ref
- Georgios Stavrou, Panagiotis Adamidis, Jason Papathanasiou, and Konstantinos Tarabanis. 2023. Team Formation: A Systematic Literature Review. (2023).Google Scholar
- Mark Sykes, Brigid M Gillespie, Wendy Chaboyer, and Evelyn Kang. 2015. Surgical team mapping: implications for staff allocation and coordination. AORN journal 101, 2 (2015), 238–248.Google ScholarCross Ref
- Madjid Tavana, Farshad Azizi, Farzad Azizi, and Majid Behzadian. 2013. A fuzzy inference system with application to player selection and team formation in multi-player sports. Sport Management Review 16, 1 (2013), 97–110.Google ScholarCross Ref
- Tzu-Liang Bill Tseng, Chun-Che Huang, How-Wei Chu, and Roger R Gung. 2004. Novel approach to multi-functional project team formation. International Journal of Project Management 22, 2 (2004), 147–159.Google ScholarCross Ref
- Siva Likitha Valluru, Biplav Srivastava, and Sai Teja Paladi. 2023. ULTRA Resources Github. In https://github.com/ai4society/ULTRA-Team- Recommendation-Resources.Google Scholar
- Siva Likitha Valluru, Biplav Srivastava, Sai Teja Paladi, Siwen Yan, and Sriraam Natarajan. 2022. ULTRA: A Data-driven Approach for Recommending Team Formation in Response to Proposal Calls. The Thirty-Sixth Annual Conference on Innovative Applications of Artificial Intelligence (IAAI/AAAI).Google Scholar
- Michael Van De Vijsel and John Anderson. 2004. Coalition formation in multi-agent systems under real-world conditions. Proceedings of association for the advancement of artificial intelligence (2004).Google Scholar
- Hyeongon Wi, Seungjin Oh, Jungtae Mun, and Mooyoung Jung. 2009. A team formation model based on knowledge and collaboration. Expert Systems with Applications 36, 5 (2009), 9121–9134.Google ScholarDigital Library
- Yu Xin 2015. Challenges in recommender systems: scalability, privacy, and structured recommendations. Ph. D. Dissertation. Massachusetts Institute of Technology.Google Scholar
- P Zainal, D Razali, and Zulkefli Mansor. 2020. Team formation for agile software development: a review. Int. J. Adv. Sci. Eng. Inf. Technol 10, 2 (2020), 555–561.Google ScholarCross Ref
- Edmundas Kazimieras Zavadskas, Algimantas Zakarevicius, and Jurgita Antucheviciene. 2006. Evaluation of ranking accuracy in multi-criteria decisions. Informatica 17, 4 (2006), 601–618.Google ScholarDigital Library
- Xi Zhu and Douglas R Wholey. 2018. Expertise redundancy, transactive memory, and team performance in interdisciplinary care teams. Health services research 53, 6 (2018), 4921–4942.Google Scholar
Index Terms
- ULTRA: Exploring Team Recommendations in Two Geographies Using Open Data in Response to Call for Proposals
Recommendations
Exploiting Linked Open Data in Cold-start Recommendations with Positive-only Feedback
CERI '16: Proceedings of the 4th Spanish Conference on Information RetrievalIn recommender systems, user preferences can be acquired either explicitly by means of ratings, or implicitly --e.g., by processing text reviews, and by mining item browsing and purchasing records. Most existing collaborative filtering approaches have ...
Naïve filterbots for robust cold-start recommendations
KDD '06: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data miningThe goal of a recommender system is to suggest items of interest to a user based on historical behavior of a community of users. Given detailed enough history, item-based collaborative filtering (CF) often performs as well or better than almost any ...
News Session-Based Recommendations using Deep Neural Networks
DLRS 2018: Proceedings of the 3rd Workshop on Deep Learning for Recommender SystemsNews recommender systems are aimed to personalize users experiences and help them to discover relevant articles from a large and dynamic search space. Therefore, news domain is a challenging scenario for recommendations, due to its sparse user profiling,...
Comments