skip to main content
research-article
Public Access

Composing Team Compositions: An Examination of Instructors' Current Algorithmic Team Formation Practices

Published: 04 October 2023 Publication History

Abstract

Instructors using algorithmic team formation tools must decide which criteria (e.g., skills, demographics, etc.) to use to group students into teams based on their teamwork goals, and have many possible sources from which to draw these configurations (e.g., the literature, other faculty, their students, etc.). However, tools offer considerable flexibility and selecting ineffective configurations can lead to teams that do not collaborate successfully. Due to such tools' relative novelty, there is currently little knowledge of how instructors choose which of these sources to utilize, how they relate different criteria to their goals for the planned teamwork, or how they determine if their configuration or the generated teams are successful. To close this gap, we conducted a survey (N=77) and interview (N=21) study of instructors using CATME Team-Maker and other criteria-based processes to investigate instructors' goals and decisions when using team formation tools. The results showed that instructors prioritized students learning to work with diverse teammates and performed "sanity checks" on their formation approach's output to ensure that the generated teams would support this goal, especially focusing on criteria like gender and race. However, they sometimes struggled to relate their educational goals to specific settings in the tool. In general, they also did not solicit any input from students when configuring the tool, despite acknowledging that this information might be useful. By opening the "black box" of the algorithm to students, more learner-centered approaches to forming teams could therefore be a promising way to provide more support to instructors configuring algorithmic tools while at the same time supporting student agency and learning about teamwork.

Supplemental Material

ZIP File
As mentioned in the paper, the supplemental/auxiliary material includes 4 files: the full text of the (1) survey, (2) follow-up survey, and (3) interview protocol, and (4) the full list of codes applied to the interviews. All are pdf files.

References

[1]
2017. Team-Maker Algorithm Detail. Retrieved 4 April 2019 from https://www.catme.org/faculty/help#TeamMakerScoring
[2]
2018. CATME Smarter Teamwork. Retrieved 31 August 2018 from http://info.catme.org/
[3]
2020. CATME User Institutions (Alphabetical by Country). Retrieved 19 August 2020 from https://info.catme.org/instructor/history-research/our-user-base/catme-user-institutions-alphabetical-by-country/
[4]
2020. Usage of CATME System. Retrieved 14 September 2020 from https://info.catme.org/instructor/history-research/
[5]
ABET. 2019. 2020--2021 Criteria for Accrediting Computing Programs. https://www.abet.org/wp-content/uploads/2019/12/C001--20--21-CAC-Criteria-MARK-UP-11--30--19-Updated-2.pdf.
[6]
Albatool A. Alamri and Brian P. Bailey. 2018. Examination of the Effectiveness of a Criteria-based Team Formation Tool. In Frontiers in Education. Ieee.
[7]
Nancy Allen, Dianne Atkinson, Meg Morgan, Teresa Moore, and Craig Snow. 1987. What Experienced Collaborators Say About Collaborative Writing. Iowa State Journal of Business and Technical Communication 1, 2 (1987), 70--90. https://doi.org/10.1177/105065198700100206 arXiv:https://doi.org/10.1177/105065198700100206
[8]
Mike Ananny and Kate Crawford. 2018. Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society 20, 3 (2018), 973--989. https://doi.org/10.1177/1461444816676645 arXiv:https://doi.org/10.1177/1461444816676645
[9]
Donald R Bacon, Kim A Stewart, and William S Silver. 1999. Lessons from the best and worst student team experiences: How a teacher can make the difference. Journal of Management Education 23, 5 (1999), 467--488.
[10]
Julia B Bear and Anita Williams Woolley. 2011. The role of gender in team collaboration and performance. Interdisciplinary science reviews 36, 2 (2011), 146--153.
[11]
Mehdi Beheshtian-Ardekani and Mo A Mahmood. 1986. Education development and validation of a tool for assigning students to groups for class projects. Decision Sciences 17, 1 (1986), 92--113.
[12]
Suzanne T Bell. 2007. Deep-level composition variables as predictors of team performance: a meta-analysis. Journal of Applied Psychology 92, 3 (2007), 595--615.
[13]
Suzanne T Bell, Shanique G Brown, Anthony Colaneri, and Neal Outland. 2018. Team composition and the ABCs of teamwork. American Psychologist 73, 4 (2018), 349.
[14]
David Boud, Ruth Cohen, et al . 2014. Peer learning in higher education: Learning from and with each other. Routledge.
[15]
Lt Col James L Brickell, Lt Col David B Porter, Lt Col Michael F Reynolds, and Capt Richard D Cosgrove. 1994. Assigning students to groups for engineering design projects: A comparison of five methods. Journal of Engineering Education 83, 3 (1994), 259--262.
[16]
Paula E Chan, Kristall J Graham-Day, Virginia A Ressa, Mary T Peters, and Moira Konrad. 2014. Beyond involvement: Promoting student ownership of learning in classrooms. Intervention in School and Clinic 50, 2 (2014), 105--113.
[17]
Prerna Chikersal, Maria Tomprou, Young Ji Kim, Anita Williams Woolley, and Laura Dabbish. 2017. Deep Structures of Collaboration: Physiological Correlates of Collective Intelligence and Group Satisfaction. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW '17). Association for Computing Machinery, New York, NY, USA, 873--888. https://doi.org/10.1145/2998181.2998250
[18]
David T Conley and Elizabeth M French. 2014. Student ownership of learning as a key component of college readiness. American Behavioral Scientist 58, 8 (2014), 1018--1034.
[19]
Henriette Cramer, Vanessa Evers, Satyan Ramlal, Maarten Van Someren, Lloyd Rutledge, Natalia Stash, Lora Aroyo, and Bob Wielinga. 2008. The effects of transparency on trust in and acceptance of a content-based art recommender. User Modeling and User-Adapted Interaction 18, 5 (2008), 455.
[20]
Curt J. Dommeyer. 1986. A Comparison of the Individual Proposal and the Team Project in the Marketing Re- search Course. Journal of Marketing Education 8, 1 (1986), 30--38. https://doi.org/10.1177/027347538600800104 arXiv:https://doi.org/10.1177/027347538600800104
[21]
Yohanan Eshel and Revital Kohavi. 2003. Perceived classroom control, self-regulated learning strategies, and academic achievement. Educational psychology 23, 3 (2003), 249--260.
[22]
Diego Gómez-Zará, Leslie A DeChurch, and Noshir S Contractor. 2020. A taxonomy of team-assembly systems: Understanding how people use technologies to form teams. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1--36.
[23]
Diego Gómez-Zará, Mengzi Guo, Leslie A DeChurch, and Noshir Contractor. 2020. The impact of displaying diversity information on the formation of self-assembling teams. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--15.
[24]
Diego Gómez-Zará, Matthew Paras, Marlon Twyman, Jacqueline N Lane, Leslie A DeChurch, and Noshir S Contractor. 2019. Who would you like to work with?. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1--15.
[25]
Hee Young Han. 2009. Relationship between students' emotional intelligence, social bond, and interactions in online learning. University of Illinois at Urbana-Champaign.
[26]
Randall S. Hansen. 2006. Benefits and Problems With Student Teams: Suggestions for Improving Team Projects. Journal of Education for Business 82, 1 (2006), 11--19. https://doi.org/10.3200/JOEB.82.1.11--19 arXiv:https://doi.org/10.3200/JOEB.82.1.11--19
[27]
Alexa M Harris, Diego Gómez-Zará, Leslie A DeChurch, and Noshir S Contractor. 2019. Joining together online: the trajectory of CSCW scholarship on group formation. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--27.
[28]
Beth Harry, Keith M Sturges, and Janette K Klingner. 2005. Mapping the process: An exemplar of process and challenge in grounded theory analysis. Educational researcher 34, 2 (2005), 3--13.
[29]
Emily M. Hastings, Albatool Alamri, Andrew Kuznetsov, Christine Pisarczyk, Karrie Karahalios, Darko Marinov, and Brian P. Bailey. 2020. LIFT: Integrating Stakeholder Voices into Algorithmic Team Formation. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (Chi '20). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3313831.3376797
[30]
Emily M. Hastings, Farnaz Jahanbakhsh, Karrie Karahalios, Darko Marinov, and Brian P. Bailey. 2018. Structure or Nurture? The Effects of Team-Building Activities and Team Composition on Team Outcomes. In Proceedings of the ACM on Human-Computer Interaction, Vol. 2. ACM.
[31]
Emily M. Hastings, Sneha R. Krishna Kumaran, Karrie Karahalios, and Brian P. Bailey. 2022. A Learner-Centered Technique for Collectively Configuring Inputs for an Algorithmic Team Formation Tool. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1 (Providence, RI, USA) (SIGCSE 2022). Association for Computing Machinery, New York, NY, USA, 969--975. https://doi.org/10.1145/3478431.3499331
[32]
Tyson R Henry. 2013. Creating effective student groups: an introduction to groupformation. org. In Proceeding of the 44th ACM technical symposium on Computer science education. ACM, 645--650.
[33]
Susan Horwitz, Susan H Rodger, Maureen Biggers, David Binkley, C Kolin Frantz, Dawn Gundermann, Susanne Hambrusch, Steven Huss-Lederman, Ethan Munson, Barbara Ryder, et al . 2009. Using peer-led team learning to increase participation and success of under-represented groups in introductory computer science. ACM SIGCSE Bulletin 41, 1 (2009), 163--167.
[34]
Sujin K Horwitz and Irwin B Horwitz. 2007. The effects of team diversity on team outcomes: A meta-analytic review of team demography. Journal of management 33, 6 (2007), 987--1015.
[35]
Roland Hubscher. 2010. Assigning students to groups using general and context-specific criteria. IEEE transactions on learning technologies 3, 3 (2010), 178--189.
[36]
Dennis Hummel and Alexander Maedche. 2019. How effective is nudging? A quantitative review on the effect sizes and limits of empirical nudging studies. Journal of Behavioral and Experimental Economics 80 (2019), 47--58. https://doi.org/10.1016/j.socec.2019.03.005
[37]
Farnaz Jahanbakhsh, Wai-Tat Fu, Karrie Karahalios, Darko Marinov, and Brian Bailey. 2017. You Want Me to Work with Who?: Stakeholder Perceptions of Automated Team Formation in Project-based Courses. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Acm, 3201--3212.
[38]
David S Jalajas and Robert I Sutton. 1984. Feuds in student groups: Coping with whiners, martyrs, saboteurs, bullies, and deadbeats. Organizational Behavior Teaching Review 9, 4 (1984), 94--102.
[39]
Jéssica Mendes JORGEª, Alexandre Crepory Abbott de OLIVEIRA, and Andrea Cristina dos SANTOS. 2020. Analyzing how university is preparing engineering students for Industry 4.0. In Transdisciplinary Engineering for Complex Socio-technical Systems--Real-life Applications: Proceedings of the 27th ISTE International Conference on Transdisciplinary Engineering, July 1--July 10, 2020, Vol. 12. IOS Press, 82.
[40]
Janna Juvonen. 2006. Sense of Belonging, Social Bonds, and School Functioning. (2006).
[41]
René F Kizilcec. 2016. How much information?: Effects of transparency on trust in an algorithmic interface. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2390--2395.
[42]
Cameron Klein, Deborah DiazGranados, Eduardo Salas, Huy Le, C Shawn Burke, Rebecca Lyons, and Gerald F Goodwin. 2009. Does team building work? Small Group Research 40, 2 (2009), 181--222.
[43]
Sneha R. Krishna Kumaran, Deana C. McDonagh, and Brian P. Bailey. 2017. Increasing Quality and Involvement in Online Peer Feedback Exchange. Proc. ACM Hum.-Comput. Interact. 1, CSCW, Article 63 (dec 2017), 18 pages. https://doi.org/10.1145/3134698
[44]
Richard A Layton, Misty L Loughry, Matthew W Ohland, and George D Ricco. 2010. Design and Validation of a Web-Based System for Assigning Members to Teams Using Instructor-Specified Criteria. Advances in Engineering Education 2, 1 (2010), n1.
[45]
Min Kyung Lee, Daniel Kusbit, Evan Metsky, and Laura Dabbish. 2015. Working with machines: The impact of algorithmic and data-driven management on human workers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1603--1612.
[46]
Susan Lerner, Diane Magrane, and Erica Friedman. 2009. Teaching teamwork in medical education. Mount Sinai Journal of Medicine: A Journal of Translational and Personalized Medicine: A Journal of Translational and Personalized Medicine 76, 4 (2009), 318--329.
[47]
Ioanna Lykourentzou, Angeliki Antoniou, Yannick Naudet, and Steven P Dow. 2016. Personality matters: Balancing for personality types leads to better outcomes for crowd teams. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 260--273.
[48]
Ioanna Lykourentzou, Robert E Kraut, and Steven P Dow. 2017. Team Dating Leads to Better Online Ad Hoc Collaborations. In CSCW. 2330--2343.
[49]
Ioanna Lykourentzou, Shannon Wang, Robert E Kraut, and Steven P Dow. 2016. Team dating: A self-organized team formation strategy for collaborative crowdsourcing. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 1243--1249.
[50]
Joseph E McGrath, Holly Arrow, and Jennifer L Berdahl. 2000. The study of groups: Past, present, and future. Personality and social psychology review 4, 1 (2000), 95--105.
[51]
Jeffrey A Mello. 1993. Improving individual member accountability in small work group settings. Journal of Management Education 17, 2 (1993), 253--259.
[52]
Hendrik Müller, Aaron Sedley, and Elizabeth Ferrall-Nunge. 2014. Survey Research in HCI. Springer New York, New York, NY, 229--266. https://doi.org/10.1007/978--1--4939-0378--8_10
[53]
Humberto Oraison, Loretta Konjarski, and Samuel Howe. 2019. Journal of Teaching and Learning for Graduate Employability 10, 1 (2019), 173--194. https://search.informit.org/doi/10.3316/informit.580981748647262
[54]
Asma Ounnas, David E Millard, and Hugh C Davis. 2007. A metrics framework for evaluating group formation. In Proceedings of the 2007 international ACM conference on Supporting group work. 221--224.
[55]
Michael A Redmond. 2001. A computer program to aid assignment of student project groups. ACM SIGCSE Bulletin 33, 1 (2001), 134--138.
[56]
Marcel M Robles. 2012. Executive perceptions of the top 10 soft skills needed in today's workplace. Business Communication Quarterly 75, 4 (2012), 453--465.
[57]
Johnny Saldaña. 2015. The coding manual for qualitative researchers. Sage.
[58]
Niloufar Salehi and Michael S Bernstein. 2018. Hive: Collective Design Through Network Rotation. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 151.
[59]
Rajiv C. Shah and Christian Sandvig. 2008. SOFTWARE DEFAULTS AS DE FACTO REGULATION The case of the wireless internet. Information, Communication & Society 11, 1 (2008), 25--46. https://doi.org/10.1080/13691180701858836 arXiv:https://doi.org/10.1080/13691180701858836
[60]
James B Shaw. 2004. A fair go for all? The impact of intragroup diversity and diversity-management skills on student experiences and outcomes in team-based class projects. Journal of Management Education 28, 2 (2004), 139--169.
[61]
Debra Smarkusky, Richard Dempsey, J Ludka, and Frouke de Quillettes. 2005. Enhancing team knowledge: instruction vs. experience. In ACM SIGCSE Bulletin, Vol. 37. ACM, 460--464.
[62]
Cass R Sunstein. 2014. Nudging: a very short guide. Journal of Consumer Policy 37, 4 (2014), 583--588.
[63]
Kristen Vaccaro, Dylan Huang, Motahhare Eslami, Christian Sandvig, Kevin Hamilton, and Karrie Karahalios. 2018. The Illusion of Control: Placebo Effects of Control Settings. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 16.
[64]
Sander Valstar, Caroline Sih, Sophia Krause-Levy, Leo Porter, and William G. Griswold. 2020. A Quantitative Study of Faculty Views on the Goals of an Undergraduate CS Program and Preparing Students for Industry. In Proceedings of the 2020 ACM Conference on International Computing Education Research (Virtual Event, New Zealand) (ICER '20). Association for Computing Machinery, New York, NY, USA, 113--123. https://doi.org/10.1145/3372782.3406277
[65]
Dai-Yi Wang, Sunny SJ Lin, and Chuen-Tsai Sun. 2007. DIANA: A computer-supported heterogeneous grouping system for teachers to conduct successful small learning groups. Computers in Human Behavior 23, 4 (2007), 1997--2010.
[66]
Miaomiao Wen, Keith Maki, Steven Dow, James D. Herbsleb, and Carolyn Rose. 2017. Supporting Virtual Team Formation through Community-Wide Deliberation. Proc. ACM Hum.-Comput. Interact. 1, CSCW, Article 109 (Dec. 2017), 19 pages. https://doi.org/10.1145/3134744
[67]
Miaomiao Wen, Keith Maki, Xu Wang, Steven Dow, James D Herbsleb, and Carolyn Penstein Rosé. 2016. Transactivity as a Predictor of Future Collaborative Knowledge Integration in Team-Based Learning in Online Courses. In EDM. 533--538.
[68]
David L. Williams, John D. Beard, and Jone Rymer. 1991. Team Projects: Achieving their Full Potential. Journal of Marketing Education 13, 2 (1991), 45--53. https://doi.org/10.1177/027347539101300208 arXiv:https://doi.org/10.1177/027347539101300208
[69]
Anita Williams Woolley, Christopher F Chabris, Alex Pentland, Nada Hashmi, and Thomas W Malone. 2010. Evidence for a collective intelligence factor in the performance of human groups. Science 330, 6004 (2010), 686--688.

Index Terms

  1. Composing Team Compositions: An Examination of Instructors' Current Algorithmic Team Formation Practices

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image Proceedings of the ACM on Human-Computer Interaction
      Proceedings of the ACM on Human-Computer Interaction  Volume 7, Issue CSCW2
      CSCW
      October 2023
      4055 pages
      EISSN:2573-0142
      DOI:10.1145/3626953
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 04 October 2023
      Published in PACMHCI Volume 7, Issue CSCW2

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. CATME
      2. algorithms
      3. collaborative learning
      4. team composition
      5. team formation

      Qualifiers

      • Research-article

      Funding Sources

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 204
        Total Downloads
      • Downloads (Last 12 months)156
      • Downloads (Last 6 weeks)24
      Reflects downloads up to 17 Jan 2025

      Other Metrics

      Citations

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Login options

      Full Access

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media