Linxuan Zhao
Dragan Gasevic
Riordan Alfredo
ABSTRACT
In co-located situations, team members use a combination of verbal and visual signals to communicate effectively, among which positional forms play a key role. The spatial patterns adopted by team members in terms of where in the physical space they are standing, and who their body is oriented to, can be key in analysing and increasing the quality of interaction during such face-to-face situations. In this paper, we model the students’ communication based on spatial (positioning) and audio (voice detection) data captured from 92 students working in teams of four in the context of healthcare simulation. We extract non-verbal events (i.e., total speaking time, overlapped speech,and speech responses to team members and teachers) and investigate to what extent they can serve as meaningful indicators of students’ performance according to teachers’ learning intentions. The contribution of this paper to multimodal learning analytics includes: i) a generic method to semi-automatically model communication in a setting where students can freely move in the learning space; and ii) results from a mixed-methods analysis of non-verbal indicators of team communication with respect to teachers’ learning design.
- Anouck Adrot and Marie Bia Figueiredo. 2019. “Lost in Digitization”: A Spatial Journey in Emergency Response and Pragmatic Legitimacy. In Materiality in Institutions, de Vaujany FX., Adrot A., Boxenbaum E., and Leca B. (Eds.). Springer, 151–181. https://doi.org/10.1007/978-3-319-97472-9_6Google Scholar
- Karan Ahuja, Dohyun Kim, Franceska Xhakaj, Virag Varga, Anne Xie, Stanley Zhang, Jay Eric Townsend, Chris Harrison, Amy Ogan, and Yuvraj Agarwal. 2019. EduSense: Practical classroom sensing at Scale. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1–26.Google ScholarDigital Library
- Carlos Prieto Alvarez, Roberto Martinez-Maldonado, and Simon Buckingham Shum. 2020. LA-DECK: A card-based learning analytics co-design tool. In Proceedings of the tenth international conference on learning analytics & knowledge. 63–72.Google ScholarDigital Library
- Haifa Alwahaby, Mutlu Cukurova, Zacharoula Papamitsiou, and Michail Giannakos. 2021. The evidence of impact and ethical considerations of multimodal learning analytics: a systematic literature review. (2021).Google Scholar
- Khaled Bachour, Frederic Kaplan, and Pierre Dillenbourg. 2010. An interactive table for supporting participation balance in face-to-face collaborative learning. IEEE Transactions on Learning Technologies 3, 3 (2010), 203–213.Google ScholarDigital Library
- Nikoletta Bassiou, Andreas Tsiartas, Jennifer Smith, Harry Bratt, Colleen Richey, Elizabeth Shriberg, Cynthia D’Angelo, and Nonye Alozie. 2016. Privacy-Preserving Speech Analytics for Automatic Assessment of Student Collaboration.. In Interspeech. 888–892.Google Scholar
- Marc Beardsley, Judit Martínez Moreno, Milica Vujovic, Patricia Santos, and Davinia Hernández-Leo. 2020. Enhancing consent forms to support participant decision making in multimodal learning data research. British Journal of Educational Technology 51, 5 (2020), 1631–1652.Google ScholarCross Ref
- Nathaniel Blanchard, Patrick Donnelly, Andrew Olney, Borhan Samei, Brooke Ward, Xiaoyi Sun, Sean Kelly, Martin Nystrand, and Sidney K D’Mello. 2016. Identifying teacher questions using automatic speech recognition in classrooms. In Proceedings of the 17th annual meeting of the special interest group on discourse and dialogue. 191–201.Google ScholarCross Ref
- Simon Buckingham Shum, Rebecca Ferguson, and Roberto Martinez-Maldonado. 2019. Human-centred learning analytics. Journal of Learning Analytics 6, 2 (2019), 1–9.Google ScholarCross Ref
- Yi Han Victoria Chua, Justin Dauwels, and Seng Chee Tan. 2019. Technologies for automated analysis of co-located, real-life, physical learning spaces: Where are we now?. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge. 11–20.Google ScholarDigital Library
- Mutlu Cukurova, Michail Giannakos, and Roberto Martinez-Maldonado. 2020. The promise and challenges of multimodal learning analytics.Google Scholar
- Cynthia D’Angelo, Jennifer Smith, Nonye Alozie, Andreas Tsiartas, Colleen Richey, and Harry Bratt. 2019. Mapping individual to group level collaboration indicators using speech data. (2019), 115–118.Google Scholar
- Sidney K D’Mello, Andrew M Olney, Nathan Blanchard, Borhan Samei, Xiaoyi Sun, Brooke Ward, and Sean Kelly. 2015. Multimodal capture of teacher-student interactions for automated dialogic analysis in live classrooms. In Proceedings of the 2015 ACM on international conference on multimodal interaction. 557–566.Google ScholarDigital Library
- Patrick J Donnelly, Nathaniel Blanchard, Borhan Samei, Andrew M Olney, Xiaoyi Sun, Brooke Ward, Sean Kelly, Martin Nystrand, and Sidney K D’Mello. 2016. Multi-sensor modeling of teacher instructional segments in live classrooms. In Proceedings of the 18th ACM international conference on multimodal interaction. 177–184.Google ScholarDigital Library
- Vanessa Echeverria, Roberto Martinez-Maldonado, and Simon Buckingham Shum. 2019. Towards collaboration translucence: Giving meaning to multimodal group data. In Proceedings of the 2019 chi conference on human factors in computing systems. 1–16.Google ScholarDigital Library
- Gloria Fernandez-Nieto, Roberto Martinez-Maldonado, Vanessa Echeverria, Kirsty Kitto, Pengcheng An, and Simon Buckingham Shum. 2021. What Can Analytics for Teamwork Proxemics Reveal About Positioning Dynamics In Clinical Simulations?Proceedings of the ACM on Human-Computer Interaction 5, CSCW1(2021), 1–24.Google Scholar
- Gloria Milena Fernandez-Nieto, Roberto Martinez-Maldonado, Kirsty Kitto, and Simon Buckingham Shum. 2021. Modelling Spatial Behaviours in Clinical Team Simulations using Epistemic Network Analysis: Methodology and Teacher Evaluation. In LAK21: 11th International Learning Analytics and Knowledge Conference. 386–396.Google ScholarDigital Library
- Marva Foster, Marie Gilbert, Darlene Hanson, Kathryn Whitcomb, and Crystal Graham. 2019. Use of simulation to develop teamwork skills in prelicensure nursing students: an integrative review. Nurse educator 44, 5 (2019), E7–E11.Google ScholarCross Ref
- Edward T Hall, Ray L Birdwhistell, Bernhard Bock, Paul Bohannan, A Richard Diebold Jr, Marshall Durbin, Munro S Edmonson, JL Fischer, Dell Hymes, Solon T Kimball, 1968. Proxemics [and comments and replies]. Current anthropology 9, 2/3 (1968), 83–108.Google Scholar
- Maria Härgestam, Marie Lindkvist, Christine Brulin, Maritha Jacobsson, and Magnus Hultin. 2013. Communication in interdisciplinary teams: exploring closed-loop communication during in situ trauma team training. BMJ open 3, 10 (2013), e003525.Google Scholar
- Donal Healion, Sam Russell, Mutlu Cukurova, and Daniel Spikol. 2017. Tracing physical movement during practice-based learning through multimodal learning analytics. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference. 588–589.Google ScholarDigital Library
- Jon Hindmarsh and Alison Pilnick. 2007. Knowing bodies at work: Embodiment and ephemeral teamwork in anaesthesia. Organization studies 28, 9 (2007), 1395–1416.Google Scholar
- Sean Kelly, Andrew M Olney, Patrick Donnelly, Martin Nystrand, and Sidney K D’Mello. 2018. Automatically measuring question authenticity in real-world classrooms. Educational Researcher 47, 7 (2018), 451–464.Google ScholarCross Ref
- Adam Kendon. 1976. The F-formation system: The spatial organization of social encounters. Man-Environment Systems 6, 01 (1976), 1976.Google Scholar
- Adam Kendon. 2010. Spacing and orientation in co-present interaction. In Development of multimodal interfaces: Active listening and synchrony. Springer, 1–15.Google Scholar
- Jaebok Kim, Khiet P Truong, Vicky Charisi, Cristina Zaga, Manja Lohse, Dirk Heylen, and Vanessa Evers. 2015. Vocal turn-taking patterns in groups of children performing collaborative tasks: an exploratory study. In Sixteenth Annual Conference of the International Speech Communication Association.Google ScholarCross Ref
- Samuel Lapkin, Tracy Levett-Jones, Helen Bellchambers, and Ritin Fernandez. 2010. Effectiveness of patient simulation manikins in teaching clinical reasoning skills to undergraduate nursing students: A systematic review. Clinical simulation in nursing 6, 6 (2010), e207–e222.Google ScholarCross Ref
- Zhao Linxuan. [n.d.]. Modelling-Co-located-Team-Communication-from-Voice-Detection-and-Positioning-Data. https://github.com/LinxZhao/Modelling-Co-located-Team-Communication-from-Voice-Detection-and-Positioning-DataGoogle Scholar
- Sharon MacLean, Michelle Kelly, Fiona Geddes, and Phillip Della. 2017. Use of simulated patients to develop communication skills in nursing education: An integrative review. Nurse education today 48(2017), 90–98.Google Scholar
- Vinicius Macuch Silva, Judith Holler, Asli Ozyurek, and Seán G Roberts. 2020. Multimodality and the origin of a novel communication system in face-to-face interaction. Royal Society open science 7, 1 (2020), 182056.Google ScholarCross Ref
- Roberto Martinez-Maldonado, Yannis Dimitriadis, Alejandra Martinez-Monés, Judy Kay, and Kalina Yacef. 2013. Capturing and analyzing verbal and physical collaborative learning interactions at an enriched interactive tabletop. International Journal of Computer-Supported Collaborative Learning 8, 4(2013), 455–485.Google ScholarCross Ref
- Roberto Martinez-Maldonado, Vanessa Echeverria, Jurgen Schulte, Antonette Shibani, Katerina Mangaroska, and Simon Buckingham Shum. 2020. Moodoo: indoor positioning analytics for characterising classroom teaching. In International Conference on Artificial Intelligence in Education. Springer, 360–373.Google ScholarDigital Library
- Margaret L McLaughlin and Michael J Cody. 1982. Awkward silences: Behavioral antecedents and consequences of the conversational lapse. Human communication research 8, 4 (1982), 299–316.Google Scholar
- Gleicia Martins de Melo, Cristiana Brasil de Almeida Rebouças, Maria Vera Lúcia Moreira Leitão Cardoso, and Leiliane Martins Farias. 2013. Nursing team communication with regard pain in newborns: a descriptive study. Brazilian Journal of Nursing(2013).Google Scholar
- Lorenza Mondada. 2013. Interactional space and the study of embodied talk-in-interaction. Space in language and linguistics: Geographical, interactional and cognitive perspectives(2013), 247–275.Google Scholar
- Alessandra Guimarães Monteiro Moreira, Albert Lengruber De Azevedo, Nébia Maria Almeida De Figueiredo, Lilian Felippe Duarte De Oliveira, and Sílvia Teresa Carvalho De Araújo. 2017. Proxemic behavior of nursing in the hemodialysis setting. ACTA Paulista de Enfermagem 30, 4 (jul 2017), 343–349. https://doi.org/10.1590/1982-0194201700051Google Scholar
- Xavier Ochoa, Katherine Chiluiza, Gonzalo Méndez, Gonzalo Luzardo, Bruno Guamán, and James Castells. 2013. Expertise estimation based on simple multimodal features. In Proceedings of the 15th ACM on International conference on multimodal interaction. 583–590.Google ScholarDigital Library
- Sharon Oviatt, Kevin Hang, Jianlong Zhou, and Fang Chen. 2015. Spoken interruptions signal productive problem solving and domain expertise in mathematics. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. 311–318.Google ScholarDigital Library
- Brian C Parker and Florence Myrick. 2009. A critical examination of high-fidelity human patient simulation within the context of nursing pedagogy. Nurse education today 29, 3 (2009), 322–329.Google Scholar
- Shiva Pedram, Stephen Palmisano, Richard Skarbez, Pascal Perez, and Matthew Farrelly. 2020. Investigating the process of mine rescuers’ safety training with immersive virtual reality: A structural equation modelling approach. Computers & Education 153 (2020), 103891. https://doi.org/10.1016/j.compedu.2020.103891Google ScholarCross Ref
- Sambit Praharaj, Maren Scheffel, Hendrik Drachsler, and Marcus Specht. 2018. Multimodal analytics for real-time feedback in co-located collaboration. In European Conference on Technology Enhanced Learning. Springer, 187–201.Google ScholarDigital Library
- Sambit Praharaj, Maren Scheffel, Hendrik Drachsler, and Marcus M Specht. 2021. Literature Review on Co-Located Collaboration Modeling Using Multimodal Learning AnalyticsCan We Go the Whole Nine Yards. IEEE Transactions on Learning Technologies(2021).Google Scholar
- Sambit Praharaj, Maren Scheffel, Marcel Schmitz, Marcus Specht, and Hendrik Drachsler. 2021. Towards automatic collaboration analytics for group speech data using learning analytics. Sensors 21, 9 (2021), 3156.Google ScholarCross Ref
- Kenneth J Rothman. 1990. No adjustments are needed for multiple comparisons. Epidemiology (1990), 43–46.Google Scholar
- Nazmus Saquib, Ayesha Bose, Dwyane George, and Sepandar Kamvar. 2018. Sensei: sensing educational interaction. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1–27.Google ScholarDigital Library
- Dave J Saville. 1990. Multiple comparison procedures: the practical solution. The American Statistician 44, 2 (1990), 174–180.Google Scholar
- Stefan Scherer, Nadir Weibel, Louis-Philippe Morency, and Sharon Oviatt. 2012. Multimodal prediction of expertise and leadership in learning groups. In Proceedings of the 1st International Workshop on Multimodal Learning Analytics. 1–8.Google ScholarDigital Library
- Patrick Schober, Christa Boer, and Lothar A Schwarte. 2018. Correlation coefficients: appropriate use and interpretation. Anesthesia & Analgesia 126, 5 (2018), 1763–1768.Google ScholarCross Ref
- Charlott Sellberg, Olle Lindmark, and Hans Rystedt. 2018. Learning to navigate: the centrality of instructions and assessments for developing students’ professional competencies in simulator-based training. WMU Journal of Maritime Affairs 17, 2 (2018), 249–265. https://doi.org/10.1007/s13437-018-0139-2Google ScholarCross Ref
- Francesco Setti, Chris Russell, Chiara Bassetti, and Marco Cristani. 2015. F-formation detection: Individuating free-standing conversational groups in images. PloS one 10, 5 (2015), e0123783.Google ScholarCross Ref
- Ingo Siegert and Julia Krüger. 2021. “Speech Melody and Speech Content Didn’t Fit Together”—Differences in Speech Behavior for Device Directed and Human Directed Interactions. Springer International Publishing, Cham, 65–95. https://doi.org/10.1007/978-3-030-51870-7_4Google Scholar
- Agnieszka Sorokowska, Piotr Sorokowski, Peter Hilpert, Katarzyna Cantarero, Tomasz Frackowiak, Khodabakhsh Ahmadi, Ahmad M Alghraibeh, Richmond Aryeetey, Anna Bertoni, Karim Bettache, 2017. Preferred interpersonal distances: a global comparison. Journal of Cross-Cultural Psychology 48, 4 (2017), 577–592.Google ScholarCross Ref
- Daniel Spikol, Emanuele Ruffaldi, and Mutlu Cukurova. 2017. Using multimodal learning analytics to identify aspects of collaboration in project-based learning. Philadelphia, PA: International Society of the Learning Sciences.Google Scholar
- Daniel Spikol, Emanuele Ruffaldi, Lorenzo Landolfi, and Mutlu Cukurova. 2017. Estimation of success in collaborative learning based on multimodal learning analytics features. In 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT). IEEE, 269–273.Google ScholarCross Ref
- Zachari Swiecki and David Williamson Shaffer. 2020. iSENS: an integrated approach to combining epistemic and social network analyses. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge. 305–313.Google ScholarDigital Library
- Wiseman. [n.d.]. Python interface to the WebRTC Voice Activity Detector. https://github.com/wiseman/py-webrtcvadGoogle Scholar
- Lixiang Yan, Roberto Martinez-Maldonado, Beatriz Gallo Cordoba, Joanne Deppeler, Deborah Corrigan, Gloria Fernandez Nieto, and Dragan Gasevic. 2021. Footprints at School: Modelling In-class Social Dynamics from Students’ Physical Positioning Traces. In LAK21: 11th International Learning Analytics and Knowledge Conference. 43–54.Google ScholarDigital Library
- Udo Zölzer, Xavier Amatriain, Daniel Arfib, Jordi Bonada, Giovanni De Poli, Pierre Dutilleux, Gianpaolo Evangelista, Florian Keiler, Alex Loscos, Davide Rocchesso, 2002. DAFX-Digital audio effects. John Wiley & Sons.Google Scholar
Recommendations
Towards Collaborative Convergence: Quantifying Collaboration Quality with Automated Co-located Collaboration Analytics
LAK22: LAK22: 12th International Learning Analytics and Knowledge ConferenceCollaboration is one of the four important 21st-century skills. With the pervasive use of sensors, interest on co-located collaboration (CC) has increased lately. Most related literature used the audio modality to detect indicators of collaboration (...
Analysing Verbal Communication in Embodied Team Learning Using Multimodal Data and Ordered Network Analysis
Artificial Intelligence in EducationAbstractIn embodied team learning activities, students are expected to learn to collaborate with others while freely moving in a physical learning space to complete a shared goal. Students can thus interact in various team configurations, resulting in ...
Co-located Collaboration Analytics
ICMI '19: 2019 International Conference on Multimodal InteractionCollaboration is an important skill of the 21st century. It can take place in an online (or remote) setting or in a co-located (or face-to-face) setting. With the large scale adoption of sensor use, studies on co-located collaboration (CC) has gained ...
Comments