skip to main content
10.1145/3577190.3614108acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

WiFiTuned: Monitoring Engagement in Online Participation by Harmonizing WiFi and Audio

Published: 09 October 2023 Publication History

Abstract

This paper proposes a multi-modal, non-intrusive and privacy preserving system WiFiTuned for monitoring engagement in online participation i.e., meeting/classes/seminars. It uses two sensing modalities i.e., WiFi CSI and audio for the same. WiFiTuned detects the head movements of participants during online participation through WiFi CSI and detects the speaker’s intent through audio. Then it correlates the two to detect engagement. We evaluate WiFiTuned with 22 participants and observe that it can detect the engagement level with an average accuracy of more than .

Supplemental Material

MP4 File
This work proposes a multi-modal, non-intrusive and privacy preserving system WiFiTuned for monitoring engagement in online participation i.e., meetings/classes/seminars. It uses two sensing modalities i.e., WiFi CSI and audio, for the same. WiFiTuned detects participants' head movements during online participation through WiFi CSI and detects the speaker?s intent through audio. Then, it correlates the two to detect engagement. We evaluated WiFiTuned with 22 participants and observed that it can detect the engagement level with an average accuracy of more than 86%.
MP4 File
This wrok proposes a multi-modal, non-intrusive and privacy preserving system WiFiTuned for monitoring engagement in online participation i.e., meetings/classes/seminars. It uses two sensing modalities i.e., WiFi CSI and audio for the same. WiFiTuned detects the head movements of participants during online participation through WiFi CSI and detects the speaker?s intent through audio. Then it correlates the two to detect engagement.We evaluate WiFiTuned with 22 participants and observe that it can detect the engagement level with an average accuracy of more than 86%.
MP4 File
This paper proposes a multi-modal, non-intrusive and privacy preserving system WiFiTuned for monitoring engagement in online participation i.e., meetings/classes/seminars. It uses two sensing modalities i.e., WiFi CSI and audio for the same. WiFiTuned detects the head movements of participants during online participation through WiFi CSI and detects the speaker?s intent through audio. Then it correlates the two to detect engagement.We evaluate WiFiTuned with 22 participants and observe that it can detect the engagement level with an average accuracy of more than 86%.

References

[1]
Arqam M Al-Nuimi and Ghassan J Mohammed. 2021. Face Direction Estimation based on Mediapipe Landmarks. In ICCITM 2021. IEEE, 185–190.
[2]
Sinem Aslan, Nese Alyuz, Cagri Tanriover, Sinem E Mete, Eda Okur, Sidney K D’Mello, and Asli Arslan Esme. 2019. Investigating the impact of a real-time, multimodal student engagement analytics technology in authentic classrooms. In CHI 2019. 1–12.
[3]
Carlos Busso, Zhigang Deng, Michael Grimm, Ulrich Neumann, and Shrikanth Narayanan. 2007. Rigid head motion in expressive speech animation: Analysis and synthesis. IEEE transactions on audio, speech, and language processing 15, 3 (2007), 1075–1086.
[4]
Hancheng Cao, Chia-Jung Lee, Shamsi Iqbal, Mary Czerwinski, Priscilla NY Wong, Sean Rintel, Brent Hecht, Jaime Teevan, and Longqi Yang. 2021. Large scale analysis of multitasking behavior during remote meetings. In CHI 2021. 1–13.
[5]
Cheng Chang, Cheng Zhang, Lei Chen, and Yang Liu. 2018. An ensemble model using face and body tracking for engagement detection. In ICMI 2018. 616–622.
[6]
Jun-Ho Choi, Marios Constantinides, Sagar Joglekar, and Daniele Quercia. 2021. KAIROS: Talking heads and moving bodies for successful meetings. In Proc. HOTMOBILE 2021. 30–36.
[7]
Snigdha Das, Sandip Chakraborty, and Bivas Mitra. 2021. Quantifying Students’ Involvement during Virtual Classrooms: A Meeting Wrapper for the Teachers. In India HCI 2021. 133–139.
[8]
Betsy DiSalvo, Dheeraj Bandaru, Qiaosi Wang, Hong Li, and Thomas Plötz. 2022. Reading the Room: Automated, Momentary Assessment of Student Engagement in the Classroom: Are We There Yet?Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1–26.
[9]
Nan Gao, Wei Shao, Mohammad Saiedur Rahaman, and Flora D Salim. 2020. n-gage: Predicting in-class emotional, behavioural and cognitive engagement in the wild. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3 (2020), 1–26.
[10]
Ruiyang Gao, Wenwei Li, Yaxiong Xie, Enze Yi, Leye Wang, Dan Wu, and Daqing Zhang. 2022. Towards Robust Gesture Recognition by Characterizing the Sensing Quality of WiFi Signals. Proc. Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 1 (2022), 1–26.
[11]
Yang Gao, Yincheng Jin, Seokmin Choi, Jiyang Li, Junjie Pan, Lin Shu, Chi Zhou, and Zhanpeng Jin. 2021. SonicFace: Tracking Facial Expressions Using a Commodity Microphone Array. Proceedings of the ACM on IMWUT 5, 4 (2021), 1–33.
[12]
Banda Gerald. 2018. A brief review of independent, dependent and one sample t-test. International journal of applied mathematics and theoretical physics 4, 2 (2018), 50–54.
[13]
Steven M. Hernandez and Eyuphan Bulut. 2020. Lightweight and Standalone IoT Based WiFi Sensing for Active Repositioning and Mobility. In WoWMoM 2020. Cork, Ireland.
[14]
Amelia R Hunt and Alan Kingstone. 2003. Covert and overt voluntary attention: linked or independent?Cognitive Brain Research 18, 1 (2003), 102–105.
[15]
Hui Jiang, Chong-Wah Ngo, and Hung-Khoon Tan. 2006. Gestalt-based feature similarity measure in trademark database. Pattern recognition 39, 5 (2006), 988–1001.
[16]
Pragma Kar, Samiran Chattopadhyay, and Sandip Chakraborty. 2020. Gestatten: Estimation of User’s Attention in Mobile MOOCs From Eye Gaze and Gaze Gesture Tracking. Proc. ACM Hum.-Comput. Interact. 4, EICS (2020), 1–32.
[17]
Amanda Lacy, Seth Polsley, Samantha Ray, and Tracy Hammond. 2022. A seat at the virtual table: Emergent inclusion in remote meetings. Proceedings of the ACM on HCI 6, CSCW2 (2022), 1–20.
[18]
Andrew Lepp, Jacob E Barkley, Aryn C Karpinski, and Shweta Singh. 2019. College students’ multitasking behavior in online versus face-to-face courses. Sage Open 9, 1 (2019), 2158244018824505.
[19]
Shuai Ma, Taichang Zhou, Fei Nie, and Xiaojuan Ma. 2022. Glancee: An Adaptable System for Instructors to Grasp Student Learning Status in Synchronous Online Classes. In CHI Conference on Human Factors in Computing Systems. 1–25.
[20]
Yongsen Ma, Gang Zhou, and Shuangquan Wang. 2019. WiFi sensing with channel state information: A survey. ACM Computing Surveys (CSUR) 52, 3 (2019), 1–36.
[21]
Arien Mack. 2003. Inattentional blindness: Looking without seeing. Current directions in psychological science 12, 5 (2003), 180–184.
[22]
Gloria Mark, Shamsi T Iqbal, Mary Czerwinski, Paul Johns, and Akane Sano. 2016. Neurotics can’t focus: An in situ study of online multitasking in the workplace. In Proceedings of the 2016 CHI conference on human factors in computing systems. 1739–1744.
[23]
Omid Mohamad Nezami, Mark Dras, Len Hamey, Deborah Richards, Stephen Wan, and Cécile Paris. 2020. Automatic recognition of student engagement using deep learning and facial expression. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 273–289.
[24]
Hamed Monkaresi, Nigel Bosch, Rafael A Calvo, and Sidney K D’Mello. 2016. Automated detection of engagement using video-based estimation of facial expressions and heart rate. IEEE Transactions on Affective Computing 8, 1 (2016), 15–28.
[25]
Kazuhiro Otsuka, Keisuke Kasuga, and Martina Köhler. 2018. Estimating visual focus of attention in multiparty meetings using deep convolutional neural networks. In ICMI 2018. 191–199.
[26]
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim, Kyungtae Lim, Jongwon Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park, Alice Oh, Jungwoo Ha, and Kyunghyun Cho. 2021. KLUE: Korean Language Understanding Evaluation. arxiv:2105.09680 [cs.CL]
[27]
Ronald K. Pearson, Yrjö Neuvo, Jaakko Astola, and Moncef Gabbouj. 2015. The class of generalized hampel filters. In 2015 23rd European Signal Processing Conference (EUSIPCO). 2501–2505.
[28]
Rupendra Raavi, Mansour Alqarni, and Patrick CK Hung. 2022. Implementation of Machine Learning for CAPTCHAs Authentication Using Facial Recognition. In ICDSIS. IEEE, 1–5.
[29]
Arta Rosaen and Lidiman Sinaga. 2012. Speech Function in Feature Stories in Reader’s Digest. Linguistica 1, 1 (2012), 146423.
[30]
Ognjen Rudovic, Meiru Zhang, Bjorn Schuller, and Rosalind Picard. 2019. Multi-modal active learning from human data: A deep reinforcement learning approach. In ICMI 2019. 6–15.
[31]
Sima Siami-Namini, Neda Tavakoli, and Akbar Siami Namin. 2019. The Performance of LSTM and BiLSTM in Forecasting Time Series. In 2019 IEEE International Conference on Big Data (Big Data). 3285–3292.
[32]
Tripti Singh, Mohan Mohadikar, Shilpa Gite, Shruti Patil, Biswajeet Pradhan, and Abdullah Alamri. 2021. Attention span prediction using head-pose estimation with deep neural networks. IEEE Access 9 (2021), 142632–142643.
[33]
Mohamed Soltani, Hafed Zarzour, and Mohamed Chaouki Babahenini. 2018. Facial emotion detection in massive open online courses. In World Conference on Information Systems and Technologies. Springer, 277–286.
[34]
Wei Sun, Yunzhi Li, Feng Tian, Xiangmin Fan, and Hongan Wang. 2019. How Presenters Perceive and React to Audience Flow Prediction In-situ: An Explorative Study of Live Online Lectures. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–19.
[35]
Jingjing Wang, Xianqing Wang, Jishen Peng, Jun Gyu Hwang, and Joon Goo Park. 2021. Indoor Fingerprinting Localization Based on Fine-grained CSI using Principal Component Analysis. In 2021 Twelfth International Conference on Ubiquitous and Future Networks (ICUFN). 322–327.
[36]
Wei Wang, Alex X Liu, Muhammad Shahzad, Kang Ling, and Sanglu Lu. 2015. Understanding and modeling of wifi signal based human activity recognition. In MobiCom 2015. 65–76.
[37]
Wei Wang, Alex X Liu, Muhammad Shahzad, Kang Ling, and Sanglu Lu. 2017. Device-free human activity recognition using commercial WiFi devices. IEEE Journal on Selected Areas in Communications 35, 5 (2017), 1118–1131.
[38]
Zhengyang Wang, Sheng Chen, Wei Yang, and Yang Xu. 2021. Environment-Independent Wi-Fi Human Activity Recognition with Adversarial Network. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 3330–3334.
[39]
Dan Wu, Daqing Zhang, Chenren Xu, Yasha Wang, and Hao Wang. 2016. WiDir: walking direction estimation using wireless signals. In Proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing. 351–362.
[40]
Yong Zhang, Qingqing Liu, Yujie Wang, and Guangwei Yu. 2022. CSI-Based Location-Independent Human Activity Recognition Using Feature Fusion. IEEE Transactions on Instrumentation and Measurement 71 (2022), 1–12.
[41]
Yue Zheng, Yi Zhang, Kun Qian, Guidong Zhang, Yunhao Liu, Chenshu Wu, and Zheng Yang. 2019. Zero-effort cross-domain gesture recognition with Wi-Fi. In MobiSys 2019. 313–325.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '23: Proceedings of the 25th International Conference on Multimodal Interaction
October 2023
858 pages
ISBN:9798400700552
DOI:10.1145/3577190
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 October 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. WiFi sensing
  2. engagement detection
  3. multimodal fusion

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ICMI '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 221
    Total Downloads
  • Downloads (Last 12 months)137
  • Downloads (Last 6 weeks)9
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media