skip to main content
research-article

SensCon: Embedding Physiological Sensing into Virtual Reality Controllers

Published: 13 September 2023 Publication History

Abstract

Virtual reality experiences increasingly use physiological data for virtual environment adaptations to evaluate user experience and immersion. Previous research required complex medical-grade equipment to collect physiological data, limiting real-world applicability. To overcome this, we present SensCon for skin conductance and heart rate data acquisition. To identify the optimal sensor location in the controller, we conducted a first study investigating users' controller grasp behavior. In a second study, we evaluated the performance of SensCon against medical-grade devices in six scenarios regarding user experience and signal quality. Users subjectively preferred SensCon in terms of usability and user experience. Moreover, the signal quality evaluation showed satisfactory accuracy across static, dynamic, and cognitive scenarios. Therefore, SensCon reduces the complexity of capturing and adapting the environment via real-time physiological data. By open-sourcing SensCon, we enable researchers and practitioners to adapt their virtual reality environment effortlessly. Finally, we discuss possible use cases for virtual reality-embedded physiological sensing.

Supplementary Material

MP4 File (v7mhci223.mp4)
Supplemental video

References

[1]
Karan Ahuja, Sven Mayer, Mayank Goel, and Chris Harrison. 2021. Pose-on-the-Go: Approximating User Pose with Smartphone Sensor Fusion and Inverse Kinematics. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 9, 12 pages. https://doi.org/10.1145/3411764.3445582
[2]
Karan Ahuja, Vivian Shen, Cathy Mengying Fang, Nathan Riopelle, Andy Kong, and Chris Harrison. 2022. Controller-Pose: Inside-Out Body Capture with VR Controller Cameras. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 108, 13 pages. https://doi.org/10.1145/3491102.3502105
[3]
Aymen A. Alian and Kirk H. Shelley. 2014. Photoplethysmography. Best Practice & Research Clinical Anaesthesiology 28, 4 (2014), 395--406. https://doi.org/10.1016/j.bpa.2019.05.008
[4]
Jennifer Allanson and Stephen H. Fairclough. 2004. A research agenda for physiological computing. Interacting with computers 16, 5 (2004), 857--878. https://doi.org/10.1016/j.intcom.2004.08.001
[5]
John Allen. 2007. Photoplethysmography and its application in clinical physiological measurement. Physiological measurement 28, 3 (2007), R1. https://doi.org/10.1088/0967--3334/28/3/r01
[6]
Douglas G Altman and J Martin Bland. 1983. Measurement in medicine: the analysis of method comparison studies. Journal of the Royal Statistical Society: Series D (The Statistician) 32, 3 (1983), 307--317. https://doi.org/10.2307/2987937
[7]
K. Carrie Armel and Vilayanur S. Ramachandran. 2003. Projecting sensations to external objects: evidence from skin conductance response. Proceedings of the Royal Society of London. Series B: Biological Sciences 270, 1523 (2003), 1499--1506. https://doi.org/10.1098/rspb.2003.2364
[8]
Ebrahim Babaei, Benjamin Tag, Tilman Dingler, and Eduardo Velloso. 2021. A Critique of Electrodermal Activity Practices at CHI. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 177, 14 pages. https://doi.org/10.1145/3411764.3445370
[9]
Anjana Bali and Amteshwar Singh Jaggi. 2015. Clinical experimental stress studies: methods and assessment. Reviews in the Neurosciences 26, 5 (2015), 555--579. https://doi.org/10.1515/revneuro-2015-0004
[10]
Thomas Baumgartner, Lilian Valko, Michaela Esslen, and Lutz Jäncke. 2006. Neural correlate of spatial presence in an arousing and noninteractive virtual reality: an EEG and psychophysiology study. CyberPsychology & Behavior 9, 1 (2006), 30--45. https://doi.org/10.1089/cpb.2006.9.30
[11]
Imogen H. Bell, Jennifer Nicholas, Mario Alvarez-Jimenez, Andrew Thompson, and Lucia Valmaggia. 2022. Virtual reality as a clinical tool in mental health research and practice. Dialogues in clinical neuroscience (2022). https://doi.org/10.31887/DCNS.2020.22.2/lvalmaggia
[12]
Gershon Ben-Shakhar. 1985. Standardization within individuals: A simple method to neutralize individual differences in skin conductance. Psychophysiology 22, 3 (1985), 292--299.
[13]
Mathias Benedek and Christian Kaernbach. 2010. Decomposition of skin conductance data by means of nonnegative deconvolution. psychophysiology 47, 4 (2010), 647--658. https://doi.org/10.1111/j.1469--8986.2009.00972.x
[14]
Guillermo Bernal, Tao Yang, Abhinandan Jain, and Pattie Maes. 2018. PhysioHMD: A Conformable, Modular Toolkit for Collecting Physiological Data from Head-Mounted Displays. In Proceedings of the 2018 ACM International Symposium on Wearable Computers (Singapore, Singapore) (ISWC '18). Association for Computing Machinery, New York, NY, USA, 160--167. https://doi.org/10.1145/3267242.3267268
[15]
J Martin Bland and Douglas G Altman. 1999. Measuring agreement in method comparison studies. Statistical Methods in Medical Research 8, 2 (1999), 135--160. https://doi.org/10.1177/096228029900800204 arXiv:https://doi.org/10.1177/096228029900800204 10501650.
[16]
Andrew Garrod Bosworth. 2022. Body pose estimation using self-tracked controllers. US Patent 11,507,203.
[17]
Gary Bradski and Adrian Kaehler. 2008. Learning OpenCV: Computer vision with the OpenCV library. O'Reilly Media, Inc., Sebastopol, California, United States.
[18]
John T. Cacioppo, Gary G. Berntson, William B. Malarkey, Janice K. Kiecolt-Glaser, John F. Sheridan, Kirsten M. Poehlmann, Mary H. Burleson, John M. Ernst, Louise C. Hawkley, and Ronald Glaser. 1998. Autonomic, neuroendocrine, and immune responses to psychological stress: The reactivity hypothesis A. Annals of the New York Academy of Sciences 840, 1 (1998), 664--673. https://doi.org/10.1111/j.1749--6632.1998.tb09605.x
[19]
John T. Cacioppo, Louis G. Tassinary, and Gary Berntson. 2007. Handbook of psychophysiology. Cambridge university press, Cambridge, UK. https://doi.org/10.1017/9781107415782
[20]
Bryson Carrier, Brenna Barrios, Brayden D Jolley, and James W Navalta. 2020. Validity and reliability of physiological data in applied settings measured by wearable technology: A rapid systematic review. Technologies 8, 4 (2020), 70. https://doi.org/10.3390/technologies8040070
[21]
Keyne Charlot, Jeremy Cornolo, Julien V Brugniaux, Jean-Paul Richalet, and Aurélien Pichon. 2009. Interchangeability between heart rate and photoplethysmography variabilities during sympathetic stimulations. Physiological measurement 30, 12 (2009), 1357. https://doi.org/10.1088/0967--3334/30/12/005
[22]
Francesco Chiossi, Thomas Kosch, Luca Menghini, Steeven Villa, and Sven Mayer. 2023. SensCon: Embedding Physiological Sensing into Virtual Reality Controllers. https://doi.org/10.17605/OSF.IO/H9MJS
[23]
Francesco Chiossi, Changkun Ou, and Sven Mayer. 2023. Exploring Physiological Correlates of Visual Complexity Adaptation: Insights from EDA, ECG, and EEG Data for Adaptation Evaluation in VR Adaptive Systems. In ACM Conference on Human-Computer Interaction (CHI '23 Extended Abstracts). Association for Computing Machinery, Hamburg, Germany. https://doi.org/10.1145/3544549.3585624
[24]
Francesco Chiossi, Yagiz Turgut, Robin Welsch, and Sven Mayer. 2023. Adapting Visual Complexity Based on Electrodermal Activity Improves Working Memory Performance in Virtual Reality. Proc. ACM Hum.-Comput. Interact. 7, MHCI, Article 296 (sep 2023), 26 pages. https://doi.org/10.1145/3604243
[25]
Francesco Chiossi, Robin Welsch, Steeven Villa, Lewis Chuang, and Sven Mayer. 2022. Virtual Reality Adaptation Using Electrodermal Activity to Support the User Experience. Big Data and Cognitive Computing 6, 2 (2022), 55. https://doi.org/10.3390/bdcc6020055
[26]
Francesco Chiossi, Johannes Zagermann, Jakob Karolus, Nils Rodrigues, Priscilla Balestrucci, Daniel Weiskopf, Benedikt Ehinger, Tiare Feuchtner, Harald Reiterer, Lewis L. Chuang, Marc Ernst, Andreas Bulling, Sven Mayer, and Albrecht Schmidt. 2022. Adapting visualizations and interfaces to the user. it - Information Technology (2022). https://doi.org/10.1515/itit-2022-0035
[27]
Dongrae Cho, Jinsil Ham, Jooyoung Oh, Jeanho Park, Sayup Kim, Nak-Kyu Lee, and Boreom Lee. 2017. Detection of stress levels from biosignals measured in virtual reality environments using a kernel-based extreme learning machine. Sensors 17, 10 (2017), 2435. https://doi.org/10.3390/s17102435
[28]
Jongyoon Choi and Ricardo Gutierrez-Osuna. 2009. Using Heart Rate Monitors to Detect Mental Stress. In 2009 Sixth International Workshop on Wearable and Implantable Body Sensor Networks. IEEE, New York, NY, USA, 219--223. https://doi.org/10.1109/BSN.2009.13
[29]
Jonny Collins, Holger Regenbrecht, Tobias Langlotz, Yekta Said Can, Cem Ersoy, and Russell Butson. 2019. Measuring Cognitive Load and Insight: A Methodology Exemplified in a Virtual Reality Learning Context. In 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR '19). IEEE, New York, NY, USA, 351--362. https://doi.org/10.1109/ISMAR.2019.00033
[30]
Elena Di Lascio, Shkurta Gashi, and Silvia Santini. 2018. Unobtrusive Assessment of Students' Emotional Engagement during Lectures Using Electrodermal Activity Sensors. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 3, Article 103 (sep 2018), 21 pages. https://doi.org/10.1145/3264913
[31]
Dennis Dietz, Carl Oechsner, Changkun Ou, Francesco Chiossi, Fabio Sarto, Sven Mayer, and Andreas Butz. 2022. Walk This Beam: Impact of Different Balance Assistance Strategies and Height Exposure on Performance and Physiological Arousal in VR. In 28th ACM Symposium on Virtual Reality Software and Technology (Tsukuba, Japan) (VRST '22). Association for Computing Machinery, New York, NY, USA, Article 32, 12 pages. https://doi.org/10.1145/3562939.3567818
[32]
Sigrun Doberenz, Walton T Roth, Eileen Wollburg, Nina I Maslowski, and Sunyoung Kim. 2011. Methodological considerations in ambulatory skin conductance monitoring. International Journal of Psychophysiology 80, 2 (2011), 87--95. https://doi.org/10.1016/j.ijpsycho.2011.02.002
[33]
Nurettin Özgür Dogan. 2018. Bland-Altman analysis: A paradigm to understand correlation and agreement. Turkish journal of emergency medicine 18, 4 (2018), 139--141. https://doi.org/10.1016/j.tjem.2018.09.001
[34]
Henrik H. Ehrsson, Katja Wiech, Nikolaus Weiskopf, Raymond J. Dolan, and Richard E. Passingham. 2007. Threatening a rubber hand that you feel is yours elicits a cortical anxiety response. Proceedings of the National Academy of Sciences 104, 23 (2007), 9828--9833. https://doi.org/10.1073/pnas.0610011104
[35]
Sergio Estupiñán, Francisco Rebelo, Paulo Noriega, Carlos Ferreira, and Emília Duarte. 2014. Can virtual reality increase emotional responses (Arousal and Valence)? A pilot study. In Design, User Experience, and Usability. User Experience Design for Diverse Interaction Platforms and Environments: Third International Conference, DUXU 2014, Held as Part of HCI International 2014, Heraklion, Crete, Greece, June 22--27, 2014, Proceedings, Part II 3. Springer, Cham, Switzerland, 541--549. https://doi.org/10.1007/978--3--319-07626--3_51
[36]
Anne M Euser, Friedo W Dekker, and Saskia Le Cessie. 2008. A practical approach to Bland-Altman plots and variation coefficients for log transformed variables. Journal of clinical epidemiology 61, 10 (2008), 978--982. https://doi.org/10.1016/j.jclinepi.2007.11.003
[37]
Society for Psychophysiological Research Ad Hoc Committee on Electrodermal Measures, Wolfram Boucsein, Don C. Fowles, Sverre Grimnes, Gershon Ben-Shakhar, Walton T. Roth, Michael E. Dawson, and Diane L. Filion. 2012. Publication Recommendations for Electrodermal Measurements. Psychophysiology 49, 8 (2012), 1017--1034. https://doi.org/10.1111/j.1469--8986.1981.tb03024.x
[38]
Jonathan Freeman, Steve E. Avons, Don E. Pearson, and Wijnand A. IJsselsteijn. 1999. Effects of sensory information and prior experience on direct subjective ratings of presence. Presence 8, 1 (1999), 1--13. https://doi.org/10.1162/105474699566017
[39]
Joachim Gavazzeni, Stefan Wiens, and Håkan Fischer. 2008. Age effects to negative arousal differ for self-report and electrodermal activity. Psychophysiology 45, 1 (2008), 148--151. https://doi.org/10.1111/j.1469--8986.2007.00596.x
[40]
Alireza Mazloumi Gavgani, Keith V Nesbitt, Karen L Blackmore, and Eugene Nalivaiko. 2017. Profiling subjective symptoms and autonomic changes associated with cybersickness. Autonomic Neuroscience 203 (2017), 41--50. https://doi.org/10.1016/j.autneu.2016.12.004
[41]
Davide Giavarina. 2015. Understanding bland altman analysis. Biochemia medica 25, 2 (2015), 141--151. https://hrcak.srce.hr/139769
[42]
Grace E. Giles, Caroline R. Mahoney, Tad T. Brunyé, Holly A. Taylor, and Robin B. Kanarek. 2014. Stress effects on mood, HPA axis, and autonomic response: comparison of three psychosocial stress paradigms. PloS one 9, 12 (2014), e113618. https://doi.org/10.1371/journal.pone.0113618
[43]
Gyeong-Nam Gimhae. 2013. Six human factors to acceptability of wearable computers. International Journal of Multimedia and Ubiquitous Engineering 8, 3 (2013), 103--114. http://www.sersc.org/journals/IJMUE/vol8_no3_2013/10.pdf
[44]
Francesco Giordana, Veselin Efremov, Gael Sourimant, Silvia Rasheva, Natasha Tatarchuk, and Callum James. 2018. Virtual Production in 'Book of the Dead': Technicolor's Genesis Platform, Powered by Unity. In ACM SIGGRAPH 2018 Real-Time Live! Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3229227.3229235
[45]
Stefan Gradl, Markus Wirth, Nico Mächtlinger, Romina Poguntke, Andrea Wonner, Nicolas Rohleder, and Bjoern M. Eskofier. 2019. The Stroop Room: A Virtual Reality-Enhanced Stroop Test. In 25th ACM Symposium on Virtual Reality Software and Technology (Parramatta, NSW, Australia) (VRST '19). Association for Computing Machinery, New York, NY, USA, Article 28, 12 pages. https://doi.org/10.1145/3359996.3364247
[46]
Klaus Gramann, Ryan Mckendrick, Carryl Baldwin, Raphaëlle Roy, Camille Jeunet, Ranjana Mehta, and Giovanni Vecchiato. 2021. Grand field challenges for cognitive neuroergonomics in the coming decade. Frontiers in Neuroergonomics 2 (2021). https://doi.org/10.3389/fnrgo.2021.643969
[47]
Catharina C. Grant, Dina C. J. van Rensburg, Nina Strydom, and Margaretha Viljoen. 2011. Importance of tachogram length and period of recording during noninvasive investigation of the autonomic nervous system. Annals of Noninvasive Electrocardiology 16, 2 (2011), 131--139. https://doi.org/10.1111/j.1542--474X.2011.00422.x
[48]
Simone Grassini and Karin Laumann. 2020. Questionnaire measures and physiological correlates of presence: A systematic review. Frontiers in psychology 11 (2020), 349. https://doi.org/10.3389/fpsyg.2020.00349
[49]
Mariam Hassib, Stefan Schneegass, Philipp Eiglsperger, Niels Henze, Albrecht Schmidt, and Florian Alt. 2017. EngageMeter: A System for Implicit Audience Engagement Sensing Using Electroencephalography. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 5114--5119. https://doi.org/10.1145/3025453.3025669
[50]
Mandy X. Hu, Femke Lamers, Eco J. C. de Geus, and Brenda WJH Penninx. 2016. Differential Autonomic Nervous System Reactivity in Depression and Anxiety During Stress Depending on Type of Stressor. Psychosomatic medicine 78, 5 (2016), 562--572. https://doi.org/10.1097/PSY.0000000000000313
[51]
Wen Huang, Rod D Roscoe, Mina C Johnson-Glenberg, and Scotty D Craig. 2021. Motivation, engagement, and performance across multiple virtual reality sessions and levels of immersion. Journal of Computer Assisted Learning 37, 3 (2021), 745--758. https://doi.org/10.1111/jcal.12520
[52]
Yi-Jheng Huang, Kang-Yi Liu, Suiang-Shyan Lee, and I-Cheng Yeh. 2021. Evaluation of a hybrid of hand gesture and controller inputs in virtual reality. International Journal of Human--Computer Interaction 37, 2 (2021), 169--180. https://doi.org/10.1080/10447318.2020.1809248
[53]
Weiwei Jiang, Kangning Yang, Maximiliane Windl, Francesco Chiossi, Benjamin Tag, Sven Mayer, and Zhanna Sarsenbayeva. 2022. Current Challenges of Using Wearable Devices for Online Emotion Sensing. arXiv preprint arXiv:2208.05206 (2022). https://doi.org/10.48550/arXiv.2208.05206
[54]
Cornelia Kappeler-Setz, Franz Gravenhorst, Johannes Schumm, Bert Arnrich, and Gerhard Tröster. 2013. Towards long term monitoring of electrodermal activity in daily life. Personal and ubiquitous computing 17, 2 (2013), 261--271. https://doi.org/10.1007/s00779-011-0463--4
[55]
Armagan Karahanoglu and Çigdem Erbug. 2011. Perceived Qualities of Smart Wearables: Determinants of User Acceptance. In Proceedings of the 2011 Conference on Designing Pleasurable Products and Interfaces (Milano, Italy) (DPPI '11). Association for Computing Machinery, New York, NY, USA, Article 26, 8 pages. https://doi.org/10.1145/2347504.2347533
[56]
Ladislav Karrach, Elena Pivarciová, and Pavol Bo?ek. 2020. Identification of QR code perspective distortion based on edge directions and edge projections analysis. Journal of imaging 6, 7 (2020), 67. https://doi.org/10.3390/jimaging6070067
[57]
Kwanguk Kim, M Zachary Rosenthal, David Zielinski, and Rachel Brady. 2012. Comparison of desktop, head mounted display, and six wall fully immersive systems using a stressful task. In 2012 IEEE Virtual Reality Workshops (VRW). IEEE, New York, NY, USA, 143--144. https://doi.org/10.1109/VR.2012.6180922
[58]
Jocelyne Kiss, Geoffrey Edwards, Ernesto Morales, Sherezada Ochoa, Bruno Bernier, et al. 2019. Enhancing Mobile VR Immersion: A Multimodal System of Neural Networks Approach to an IMU Gesture Controller. Electronic Imaging 2019, 2 (2019), 184--1. https://doi.org/10.2352/ISSN.2470--1173.2019.2.ERVR-184
[59]
Thomas Kosch, Markus Funk, Albrecht Schmidt, and Lewis L. Chuang. 2018. Identifying Cognitive Assistance with Mobile Electroencephalography: A Case Study with In-Situ Projections for Manual Assembly. Proc. ACM Hum.-Comput. Interact. 2, EICS, Article 11 (jun 2018), 20 pages. https://doi.org/10.1145/3229093
[60]
Thomas Kosch, Jakob Karolus, Havy Ha, and Albrecht Schmidt. 2019. Your Skin Resists: Exploring Electrodermal Activity As Workload Indicator During Manual Assembly. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (Valencia, Spain) (EICS '19). ACM, New York, NY, USA, Article 8, 5 pages. https://doi.org/10.1145/3319499.3328230
[61]
Thomas Kosch, Jakob Karolus, Johannes Zagermann, Harald Reiterer, Albrecht Schmidt, and Pawel Wozniak. 2023. A Survey on Measuring Cognitive Workload in Human-Computer Interaction. Comput. Surveys, Article 1 (jan 2023), 37 pages. https://doi.org/10.1145/3582272
[62]
George Alex Koulieris, Kaan Aksit, Michael Stengel, Rafal K Mantiuk, Katerina Mania, and Christian Richardt. 2019. Near-eye display and tracking technologies for virtual and augmented reality. In Computer Graphics Forum, Vol. 38. Wiley Online Library, Hoboken, New Jersey, USA, 493--519. https://doi.org/10.1111/cgf.13654
[63]
Peter J Lang, Mark K Greenwald, Margaret M Bradley, and Alfons O Hamm. 1993. Looking at pictures: Affective, facial, visceral, and behavioral reactions. Psychophysiology 30, 3 (1993), 261--273. https://doi.org/10.1111/j.1469--8986.1993.tb03352.x
[64]
László Laufer and Bottyán Németh. 2008. Predicting User Action from Skin Conductance. In Proceedings of the 13th International Conference on Intelligent User Interfaces (Gran Canaria, Spain) (IUI '08). Association for Computing Machinery, New York, NY, USA, 357--360. https://doi.org/10.1145/1378773.1378829
[65]
Huy Viet Le, Sven Mayer, Katrin Wolf, and Niels Henze. 2016. Finger Placement and Hand Grasp during Smartphone Interaction. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (San Jose, California, USA) (CHI EA '16). Association for Computing Machinery, New York, NY, USA, 2576--2584. https://doi.org/10.1145/2851581.2892462
[66]
James R Lewis. 2018. The system usability scale: past, present, and future. International Journal of Human--Computer Interaction 34, 7 (2018), 577--590. https://doi.org/10.1080/10447318.2018.1455307
[67]
Yue Li, Lu Zheng, and Xuewen Wang. 2019. Flexible and wearable healthcare sensors for visual reality health-monitoring. Virtual Reality & Intelligent Hardware 1, 4 (2019), 411--427. https://doi.org/10.1016/j.vrih.2019.08.001
[68]
Wan-Hua Lin, Dan Wu, Chunyue Li, Heye Zhang, and Yuan-Ting Zhang. 2014. Comparison of heart rate variability from PPG with that from ECG. In The international conference on health informatics. Springer, Cham, Switzerland, 213--215. https://doi.org/10.1007/978--3--319-03005-0_54
[69]
Tiffany Luong, Ferran Argelaguet, Nicolas Martin, and Anatole Lécuyer. 2020. Introducing mental workload assessment for the design of virtual reality training scenarios. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, New York, NY, USA, 662--671. https://doi.org/10.1109/VR46266.2020.00089
[70]
Tiffany Luong, Nicolas Martin, Anais Raison, Ferran Argelaguet, Jean-Marc Diverrez, and Anatole Lécuyer. 2020. Towards real-time recognition of users mental workload using integrated physiological sensors into a VR HMD. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, New York, NY, USA, 425--437. https://doi.org/10.1109/ISMAR50242.2020.00068
[71]
David T Lykken and Peter H Venables. 1971. Direct measurement of skin conductance: A proposal for standardization. Psychophysiology 8, 5 (1971), 656--672. https://doi.org/10.1111/j.1469--8986.1971.tb00501.x
[72]
Dominique Makowski, Tam Pham, Zen J. Lau, Jan C. Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and SH Chen. 2021. NeuroKit2: A Python toolbox for neurophysiological signal processing. Behavior research methods 53, 4 (2021), 1689--1696. https://doi.org/10.3758/s13428-020-01516-y
[73]
Srinivas Mantha, Michael F Roizen, Lee A Fleisher, Ronald Thisted, and Joseph Foss. 2000. Comparing methods of clinical measurement: reporting standards for Bland and Altman analysis. Anesthesia & Analgesia 90, 3 (2000), 593--602. https://doi.org/10.1097/00000539--200003000-00018
[74]
Javier Marín-Morales, Carmen Llinares, Jaime Guixeres, and Mariano Alcañiz. 2020. Emotion recognition in immersive virtual reality: From statistics to affective computing. Sensors 20, 18 (2020), 5163. https://doi.org/10.3390/s20185163
[75]
Michael Meehan, Brent Insko, Mary Whitton, and Frederick P. Brooks. 2002. Physiological Measures of Presence in Stressful Virtual Environments. ACM Trans. Graph. 21, 3 (jul 2002), 645--652. https://doi.org/10.1145/566654.566630
[76]
Luca Menghini, Nicola Cellini, Aimee Goldstone, Fiona C Baker, and Massimiliano de Zambotti. 2020. A standardized framework for testing the performance of sleep-tracking technology: step-by-step guidelines and open-source code. Sleep 44, 2 (10 2020). https://doi.org/10.1093/sleep/zsaa170 arXiv:https://academic.oup.com/sleep/article-pdf/44/2/zsaa170/36244924/zsaa170.pdf zsaa170.
[77]
Luca Menghini, Evelyn Gianfranchi, Nicola Cellini, Elisabetta Patron, Mariaelena Tagliabue, and Michela Sarlo. 2019. Stressing the accuracy: Wrist-worn wearable sensor validation over different conditions. Psychophysiology 56, 11 (2019), e13441. https://doi.org/10.1111/psyp.13441
[78]
Nir Milstein and Ilanit Gordon. 2020. Validating measures of electrodermal activity and heart rate variability derived from the empatica e4 utilized in research settings that involve interactive dyadic states. Frontiers in Behavioral Neuroscience (2020), 148. https://doi.org/10.3389/fnbeh.2020.00148
[79]
John E. Muñoz, M. Cameirão, S. Bermúdez i Badia, and E. Rubio Gouveia. 2018. Closing the Loop in Exergaming - Health Benefits of Biocybernetic Adaptation in Senior Adults. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (Melbourne, VIC, Australia) (CHI PLAY '18). Association for Computing Machinery, New York, NY, USA, 329--339. https://doi.org/10.1145/3242671.3242673
[80]
Zhigeng Pan, Adrian David Cheok, Hongwei Yang, Jiejie Zhu, and Jiaoying Shi. 2006. Virtual reality and mixed reality for virtual learning environments. Computers & Graphics 30, 1 (2006), 20--28. https://doi.org/10.1016/j.cag.2005.10.004
[81]
Andrew F. H. Payne, Anne M. Schell, and Michael E. Dawson. 2016. Lapses in skin conductance responding across anatomical sites: Comparison of fingers, feet, forehead, and wrist. Psychophysiology 53, 7 (2016), 1084--1092. https://doi.org/10.1111/psyp.12643
[82]
Riccardo Pernice, Michal Javorka, Jana Krohova, Barbora Czippelova, Zuzana Turianikova, Alessandro Busacca, and Luca Faes. 2018. Reliability of short-term heart rate variability indexes assessed through photoplethysmography. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC'18). IEEE, New York, NY, USA, 5610--5513. https://doi.org/10.1109/EMBC.2018.8513634
[83]
Livia Petrescu, Catalin Petrescu, Oana Mitru t, Gabriela Moise, Alin Moldoveanu, Florica Moldoveanu, and Marius Leordeanu. 2020. Integrating biosignals measurement in virtual reality environments for anxiety detection. Sensors 20, 24 (2020), 7088. https://doi.org/10.3390/s20247088
[84]
C.C.Y. Poon and Y.T. Zhang. 2005. Cuff-less and Noninvasive Measurements of Arterial Blood Pressure by Pulse Transit Time. In 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference. IEEE, New York, NY, USA, 5877--5880. https://doi.org/10.1109/IEMBS.2005.1615827
[85]
Hugo F Posada-Quintero and Jeffrey B Bolkhovsky. 2019. Machine learning models for the identification of cognitive tasks using autonomic reactions from heart rate variability and electrodermal activity. Behavioral Sciences 9, 4 (2019), 45. https://doi.org/10.3390/bs9040045
[86]
Alan Poston. 2000. Human engineering design data digest. Washington, DC: Department of Defense Human Factors Engineering Technical Advisory Group (2000), 61--75.
[87]
Ramesh Kumar Ramakrishnan, Rahul Gavas, Venkata Subramaninan Viraraghavan, Lalit Kumar Hissaria, Arpan Pal, and P. Balamuralidhar. 2020. Monitoring The Early Physiological Effects of Fitness Regimes Using PPG Signals. In 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). IEEE, New York, NY, USA, 1--5. https://doi.org/10.1109/PerComWorkshops48775.2020.9156258
[88]
Ian M. Raugh, Hannah C. Chapman, Lisa A. Bartolomeo, Cristina Gonzalez, and Gregory P. Strauss. 2019. A comprehensive review of psychophysiological applications for ecological momentary assessment in psychiatric populations. Psychological assessment 31, 3 (2019), 304. https://doi.org/10.1037/pas0000651
[89]
Giuseppe Riva, Fabrizia Mantovani, Claret Samantha Capideville, Alessandra Preziosa, Francesca Morganti, Daniela Villani, Andrea Gaggioli, Cristina Botella, and Mariano Alcañiz. 2007. Affective interactions using virtual reality: the link between presence and emotions. Cyberpsychology & behavior 10, 1 (2007), 45--56. https://doi.org/10.1089/cpb.2006.9993
[90]
Justas alkevicius, Robertas Dama?evicius, Rytis Maskeliunas, and Ilona Laukiene. 2019. Anxiety level recognition for virtual reality therapy system using physiological signals. Electronics 8, 9 (2019), 1039. https://doi.org/10.3390/electronics8091039
[91]
Axel Schäfer and Jan Vagedes. 2013. How accurate is pulse rate variability as an estimate of heart rate variability?: A review on studies comparing photoplethysmographic technology with an electrocardiogram. International journal of cardiology 166, 1 (2013), 15--29. https://doi.org/10.1016/j.ijcard.2012.03.119
[92]
Valentin Schwind, Pascal Knierim, Nico Haas, and Niels Henze. 2019. Using Presence Questionnaires in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300590
[93]
Nandakumar Selvaraj, Ashok Jaryal, Jayashree Santhosh, Kishore K. Deepak, and Sneh Anand. 2008. Assessment of heart rate variability derived from finger-tip photoplethysmography as compared to electrocardiography. Journal of medical engineering & technology 32, 6 (2008), 479--484. https://doi.org/10.1080/03091900701781317
[94]
Cornelia Setz, Bert Arnrich, Johannes Schumm, Roberto La Marca, Gerhard Tröster, and Ulrike Ehlert. 2010. Discriminating Stress From Cognitive Load Using a Wearable EDA Device. IEEE Transactions on Information Technology in Biomedicine 14, 2 (2010), 410--417. https://doi.org/10.1109/TITB.2009.2036164
[95]
Mel Slater. 2004. How colorful was your day? Why questionnaires cannot assess presence in virtual environments. Presence 13, 4 (2004), 484--493. https://doi.org/10.1162/1054746041944849
[96]
Mel Slater. 2018. Immersion and the illusion of presence in virtual reality. British journal of psychology 109, 3 (2018), 431--433. https://doi.org/10.1111/bjop.12305
[97]
Phyllis K Stein, Ali A Ehsani, Peter P Domitrovich, Robert E Kleiger, and Jeffrey N Rottman. 1999. Effect of exercise training on heart rate variability in healthy older adults. American heart journal 138, 3 (1999), 567--576. https://doi.org/10.1007/s00421-009--1089--3
[98]
Paul Streli, Rayan Armani, Yi Fei Cheng, and Christian Holz. 2023. HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction Using Inertial Sensing. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 310, 16 pages. https://doi.org/10.1145/3544548.3581468
[99]
Erin Treacy Solovey, Daniel Afergan, Evan M. Peck, Samuel W. Hincks, and Robert J. K. Jacob. 2015. Designing Implicit Interfaces for Physiological Computing: Guidelines and Lessons Learned Using FNIRS. ACM Trans. Comput.-Hum. Interact. 21, 6, Article 35 (jan 2015), 27 pages. https://doi.org/10.1145/2687926
[100]
Marieke van Dooren, Joris H Janssen, et al. 2012. Emotional sweating across the body: Comparing 16 different skin conductance measurement locations. Physiology & behavior 106, 2 (2012), 298--304. https://doi.org/10.1016/j.physbeh.2012.01.020
[101]
Sjoerd van Halem, Eeske Van Roekel, Lara Kroencke, Niclas Kuper, and Jaap Denissen. 2020. Moments that matter? On the complexity of using triggers based on skin conductance to sample arousing events within an experience sampling framework. European Journal of Personality 34, 5 (2020), 794--807. https://doi.org/10.1002/per.2252
[102]
Don van Ravenzwaaij, Rei Monden, Jorge N Tendeiro, and John Ioannidis. 2019. Bayes factors for superiority, non-inferiority, and equivalence designs. BMC medical research methodology 19, 1 (2019), 1--12. https://doi.org/10.1186/s12874-019-0699--7
[103]
José Varela-Aldás, Esteban M Fuentes, Guillermo Palacios-Navarro, and Iván García-Magariño. 2019. A comparison of heart rate in normal physical activity vs. immersive virtual reality exergames. In International Conference on Human Systems Engineering and Design: Future Trends and Applications. Springer, Cham, Switzerland, 684--689. https://doi.org/10.1007/978--3-030--27928--8_104
[104]
Viswanath Venkatesh, Michael G. Morris, Gordon B. Davis, and Fred D. Davis. 2003. User acceptance of information technology: Toward a unified view. MIS quarterly (2003), 425--478. https://doi.org/10.2307/30036540
[105]
Roger Vilardaga, Jonathan B. Bricker, and Michael G. McDonell. 2014. The promise of mobile technologies and single case designs for the study of individuals in their natural environment. Journal of Contextual Behavioral Science 3, 2 (2014), 148--153. https://doi.org/10.1016/j.jcbs.2014.03.003
[106]
Francesco Walker, Debargha Dey, Marieke Martens, Bastian Pfleging, Berry Eggen, and Jacques Terken. 2019. Feeling-of-Safety Slider: Measuring Pedestrian Willingness to Cross Roads in Field Interactions with Vehicles. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA '19). Association for Computing Machinery, New York, NY, USA, 1--6. https://doi.org/10.1145/3290607.3312880
[107]
Michael Wilkinson, Zachary H. Pugh, Aaron Crowson, Jing Feng, Christopher B. Mayhorn, and Douglas J. Gillan. 2019. Seeing in slow motion: manipulating arousal in virtual reality. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 63. SAGE Publications Sage CA, Los Angeles, CA, 1649--1653. https://doi.org/10.1177/1071181319631512
[108]
Mark G. Williams, Andrew Mathews, and Colin MacLeod. 1996. The emotional Stroop task and psychopathology. Psychological bulletin 120, 1 (1996), 3. https://doi.org/10.1037/0033--2909.120.1.3
[109]
Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI '11). Association for Computing Machinery, New York, NY, USA, 143--146. https://doi.org/10.1145/1978942.1978963
[110]
Ruiyang Yin, Depeng Wang, Shufang Zhao, Zheng Lou, and Guozhen Shen. 2021. Wearable sensors-enabled human--machine interaction systems: from design to application. Advanced Functional Materials 31, 11 (2021), 2008936. https://doi.org/10.1002/adfm.202008936
[111]
Tomas Ysehak Abay, Kamran Shafqat, and Panayiotis A Kyriacou. 2019. Perfusion changes at the forehead measured by photoplethysmography during a head-down tilt protocol. Biosensors 9, 2 (2019), 71. https://doi.org/10.3390/bios9020071
[112]
Clint Zeagler. 2017. Where to Wear It: Functional, Technical, and Social Considerations in on-Body Location for Wearable Technology 20 Years of Designing for Wearability. In Proceedings of the 2017 ACM International Symposium on Wearable Computers (Maui, Hawaii) (ISWC '17). Association for Computing Machinery, New York, NY, USA, 150--157. https://doi.org/10.1145/3123021.3123042
[113]
Yang Zhang, Wolf Kienzle, Yanjun Ma, Shiu S. Ng, Hrvoje Benko, and Chris Harrison. 2019. ActiTouch: Robust Touch Detection for On-Skin AR/VR Interfaces. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST '19). Association for Computing Machinery, New York, NY, USA, 1151--1159. https://doi.org/10.1145/3332165.3347869

Cited By

View all
  • (2024)Towards Everyday Physiological Monitoring: A Sock Prototype for Electrodermal Activity Measurements2024 9th International Conference on Smart and Sustainable Technologies (SpliTech)10.23919/SpliTech61897.2024.10612408(1-6)Online publication date: 25-Jun-2024
  • (2024)Evaluating Typing Performance in Different Mixed Reality Manifestations using Physiological FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36981428:ISS(377-406)Online publication date: 24-Oct-2024
  • (2024)An Empirical Study on Social Anxiety in a Virtual Environment through Mediating Variables and Multiple Sensor DataProceedings of the ACM on Human-Computer Interaction10.1145/36869778:CSCW2(1-24)Online publication date: 8-Nov-2024
  • Show More Cited By

Index Terms

  1. SensCon: Embedding Physiological Sensing into Virtual Reality Controllers

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Human-Computer Interaction
    Proceedings of the ACM on Human-Computer Interaction  Volume 7, Issue MHCI
    MHCI
    September 2023
    1017 pages
    EISSN:2573-0142
    DOI:10.1145/3624512
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 13 September 2023
    Published in PACMHCI Volume 7, Issue MHCI

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. controller
    2. embedded systems
    3. physiological computing
    4. physiological interaction
    5. virtual reality
    6. wearable computing

    Qualifiers

    • Research-article

    Funding Sources

    • Deutsche Forschungsgemeinschaft (DFG)

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)116
    • Downloads (Last 6 weeks)13
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Towards Everyday Physiological Monitoring: A Sock Prototype for Electrodermal Activity Measurements2024 9th International Conference on Smart and Sustainable Technologies (SpliTech)10.23919/SpliTech61897.2024.10612408(1-6)Online publication date: 25-Jun-2024
    • (2024)Evaluating Typing Performance in Different Mixed Reality Manifestations using Physiological FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36981428:ISS(377-406)Online publication date: 24-Oct-2024
    • (2024)An Empirical Study on Social Anxiety in a Virtual Environment through Mediating Variables and Multiple Sensor DataProceedings of the ACM on Human-Computer Interaction10.1145/36869778:CSCW2(1-24)Online publication date: 8-Nov-2024
    • (2024)Comparing the Effects of Visual, Haptic, and Visuohaptic Encoding on Memory Retention of Digital Objects in Virtual RealityProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685349(1-13)Online publication date: 13-Oct-2024
    • (2024)Your Eyes on Speed: Using Pupil Dilation to Adaptively Select Speed-Reading Parameters in Virtual RealityProceedings of the ACM on Human-Computer Interaction10.1145/36765318:MHCI(1-17)Online publication date: 24-Sep-2024
    • (2024)Multimodal Detection of External and Internal Attention in Virtual Reality using EEG and Eye Tracking FeaturesProceedings of Mensch und Computer 202410.1145/3670653.3670657(29-43)Online publication date: 1-Sep-2024
    • (2024)3DA: Assessing 3D-Printed Electrodes for Measuring Electrodermal ActivityExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650938(1-7)Online publication date: 11-May-2024
    • (2024)Closing the Loop: The Effects of Biofeedback Awareness on Physiological Stress Response Using Electrodermal Activity in Virtual RealityExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650830(1-7)Online publication date: 11-May-2024
    • (2024)PhysioCHI: Towards Best Practices for Integrating Physiological Signals in HCIExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636286(1-7)Online publication date: 11-May-2024
    • (2023)Adapting Visual Complexity Based on Electrodermal Activity Improves Working Memory Performance in Virtual RealityProceedings of the ACM on Human-Computer Interaction10.1145/36042437:MHCI(1-26)Online publication date: 13-Sep-2023

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media