Skip to main content
SpringerLink
Log in

Processing Physiological Sensor Data in Near Real-Time as Social Signals for Their Use on Social Virtual Reality Platforms

  • Conference paper
  • First Online:
Extended Reality (XR Salento 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13446))

Included in the following conference series:

Abstract

Social interactions increasingly shift to computer-mediated communication channels. Compared to face-to-face communication, their use suffers from a loss or distortion in the transmission of social signals, which are prerequisites of social interactions. Social virtual reality platforms offer users a variety of possibilities to express themselves verbally as well as non-verbally. Although these platforms take steps towards compensating the addressed communication gap, there is still high demand to ensure and further improve the correct transmission of social signals. To address this issue, we investigate the processing of physiological sensor data as social signals. This paper provides two major contributions: Firstly, a concept for processing physiological sensor data in near real-time as social signals. The concept enables the processing of physiological sensor data on an individual level as well as across all users. For both the individual user and the collective, single sensors or the data from the whole sensor cluster can be analysed, resulting in four ways of analysis. Secondly, we provide concrete suggestions for a software setup, based on an extensive analysis of available open source software, to support a potential future implementation of the proposed concept. The results of this work are highly relevant for social virtual reality platforms, especially since modern head-mounted displays are often already equipped with appropriate measurement sensors. Moreover, the results can also be transferred to numerous other media, applications and research fields concerned with processing physiological sensor data, which reinforces the provided added value.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Acampora, G., Cook, D.J., Rashidi, P., Vasilakos, A.V.: A survey on ambient intelligence in healthcare. Proc. IEEE 101(12), 2470–2494 (2013)

    Article  Google Scholar 

  2. Albahri, A.S., et al.: Iot-based telemedicine for disease prevention and health promotion: state-of-the-art. J. Netw. Comput. Appl. 173, 102873 (2021)

    Google Scholar 

  3. Amazon: Kinesis streams (2013). https://aws.amazon.com/de/kinesis/. (Accessed 21 Feb 2022)

  4. Apache-Software-Foundation: Acitvemq (2007). https://github.com/apache/activemq. (Accessed 03 Mar 2022)

  5. Apache-Software-Foundation: Camel (2007). https://github.com/apache/camel. (Accessed 22 Feb 2022)

  6. Apache-Software-Foundation: Flink (2011). https://github.com/apache/flink. (Accessed 21 Feb 2022)

  7. Apache-Software-Foundation: Kafka (2011). https://github.com/apache/kafka. (Accessed 21 Feb 2022)

  8. Apache-Software-Foundation: Flume (2012). https://github.com/apache/flume. (Accessed 25 Feb 2022)

  9. Apache-Software-Foundation: Spark streaming (2012). https://github.com/apache/spark/tree/master/streaming. (Accessed 21 Feb 2022)

  10. Apache-Software-Foundation: Bookkeeper (2014). https://github.com/apache/bookkeeper. (Accessed 21 Feb 2022)

  11. Apache-Software-Foundation: Beam (2016). https://github.com/apache/beam. (Accessed 21 Feb 2022)

  12. Arasu, A., et al.: Linear road: a stream data management benchmark. In: Proceedings of the Thirtieth International Conference on Very Large Data Bases, vol. 30, pp. 480–491 (2004)

    Google Scholar 

  13. Balazinska, M., Balakrishnan, H., Madden, S., Stonebraker, M.: Fault-tolerance in the borealis distributed stream processing system. In: Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, pp. 13–24 (2005)

    Google Scholar 

  14. Bao, L., Intille, S.S.: Activity recognition from user-annotated acceleration data. In: Ferscha, A., Mattern, F. (eds.) Pervasive 2004. LNCS, vol. 3001, pp. 1–17. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24646-6_1

    Chapter  Google Scholar 

  15. Benssassi, E.M., Ye, J.: Investigating multisensory integration in emotion recognition through bio-inspired computational models. IEEE Trans. Affect. Comput. 1 (2021). https://doi.org/10.1109/taffc.2021.3106254

  16. Bonér, J.: Akka (2009). https://github.com/akka/akka. (Accessed 24 Feb 2022)

  17. Brewer, E.: Cap twelve years later: How the “rules" have changed. Computer 45(2), 23–29 (2012)

    Article  Google Scholar 

  18. Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache flink: Stream and batch processing in a single engine. Bull. IEEE Comput. Soc. Tech. Committee Data Eng. 36(4), 28–38 (2015)

    Google Scholar 

  19. Chanel, G., Mühl, C.: Connecting brains and bodies: applying physiological computing to support social interaction. Interact. Comput. 27(5), 534–550 (2015). https://doi.org/10.1093/iwc/iwv013

    Article  Google Scholar 

  20. Chintapalli, S., et al.: Benchmarking streaming computation engines: Storm, flink and spark streaming. In: 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 1789–1792. IEEE (2016)

    Google Scholar 

  21. Del Monte, B., Zeuch, S., Rabl, T., Markl, V.: Rhino: efficient management of very large distributed state for stream processing engines. In: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pp. 2471–2486 (2020)

    Google Scholar 

  22. Deleuze, G.: Postscript on the Societies of Control. Routledge (2017)

    Google Scholar 

  23. Desnoyers-Stewart, J., Stepanova, E., Pasquier, P., Riecke, B.E.: JeL: Connecting Through Breath in Virtual Reality (2019)

    Google Scholar 

  24. Dey, A., Chen, H., Hayati, A., Billinghurst, M., Lindeman, R.W.: Sharing manipulated heart rate feedback in collaborative virtual environments. In: 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE (2019). https://doi.org/10.1109/ismar.2019.00022

  25. Dishongh, T.J., McGrath, M.: Wireless sensor networks for healthcare applications. Artech House (2010)

    Google Scholar 

  26. Doan, Q.T., Kayes, A., Rahayu, W., Nguyen, K.: Integration of iot streaming data with efficient indexing and storage optimization. IEEE Access 8, 47456–47467 (2020)

    Article  Google Scholar 

  27. Dumka, A., Chaurasiya, S.K., Biswas, A., Mandoria, H.L.: A Complete Guide to Wireless Sensor Networks: From Inception to Current Trends. CRC Press (2019)

    Google Scholar 

  28. Dzardanova, E., Kasapakis, V., Gavalas, D.: Social Virtual Reality. Encyclopedia of Computer Graphics and Games (2018)

    Google Scholar 

  29. Elastic: Logstash (2016). https://github.com/elastic/logstash. (Accessed 21 Feb 2022)

  30. Forsberg, K., Mooz, H.: The relationship of system engineering to the project cycle. In: INCOSE International Symposium, vol. 1(1), 57–65 (1991). https://doi.org/10.1002/j.2334-5837.1991.tb01484.x

  31. Fu, Y., Soman, C.: Real-time data infrastructure at uber. In: Proceedings of the 2021 International Conference on Management of Data, pp. 2503–2516 (2021)

    Google Scholar 

  32. Genz, F., Hufeld, C., Müller, S., Kolb, D., Starck, J., Kranzlmüller, D.: Replacing EEG sensors by AI based emulation. In: De Paolis, L.T., Arpaia, P., Bourdot, P. (eds.) AVR 2021. LNCS, vol. 12980, pp. 66–80. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87595-4_6

    Chapter  Google Scholar 

  33. GitHub: Hadoop (2006). https://github.com/apache/hadoop, (Accessed 21 Feb 2022)

  34. Google: Google cloud dataflow (2015). https://cloud.google.com/dataflow. (Accessed 21 Feb 2022)

  35. Gopalakrishna, K., Fu, X.: Pinot (2014). https://github.com/apache/pinot. (Accessed 21 Feb 2022)

  36. Gozuacik, N., Oktug, S.: Parent-aware routing for IoT networks. In: Balandin, S., Andreev, S., Koucheryavy, Y. (eds.) ruSMART 2015. LNCS, vol. 9247, pp. 23–33. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23126-6_3

    Chapter  Google Scholar 

  37. Guzel, M., Kok, I., Akay, D., Ozdemir, S.: Anfis and deep learning based missing sensor data prediction in IoT. Concurrency Comput. Pract. Experience 32(2), e5400 (2020)

    Google Scholar 

  38. Halbig, A., Latoschik, M.E.: A Systematic Review of Physiological Measurements, Factors, Methods, and Applications in Virtual Reality (2021)

    Google Scholar 

  39. Harper, R.: Human expression in the age of communications overload (2010)

    Google Scholar 

  40. He, P., Zhu, J., Xu, P., Zheng, Z., Lyu, M.R.: A directed acyclic graph approach to online log parsing. arXiv preprint arXiv:1806.04356 (2018)

  41. Hesse, G., Matthies, C., Perscheid, M., Uflacker, M., Plattner, H.: Espbench: the enterprise stream processing benchmark. In: Proceedings of the ACM/SPEC International Conference on Performance Engineering, pp. 201–212 (2021)

    Google Scholar 

  42. Huang, X.Y., et al.: Multi-matrices factorization with application to missing sensor data imputation. Sensors 13(11), 15172–15186 (2013)

    Article  Google Scholar 

  43. Hwang, J.H., Balazinska, M., Rasin, A., Cetintemel, U., Stonebraker, M., Zdonik, S.: High-availability algorithms for distributed stream processing. In: 21st International Conference on Data Engineering (ICDE 2005), pp. 779–790. IEEE (2005)

    Google Scholar 

  44. Im, J.F., et al.: Pinot: realtime olap for 530 million users. In: Proceedings of the 2018 International Conference on Management of Data, pp. 583–594 (2018)

    Google Scholar 

  45. Isah, H., Zulkernine, F.: A scalable and robust framework for data stream ingestion. In: 2018 IEEE International Conference on Big Data (Big Data), pp. 2900–2905. IEEE (2018)

    Google Scholar 

  46. Jennings, J.R., Berg, W.K., Hutcheson, J.S., Obrist, P., Porges, S., Turpin, G.: Committee report. publication guidelines for heart rate studies in man. Psychophysiology 18(3), 226–231 (1981). https://doi.org/10.1111/j.1469-8986.1981.tb03023.x

    Article  Google Scholar 

  47. Jiang, X., Tian, Z., Li, K.: A graph-based approach for missing sensor data imputation. IEEE Sens. J. 21(20), 23133–23144 (2021)

    Article  Google Scholar 

  48. Jonell, P.: Using Social and Physiological Signals for User Adaptation in Conversational Agents (2019)

    Google Scholar 

  49. Kopetz, H.: The real-time environment. Real-Time Systems: Design Principles for Distributed Embedded Applications, pp. 1–28 (2011)

    Google Scholar 

  50. Korzun, D.G., Nikolaevskiy, I., Gurtov, A.: Service Intelligence support for medical sensor networks in personalized mobile health systems. In: Balandin, S., Andreev, S., Koucheryavy, Y. (eds.) ruSMART 2015. LNCS, vol. 9247, pp. 116–127. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23126-6_11

    Chapter  Google Scholar 

  51. Lazer, D., et al.: Social science. computational social science. Science 323(5915), 721–723 (2009). https://doi.org/10.1126/science.1167742

    Article  Google Scholar 

  52. Lee, M., Kolkmeier, J., Heylen, D., IJsselsteijn, W.: Who Makes Your Heart Beat? What Makes You Sweat? Social Conflict in Virtual Reality for Educators (2021)

    Google Scholar 

  53. Leventov, R.: Comparison of the open source olap systems for big data: Clickhouse, druid, and pinot, Feb 2018. (Accessed 21 Feb 2022)

    Google Scholar 

  54. LinkedIn: Samza (2013). https://github.com/apache/samza. (Accessed 23 Feb 2022)

  55. LinkedIn: Gobblin (2015). https://github.com/apache/gobblin. (Accessed 25 Feb 2022)

  56. Lou, J., et al.: Realistic facial expression reconstruction for vr hmd users. IEEE Trans. Multimedia 22(3), 730–743 (2020). https://doi.org/10.1109/tmm.2019.2933338

    Article  Google Scholar 

  57. Lu, R., Wu, G., Xie, B., Hu, J.: Stream bench: Towards benchmarking modern distributed stream computing frameworks. In: 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, pp. 69–78. IEEE (2014)

    Google Scholar 

  58. Marcu, O.C., et al.: Kera: Scalable data ingestion for stream processing. In: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), pp. 1480–1485. IEEE (2018)

    Google Scholar 

  59. Marz, N.: How to beat the cap theorem, Oct 2011. http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html. (Accessed 21 Feb 2022)

  60. Marz, N.: Storm (2011). https://github.com/apache/storm. (Accessed 25 Feb 2022)

  61. McStay, A.: Emotional AI: The rise of empathic media. Sage (2018)

    Google Scholar 

  62. Meehan, J., Aslantas, C., Zdonik, S., Tatbul, N., Du, J.: Data ingestion for the connected world. In: CIDR (2017)

    Google Scholar 

  63. Mehmood, E., Anees, T.: Challenges and solutions for processing real-time big data stream: a systematic literature review. IEEE Access 8, 119123–119143 (2020)

    Article  Google Scholar 

  64. Metamarkets: Druid (2014). https://github.com/apache/druid. (Accessed 21 Feb 2022)

  65. Microsoft: Azure stream analytics (2015). https://azure.microsoft.com/en-us/services/stream-analytics/. (Accessed 21 Feb 2022)

  66. Mukhopadhyay, S.C.: Wearable sensors for human activity monitoring: a review. IEEE Sens. J. 15(3), 1321–1330 (2014)

    Article  Google Scholar 

  67. Pentland, A.: Social signal processing [exploratory dsp]. IEEE Signal Process. Mag. 24(4), 108–111 (2007). https://doi.org/10.1109/msp.2007.4286569

    Article  Google Scholar 

  68. Prokopowicz, D., Golebiowska, A., Matosek, M.: Growing importance of digitization of remote communication processes and the internetization of economic processes and the impact of the sars-cov-2 (covid-19) coronavirus pandemic on the economy. In: Socio-Economic and Legal Dimensions of Digital Transformation, pp. 221–250. SGSP, Warsaw (2021)

    Google Scholar 

  69. Reeves, B., Nass, C.: The media equation: How people treat computers, television, and new media like real people. Cambridge, UK. (1996)

    Google Scholar 

  70. Reiss, A., Stricker, D.: Introducing a new benchmarked dataset for activity monitoring. In: 2012 16th International Symposium on Wearable Computers, pp. 108–109. IEEE (2012)

    Google Scholar 

  71. Salminen, M.: Evoking physiological synchrony and empathy using social vr with biofeedback. IEEE Trans. Affect. Comput. 13(2), 746–755 (2022). https://doi.org/10.1109/taffc.2019.2958657

    Article  Google Scholar 

  72. Schultz, R.: Welcome to the metaverse: A comprehensive list of social vr/ar platforms and virtual worlds (2022). https://ryanschultz.com/list-of-social-vr-virtual-worlds/. (Accessed 21 Feb 2022)

  73. Shah, M.A., Hellerstein, J.M., Brewer, E.: Highly available, fault-tolerant, parallel dataflows. In: Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data, pp. 827–838 (2004)

    Google Scholar 

  74. Shahrivari, S.: Beyond batch processing: towards real-time and streaming big data. Computers 3(4), 117–129 (2014)

    Article  Google Scholar 

  75. Shukla, A., Chaturvedi, S., Simmhan, Y.: Riotbench: an iot benchmark for distributed stream processing systems. Concurrency Comput. Pract. Exp. 29(21), e4257 (2017)

    Google Scholar 

  76. Silvestre, P.F., Fragkoulis, M., Spinellis, D., Katsifodimos, A.: Clonos: consistent causal recovery for highly-available streaming dataflows. In: Proceedings of the 2021 International Conference on Management of Data, pp. 1637–1650 (2021)

    Google Scholar 

  77. Stanford, C., Kallas, K., Alur, R.: Correctness in stream processing: Challenges and opportunities. In: Conference on Innovative Data Systems Research (CIDR) (2022)

    Google Scholar 

  78. Twitter: Heron (2015). https://github.com/apache/heron (Accessed 21 Feb 2022)

  79. Uber: Athenax (2017). https://github.com/uber-archive/AthenaX. (Accessed 21 Feb 2022)

  80. Van Dongen, G., Van den Poel, D.: Evaluation of stream processing frameworks. IEEE Trans. Parallel Distrib. Syst. 31(8), 1845–1858 (2020)

    Article  Google Scholar 

  81. Vinciarelli, A., et al.: Bridging the gap between social animal and unsocial machine: a survey of social signal processing. IEEE Trans. Affect. Comput. 3(1), 69–87 (2012). https://doi.org/10.1109/T-AFFC.2011.27

    Article  Google Scholar 

  82. Vinciarelli, A., Pantic, M., Bourlard, H.: Social signal processing: survey of an emerging domain. Image Vis. Comput. 27(12), 1743–1759 (2009)

    Article  Google Scholar 

  83. Vinciarelli, A., Pentland, A.S.: New social signals in a new interaction world: the next frontier for social signal processing. IEEE Syst. Man Cybern. Mag. 1(2), 10–17 (2015). https://doi.org/10.1109/MSMC.2015.2441992

    Article  Google Scholar 

  84. Wagner, J., Lingenfelser, F., Baur, T., Ionut, D., Kistler, F., André, E.: The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time (2013)

    Google Scholar 

  85. Williamson, J., Li, J., Vinayagamoorthy, V., Shamma, D.A., Cesar, P.: Proxemics and social interactions in an instrumented virtual reality workshop. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York (2021). https://doi.org/10.1145/3411764.3445729

  86. Wingerath, W., Ritter, N., Gessert, F.: Real-Time & Stream Data Management. SCS, Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10555-6

    Book  Google Scholar 

  87. Yahoo: Pulsar (2016). https://github.com/apache/pulsar. (Accessed 22 Feb 2022)

  88. Yang, F., Tschetter, E., Léauté, X., Ray, N., Merlino, G., Ganguli, D.: Druid: a real-time analytical data store. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 157–168 (2014)

    Google Scholar 

  89. Zhang, Y.F., Thorburn, P.J., Xiang, W., Fitch, P.: Ssim-a deep learning approach for recovering missing time series sensor data. IEEE Internet Things J. 6(4), 6618–6628 (2019)

    Article  Google Scholar 

  90. Zimmermann, L.: Sensor Fusion in Human Activity Recognition and Occupancy Detection. Ph.D. thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) (2020)

    Google Scholar 

Download references

Acknowledgement

We would like to thank Thomas Odaker, Elisabeth Mayer, Simone Müller and Daniel Kolb who supported this work with helpful discussions and feedback.

Author information

Authors and Affiliations

  1. Ludwig-Maximilians-Universität München, Munich, Germany

    Fabio Genz, Clemens Hufeld & Dieter Kranzlmüller

  2. Leibniz Supercomputing Centre, Munich, Germany

    Dieter Kranzlmüller

Authors
  1. Fabio Genz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Clemens Hufeld
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dieter Kranzlmüller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fabio Genz .

Editor information

Editors and Affiliations

  1. University of Salento, Lecce, Italy

    Lucio Tommaso De Paolis

  2. Università di Napoli Federico II, Naples, Italy

    Pasquale Arpaia

  3. CNR-STIIMA, Lecco, Italy

    Marco Sacco

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Genz, F., Hufeld, C., Kranzlmüller, D. (2022). Processing Physiological Sensor Data in Near Real-Time as Social Signals for Their Use on Social Virtual Reality Platforms. In: De Paolis, L.T., Arpaia, P., Sacco, M. (eds) Extended Reality. XR Salento 2022. Lecture Notes in Computer Science, vol 13446. Springer, Cham. https://doi.org/10.1007/978-3-031-15553-6_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-15553-6_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-15552-9

  • Online ISBN: 978-3-031-15553-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation