Skip to main content

Towards a Semantic Classification of Possible Human-to-Environment Interactions in IoT

  • Conference paper
  • First Online:
Distributed, Ambient and Pervasive Interactions (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12782))

Included in the following conference series:

  • 836 Accesses

Abstract

When looking at the way in which humans choose to participate in and interact with cyber physical-embedded environments such as Internet of Things (IoT), one could assume that such environments are permeated with ‘ad hoc’, ‘heterogeneous’ and ‘dynamic’ interactions in them. Existing literature on human to computer interactions, their types and definitions fail to provide a concrete understanding of these dimensions in cyber physical-embedded environments. Therefore, this paper presents the results of investigating existing categories of Human Computer Interaction (HCI) to make sense of these interactions and the inherent heterogeneity they carry. An integrative literature review using the PRISMA model to locate, select, and include 120 relevant articles has been carried out. The main finding of this review is a semantic classification of possible Human-to-Environment Interactions (HEI). The classification plays an important role as a starting point when looking at the current and future offerings of the HEI in the IoT. The classification also serves input as formal knowledge representations, such as Ontology Web Language (OWL) ontologies, which could assist in creating explicit representations of interaction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    where objects denote sensors, actuators, phones, devices, tablets, computers and smart wearables, to name but a few.

  2. 2.

    which may be needed for their implementation and/or acceptance.

References

  1. Weiser, M.: The computer for the 21st century. Sci. Am. 265(3), 94–104 (1991)

    Article  Google Scholar 

  2. Frauenberger, C.: Entanglement HCI the next wave? ACM Trans. Comput. -Hum. Interact. 27(1), 2:1–2:27 (2019)

    Google Scholar 

  3. Stephanidis, C., et al.: Seven HCI grand challenges. Int. J. Hum. Comput. Interact. 35(14), 1229–1269 (2019)

    Article  Google Scholar 

  4. Harper, R.H.R.: The role of HCI in the age of AI. Int. J. Hum. Comput. Interact. 35(15), 1331–1344 (2019)

    Article  Google Scholar 

  5. Dix, A.: Human-computer interaction, foundations and new paradigms. J. Vis. Lang. Comput. 42, 122–134 (2017)

    Article  Google Scholar 

  6. Earnshaw, R.A., De Silva, M., Excell, P.S.: Ten unsolved problems with the internet of things. In: International Conference on Cyber Worlds 2015 Proceedings, pp. 1–7. IEEE, Sweden (2015)

    Google Scholar 

  7. Juric, R., McClenaghan, K.M.: Towards the semantic classification of constituent parts of the internet-of-vehicles. In: 24th International Conference for the Society for Design and Process Science Transformative Research and Education through Transdisciplinary Means Proceedings, pp. 17–22. SDPS, Taiwan (2019)

    Google Scholar 

  8. Gheisar, M., Wang, G., Chen, S.: An edge computing-enhanced internet of things framework for privacy-preserving in Smart City. J. Comput. Electr. Eng. 81, (2020)

    Article  Google Scholar 

  9. Gulati, N., Kaur, P.D.: Towards socially enabled internet of industrial things: architecture, semantic model and relationship management. J. Ad Hoc Netw. 91, (2019)

    Article  Google Scholar 

  10. Zhu, T., Dhelim, S., Zhou, Z., Yang, S., Ning, H.: An architecture for aggregating information from distributed data nodes for industrial internet of things. J. Comput. Electr. Eng. 58, 337–349 (2017)

    Article  Google Scholar 

  11. Kataria, P., Juric, R., Mandani, K.: Go-CID: generic ontology for context-aware, interoperable and data sharing applications. In: Smith, J.E, (eds.), SEA 2007, pp. 439–444. ACTA Press (2007)

    Google Scholar 

  12. Gomez, J., Oviedo, B., Fernandez, A., Sanchez, M.A.Z., Viteri, J.T.M., Leon, A.R.E.: Semantic representation models of sensor data for monitoring agricultural crops. In: Botto-Tobar, M., León-Acurio, J., Díaz Cadena, A., Montiel Díaz, P. (eds.) Advances in Emerging Trends and Technologies (ICAETT 2019). Advances in Intelligent Systems and Computing, vol. 1066, pp. 33–41. Springer, Cham (2019)

    Chapter  Google Scholar 

  13. Ning, H., Shi, F., Zhu, T., Li, Q., Chen, L.: A novel ontology consistent with acknowledged standards in smart homes. J. Comput. Netw. 148, 101–107 (2019)

    Article  Google Scholar 

  14. Pahal, N., Mallik, A., Chaudhury, S.: An ontology- based context-aware IoT framework for smart surveillance. In: Mohamed, B., Abdelhakim, B.A., Ali, Y. (eds.) International Conference on Smart City applications 2018, pp. 1–7. Association for Computer Machinery (2018)

    Google Scholar 

  15. Veiga, E.F., Arruda, M.K., Neto, J.A.B., Bulcão-Neto, R.: An ontology-based representation service of context information for the internet of things. In: Roesler, V., Valdeni de Lima, J., (eds.) Webmedia 2017. 23rd Brazillian Symposium on Multimedia and the Web, 2017, pp. 301–308. Association for Computer Machinery (2017)

    Google Scholar 

  16. Horbaek, K., Oulasvirta, A.: What is interaction? In: 2017 CHI Conference of Human Factors in Computing Systems Proceedings, pp. 5040–5052. Association for Computer Machinery, New York (2017)

    Google Scholar 

  17. Kim, K.J.: Interacting socially with the internet of things (IoT): effects of source attribution and specialisation in human - IoT interaction. J. Comput. Mediated Commun. 21(6), 420–435 (2016)

    Article  Google Scholar 

  18. Bakker, S., Niemantsverdriet, K.: The interaction-attention continuum: considering various levels of human attention in interaction design. Int. J. Des. 10(2), 1–14 (2016)

    Google Scholar 

  19. Kostakos, V., Musolesi, M.: Introduction to the special issue on social networks and ubiquitous interactions. Int. J. Hum. Comput Stud. 71(9), 859–861 (2013)

    Article  Google Scholar 

  20. Reeves, D., Serafin, S.: Sonic interaction design. Int. J. Hum. Comput. Interact. Stud. 6(9), 905–906 (2009)

    Google Scholar 

  21. Reeves, S., Beck, J.: Talking about interaction. Int. J. Hum. Comput. Interact. Stud. 131, 144–151 (2019)

    Article  Google Scholar 

  22. Mechant, P., All, A., De Marez, L.: Evaluating user experience in smart home contexts: a methodological framework. In: Streitz, N., Konomi, S. (eds.) Distributed, Ambient and Pervasive Interactions: Understanding Humans (DAPI 2018). LNCS, vol. 10921, pp. 91–102. Springer, Cham (2018)

    Chapter  Google Scholar 

  23. Pallot, M. Pawar, K.: A holistic model of user experience for living lab experimental design. In: Katzy, B., Holzmann, T., Sailer, K., Thobens, K.D. (eds.) 18th International Conference on Engineering, Technology and Innovation, pp. 1–15. IEEE (2012)

    Google Scholar 

  24. Jordan, P.: Designing Pleasurable Products. An Introduction to the New Human Factors, 1st edn. Taylor and Francis, London, New York (2000)

    Google Scholar 

  25. Gaver, W.W., Martin, H.: Alternatives: exploring information appliances through conceptual design proposals. In: SIGCHI 2000 Conference on Human factors in Computing Systems, pp. 209–216. Association for Computer Machinery, New York (2000)

    Google Scholar 

  26. Stumpf, T., Califf, C.B., Frye, J.J.: The conceptualisation and uses of technological metaworlds in travel. In: 52nd Hawaii International Conference on Systems Sciences Proceedings, pp. 6937–6946, HICSS, Hawaii (2019)

    Google Scholar 

  27. Janlert, L.E., Stolterman, E.: Faceless interaction - a conceptual examination of the notion of interface: past, present, and future. J. Hum. Comput. Interact. 30(6), 507–539 (2015)

    Article  Google Scholar 

  28. Bibri, S.E.: The human face of ambient intelligence: cognitive, emotional, affective, behavioral and conversational aspects. In: Atlantis Ambient and Pervasive Intelligence, vol. 9. Atlantis Press, Paris (2015)

    Google Scholar 

  29. Harper, R.H.R.: The role of HCI in the age of AI. Int. J. Hum. Comput. Interact. 35(15), 1331–1344 (2019)

    Article  Google Scholar 

  30. Patterson, R.E.: Intuitive cognition and models of human-automation interaction. J. Hum. –Autom. Interact. 59(1), 101–115 (2017)

    Google Scholar 

  31. Glodek, M., et al.: Fusion paradigms in cognitive technical systems for human-computer interaction. J. Neurocomput. 161, 17–32 (2015)

    Article  Google Scholar 

  32. Dorsemaine, B., Gaulier, J.P., Wary, J.P., Kheir, K., Urien, P.: Internet of things: a definition & taxonomy. In: 9th International Conference on Next Generation Mobile Applications, Services and Technologies Proceedings, pp. 9–11, NGMAST, Cambridge (2015)

    Google Scholar 

  33. Barker, L., et al.: Taxonomy for internet of things: tools for monitoring personal effects. In: International Conference on Pervasive and Embedded Computing and Communication Systems Proceedings (PECCS 2014), Portugal, vol. 1 (2014)

    Google Scholar 

  34. Fortino, G., Rovella, A., Russo, W., Savaglio, C.: On the classification of cyber physical smart objects in the internet of things. In: International Workshop on Networks of Cooperating Objects for Smart Cities, vol. 1156, pp. 86–94, UBICITEC, Germany (2014)

    Google Scholar 

  35. Mamo, K., Nieto, J.I., Leon, M.D.C., Vazquez, M., López, J.D.S., Buenrostro, R.: Major existing classification matrices and future directions for internet of things. J. Adv. Internet of Things 07(04), 112–120 (2017)

    Article  Google Scholar 

  36. Liberati, A., et al.: The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 6(7), (2009)

    Article  Google Scholar 

  37. Popovici, I., Schipor, O.A., Vatavu, R.D.: Hover: exploring cognitive maps and mid-air pointing for television control. Int. J. Hum. Comput. Stud. 129, 95–107 (2019)

    Article  Google Scholar 

  38. Ardito, C., Buono, P., Desolda, G., Matera, M.: From Smart objects to smart experiences: an end user development approach. Int. J. Hum. Comput. Stud. 114, 52–68 (2018)

    Article  Google Scholar 

  39. Xiang, A.C., Li, Y.: Improv: an input framework for improvising cross-device interaction by demonstration. ACM Trans. Comput. Hum. Interact. 24(2), 15 (2017)

    Google Scholar 

  40. Sutcliffe, A., Hart, J.: Analysing the role of interactivity in user experience. Int. J. Hum. Comput. Interact. 33(3), 229–240 (2017)

    Article  Google Scholar 

  41. Loke, L., Robertson, T.: Moving and making strange: an embodied approach to movement-based interaction design. ACM Trans. Comput. Hum. Interact. 20(1), 7 (2013)

    Article  Google Scholar 

  42. Bilandzic, M., Foth, M.: A review of locative media, mobile and embodied spatial interaction. Int. J. Hum. Comput. Stud. 70(1), 66–71 (2012)

    Article  Google Scholar 

  43. McGrenere, J., Baecker, R.M., Booth, K.S.: A field evaluation of an adaptable two-interface design of feature-rich software. ACM Trans. Comput. Hum. Interact. 14(1), 3 (2007)

    Article  Google Scholar 

  44. Benford, S., et al.: Expected, sensed and desired: a framework for designing sensing-based interaction. ACM Trans. Comput. Hum. Interact. 12(1), 3–30 (2005)

    Article  Google Scholar 

  45. Zhai, S., Bellotti, V.: Introduction to sensing-based interaction. ACM Trans. Comput. Hum. Interact. 12(1), 1–2 (2005)

    Article  Google Scholar 

  46. Quek, F., et al.: Multimodal human discourse: gesture and speech. ACM Trans. Comput. Hum. Interact. 9(3), 171–193 (2002)

    Article  Google Scholar 

  47. Benford, S., Dourish, P., Rodden, T.: Introduction to the special issue on human-computer interaction and collaborative virtual environments. ACM Trans. Comput. Hum. Interact. 7(4), 439–441 (2000)

    Article  Google Scholar 

  48. Velloso, E., Carter, M., Newn, J., Esteves, A., Clarke, C., Gellersen, H.: Motion correlation: selecting objects by matching their movement. ACM Trans. Comput. Hum. Interact. 24(3), 22 (2017)

    Article  Google Scholar 

  49. Hornbaek, K., Mottelson, A., Knibbe, J., Vogel, D.: What do we mean by “Interaction”? an analysis of 35 years of CHI. ACM Trans. Comput. Hum. Interact. 26(4), 27 (2019)

    Article  Google Scholar 

  50. Oviatt, S., Seneff, S.: Introduction to mobile and adaptive conversational interfaces. ACM Trans. Comput. Hum. Interact. 11(3), 237–240 (2004)

    Article  Google Scholar 

  51. Truschin, S., Schermann, M., Goswami, S., Krcmar, H.: Designing interfaces for multiple-goal environments: experimental insights from in-vehicle speech interfaces. ACM Trans. Comput. Hum. Interact. 21(1), 7 (2014)

    Article  Google Scholar 

  52. Nansen, B., Vetere, F., Robertson, T., Downs, J., Brereton, M., Durick, J.: Reciprocal habituation: a study of older people and the Kinect. ACM Trans. Comput. Hum. Interact. 21(3), 18 (2014)

    Article  Google Scholar 

  53. Kosmyna, N., Tarpin-Bernard, F., Rivet, B.: Adding human learning in brain-computer interfaces (BCIs): towards a practical control modality. ACM Trans. Comput. Hum. Interact. 22(3), 12 (2015)

    Article  Google Scholar 

  54. Solovey, E.T., Afergan, D., Peck, E.M., Hincks, S.W., Jacob, R.J.K.: Designing Implicit Interfaces for psychological computing: guidelines and lessons learned using fNIRS. ACM Trans. Comput. Hum. Interact. 21(6), 35 (2015)

    Google Scholar 

  55. Kirsh, D.: Embodied cognition and the magical future of interaction design. ACM Trans. Comput. Hum. Interact. 20(1), 3 (2013)

    Article  MathSciNet  Google Scholar 

  56. Benford, S., Calder, M., Rodden, T., Sevegnani, M.: On Lions, Impala, and Bigraphs: modelling interactions and physical virtual spaces. ACM Trans. Comput. Hum. Interact. 23(2), 9 (2016)

    Article  Google Scholar 

  57. Vasquez-Alverez, Y., Aylett, M.P., Brewster, R., Von Jungenfeld, S.A., Virolainen, A.: Designing interactions with multilevel auditory displays in mobile audio-augmented reality. ACM Trans. Comput. Hum. Interact. 23(1), 3 (2015)

    Google Scholar 

  58. Mehra, S., Werkhoven, P., Worring, M.: Navigating on handheld displays: dynamic versus static peephole navigation. ACM Trans. Comput. Hum. Interact. 13(4), 448–457 (2006)

    Article  Google Scholar 

  59. Benford, S., GreenHalgh, C., Reynard, G., Brown, C., Koleva, B.: Understanding and constructing shared spaces with mixed reality boundaries. ACM Trans. Comput. Hum. Interact. 5(3), 185–223 (1998)

    Article  Google Scholar 

  60. Hornbaek, K., Hertzum, M.: Technology acceptance and user experience: a review of experiential components in HCI. ACM Trans. Comput. Hum. Interact. 24(5), 33 (2017)

    Article  Google Scholar 

  61. Paik, J., Kim, J.W., Ritter, F.E., Reitter, D.: Predicting user performance and learning in human-computer interaction with the herbal compiler. ACM Trans. Comput. Hum. Interact. 22(5), 25 (2015)

    Article  Google Scholar 

  62. Borsci, S., Macredie, R.D., Barnett, J., Martin, J., Kuljis, J., Young, T.: Reviewing and extending the five-user assumption: a grounded procedure for interaction evaluation. ACM Trans. Comput. Hum. Interact. 20(5), 29 (2013)

    Article  Google Scholar 

  63. Rapp, A., Tirassa, M., Tirabeni, L.: Rethinking technologies for behaviour change: a view from the inside of human change. ACM Trans. Comput. Hum. Interact. 26(4), 22 (2019)

    Article  Google Scholar 

  64. Rozendaal, M.C., Boon, B., Kaptelinin, V.: Objects with intent: designing everyday things as collaborative partners. ACM Trans. Comput. Hum. Interact. 26(4), 26 (2019)

    Article  Google Scholar 

  65. Dalsgaard, P., Hansen, L.K.: Performing perception - staging aesthetics of interaction. ACM Trans. Comput. Hum. Interact. 15(3), 13 (2008)

    Article  Google Scholar 

  66. Raisamo, R., Rakkolainen, I., Majaranta, P., Salminen, K., Rantala, J., Farooq, A.: Human augmentation: past, present and future. Int. J. Hum Comput Stud. 131, 131–143 (2019)

    Article  Google Scholar 

  67. Rhiu, I., Hwan Yun, M.: Exploring user experience of smartphones in social media: a mixed-method analysis. Int. J. Hum. Comput. Interact. 34(10), 960–969 (2018)

    Article  Google Scholar 

  68. Mealla Cincuegrani, S., Jorda, S., Valjamae, A.: Physiopucks: increasing user motivation by combining tangible and implicit physiological interaction. ACM Trans. Comput. Hum. Interact. 23(1), 4 (2016)

    Google Scholar 

  69. Poor, G.M., et al.: Applying the Norman 1986 user-centered model to post-WIMP UIs: theoretical predictions and empirical outcomes. ACM Trans. Comput. Hum. Interact. 23(5), 30 (2016)

    Article  Google Scholar 

  70. Bilandzic, M., Foth, M.: Embedded, embodied and situated contexts in interaction with technologies. Int. J. Hum. Comput. Stud. 70(1), 66–71 (2012)

    Article  Google Scholar 

  71. Kjeldskov, J., Paay, J.: Indexicality: understanding mobile human-computer interaction in context. ACM Trans. Comput. Hum. Interact. 17(4), 14 (2010)

    Article  Google Scholar 

  72. Blackwell, A.F., Rode, J.A., Toye, E.F.: How do we program the home? Gender, attention investment, and the psychology of programming at home. Int. J. Hum. Comput. Stud. 67(4), 324–341 (2009)

    Article  Google Scholar 

  73. Hinckley, K., Pierce, J., Horvitz, E., Sinclair, M.: Foreground and background interaction with sensor-enhanced mobile devices. ACM Trans. Comput. Hum. Interact. 12(1), 31–52 (2005)

    Article  Google Scholar 

  74. Liu, Y., Feyen, R., Tsimhoni, O.: Queueing network-model human processor (QN-MHP): a computational architecture for multitask performance in human-machine systems. ACM Trans. Comput. Hum. Interact. 13(1), 37–90 (2006)

    Article  Google Scholar 

  75. Ruddle, R.A., Savage, J.C.D., Jones, D.M.: Symmetric and asymmetric action integration during cooperative object manipulation in virtual environments. ACM Trans. Comput. Hum. Interact. 9(4), 285–308 (2002)

    Article  Google Scholar 

  76. Zanden, B.V., Myers, B.A.: Demonstrational and constraint-based techniques for pictorially specifying application objects and behaviours. ACM Trans. Comput. Hum. Interact. 2(4), 308–356 (1995)

    Article  Google Scholar 

  77. Jacob, R.J.K., Sibert, L.E., McFarlane, D.C., Mullen, M.P.: Integrality and separability of input devices. ACM Trans. Comput. Hum. Interact. 1(1), 3–26 (1994)

    Article  Google Scholar 

  78. Kang, H., Kim, K.J.: Feeling connected to smart objects? A moderated mediation model of locus of agency, anthropomorphism, and sense of connectedness. Int. J. Hum. Comput. Stud. 133, 45–55 (2020)

    Article  Google Scholar 

  79. Janssen, C.P., Boyle, L.N., Kun, A.L., Ju, W., Chuang, L.L.: A hidden Markov framework to capture human machine interaction in automated vehicles. Int. J. Hum. Comput. Interact. 35(11), 947–955 (2019)

    Article  Google Scholar 

  80. Victorelli, E.Z., Reis, J.C.D., Hornung, H., Prado, A.B.: Understanding human-data interaction: literature review and recommendations for design. Int. J. Hum. Comput. Stud. 134, 13–21 (2019)

    Article  Google Scholar 

  81. Alan, A.T., Costanza, E., Ramchurn, S.D., Fischer, J., Rodden, T., Jennings, N.R.: Tariff agent: interacting with a future smart energy system at home. ACM Trans. Comput. Hum. Interact. 23(4), 25 (2016)

    Article  Google Scholar 

  82. Baljko, M., Tenhaaf, N.: The Aesthetics of emergence: co-constructed interactions. ACM Trans. Comput. Hum. Interact. 15(3), 11 (2008)

    Article  Google Scholar 

  83. McGuffic, M.J., Balakrishnan, R.: Fitts’ law and expanding targets: experimental studies and designs for user interfaces. ACM Trans. Comput. Hum. Interact. 12(4), 388–422 (2005)

    Article  Google Scholar 

  84. Abowd, G.D., Mynatt, E.D.: Charting past, present, and future research in ubiquitous computing. ACM Trans. Comput. Hum. Interact. 7(1), 3–28 (2000)

    Article  Google Scholar 

  85. Kieras, D.E., Wood, S.D., Meyer, D.E.: Predictive engineering models based on the EPIC architecture for a multimodal high-performance human computer interaction task. ACM Trans. Comput. Hum. Interact. 4(3), 230–275 (1997)

    Article  Google Scholar 

  86. Dix, A., Rodden, T., Davies, N., Trevor, J., Friday, A., Palfreyman, K.: Exploiting space and location as a design framework for interactive mobile systems. ACM Trans. Comput. Hum. Interact. 7(3), 285–321 (2000)

    Article  Google Scholar 

  87. Lee, M.: An empirical study of home IoT services in South Korea: the moderating effect of the usage experience. Int. J. Hum. Comput. Interact. 35(7), 535–547 (2019)

    Article  Google Scholar 

  88. Bickmore, T.W., Picard, R.W.: Establishing and maintaining long-term human-computer relationships. ACM Trans. Comput. Hum. Interact. 12(2), 293–327 (2005)

    Article  Google Scholar 

  89. Bermudez-Edo, M., Elsaleh, T., Barnaghi, P., Taylor, K.: IoT-Lite: a lightweight semantic model for the internet of things. J. Pers. Ubiquit. Comput. 21(3), 475–487 (2017)

    Article  Google Scholar 

  90. Compton, M., et al.: The SSN ontology of the W3C semantic sensor network incubator group. J. Web Semant. Sci. Serv. Agents World Wide Web 17(C), 25–32 (2012)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Pavandeep Kataria .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kataria, P. (2021). Towards a Semantic Classification of Possible Human-to-Environment Interactions in IoT. In: Streitz, N., Konomi, S. (eds) Distributed, Ambient and Pervasive Interactions. HCII 2021. Lecture Notes in Computer Science(), vol 12782. Springer, Cham. https://doi.org/10.1007/978-3-030-77015-0_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77015-0_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77014-3

  • Online ISBN: 978-3-030-77015-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics