Skip to main content
SpringerLink
Log in

Rethinking User Interaction with Smart Environments—A Comparative Study of Four Interaction Modalities

  • Conference paper
  • First Online:
Distributed, Ambient and Pervasive Interactions (HCII 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12203))

Included in the following conference series:

Abstract

Smart environments, comprised of networked embedded devices, improve the lives of their users by providing them with a variety of assistive services that traditional built environments are incapable of supporting. However, as the number of connected devices in smart environments continue to increase, so does the level of complexity involved in interacting with these environments. Traditional human-computer interaction techniques are not always well-suited for smart environments and this poses some unique usability challenges. To facilitate interactions within such technology-rich smart environments, new models and interaction interfaces need to be developed. In this paper we propose a multi-modal approach to smart environment interaction and explore two novel interaction interfaces: gesture-based interface and mixed-reality-based interface. We also conducted a user study to compare the learnability, efficiency and memorability of these new interfaces to two more commonly used interfaces: voice-based interface and a smartphone GUI-based interface. Our user study experiment involved four light control tasks that subjects were asked to complete using the four interaction interfaces. Study subjects found different interaction techniques to be more suitable for different tasks based on the type, complexity and context of the task. Our analysis of the study results and subject feedback suggest that a multi-modal approach is preferable to a uni-modal approach for interacting with smart environments. We suggest that novel interaction techniques be further explored in order to develop efficient multi-modal approaches along with the widely used techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Alam, M.R., Reaz, M.B.I., Ali, M.A.M.: A review of smart homes–past, present, and future. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 42(6), 1190–1203 (2012)

    Google Scholar 

  2. Ando, H., Kitahara, Y., Hataoka, N.: Evaluation of multimodal interface using spoken language and pointing gesture on interior design system. In: Third International Conference on Spoken Language Processing (1994)

    Google Scholar 

  3. Bien, Z.Z., Do, J.H., Kim, J.B., Stefanov, D., Park, K.H.: User-friendly interaction/interface control of intelligent home for movement-disabled people. In: Proceedings of the 10th International Conference on Human-Computer Interaction, pp. 304–308 (2003)

    Google Scholar 

  4. Brooke, J., et al.: SUS–a quick and dirty usability scale. In: Usability Evaluation in Industry, vol. 189, no. 194, pp. 4–7 (1996)

    Google Scholar 

  5. Brumitt, B., Cadiz, J.J.: “Let there be light”: examining interfaces for homes of the future. In: INTERACT, vol. 1, pp. 375–382 (2001)

    Google Scholar 

  6. Cohen, P.R., et al.: QuickSet: multimodal interaction for distributed applications. ACM Multimedia, vol. 97, pp. 31–40 (1997)

    Google Scholar 

  7. Cook, D., Das, S.K.: Smart Environments: Technology, Protocols, and Applications, vol. 43. Wiley (2004)

    Google Scholar 

  8. Coppens, A.: Merging real and virtual worlds: an analysis of the state of the art and practical evaluation of microsoft hololens. arXiv preprint arXiv:1706.08096 (2017)

  9. Dasgupta, A., Handosa, M., Manuel, M., Gračanin, D.: A user-centric design framework for smart built environments. In: Streitz, N., Konomi, S. (eds.) HCII 2019. LNCS, vol. 11587, pp. 124–143. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-21935-2_11

    Chapter  Google Scholar 

  10. Froiz-Míguez, I., Fernández-Caramés, T., Fraga-Lamas, P., Castedo, L.: Design, implementation and practical evaluation of an iot home automation system for fog computing applications based on mqtt and zigbee-wifi sensor nodes. Sensors 18(8), 2660 (2018)

    Article  Google Scholar 

  11. Gieselmann, P., Denecke, M.: Towards multimodal interaction with an intelligent room. In: Eighth European Conference on Speech Communication and Technology (2003)

    Google Scholar 

  12. Handosa, M., Gračanin, D., Elmongui, H.G., Ciambrone, A.: Painting with light: gesture based light control in architectural settings. In: Proceedings of the 2017 IEEE Symposium on 3D User Interfaces (3DUI 2017), pp. 249–250, 18–19 March 2017

    Google Scholar 

  13. Jeon, Y., et al.: Iot-based occupancy detection system in indoor residential environments. Build. Environ. 132, 181–204 (2018)

    Article  Google Scholar 

  14. Jo, D., Kim, G.J.: ARIoT: scalable augmented reality framework for interacting with internet of things appliances everywhere. IEEE Trans. Consum. Electron. 62(3), 334–340 (2016)

    Article  Google Scholar 

  15. Koskela, T., Väänänen-Vainio-Mattila, K.: Evolution towards smart home environments: empirical evaluation of three user interfaces. Pers. Ubiquit. Comput. 8(3), 234–240 (2004)

    Google Scholar 

  16. Kühnel, C., Westermann, T., Hemmert, F., Kratz, S., Müller, A., Möller, S.: I’m home: defining and evaluating a gesture set for smart-home control. Int. J. Hum Comput Stud. 69(11), 693–704 (2011)

    Article  Google Scholar 

  17. Kühnel, C., Westermann, T., Weiss, B., Möller, S.: Evaluating multimodal systems: a comparison of established questionnaires and interaction parameters. In: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, pp. 286–294. ACM (2010)

    Google Scholar 

  18. Ladwig, P., Geiger, C.: A literature review on collaboration in mixed reality. In: Auer, M.E., Langmann, R. (eds.) REV 2018. LNNS, vol. 47, pp. 591–600. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-95678-7_65

    Chapter  Google Scholar 

  19. Lee, Y., Hsiao, W., Huang, C., Chou, S.T.: An integrated cloud-based smart home management system with community hierarchy. IEEE Trans. Consum. Electron. 62(1), 1–9 (2016)

    Article  Google Scholar 

  20. Luria, M., Hoffman, G., Zuckerman, O.: Comparing social robot, screen and voice interfaces for smart-home control. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 580–628. ACM (2017)

    Google Scholar 

  21. Mrazovac, B., Bjelica, M.Z., Simić, D., Tikvić, S., Papp, I.: Gesture based hardware interface for RF lighting control. In: 2011 IEEE 9th International Symposium on Intelligent Systems and Informatics, pp. 309–314. IEEE (2011)

    Google Scholar 

  22. Norouzi, N., Bruder, G., Belna, B., Mutter, S., Turgut, D., Welch, G.: A systematic review of the convergence of augmented reality, intelligent virtual agents, and the internet of things. In: Al-Turjman, F. (ed.) Artificial Intelligence in IoT. TCSCI, pp. 1–24. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-04110-6_1

    Chapter  Google Scholar 

  23. Oviatt, S.: Mutual disambiguation of recognition errors in a multimodel architecture. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 576–583. ACM (1999)

    Google Scholar 

  24. Petersen, N., Stricker, D.: Continuous natural user interface: Reducing the gap between real and digital world. In: ISMAR, pp. 23–26 (2009)

    Google Scholar 

  25. Rashidi, P., Cook, D.J.: Keeping the resident in the loop: adapting the smart home to the user. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 39(5), 949–959 (2009)

    Article  Google Scholar 

  26. Rosendahl, A., Hampe, J.F., Botterweck, G.: Mobile home automation – merging mobile value added services and home automation technologies. In: Proceedings of the International Conference on the Management of Mobile Business (ICMB 2007), pp. 31:1–31:8. IEEE, 9–11 July 2007

    Google Scholar 

  27. Swaminathan, R., Nischt, M., Kuhnel, C.: Localization based object recognition for smart home environments. In: 2008 IEEE International Conference on Multimedia and Expo, pp. 921–924. IEEE (2008)

    Google Scholar 

  28. Tasooji, R., Dasgupta, A., Gračanin, D., LaGro, M., Matković, K.: A multi-purpose IoT framework for smart built environments. In: Proceedings of the 2018 Winter Simulation Conference, pp. 4240–4241. IEEE Press (2018)

    Google Scholar 

  29. Wagner, D., Schmalstieg, D.: History and future of tracking for mobile phone augmented reality. In: 2009 International Symposium on Ubiquitous Virtual Reality, pp. 7–10, July 2009

    Google Scholar 

  30. Wobbrock, J.O., Morris, M.R., Wilson, A.D.: User-defined gestures for surface computing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1083–1092. ACM (2009)

    Google Scholar 

  31. Yamazaki, T.: Beyond the smart home. In: 2006 International Conference on Hybrid Information Technology, vol. 2, pp. 350–355. IEEE (2006)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Mansoura University, Mansoura, Egypt

    Mohamed Handosa

  2. Virginia Tech, Blacksburg, VA, 24060, USA

    Archi Dasgupta, Mark Manuel & Denis Gračanin

Authors
  1. Mohamed Handosa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Archi Dasgupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mark Manuel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Denis Gračanin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mark Manuel .

Editor information

Editors and Affiliations

  1. Smart Future Initiative, Frankfurt/Main, Germany

    Norbert Streitz

  2. Kyushu University, Fukuoka, Japan

    Shin'ichi Konomi

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Handosa, M., Dasgupta, A., Manuel, M., Gračanin, D. (2020). Rethinking User Interaction with Smart Environments—A Comparative Study of Four Interaction Modalities. In: Streitz, N., Konomi, S. (eds) Distributed, Ambient and Pervasive Interactions. HCII 2020. Lecture Notes in Computer Science(), vol 12203. Springer, Cham. https://doi.org/10.1007/978-3-030-50344-4_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-50344-4_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-50343-7

  • Online ISBN: 978-3-030-50344-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation