Skip to main content
SpringerLink
Log in

Holistic Multimodal Interaction and Design

  • Conference paper
  • First Online:
HCI International 2022 - Late Breaking Papers. Design, User Experience and Interaction (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13516))

Included in the following conference series:

  • 1306 Accesses

Abstract

This paper addresses the lack of an HCI framework that examines multimodal experiences in a holistic way to help design interactions where modalities are properly integrated. Such a holistic approach is important as new emerging technologies and experiences are being introduced and used frequently. The resulting multimodal experiences present new requirements and challenges which are not managed by existing HCI frameworks. Through a literature review, we propose a set of principles that define the Holistic Multimodal Interaction and Design (HMID) framework. We perform a user study as the initial evaluation of the effectiveness of this framework. The study findings showed the potential value of HMID and suggested improvements to its guiding principles.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Nigay, L., Coutaz, J.: A design space for multimodal systems. In: Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’93. ACM Press, New York, New York, USA, pp 172–178 (1993)

    Google Scholar 

  2. Oviatt, S.: Multimodal Interfaces. human-computer Interact Handb Fundam Evol Technol Emerg Appl 14, 405–430 (2012). https://doi.org/10.1201/b11963-22

  3. Kruschitz, C., Hitz, M.: Human-computer interaction design patterns: structure, methods, and tools. Int J Adv Softw 3, 225–237 (2010)

    Google Scholar 

  4. Ishii, H., Ullmer, B.: Emerging frameworks for tangible user interfaces. IBM Syst J 39, 915–931 (2000)

    Article  Google Scholar 

  5. Weiser, M.: The computer for the 21st century. Sci Am 265, 94–104 (1991). https://doi.org/10.1038/scientificamerican0991-94

    Article  Google Scholar 

  6. Jain, J., Lund, A., Wixon, D.: The future of natural user interfaces. Conf Hum Factors Comput Syst - Proc 211–214 (2011). https://doi.org/10.1145/1979742.1979527

  7. Gorlewicz, J.L., Tennison, J.L., Uesbeck, P.M., et al.: Design guidelines and recommendations for multimodal, touchscreen-based graphics. ACM Trans. Access Comput. 13 (2020). https://doi.org/10.1145/3403933

  8. Reeves, L.M., Martin, J.C., McTear, M., et al.: Guidelines for multimodal user interface design. Commun ACM 47, 57–59 (2004)

    Article  Google Scholar 

  9. Sarter, N.B.: Multimodal information presentation: design guidance and research challenges. Int J Ind Ergon 36, 439–445 (2006). https://doi.org/10.1016/j.ergon.2006.01.007

    Article  Google Scholar 

  10. Snyder, L.J.: Stanford Encyclopedia of Philosophy Stanford Encyclopedia of Philosophy: Implicit Bias. 1–22 (2017)

    Google Scholar 

  11. Smuts, J.C.: Holism and Evolution. Macmillan (1926)

    Google Scholar 

  12. Pallot, M., Pawar, K.: A holistic model of user experience for living lab experiential design. 2012 18th Int Conf Eng Technol Innov ICE 2012 - Conf Proc. (2012) https://doi.org/10.1109/ICE.2012.6297648

  13. Luxoft Developed a Holistic Automotive User Experience Solution Based on SmartDeviceLink | Business Wire. https://www.businesswire.com/news/home/20131111006225/en/Luxoft-Developed-a-Holistic-Automotive-User-Experience-Solution-Based-on-SmartDeviceLink. Accessed 9 Jun 2022

  14. Turk, M.: Multimodal interaction: a review. Pattern Recognit Lett 36, 189–195 (2014). https://doi.org/10.1016/j.patrec.2013.07.003

    Article  Google Scholar 

  15. Truong, K.N., Hayes, G.R., Abowd, G.D.: Storyboarding: an empirical determination of best practices and effective guidelines. Proc. Conf. Des. Interact Syst. Process. Pract. Methods, Tech. DIS 2006, 12–21 (2006)

    Google Scholar 

  16. Mayhew, R.: Part and whole in aristotle’s political philosophy. J Ethics 1, 325–340 (1997). https://doi.org/10.1023/A:1009743012461

    Article  Google Scholar 

  17. Heller, M., Simons, P.: Parts: a study in ontology. Philos. Rev. 100, 488 (1991). https://doi.org/10.2307/2185078

    Article  Google Scholar 

  18. Mather, G.: Foundations of perception. 404–427 (2006)

    Google Scholar 

  19. Van Dam, A.: Post-WIMP user interfaces. Commun ACM 40, 63–67 (1997). https://doi.org/10.1145/253671.253708

    Article  Google Scholar 

  20. Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Mixed Reality (MR) Reality-Virtuality (RV) Continuum. Syst Res 2351, 282–292 (1994)

    Google Scholar 

  21. Raheel, S.: Improving the user experience using an intelligent Adaptive User Interface in mobile applications. 2016 IEEE Int Multidiscip Conf Eng Technol IMCET 2016 64–68 (2016). https://doi.org/10.1109/IMCET.2016.7777428

  22. Hui, K.K., Liu, J., Wong, C.K.: An adaptive user interface based on personalized learning. IEEE Intell. Syst. 18, 52–57 (2003)

    Article  Google Scholar 

  23. Krishnaswamy, N., Pustejovsky, J.: An evaluation framework for multimodal interaction. Lr 2018 - 11th Int Conf Lang Resour Eval 2127–2134 (2019)

    Google Scholar 

  24. Rakkolainen, I., Farooq, A., Kangas, J., et al.: Technologies for multimodal interaction in extended reality—a scoping review. Multimodal Technol. Interact 5 (2021). https://doi.org/10.3390/mti5120081

  25. Keith, M.G., Tay, L., Harms, P.D.: Systems perspective of amazon mechanical turk for organizational research: review and recommendations. Front Psychol 8 (2017). https://doi.org/10.3389/fpsyg.2017.01359

  26. Difallah, D.E., Catasta, M., Demartini, G., et al.: The dynamics of micro-Task crowdsourcing: The case of amazon MTurk. WWW 2015 - Proc 24th Int Conf World Wide Web 238–247 (2015). https://doi.org/10.1145/2736277.2741685

  27. Orji, R., Tondello, G.F., Nacke, L.E.: Personalizing persuasive strategies in gameful systems to gamification user types. Conf Hum Factors Comput Syst - Proc 2018-April (2018). https://doi.org/10.1145/3173574.3174009

  28. Hirsh, J.B., Kang, S.K., Bodenhausen, G.V.: Personalized persuasion: tailoring persuasive appeals to recipients’ personality traits. Psychol. Sci. 23, 578–581 (2012). https://doi.org/10.1177/0956797611436349

    Article  Google Scholar 

  29. Jia, Y., Xu, B., Karanam, Y., Voida, S.: Personality-targeted gamification: a survey study on personality traits and motivational affordances. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16. ACM Press, New York, New York, USA, pp 2001–2013 (2016)

    Google Scholar 

  30. Buhrmester, M., Kwang, T., Gosling, S.D.: Amazon’s mechanical turk: a new source of inexpensive, yet high-quality data? Methodol issues Strateg Clin Res (4th ed) 133–139 (2015). https://doi.org/10.1037/14805-009

  31. Marjanovic, Z., Struthers, C.W., Cribbie, R., Greenglass, E.R.: The conscientious responders scale: a new tool for discriminating between conscientious and random responders. SAGE Open 4 (2014). https://doi.org/10.1177/2158244014545964

  32. Mohd, N., Bee, Y.: Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests. J. Stat. Model Anal. 2, 13–14 (2011)

    Google Scholar 

  33. Mills, A., Durepos, G., Wiebe, E.: Coding: open coding. In: Encyclopedia of Case Study Research. SAGE Publications, Inc., 2455 Teller Road, Thousand Oaks California 91320 United States (2013)

    Google Scholar 

  34. Allen, M.: Axial coding. In: The SAGE Encyclopedia of Communication Research Methods. SAGE Publications, Inc, 2455 Teller Road, Thousand Oaks California 91320 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Information Technology, Carleton University, Ottawa, Canada

    Eric Chan, Gerry Chan, Assem Kroma & Ali Arya

Authors
  1. Eric Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gerry Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Assem Kroma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ali Arya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ali Arya .

Editor information

Editors and Affiliations

  1. The Open University of Japan, Chiba, Japan

    Masaaki Kurosu

  2. Tokyo University of Science, Tokyo, Saitama, Japan

    Sakae Yamamoto

  3. Tokyo City University, Tokyo, Japan

    Hirohiko Mori

  4. Southern University of Science and Technology, Shenzhen, China

    Marcelo M. Soares

  5. World Usability Day and Bubble Mountain Consulting, Newton Center, MA, USA

    Elizabeth Rosenzweig

  6. Aaron Marcus and Associates, Berkeley, CA, USA

    Aaron Marcus

  7. Tsinghua University, Beijing, China

    Pei-Luen Patrick Rau

  8. Coventry University, Coventry, UK

    Don Harris

  9. Cranfield University, Cranfield, UK

    Wen-Chin Li

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chan, E., Chan, G., Kroma, A., Arya, A. (2022). Holistic Multimodal Interaction and Design. In: Kurosu, M., et al. HCI International 2022 - Late Breaking Papers. Design, User Experience and Interaction. HCII 2022. Lecture Notes in Computer Science, vol 13516. Springer, Cham. https://doi.org/10.1007/978-3-031-17615-9_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-17615-9_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-17614-2

  • Online ISBN: 978-3-031-17615-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation