skip to main content
10.1145/3673805.3673818acmotherconferencesArticle/Chapter ViewAbstractPublication PagesecceConference Proceedingsconference-collections
research-article

Approaching Intelligent In-vehicle Infotainment Systems through Fusion Visual-Speech Multimodal Interaction: A State-of-the-Art Review

Published: 08 October 2024 Publication History

Abstract

Advanced in-vehicle infotainment systems are part of the future’s intelligent autonomous vehicles. Developing such systems requires advanced interaction modalities to be utilized. On the other hand, using sophisticated applications on an unimodal touch display for manual driving vehicles could endanger drivers’ lives and result in a poor user experience. Offering in-vehicle fusion multimodal interaction could broaden the types of applications and enhance user experience while keeping safety and low distraction into account. Since in-vehicle interaction is bi-directional, both driver-vehicle and vehicle-driver sides are equally important to achieve and develop advanced infotainment systems. Searching in related literature, it has been found that there is good progress in driver-vehicle fusion multimodal interaction; in comparison, only a scarce amount of research on the vehicle-driver side is available. This paper presents the state-of-the-art in vehicle-driver fusion multimodal interaction for infotainment systems. This type of interaction is essentially a form of human-computer interaction. However, when we specify that the computer is a vehicle computer, certain specific factors become essential to consider when developing the user interface for this type of computer. Furthermore, a research agenda together with challenges and opportunities is proposed.

References

[1]
Abdul Rafey Aftab. 2019. Multimodal Driver Interaction with Gesture, Gaze and Speech. In 2019 International Conference on Multimodal Interaction(ICMI ’19). Association for Computing Machinery, New York, NY, USA, 487–492. https://doi.org/10.1145/3340555.3356093
[2]
Android. 2024. Android Auto. https://www.android.com/auto/
[3]
Apple. 2024. iOS - CarPlay. https://www.apple.com/ios/carplay/
[4]
Grace M. Begany, Ning Sa, and Xiaojun Yuan. 2016. Factors Affecting User Perception of a Spoken Language vs. Textual Search Interface: A Content Analysis. Interacting with Computers 28, 2 (March 2016), 170–180. https://doi.org/10.1093/iwc/iwv029
[5]
Michael Braun, Jingyi Li, Florian Weber, Bastian Pfleging, Andreas Butz, and Florian Alt. 2020. What If Your Car Would Care? Exploring Use Cases For Affective Automotive User Interfaces. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services(MobileHCI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3379503.3403530
[6]
Nabil Al Nahin Ch, Diana Tosca, Tyanna Crump, Alberta Ansah, Andrew Kun, and Orit Shaer. 2022. Gesture and Voice Commands to Interact With AR Windshield Display in Automated Vehicle: A Remote Elicitation Study. In Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications(AutomotiveUI ’22). Association for Computing Machinery, New York, NY, USA, 171–182. https://doi.org/10.1145/3543174.3545257
[7]
Alan Dix. 2007. Human computer interaction (3. ed., [nachdr.] ed.). Pearson Prentice Hall, Harlow, England [u.a.
[8]
Sandro Rodriguez Garzon. 2012. Intelligent In-Car-Infotainment Systems: A Contextual Personalized Approach. In 2012 Eighth International Conference on Intelligent Environments. IEEE, Guanajuato, Mexico, 315–318. https://doi.org/10.1109/IE.2012.70
[9]
Renate Haeuslschmid, Susanne Forster, Katharina Vierheilig, Daniel Buschek, and Andreas Butz. 2017. Recognition of Text and Shapes on a Large-Sized Head-Up Display. In Proceedings of the 2017 Conference on Designing Interactive Systems(DIS ’17). Association for Computing Machinery, New York, NY, USA, 821–831. https://doi.org/10.1145/3064663.3064736
[10]
Hansjörg Hofmann, Vanessa Tobisch, Ute Ehrlich, André Berton, and Angela Mahr. 2014. Comparison of speech-based in-car HMI concepts in a driving simulation study. In Proceedings of the 19th international conference on Intelligent User Interfaces(IUI ’14). Association for Computing Machinery, New York, NY, USA, 215–224. https://doi.org/10.1145/2557500.2557509
[11]
SAE International. 2024. SAE Levels of Driving Automation™ Refined for Clarity and International Audience. https://www.sae.org/site/blog/sae-j3016-update
[12]
Grega Jakus, Christina Dicke, and Jaka Sodnik. 2015. A user study of auditory, head-up and multi-modal displays in vehicles. Applied Ergonomics 46 (Jan. 2015), 184–192. https://doi.org/10.1016/j.apergo.2014.08.008
[13]
Pascal Jansen, Mark Colley, and Enrico Rukzio. 2022. A Design Space for Human Sensor and Actuator Focused In-Vehicle Interaction Based on a Systematic Literature Review. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 2 (July 2022), 56:1–56:51. https://doi.org/10.1145/3534617
[14]
Jingun Jung, Sangyoon Lee, Jiwoo Hong, Eunhye Youn, and Geehyuk Lee. 2020. Voice+Tactile: Augmenting In-vehicle Voice User Interface with Tactile Touchpad Interaction. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376863
[15]
Jeff Smith Kia. 2022. The History and Evolution of In-Car Digital Display Systems. https://www.kiakool.com/blog/2022/april/20/the-history-and-evolution-of-in-car-digital-display-systems.htm
[16]
Raymond J. Kiefer. 1991. Effect of a Head-Up Versus Head-Down Digital Speedometer on Visual Sampling Behavior and Speed Control Performance During Daytime Automobile Driving. SAE Transactions 100 (1991), 82–93. https://www.jstor.org/stable/44632015 Publisher: SAE International.
[17]
Kalmanje Krishnakumar. 2002. Intelligent Systems for Aerospace Engineering: An Overview. Brussels. https://ntrs.nasa.gov/citations/20020065377 NTRS Author Affiliations: NASA Ames Research Center NTRS Document ID: 20020065377 NTRS Research Center: Ames Research Center (ARC).
[18]
Guofa Li, Fangping Zhu, Tingru Zhang, Ying Wang, Shengfan He, and Xingda Qu. 2018. Evaluation of Three In-Vehicle Interactions from Drivers’ Driving Performance and Eye Movement behavior. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC). 2086–2091. https://doi.org/10.1109/ITSC.2018.8569917 ISSN: 2153-0017.
[19]
Mercedes-Benz. 2020. The new Mercedes-Benz S-Class. http://media.mercedes-benz.ca/releases/the-new-mercedes-benz-s-class
[20]
Prajval Kumar Murali, Mohsen Kaboli, and Ravinder Dahiya. 2022. Intelligent In-Vehicle Interaction Technologies. Advanced Intelligent Systems 4, 2 (2022), 2100122. https://doi.org/10.1002/aisy.202100122 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/aisy.202100122.
[21]
Christian Müller and Garrett Weinberg. 2011. Multimodal Input in the Car, Today and Tomorrow. IEEE MultiMedia 18, 1 (Jan. 2011), 98–103. https://doi.org/10.1109/MMUL.2011.14 Conference Name: IEEE MultiMedia.
[22]
Matthias Peissner, Vanessa Doebler, and Florian Metze. 2011. Can voice interaction help reducing the level of distraction and prevent accidents? (2011). https://publica.fraunhofer.de/handle/publica/295615 Publisher: Fraunhofer IAO.
[23]
F. Schewe and M. Vollrath. 2020. Ecological interface design effectively reduces cognitive workload – The example of HMIs for speed control. Transportation Research Part F: Traffic Psychology and Behaviour 72 (July 2020), 155–170. https://doi.org/10.1016/j.trf.2020.05.009
[24]
Tanja Schneeberger, Simon von Massow, Mohammad Mehdi Moniri, Angela Castronovo, Christian Müller, and Jan Macek. 2015. Tailoring mobile apps for safe on-road usage: how an interaction concept enables safe interaction with hotel booking, news, Wolfram Alpha and Facebook. In Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications(AutomotiveUI ’15). Association for Computing Machinery, New York, NY, USA, 241–248. https://doi.org/10.1145/2799250.2799264
[25]
Missie Smith, Joseph L. Gabbard, Gary Burnett, and Nadejda Doutcheva. 2017. The Effects of Augmented Reality Head-Up Displays on Drivers’ Eye Scan Patterns, Performance, and Perceptions. International Journal of Mobile Human Computer Interaction (IJMHCI) 9, 2 (2017), 1–17. https://doi.org/10.4018/IJMHCI.2017040101 Publisher: IGI Global.
[26]
Bethan Hannah Topliss, Sanna M Pampel, Gary Burnett, Lee Skrypchuk, and Chrisminder Hare. 2018. Establishing the Role of a Virtual Lead Vehicle as a Novel Augmented Reality Navigational Aid. In Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications(AutomotiveUI ’18). Association for Computing Machinery, New York, NY, USA, 137–145. https://doi.org/10.1145/3239060.3239069
[27]
Oliver M. Winzer, Antonia S. Conti-Kufner, and Klaus Bengler. 2018. Intersection Traffic Light Assistant – An Evaluation of the Suitability of two Human Machine Interfaces. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC). 261–265. https://doi.org/10.1109/ITSC.2018.8569708 ISSN: 2153-0017.

Index Terms

  1. Approaching Intelligent In-vehicle Infotainment Systems through Fusion Visual-Speech Multimodal Interaction: A State-of-the-Art Review

          Recommendations

          Comments

          Information & Contributors

          Information

          Published In

          cover image ACM Other conferences
          ECCE '24: Proceedings of the European Conference on Cognitive Ergonomics 2024
          October 2024
          286 pages
          ISBN:9798400718243
          DOI:10.1145/3673805
          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          Published: 08 October 2024

          Permissions

          Request permissions for this article.

          Check for updates

          Author Tags

          1. HUD
          2. In-vehicle interaction
          3. fusion multimodal interaction
          4. head-up display
          5. vehicle-driver interaction
          6. visual-speech interaction

          Qualifiers

          • Research-article
          • Research
          • Refereed limited

          Conference

          ECCE 2024

          Acceptance Rates

          Overall Acceptance Rate 56 of 91 submissions, 62%

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • 0
            Total Citations
          • 37
            Total Downloads
          • Downloads (Last 12 months)37
          • Downloads (Last 6 weeks)9
          Reflects downloads up to 20 Jan 2025

          Other Metrics

          Citations

          View Options

          Login options

          View options

          PDF

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format.

          HTML Format

          Media

          Figures

          Other

          Tables

          Share

          Share

          Share this Publication link

          Share on social media