demonstration

MIDriveSafely: Multimodal Interaction for Drive Safely

Authors:

Alexey KarpovAuthors Info & Claims

ICMI '22: Proceedings of the 2022 International Conference on Multimodal Interaction

Pages 733 - 735

https://doi.org/10.1145/3536221.3557037

Published: 07 November 2022 Publication History

Get Access

Abstract

In this paper, we present a novel multimodal interaction application to help car drivers and increase their road safety. MIDriveSafely is a mobile application that provides the following functions: (1) detect dangerous situations based on video information from a smartphone front-facing camera, such as drowsiness/sleepiness, phone usage while driving, eating, smoking, unfastened seat belt, etc.; gives a feedback to the driver (2) provide entertainment (e.g. rock-paper-scissors game, based on automatic speech recognition), (3) provide voice control capabilities to navigation/multimedia systems of a smartphone (potentially vehicle systems such as lighting conditions/climate control). Speech recognition in driving conditions is highly challenging due to acoustic noises, active head turns, pose variations, distance to recording devices, etc. MIDriveSafely incorporates driver's audio-visual speech recognition (DAVIS) system and uses it for multimodal interaction. Along with this, the original DriveSafely system is used for dangerous state detection. MIDriveSafely improves upon existing driver monitoring applications using multimodal (mainly audio-visual) information. MIDriveSafely motivates people to drive in a safer manner by providing the feedback to the drivers and by creating a fun user experience.

References

[1]

Das, Kapotaksha, 2021. Multimodal Detection of Drivers Drowsiness and Distraction. In Proceedings of the 2021 International Conference on Multimodal Interaction (ICMI), 416-424.

Digital Library

Google Scholar

[2]

Aftab, Abdul Rafey. 2019. Multimodal Driver Interaction with Gesture, Gaze and Speech. In 2019 International Conference on Multimodal Interaction (ICMI), 487-492.

Digital Library

Google Scholar

[3]

Stappen, Lukas, Georgios Rizos, and Björn Schuller. 2020. X-aware: Context-aware human-environment attention fusion for driver gaze prediction in the wild. Proceedings of the 2020 International Conference on Multimodal Interaction (ICMI), 858-867.

Digital Library

Google Scholar

[4]

Du, Yulun, 2018. Multimodal polynomial fusion for detecting driver distraction. arXiv preprint arXiv:1810.10565.

Google Scholar

[5]

Roider, Florian, 2019. Investigating the effects of modality switches on driver distraction and interaction efficiency in the car. Journal on Multimodal User Interfaces 13.2 (2019), 89-97.

Google Scholar

[6]

Mou, Luntian, 2021. Driver stress detection via multimodal fusion using attention-based CNN-LSTM. Expert Systems with Applications 173 (2021), 114693.

Digital Library

Google Scholar

[7]

Bylykbashi, Kevin, 2020. Fuzzy-based Driver Monitoring System (FDMS): Implementation of two intelligent FDMSs and a testbed for safe driving in VANETs. Future Generation Computer Systems 105, 2020, 665-674.

Digital Library

Google Scholar

[8]

Xie, Yadong, Real-time detection for drowsy driving via acoustic sensing on smartphones. IEEE Transactions on Mobile Computing 20.8, 2020, 2671-2685.

Google Scholar

[9]

Kashevnik, Alexey, 2020. Cloud-Based Driver Monitoring System Using a Smartphone. IEEE Sensors, vol. 20(12), 6701–6715.

Crossref

Google Scholar

[10]

Ivanko, Denis, 2022. Multi-Speaker Audio-Visual Corpus RUSAVIC: Russian Audio-Visual Speech in Cars. In Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), 1555-1559.

Google Scholar

[11]

Ivanko, Denis, 2022. DAVIS: Driver's Audio-Visual Speech Recognition. Accepted for ISCA Annual Conference Interspeech 2022.

Google Scholar

Cited By

View all

Rathipriya NMaheswari N(2024)A Comprehensive Review of Recent Advances in Deep Neural Networks for Lipreading With Sign Language RecognitionIEEE Access10.1109/ACCESS.2024.346396912(136846-136879)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3463969
Ryumin DAxyonov ARyumina EIvanko DKashevnik AKarpov A(2024)Audio–visual speech recognition based on regulated transformer and spatio–temporal fusion strategy for driver assistive systemsExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.124159252:PAOnline publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.124159
Ryumin DIvanko DRyumina E(2023)Audio-Visual Speech and Gesture Recognition by Sensors of Mobile DevicesSensors10.3390/s2304228423:4(2284)Online publication date: 17-Feb-2023
https://doi.org/10.3390/s23042284
Show More Cited By

Recommendations

The Paradox of Driving-Related Distraction: Driver Monitoring Systems Trigger Warnings During Overtaking Maneuvers
AutomotiveUI '24 Adjunct: Adjunct Proceedings of the 16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

Driver Monitoring Systems (DMS) detect visual distraction and alert drivers to maintain focus on the road. The European New Car Assessment Programme developed an algorithm that triggers warnings upon detecting distraction. Previous research showed that ...
How Do Automation Modes Influence the Frequency of Advanced Driver Distraction Warnings? A Simulator Study
AutomotiveUI '23 Adjunct: Adjunct Proceedings of the 15th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

Driver monitoring systems detect driver distraction and will become a standard safety feature in vehicles soon. Advanced driver distraction warnings prompt drivers through different modalities to keep their attention on the road. With added driving ...
Multimodal inference for driver-vehicle interaction
ICMI-MLMI '09: Proceedings of the 2009 international conference on Multimodal interfaces

In this paper we present a novel system for driver-vehicle interaction which combines speech recognition with facial-expression recognition to increase intention recognition accuracy in the presence of engine- and road-noise. Our system would allow ...

Comments

Information & Contributors

Information

Published In

ICMI '22: Proceedings of the 2022 International Conference on Multimodal Interaction

November 2022

830 pages

ISBN:9781450393904

DOI:10.1145/3536221

Editors:
Raj Tumuluri
Openstream
,
Nicu Sebe
University of Trento
,
Gopal Pingali
Accenture
,
Dinesh Babu Jayagopi
IIIT Bangalore
,
Abhinav Dhall
IIT Ropar
,
Richa Singh
IIT Jodhpur
,
Lisa Anthony
University of Florida
,
Albert Ali Salah
Utrecht University and Boğaziçi University

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2022

Check for updates

Author Tags

Qualifiers

Demonstration
Research
Refereed limited

Conference

ICMI '22

Sponsor:

SIGCHI

ICMI '22: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 7 - 11, 2022

Bengaluru, India

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
173
Total Downloads

Downloads (Last 12 months)46
Downloads (Last 6 weeks)5

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Rathipriya NMaheswari N(2024)A Comprehensive Review of Recent Advances in Deep Neural Networks for Lipreading With Sign Language RecognitionIEEE Access10.1109/ACCESS.2024.346396912(136846-136879)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3463969
Ryumin DAxyonov ARyumina EIvanko DKashevnik AKarpov A(2024)Audio–visual speech recognition based on regulated transformer and spatio–temporal fusion strategy for driver assistive systemsExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.124159252:PAOnline publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.124159
Ryumin DIvanko DRyumina E(2023)Audio-Visual Speech and Gesture Recognition by Sensors of Mobile DevicesSensors10.3390/s2304228423:4(2284)Online publication date: 17-Feb-2023
https://doi.org/10.3390/s23042284
Ivanko DRyumin DKarpov A(2023)A Review of Recent Advances on Deep Learning Methods for Audio-Visual Speech RecognitionMathematics10.3390/math1112266511:12(2665)Online publication date: 12-Jun-2023
https://doi.org/10.3390/math11122665

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Recommendations

The Paradox of Driving-Related Distraction: Driver Monitoring Systems Trigger Warnings During Overtaking Maneuvers

How Do Automation Modes Influence the Frequency of Advanced Driver Distraction Warnings? A Simulator Study

Multimodal inference for driver-vehicle interaction

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Share

Share this Publication link

Share on social media

Affiliations