Skip to main content
SpringerLink
Log in

DSDCLA: driving style detection via hybrid CNN-LSTM with multi-level attention fusion

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Driving style detection is an essential real-world requirement in diverse contexts, such as traffic safety, car insurance and fuel consumption optimization. However, the existing methods either rely on handcrafted features or fail to explore deep spatial-temporal features from multi-modal sensing signals. In this paper, we propose a novel attention-based hybrid convolutional neural network (CNN) and long short-term memory (LSTM) framework named DSDCLA to address these problems. Specifically, DSDCLA first introduces CNN and self-attention for extracting local spatial features from multi-modal driving sequences. Then, we utilize LSTM and multi-head attention to explore the long-term temporal relationships between timesteps. Therefore, DSDCLA can identify driving style by short- and long-term spatial-temporal features. Furthermore, we design three variants with different levels of fusion, which shows the advantage of selecting components and improves the interpretability. We extensively evaluated the proposed DSDCLA on two public real-world datasets, and the experimental results show that DSDCLA outperforms the current state-of-the-art methods, achieving the F1-scores of 97.03% and 97.65%. Numerous ablation studies and visualizations indicate the effectiveness of the model and the importance of multi-level attention fusion for identifying driving style between timesteps.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. This dataset is available at https://github.com/fdjingliu/FD-Driveset.

  2. For better visualization, we process the raw data at high frequencies by low-pass filtering [32] to make the curves look smoother rather than violently jittery.

References

  1. Azadani M N, Boukerche A (2021) Driving behavior analysis guidelines for intelligent transportation systems. IEEE Trans Intell Transp Syst:1–19, https://doi.org/10/gmwv7k

  2. Bejani M M, Ghatee M (2018) A context aware system for driving style evaluation by an ensemble learning on smartphone sensors data. Transportation Research Part C: Emerging Technologies 89:303–320. https://doi.org/10/gdcv2s

    Article  Google Scholar 

  3. Bejani M M, Ghatee M (2020) Convolutional neural network with adaptive regularization to classify driving styles on smartphones. IEEE Trans Intell Transp Syst 21(2):543–552. https://doi.org/10/gnkknp

    Article  Google Scholar 

  4. Chan T K, Chin C S, Chen H et al (2020) A comprehensive review of driver behavior analysis utilizing smartphones. IEEE Trans Intell Transp Syst 21(10):4444–4475. https://doi.org/10/ghnt9h

    Article  Google Scholar 

  5. Chechetka A (2019) Pilotguru. https://github.com/waiwnf/pilotguru

  6. Chen J, Wu Z, Zhang J (2019) Driving safety risk prediction using cost-sensitive with nonnegativity-constrained autoencoders based on imbalanced naturalistic driving data. IEEE Trans Intell Transp Syst 20 (12):4450–4465. https://doi.org/10/gmwv65

    Article  Google Scholar 

  7. Dhal P, Azad C (2022) A comprehensive survey on feature selection in the various fields of machine learning. Appl Intell 52(4):4543–4581. https://doi.org/10/gqqvrd

    Article  Google Scholar 

  8. Dong W, Yuan T, Yang K et al (2017) Autoencoder regularized network for driving style representation learning. In: Proceedings of the twenty-sixth international joint conference on artificial intelligence, pp 1603–1609, https://doi.org/10/gqfpkp

  9. Jain N, Mittal S (2021) Bayesian nash equilibrium based gaming model for eco-safe driving. J King Saud University - Comput Inf Sci, https://doi.org/10/gpktn8

  10. Kenkar Z, AlHalawani S et al (2019) Event-based driving style analysis. In: Alfaries A, Mengash H, Yasar A (eds) Advances in data science, cyber security and IT applications. Springer international publishing, Cham, Communications in computer and information science, pp 170-182. https://doi.org/10/gm57kn

  11. Khodairy M A, Abosamra G (2021) Driving behavior classification based on oversampled signals of smartphone embedded sensors using an optimized stacked-lstm neural networks. IEEE Access 9:4957–4972. https://doi.org/10/gmv3r7

    Article  Google Scholar 

  12. Kieu T, Yang B, Jensen CS (2018) Outlier detection for multidimensional time series using deep neural networks. In: 2018 19th IEEE international conference on mobile data management (MDM), pp 125–134, https://doi.org/10/gmv7kr

  13. Lee D H, Chen K L, Liou K H et al (2021) Deep learning and control algorithms of direct perception for autonomous driving. Appl Intell 51(1):237–247. https://doi.org/10/gqqvqr

    Article  Google Scholar 

  14. Li F, Gui Z, Zhang Z et al (2020) A hierarchical temporal attention-based lstm encoder-decoder model for individual mobility prediction. Neurocomputing 403:153–166. https://doi.org/10/gpktpb

    Article  Google Scholar 

  15. Li H S, Fan P, Hy Xia et al (2019) Quantum multi-level wavelet transforms. Inf Sci 504:113–135. https://doi.org/10/gqqs6c

    Article  MathSciNet  MATH  Google Scholar 

  16. Lin X, Zhang G, Wei S (2021) Velocity prediction using markov chain combined with driving pattern recognition and applied to dual-motor electric vehicle energy consumption evaluation. Appl Soft Comput 101:106,998. https://doi.org/10/gpktn6

    Article  Google Scholar 

  17. Liu C, Zhang L, Niu J et al (2020a) Intelligent prognostics of machining tools based on adaptive variational mode decomposition and deep learning method with attention mechanism. Neurocomputing 417:239–254. https://doi.org/10/gjq5dr

    Article  Google Scholar 

  18. Liu W, Deng K, Zhang X et al (2020b) A semi-supervised tri-catboost method for driving style recognition. Symmetry 12(3):336. https://doi.org/10/gmwcnj

    Article  Google Scholar 

  19. Liu Y, Liu J, Lin J et al (2022a) Appearance-motion united auto-encoder framework for video anomaly detection. IEEE Trans Circuits Syst II: Express Briefs (TCAS-II):5, https://doi.org/10/gpwbmr

  20. Liu Y, Liu J, Zhao M et al (2022b) Collaborative normality learning framework for weakly supervised video anomaly detection. IEEE Trans Circuits Syst II: Express Briefs (TCAS-II):5, https://doi.org/10/gpwbmq

  21. Ma C, Dai X, Zhu J et al (2017) Drivingsense: Dangerous driving behavior identification based on smartphone autocalibration. Mobile Inf Syst 2017:e9075,653. https://doi.org/10.1155/2017/9075653

    Google Scholar 

  22. Ma H, Li W, Zhang X et al (2019) Attnsense: multi-level attention mechanism for multimodal human activity recognition. In: Proceedings of the twenty-eighth international joint conference on artificial intelligence. International joint conferences on artificial intelligence organization, Macao, pp 3109-3115. https://doi.org/10/gjgc7v

  23. Ma Y, Li W, Tang K et al (2021) Driving style recognition and comparisons among driving tasks based on driver behavior in the online car-hailing industry. Accident Anal Prevention 154:106,096. https://doi.org/10/gpktpm

    Article  Google Scholar 

  24. Manaswi N K (2018) Understanding and working with keras. In: Manaswi NK (ed) Deep learning with applications using python: chatbots and face, Object, and speech recognition with TensorFlow and Keras, pp 31-43. https://doi.org/10.1007/978-1-4842-3516-4_2

  25. Martinelli F, Marulli F, Mercaldo F et al (2021) Neural networks for driver behavior analysis. Electronics 10(3):342. https://doi.org/10/gmwv7d

    Article  Google Scholar 

  26. Moreira-Matias L, Farah H (2017) On developing a driver identification methodology using in-vehicle data recorders. IEEE Trans Intell Transp Syst 18(9):2387–2396. https://doi.org/10/gbwkgg

    Article  Google Scholar 

  27. Mou L, Zhou C, Zhao P et al (2021) Driver stress detection via multimodal fusion using attention-based cnn-lstm. Expert Syst Appl 173:114,693. https://doi.org/10/gkxx56

    Article  Google Scholar 

  28. Moujahid A, Dornaika F, Arganda-Carreras I et al (2021) Efficient and compact face descriptor for driver drowsiness detection. Expert Syst Appl 168:114,334. https://doi.org/10/gpktn9

    Article  Google Scholar 

  29. Moukafih Y, Hafidi H, Ghogho M (2019) Aggressive driving detection using deep learning-based time series classification. In: 2019 IEEE international symposium on INnovations in intelligent SysTems and applications (INISTA), pp 1–5, https://doi.org/10/gmv3wf

  30. Najah Ahmed A, Binti Othman F, Abdulmohsin Afan H et al (2019) Machine learning methods for better water quality prediction. J Hydrology 578:124,084. https://doi.org/10/ghp5w8

    Article  Google Scholar 

  31. Niu Z, Zhong G, Yu H (2021) A review on the attention mechanism of deep learning. Neurocomputing 452:48–62. https://doi.org/10/gk8br9

    Article  Google Scholar 

  32. Ouyang K, Hou Y, Zhang Y et al (2022) Knowledge transfer via distillation from time and frequency domain for time series classification. Appl Intell, https://doi.org/10/gqqvrf

  33. Poernomo A, Kang DK (2018) Biased dropout and crossmap dropout: learning towards effective dropout regularization in convolutional neural network. Neural Netw 104:60–67. https://doi.org/10/gnb9z2

    Article  Google Scholar 

  34. Rashid KM, Louis J (2019) Times-series data augmentation and deep learning for construction equipment activity recognition. Adv Eng Inf 42:100,944. https://doi.org/10/ghs4bx

    Article  Google Scholar 

  35. Rastgoo M N, Nakisa B, Maire F et al (2019) Automatic driver stress level classification using multimodal deep learning. Expert Syst Appl 138:112,793. https://doi.org/10/gjvgh4

    Article  Google Scholar 

  36. Rodríguez P, Bautista M A, Gonzàlez J et al (2018) Beyond one-hot encoding: lower dimensional target embedding. Image Vis Comput 75:21–31. https://doi.org/10/gdw9pf

    Article  Google Scholar 

  37. Romera E, Bergasa LM, Arroyo R (2016) Need data for driver behaviour analysis? presenting the public uah-driveset. In: 2016 IEEE 19th international conference on intelligent transportation systems (ITSC), pp 387–392, https://doi.org/10/ggwdcb

  38. Saleh K, Hossny M, Nahavandi S (2017) Driving behavior classification based on sensor data fusion using lstm recurrent neural networks. In: 2017 IEEE 20th international conference on intelligent transportation systems (ITSC), pp 1–6, https://doi.org/10/gktnfr

  39. Savelonas M, Vernikos I, Mantzekis D et al (2021) Hybrid representation of sensor data for the classification of driving behaviour. Appl Sci 11(18):8574. https://doi.org/10/gmwwxv

    Article  Google Scholar 

  40. Song L, Hu X, Zhang G et al (2022) Networking systems of ai: on the convergence of computing and communications. IEEE Internet of Things J:1–1, https://doi.org/10/gp35v3

  41. Suzdaleva E, Nagy I (2018) An online estimation of driving style using data-dependent pointer model. Transport Res Part C: Emerg Technol 86:23–36. https://doi.org/10/gc49f5

    Article  Google Scholar 

  42. van de Ruit M, Billeter M, Eisemann E (2022) An efficient dual-hierarchy t-sne minimization. IEEE Trans Vis Comput Graph 28(1):614–622. https://doi.org/10/gpg323

    Article  Google Scholar 

  43. Wang J, Zhang Z, Lu G (2021) A bayesian inference based adaptive lane change prediction model. Transport Res Part C: Emerg Technol 132:103,363. https://doi.org/10.1016/j.trc.2021.103363

    Article  Google Scholar 

  44. Wang Y, Song W, Tao W et al (2022a) A systematic review on affective computing: emotion models, databases, and recent advances. Inf Fusion 83–84:19–52. 10.1016/j.inffus.2022.03.009

    Article  Google Scholar 

  45. Wang Y, Sun Y, Huang Y et al (2022b) Ferv39k: a large-scale multi-scene dataset for facial expression recognition in videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 20,922–20,931, https://doi.org/10/gqqvvt

  46. Wei DL, Liu CG, Liu Y et al (2022) Look, listen and pay more attention: Fusing multi-modal information for video violence detection. In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 1980-1984. https://doi.org/10/gqqs6f

  47. Würtz S, Göhner U (2020) Driving style analysis using recurrent neural networks with lstm cells. J Adv Inf Technol:1–9, https://doi.org/10.12720/jait.11.1.1-9

  48. Xie J, Hu K, Li G et al (2021) Cnn-based driving maneuver classification using multi-sliding window fusion. Expert Syst Appl 169:114,442. https://doi.org/10/gmwv7j

    Article  Google Scholar 

  49. Xie Y, He M, Ma T et al (2022) Optimal distributed parallel algorithms for deep learning framework tensorflow. Appl Intell 52(4):3880–3900. https://doi.org/10/gqqs6d

    Article  Google Scholar 

  50. Yang L, Ma R, Zhang HM et al (2018) Driving behavior recognition using eeg data from a simulated car-following experiment. Accident Anal Prevention 116:30–40. https://doi.org/10/gdpvgk

    Article  Google Scholar 

  51. Yu J, Chen Z, Zhu Y et al (2017) Fine-grained abnormal driving behaviors detection and identification with smartphones. IEEE Trans Mob Comput 16(8):2198–2212. https://doi.org/10/ghnt9g

    Article  Google Scholar 

  52. Yuan W, Hu F, Lu L (2022) A new non-adaptive optimization method: Stochastic gradient descent with momentum and difference. Appl Intell 52(4):3939–3953. https://doi.org/10/gn8bjg

    Article  Google Scholar 

  53. Yuan Y, Lu Y, Wang Q (2020) Adaptive forward vehicle collision warning based on driving behavior. Neurocomputing 408:64–71. https://doi.org/10/gnz84b

    Article  Google Scholar 

  54. Zhang J, Wu Z, Li F et al (2019a) A deep learning framework for driving behavior identification on in-vehicle can-bus sensor data. Sensors 19(6):1356. https://doi.org/10/ggsqcr

    Article  Google Scholar 

  55. Zhang Y, Li J, Guo Y et al (2019b) Vehicle driving behavior recognition based on multi-view convolutional neural network with joint data augmentation. IEEE Trans Veh Technol 68(5):4223–4234. https://doi.org/10/gktj5w

    Article  Google Scholar 

  56. Zheng Y, Hansen JHL (2017) Lane-change detection from steering signal using spectral segmentation and learning-based classification. IEEE Trans Intell Vehicles 2(1):14–24. https://doi.org/10/gnm68j

    Article  Google Scholar 

Download references

Acknowledgements

This research is funded by the China Mobile Research Fund of Chinese Ministry of Education (Grant No. KEH2310029). The work is also supported by the Shanghai Key Research Lab. of NSAI and the Joint Lab. on Networked AI Edge Computing Fudan University-Changan. We would like to thank Prof. Xing Hu of the University of Shanghai for Science and Technology and Dr. Xiaoguang Zhu of the Fudan University for their help in checking and polishing this paper. We sincerely thank all the editors and anonymous reviewers for their careful work and thoughtful suggestions that have helped improve this paper substantially.

Author information

Authors and Affiliations

  1. Academy for Engineering & Technology, Fudan University, Shanghai, Shanghai, China

    Jing Liu, Yang Liu, Hanqi Wang & Liang Song

  2. Shanghai East-bund Research Institute on NSAI, Shanghai, Shanghai, China

    Jing Liu, Di Li & Liang Song

  3. School of Information Management & Engineering, Shanghai University of Finance and Economics, Shanghai, Shanghai, China

    Xiaohong Huang

Authors
  1. Jing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Di Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hanqi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaohong Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Liang Song
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liang Song.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, J., Liu, Y., Li, D. et al. DSDCLA: driving style detection via hybrid CNN-LSTM with multi-level attention fusion. Appl Intell 53, 19237–19254 (2023). https://doi.org/10.1007/s10489-023-04451-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04451-5

Keywords

Search

Navigation