Skip to main content
SpringerLink
Log in

A Multimodal Hypovigilance Detection System Based on Fuzzy Logic and Transfer Learning

  • Published:
Journal of Signal Processing Systems Aims and scope Submit manuscript

Abstract

The problem of road safety is an important issue for human society. The challenge is to reduce the number of accidents, and consequently the number of fatalities and injuries on roads caused by the hypovigilance. The objective of this work is to concept a multimodal driving assistance system able to detect different categories of lack of vigilance and alert the driver at an early stage to ensure his safety and avoid road accidents. In fact, our system uses a video approach through eyes blinking and head movement analysis, which allows us to detect many vigilance states. Our first contribution is manifested by the invention of a hybrid eyes classifier using the fast wavelet transform (FWT-F) which includes a fuzzy decision part involved in the test phase to compute the similarity degree between training images and testing ones. It is more efficient than using the traditional fast wavelet network (FWT), which is based on the Euclidean distance for similarity computing. The second contribution resides in the proposition of a novel architecture of eyes classifier system based on Convolution Neural Network (CNN) feature extraction and wavelet network (WN) classifier. The last contribution is manifested by the final system, which provides seven different vigilance levels.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Figure 18
Figure 19
Figure 20
Figure 21
Figure 22

Similar content being viewed by others

References

  1. Shin, H., Jung, S., Kim, J., & Chung, W. Y. (2010). Real time car driver’s condition monitoring system. In Proceeding of IEEE Sensors conference, pp. 951 - 954, Kona, HT, USA, November 1 - 4.

  2. Klein, R., Allen, R., & Miller, J. (1980). Relationship between truck ride quality and safety of operations: methodology development. System technology: Technical report.

    Google Scholar 

  3. Liang, Y., & Lee, J. D. (2014). A hybrid Bayesian Network approach to detect driver cognitive distraction. Transportation Research Part C: Emerging Technologies (TRANSPORT RES C-EMER), vol. 38 , pp. 146 - 155.

  4. Picot, A., Caplier A., & Charbonnier, S. (2009). Driver drowsiness detection using both physiological and video information, Institut National Polytechnique de Grenoble - INPG, French, Thesis report.

  5. Akrout, B., & Mahdi, W. (2015). Spatio-temporal features for the automatic control of driver drowsiness state and lack of concentration. Machine Vision and Applications (MVA), Springer-Verlag, vol. 26 (1), pp. 1 - 13.

  6. Teyeb, I., Jemai, O., Zaied, M., & Ben Amar, C. (2014). A Novel Approach for Drowsy Driver Detection Using Head Posture Estimation and Eyes Recognition System Based on Wavelet Network. The Fifth International Conference on Information, Intelligence, Systems and Applications (IISA) proceedings (pp. 379–384). Crete: Chania.

    Google Scholar 

  7. Teyeb, I., Jemai, O., Zaied, M., & Ben Amar, C. (2014). A Drowsy Driver Detection System Based on a New Method of Head Posture Estimation, The 15th International Conference on Intelligent Data Engineering and Automated Learning (IDEAL) proceedings, vol. 8669, pp. 362 - 369, Salamanca, Spain.

  8. Teyeb, I., Jemai, O., Zaied, M., & Ben Amar, C. (2015). A multi level system design for vigilance measurement based on head posture estimation and eyes blinking, Proc. SPIE 9875, Eighth International Conference on Machine Vision (ICMV), pp. 987 - 51P, Barcelone, Spain.

  9. Teyeb, I., Jemai, O., Zaied, M., & Ben Amar, C. (2015). Vigilance Measurement System Through Analysis of Visual and Emotional Drivers Signs Using Wavelet Networks, Proceeding of The 15th International Conference on Intelligent Systems Design and Applications (ISDA), pp. 140 - 147, Marrakech, Morocco.

  10. Viola, P., & Jones, M. (2001). Robust real-time object detection. International Journal of Computer Vision, 57(2), 137–154.

    Article  Google Scholar 

  11. Teyeb, I., Jemai, O., Bouchrika, T., & Ben Amar, C. (2013). Detecting Driver Drowsiness Using Eyes Recognition System Based on Wavelet Network, 5th International Conference on Web and Information Technologies (ICWIT), pp. 245 - 254, Hammamat, Tunisia.

  12. Jemai, O., Teyeb, I., Bouchrika, T., & Ben Amar, C. (2013). A novel approach for drowsy driver detection using eyes recognition system based on wavelet network. International Journal of Recent Contributions from Engineering, Science IT (IJES), vol. 1 (1), pp. 46 - 52, July 07 - 08, LNCS 2095.

  13. Triyanti, V., & Iridiastadi, H. (2017). Challenges in detecting drowsiness based on driver’s behavior. IOP Conference Series: Materials Science and Engineering, 277, 012042.

  14. Ben Amar, C., & Jemai, O. (2007). Wavelet networks approach for image compression. ICGST: The International Congress for global Science and Technology, GVIP International Journal on Graphics, Vision and Image Processing, GVIP Special Issue on Image Compression, pp. 1523.

  15. Bouchrika, T., Zaied, M., Jemai, O., & Ben Amar, C. (2012). Ordering computers by hand gestures recognition based on wavelet networks, 2nd International Conference on Communications, Computing and Control Applications proceedings, December, 06-08, Marseilles, France, https://doi.org/10.1109/CCCA.2012.6417911,pp. 1-6.

  16. Said, S., Ben Amor, B., Zaied, M., Ben Amar, C., & Daoudi, M. (2009). Fast and efficient 3D face recognition using wavelet networks. 16th IEEE International Conference on Image Processing, pp. 4153-4156.

  17. Zaied, M., Said, S., Jemai, O., & Ben Amar, C. (2011). A novel approach for face recognition based on fast learning algorithm and wavelet network theory, IJWMIP: International Journal of Wavelets. Multiresolution and Information Processing, 9(6), 923–945.

    Article  MathSciNet  MATH  Google Scholar 

  18. Zaied, M., Jemai, O., & Ben Amar, C. (2008). Training of the Beta wavelet networks by the frames theory: Application to face recognition, IPTA: Image Processing Theory, Tools Applications, pp. 165-170, Tunisia November.

  19. Zaied, M., Ben Amar, C., & Alimi, A. M. (2005). Beta Wavelet Networks for Face Recognition, Journal Of Decision Systems JDS, Lavoisier 2005 Edition, Vol. 14, N. 1-2, pages 109-122.

  20. Guedri, B., Zaied, M. & Ben Amar, C. (2011). Indexing and images retrieval by content, IEEE 2011 International Conference on High Performance Computing Simulation (HPCS2011), Istanbul Turkey, pp. 369-375, July 4-8.

  21. Ejbeli, R., Jemai, O., Zaied, M., & Ben Amar, C. (2015). A speech recognition system using fast learning algorithm and beta wavelet network , Proc. of The 15th International Conference on Intelligent Systems Design and Applications (ISDA 2015), pp. 14-18, Marrakech, Morocco, December 14-16.

  22. Jemai, O., Zaied, M., Ben Amar, C., & Alimi, A. M. (2011). Pyramidal Hybrid Approach : Wavelet Network with OLS Algorithm Based-Image Classification. IJWMIP : International Journal of Wavelets, Multiresolution and Information Processing, 9(1), 111–130.

    MathSciNet  MATH  Google Scholar 

  23. Jemai, O., Zaied, M., Ben Amar, C., & Alimi, A. M. (2010). FBWN: an architecture of Fast Beta Wavelet Networks for Image Classification, in 2010 IEEE World Congress on Computational Intelligence (IEEE WCCI 2010), the 2010 International Joint Conference on Neural Networks (IJCNN 2010), pp. 1953-1960, CCIB, Barcelona, Espagne, Juillet, 18-23.

  24. Bouchrika, T., Jemai, O., Zaied, M., & Ben Amar, C. (2014). Cascaded Hybrid Wavelet Network for Hand Gestures Recognition, in IEEE International Conference on Systems, Man and Cybernetics (IEE SMC 2014),San Diego, CA, USA, pp. 1379-1384, October, 05-08.

  25. Jarray, R., Snoun, A., Bouchrika, T., & Jemai, O. (2021). Deep Human Action Recognition System for Assistance of Alzheimer’s Patients. In: Abraham A., Hanne T., Castillo O., Gandhi N., Nogueira Rios T., Hong TP. (eds) Hybrid Intelligent Systems. HIS 2020. Advances in Intelligent Systems and Computing, vol 1375. Springer, Cham. https://doi.org/10.1007/978-3-030-73050-5_49

  26. Jlidi, N., Snoun, A., Bouchrika, T., Jemai, O., & Zaied, M. (2020). PTLHAR: PoseNet and transfer learning for human activities recognition based on body articulations. Proc. SPIE 11433, Twelfth International Conference on Machine Vision (ICMV), 114330Q. https://doi.org/10.1117/12.2559567

  27. Hassairi, S., Ejbali, R., & Zaied, M. (2015). Supervised image classification using deep convolutional wavelets network, in IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI), pp. 265 - 271.

  28. Hassairi, S., Ejbali, R., & Zaied, M. (2015). A deep convolutional neural wavelet network to supervised Arabic letter image classification, 2015 15th International Conference on Intelligent Systems Design and Applications (ISDA), pp. 207 - 212.

  29. S. Said, O. Jemai, S. Hassairi, R. Ejbali, M. Zaied, and C. BenAmar (2015). Deep wavelet network for image classification, IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 000922 - 000927.

  30. Hassairi, S., Ejbali, R., & Zaied, M. (2018). A deep stacked wavelet auto-encoders to supervised feature extraction to pattern classification. Journal of Multimedia Tools and Applications, 77(5), 5443–5459.

    Article  Google Scholar 

  31. Hassairi, S., Ejbali, R., & Zaied, M. (2016). Sparse wavelet auto-encoders for image classification, International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1 - 6.

  32. Bouallégue, A., Hassairi, S., Ejbali, R., & Zaied, M. (2016). Learning Deep Wavelet Networks for Recognition System of Arabic Words, International Joint Conference SOCO’16-CISIS’16-ICEUTE’16, pp. 498 - 507.

  33. Snoun, A., Jlidi, N., Bouchrika, T., & Jemai, O. (2021). Towards a deep human activity recognition approach based on video to image transformation with skeleton data. Multimedia Tools and Applications, 80, 29675–29698. https://doi.org/10.1007/s11042-021-11188-1

    Article  Google Scholar 

  34. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks, Advances in Neural Information Processing Systems (NIPS), pp. 1097 - 1105.

  35. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition, In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778.

  36. Sarbjit, S., & Nikolaos, P. (1999). Monitoring driver fatigue using facial analysis techniques/ Proceedings 199 IEEE/IEEJ/JSAI International Conference on Intelligent Transportation Systems (ITSC), pp. 314 - 318, Tokyo, Japan, October 5 - 8, 1999.

  37. Horng, W., Chen, C., & Chang, Y. (2004). Driver fatigue detection based on eye tracking and dynamic template matching. Proceeding of the EEE International Conference on Networking, Sensing and Control (ICNSC), pp. 7 - 12, Taipei, Taiwan.

  38. Sharabaty, H., Jammes, B., & Esteve, D. (2008). Eeg analysis using hht: One step toward automatic drowsiness scoring. The 22 nd International Conference on Advanced Information Networking and Applications - Workshops (AINA workshops), pp. 826 - 831, Okinawa, Japan.

  39. Bhler, H. (1993). Reglage par logique floue. Lausanne: Presses polytechniques et universitaires romandes.

    Google Scholar 

  40. [Retrieved: 17-08-2015]. https://www.bioid.com/About/BioID-Face-Database

  41. [Retrieved: 10-04-2017]. http://parnec.nuaa.edu.cn/xtan/data/ClosedEyeData- bases.html

  42. Song, F., Tan, X., Liu, X., & Chen, S. (2015). Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients. Pattern Recognition, vol. 47 (9), pp. 2825 - 2838, 2015.

  43. Akrout, B. (2016). Détection d’Hypovigilance chez le Conducteur par Analyse de Caractéristiques Visuelles, Faculty of Economics Sciences and Management, Sfax, Thesis report.

  44. Craye, C., Rashwan, A., Kamel, M. S., & Karray, F. (2015). A multi-modal driver fatigue and distraction assessment system. International Journal of Intelligent Transportation Systems Research (IJITSR), 14(3), 1–22.

    Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge the financial support of this work by grants from General Direction of Scientific Research (DGRST), Tunisia, under the ARUB program.

Author information

Authors and Affiliations

  1. Research Team in Intelligent Machines (RTIM), National Engineering School of Gabes (ENIG) - University of Gabes, Gabes, Tunisia

    Ahmed Snoun, Tahani Bouchrika, Ines Teyeb & Olfa Jemai

Authors
  1. Ahmed Snoun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tahani Bouchrika
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ines Teyeb
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Olfa Jemai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ahmed Snoun.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Snoun, A., Bouchrika, T., Teyeb, I. et al. A Multimodal Hypovigilance Detection System Based on Fuzzy Logic and Transfer Learning. J Sign Process Syst 94, 1411–1427 (2022). https://doi.org/10.1007/s11265-022-01813-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11265-022-01813-z

Keywords

Search

Navigation