Skip to main content
SpringerLink
Log in

Detection of Pilot’s Drowsiness Based on Multimodal Convolutional Bidirectional LSTM Network

  • Conference paper
  • First Online:
Pattern Recognition (ACPR 2019)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12047))

Included in the following conference series:

  • 1344 Accesses

Abstract

The drowsiness of pilot causes the various aviation accidents such as an aircraft crash, breaking away airline, and passenger safety. Therefore, detecting the pilot’s drowsiness is one of the critical issues to prevent huge aircraft accidents and to predict pilot’s mental states. Conventional studies have been investigated using physiological signals such as brain signals, electrodermal activity (EDA), electrocardiogram (ECG), respiration (RESP) for detecting pilot’s drowsiness. However, these studies have not sufficient performance to prevent sudden aviation accidents yet because it could detect the mental states after drowsiness occurred and only focus on whether drowsiness or not. To overcome the limitations, in this paper, we propose a multimodal convolutional bidirectional LSTM network (MCBLN) to detect drowsiness or not as well as drowsiness level using the fused physiological signals (electroencephalography (EEG), EDA, ECG, and RESP) for the pilot’s environment. We acquired the physiological signals for the pilot’s simulated aircraft environment across seven participants. The proposed MCBLN extracted the features considering the spatial-temporal correlation of between EEG signals and peripheral physiological measures (PPMs) (EDA, ECG, RESP) to detect the current pilot’s drowsiness level. Our proposed method achieved the grand-averaged 45.16% (±1.01) classification accuracy for 9-level of drowsiness. Also, we obtained 84.41% (±1.34) classification accuracy for whether the drowsiness or not across all participants. Hence, we have demonstrated the possibility of the not only drowsiness detection but also 9-level of drowsiness for the pilot’s aircraft environment.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Han, S.-Y., Kim, J.-W., Lee, S.-W.: Recognition of pilot’s cognitive states based on combination of physiological signals. In: Proceedings of the 7th International Winter Conference on Brain-Computer Interface (BCI), Korea, pp. 1–5 (2019)

    Google Scholar 

  2. Lee, M., et al.: Connectivity differences between consciousness and unconsciousness in non-rapid eye movement sleep: a TMS–EEG study. Sci. Rep. 9, 1–9 (2019)

    Article  Google Scholar 

  3. Balandong, R.P., Ahmad, R.F., Saad, M.N.M., Malik, A.S.: A review on EEG-based automatic sleepiness detection systems for driver. IEEE Access 6, 22908–22919 (2018)

    Article  Google Scholar 

  4. Won, D.O., Hwang, H.J., Dähne, S., Müller, K.R., Lee, S.-W.: Effect of higher frequency on the classification of steady-state visual evoked potentials. J. Neural Eng. 13(1), 1–11 (2016)

    Article  Google Scholar 

  5. Kim, I.-H., Kim, J.-W., Haufe, S., Lee, S.-W.: Detection of braking intention in diverse situations during simulated driving based on EEG feature combination. J. Neural Eng. 12(1), 1–12 (2015)

    Article  Google Scholar 

  6. Roh, M.C., Shin, H.K., Lee, S.-W.: View-independent human action recognition with volume motion template on single stereo camera. Pattern Recognit. Let. 31(7), 639–647 (2010)

    Article  Google Scholar 

  7. Jeong, J.-H., Shim, K.-H., Cho, J.-H., Lee, S.-W.: Trajectory decoding of arm reaching movement imageries for brain-controlled robot arm system. In: Conference Proceedings Engineering in Medicine and Biology Society (EMBC), Germany, pp. 1–4 (2019)

    Google Scholar 

  8. Choi, I.-H., Kim, Y.-G.: Head pose and gaze direction tracking for detecting a drowsy driver. Appl. Math. Inf. Sci. 9, 505–512 (2015)

    Google Scholar 

  9. Kwak, N.-S., Muller, K.-R., Lee, S.-W.: A lower limb exoskeleton control system based on steady state visual evoked potentials. J. Neural Eng. 12(5), 1–14 (2015)

    Article  Google Scholar 

  10. Lee, M.-H., Williamson, J., Won, D.-O., Fazli, S., Lee, S.-W.: A high performance spelling system based on EEG-EOG signals with visual feedback. IEEE Trans. Neural Syst. Rehabil. Eng. 26(7), 1443–1459 (2018)

    Article  Google Scholar 

  11. Kim, J.-H., Bießmann, F., Lee, S.-W.: Decoding three-dimensional trajectory of executed and imagined arm movements from electroencephalogram signals. IEEE Trans. Neural Syst. Rehabil. Eng. 23(5), 867–876 (2014)

    Article  Google Scholar 

  12. Wei, C.-S., Wang, Y.-T., Lin, C.-T., Jung, T.-P.: Toward drowsiness detection using non-hair-bearing EEG-based BCI. IEEE Trans. Neural Syst. Rehabil. Eng. 26(2), 400–406 (2018)

    Article  Google Scholar 

  13. Zhang, X., et al.: Design of a fatigue detection system for high speed trains based on driver vigilance using a wireless wearable EEG. Sensors 17, 1–21 (2017)

    Article  Google Scholar 

  14. Kim, K.-T., Suk, H.-I., Lee, S.-W.: Commanding a brain-controlled wheelchair using steady-state somatosensory evoked potentials. IEEE Trans. Neural Syst. Rehabil. Eng. 26(3), 654–665 (2016)

    Article  Google Scholar 

  15. Rumagit, A.M., Akbar, I.A., Igasaki, T.: Gazing time analysis for drowsiness assessment using eye gaze tracker. Telkomnika 15(2), 919–925 (2017)

    Article  Google Scholar 

  16. Lee, M.-H., et al.: EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy. Gigascience 8(5), 1–16 (2019)

    Article  Google Scholar 

  17. Yen, J.-R., Hsu, C.-C., Yang, H., Ho, H.: An investigation of fatigue issues on different flight operations. J. Air Transp. Manag. 15, 236–240 (2009)

    Article  Google Scholar 

  18. Lee, M.-H., Fazli, S., Mehnert, J., Lee, S.-W.: Subject-dependent classification for robust idle state detection using multi-modal neuroimaging and data-fusion techniques in BCI. Pattern Recognit. 48(8), 2725–2737 (2015)

    Article  Google Scholar 

  19. Lee, M.-H., Fazli, S., Mehnert, J., Lee, S.-W.: Hybrid brain-computer interface based on EEG and NIRS modalities. In: Proceedings of the 2nd International Winter Conference on Brain-Computer Interface (BCI), Korea, pp. 1–15 (2014)

    Google Scholar 

  20. Fazli, S., Lee, S.-W.: Brain computer interfacing: a multi-modal perspective. J. Comput. Sci. Eng. 7(2), 132–138 (2013)

    Article  Google Scholar 

  21. Yeom, S.-K., et al.: Spatio-temporal dynamics of multimodal EEG-fNIRS signals in the loss and recovery of consciousness under sedation using midazolam and propofol. PLoS ONE 12(11), 1–22 (2017)

    Article  Google Scholar 

  22. Nguyen, T., Ahn, S., Jang, H., Jun, S.C., Kim, J.G.: Utilization of a combined EEG/NIRS system to predict driver drowsiness. Sci. Rep. 7, 1–10 (2017)

    Article  Google Scholar 

  23. Ahn, S., Nguyen, T., Jang, H., Kim, J.G., Jun, S.C.: Exploring neuro-physiological correlates of drivers’ mental fatigue using simultaneous EEG, ECG, and fNIRS. Front. Hum. Neurosci. 10, 1–14 (2016)

    Google Scholar 

  24. Park, U., Choi, H.C., Jain, A.K., Lee, S.-W.: Face tracking and recognition at a distance: a coaxial and concentric PTZ camera system. IEEE Trans. Inf. Forensics Secur. 8(10), 1665–1677 (2013)

    Article  Google Scholar 

  25. Dimitrakopoulos, G.N., et al.: Functional connectivity analysis of fatigue reveals different network topological alterations. IEEE Trans. Neural Syst. Rehabil. Eng. 26(4), 1–14 (2018)

    Article  Google Scholar 

  26. Hong, S., Kwon, H., Choi, S.H., Park, K.S.: Intelligent system for drowsiness recognition based on ear canal EEG with PPG and ECG. Inf. Sci. 453, 302–322 (2018)

    Article  Google Scholar 

  27. Mårtensson, H., Keelan, O., Ahlström, C.: Driver sleepiness classification based on physiological data and driving performance from real road driving. IEEE Trans. Intell. Transp. Syst. 20, 421–430 (2018)

    Article  Google Scholar 

  28. Chen, J., Wang, H., Hua, C.: Assessment of driver drowsiness using electroencephalogram signals based on multiple functional brain networks. Int. J. Psychophysiol. 133, 120–130 (2018)

    Article  Google Scholar 

  29. De Naurois, C.J., Bourdin, C., Stratulat, A., Diaz, E., Vercher, J.-L.: Detection and prediction of driver drowsiness using artificial neural network models. Accid. Anal. Prev. 126, 95–104 (2019)

    Article  Google Scholar 

  30. Wu, E.Q., Peng, X.Y., Zhang, C.Z., Lin, J.X., Sheng, R.S.F.: Pilot’s fatigue status recognition using deep contractive autoencoder network. IEEE Trans. Instrum. Meas. 68, 3907–3919 (2019)

    Article  Google Scholar 

  31. Hajinoroozi, M., Mao, Z., Jung, T.-P., Lin, C.-T., Huang, Y.: EEG based prediction of driver’s cognitive performance by deep convolutional neural network. Signal Process. Image Commun. 47, 549–555 (2016)

    Article  Google Scholar 

  32. Lawhern, V.J., Solon, A.J., Waytowich, N.R., Goedon, S.M., Hung, C.P., Lance, B.J.: EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. arXiv, 1–30 (2018)

    Google Scholar 

  33. Sakhavi, S., Guan, C., Yan, S.: Learning temporal information for brain-computer interface using convolutional neural networks. IEEE Trans. Neural Netw. Learn. Syst. 29(11), 5619–5629 (2018)

    Article  MathSciNet  Google Scholar 

  34. Hefron, R.G., Borghetti, B.J., Christensen, J.C., Kabban, C.M.S.: Deep long short-term memory structures model temporal dependencies improving cognitive workload estimation. Pattern Recognit. Lett. 94, 96–104 (2017)

    Article  Google Scholar 

  35. Lee, M., et al.: Network properties in transitions of consciousness during propofol-induced sedation. Sci. Rep. 1(7), 16791 (2017)

    Article  Google Scholar 

  36. Bulthoff, H.H., Lee, S.-W., Poggio, T.A., Wallraven, C.: Biologically motivated computer vision. Lecture Notes in Computer Science, vol. 2525. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-36181-2

    Book  MATH  Google Scholar 

  37. Jeong, J.-H., Lee, M.-H., Kwak, N.-S., Lee, S.-W.: Single-trial analysis of readiness potentials for lower limb exoskeleton control. In: Proceedings of the 7th International Winter Conference on Brain-Computer Interface (BCI), Korea, pp. 50–52 (2017)

    Google Scholar 

  38. Lee, S.-H., Lee, M., Jeong, J.-H., Lee, S.-W.: Towards an EEG-based intuitive BCI communication system using imagined speech and visual imagery. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, pp. 4409–4414. IEEE, Bari (2019)

    Google Scholar 

Download references

Acknowledgement

This work was supported by Defense Acquisition Program Administration (DAPA) and Agency for Defense Development (ADD) of Korea (06-201-305-001, A Study on Human-Computer Interaction Technology for the Pilot Status Recognition).

Author information

Authors and Affiliations

  1. Korea University, Seoul, 02841, Republic of Korea

    Baek-Woon Yu, Ji-Hoon Jeong, Dae-Hyeok Lee & Seong-Whan Lee

Authors
  1. Baek-Woon Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ji-Hoon Jeong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dae-Hyeok Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Seong-Whan Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seong-Whan Lee .

Editor information

Editors and Affiliations

  1. University of Malaya, Kuala Lumpur, Malaysia

    Shivakumara Palaiahnakote

  2. Consiglio Nazionale delle Ricerche, ICAR, Naples, Italy

    Gabriella Sanniti di Baja

  3. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  4. Auckland University of Technology, Auckland, New Zealand

    Wei Qi Yan

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yu, BW., Jeong, JH., Lee, DH., Lee, SW. (2020). Detection of Pilot’s Drowsiness Based on Multimodal Convolutional Bidirectional LSTM Network. In: Palaiahnakote, S., Sanniti di Baja, G., Wang, L., Yan, W. (eds) Pattern Recognition. ACPR 2019. Lecture Notes in Computer Science(), vol 12047. Springer, Cham. https://doi.org/10.1007/978-3-030-41299-9_41

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-41299-9_41

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-41298-2

  • Online ISBN: 978-3-030-41299-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation