Skip to main content
SpringerLink
Log in

A Multi-Modal Driver Fatigue and Distraction Assessment System

  • Published:
International Journal of Intelligent Transportation Systems Research Aims and scope Submit manuscript

Abstract

In this paper, we present a multi-modal approach for driver fatigue and distraction detection. Based on a driving simulator platform equipped with several sensors, we have designed a framework to acquire sensor data, process and extract features related to fatigue and distraction. Ultimately the features from the different sources are fused to infer the driver’s state of inattention. In our work, we extract audio, color video, depth map, heart rate, and steering wheel and pedals positions. We then process the signals according to three modules, namely the vision module, audio module, and other signals module. The modules are independent from each other and can be enabled or disabled at any time. Each module extracts relevant features and, based on hidden Markov models, produces its own estimation of driver fatigue and distraction. Lastly, fusion is done using the output of each module, contextual information, and a Bayesian network. A dedicated Bayesian network was designed for both fatigue and distraction. The complementary information extracted from all the mod- ules allows a reliable estimation of driver inattention. Our experimental results show that we are able to detect fatigue with 98.4 % accuracy and distraction with 90.5 %.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

References

  1. Stanton, N.A., Salmon, P.M.: Human error taxonomies applied to driving: A generic driver error taxonomy and its implications for intelligent transport systems. Saf. Sci. 47(2), 227–237 (2009)

    Article  Google Scholar 

  2. NHTSA: Distraction.gov, official us government website for distracted driving. http://www.distraction.gov (2013)

  3. Lee, J.D., Young, K.L., Regan, M.A.: Defining driver distraction: Driver distraction: Theory, effects, and mitigation (2008)

  4. Lal, S.K., Craig, A.: A critical review of the psychophysiology of driver fatigue. Biol. Psychol. 55(3), 173–194 (2001)

    Article  Google Scholar 

  5. Strohl, K., Merritt, S., Blatt, J., Pack, A., Council, F., Rogus, S.: Drowsy driving and automobile crashes. nccdr/nhtsa expert panel on driver fatigue and sleepiness. http://www.nhtsa.gov/people/injury/drowsy_driving1/Drowsy.html (2004)

  6. Klauer, S.G., Dingus, T.A., Neale, V.L., Sudweeks, J., Ramsey, D.: The impact of driver inattention on near-crash/crash risk: An analysis using the 100-car naturalistic driving study data, Washington DC: National Highway Traffic Safety Administration. Tech. Rep. HS-810, 594 (2006)

    Google Scholar 

  7. Brock, J., Robinson, A., Robinson, B., Percer, J.: Traffic safety facts, research note,” Washington DC: National Highway Traffic Safety Administration. Tech. Rep. HS-811, 737 (2013)

    Google Scholar 

  8. Olson, R.L., Hanowski, R.J., Hickman, J.S., Bocanegra, J.L.: Driver distraction in commercial vehicle operations, federal motor carrier safety administration, USDOT, Tech. Rep. FMCSA-RRT-09-042 (2009)

  9. Williamson, A., Chamberlain, T.: Review of on-road driver fatigue monitoring devices. NSW Injury Risk Management Research Center, University of New South Wales,Tech. Rep. (2005)

  10. Young, K., Regan, M., Hammer, M.: Driver distraction: A review of the literature, pp. 379–405 (2007)

  11. Dong, Y., Hu, Z., Uchimura, K., Murayama, N.: Driver inattention monitoring system for intelligent vehicles: A review. Intell Trans Syst IEEE Trans on 12(2), 596–614 (2011)

    Article  Google Scholar 

  12. EyeAlert: Distracted driving and fatigue sentinels. http://www.eyealert.com/ (2012)

  13. Edenborough, N., Hammoud, R., Harbach, A., Ingold, A., Kisacanin, B., Malawey, P., Newman, T., Scharenbroch, G., Skiver, S., Smith, M., Wilhelm, A., Witt, G., Yoder, E., Zhang, H.: Driver state monitor from delphi, in Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Comput Soc Conf on 2, 1206–1207 (2005)

    Google Scholar 

  14. Lexus: Driver monitoring system. http://www.lexus.eu/car-models/ls/ls-600h/index.tmex (2006)

  15. InSight: Sensomotoric instruments gmbh. http://www.smivision.com/en/gaze-and-eye-tracking-systems/services/smi-eye-tracking-roadshow.html (2011)

  16. Car, M.V.: Driver alert control. http://www.media.volvocars.com (2007)

  17. Alert, F.D.: Ford’s wake-up call for europe’s sleepy drivers. http://media.ford.com (2011)

  18. Volkswagen: Driver alert, driver assistance, experience. http://www.volkswagen.co.uk/new/passat-vii/explore/experience/driver-assistance/driver-alert (2011)

  19. Healey, J.A., Picard, R.W.: Detecting stress during real-world driving tasks using physiological sensors. Intell. Trans. Syst. IEEE Trans. on 6(2), 156–166 (2005)

    Article  Google Scholar 

  20. Egelund, N.: Spectral analysis of heart rate variability as an indicator of driver fatigue. Ergonomics 25(7), 663–672 (1982)

    Article  Google Scholar 

  21. Patel, M., Lal, S., Kavanagh, D., Rossiter, P.: Applying neural network analysis on heart rate variability data to assess driver fatigue. Expert Syst. Appl. 38(6), 7235–7242 (2011)

    Article  Google Scholar 

  22. Miyaji, M., Danno, M., Kawanaka, H., Oguri, K.: Driver? cognitive distraction detection using adaboost on pattern recognition basis, in Vehicular Electronics and Safety, 2008. ICVES 2008, IEEE International Conference on, pp. 51–56. IEEE (2008)

  23. Shiwu, L., Linhong, W., Zhifa, Y., Bingkui, J., Feiyan, Q., Zhongkai, Y.: An active driver fatigue identification technique using multiple physiological features, in Mechatronic Science, Electric Engineering and Computer (MEC), 2011 International Conference on, pp. 733–737. IEEE (2011)

  24. Lal, S.K., Craig, A.: Driver fatigue: electroencephalography and psychological assessment. Psychophysiology 39(3), 313–321 (2002)

    Article  Google Scholar 

  25. Jones, C.M., Jonsson, I.-M.: Automatic recognition of affective cues in the speech of car drivers to allow appropriate responses. In: Proceedings of the 17th Australia conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future, ser. OZCHI ’05. Computer-Human Interaction Special Interest Group (CHISIG) of Australia, pp. 1–10 (2005). Available: http://dl.acm.org/citation.cfm?id=1108368.1108397

  26. Tawari, A., Trivedi, M.: Speech based emotion classification framework for driver assistance system, in Intelligent Vehicles Symposium (IV), 2010 IEEE, pp. 174– 178. IEEE (2010)

  27. Lugger, M., Yang, B. In: Mihelic, F., Zibert, J. (eds.): Psychological motivated multi-stage emotion classification exploiting voice quality features, in Speech Recognition, Technologies and Applications (2008). intech

  28. Greeley, H.P., Friets, E., Wilson, J.P., Raghavan, S., Picone, J. , Berg, J.: Detecting fatigue from voice using speech recognition, in Signal Processing and Information Technology, 2006 IEEE International Symposium on, pp. 567–571 (2006)

  29. Krajewski, J., Trutschel, U., Golz, M., Sommer, D., Edwards, D.: Estimating fatigue from predetermined speech samples transmitted by operator communication systems. In: Proceedings of the International Driving Symposium on Human Factors in Driver Assessment. Train. Veh. Des. 5, 468– 474 (2009)

    Google Scholar 

  30. Dhupati, L., Kar, S., Rajaguru, A., Routray, A.: A novel drowsiness detection scheme based on speech analysis with validation using simultaneous eeg recordings, in, Automation Science and Engineering (CASE), 2010 IEEE Conference on (2010)

  31. Bergasa, L.M., Buenaposada, J.M., Nuevo, J., Jimenez, P., Baumela, L.: Analysing driver’s attention level using computer vision, in Intelligent Transportation Systems, 2008. ITSC 2008. 11th International IEEE Conference on, 1149–1154 (2008)

  32. Ji, Q., Zhu, Z., Lan, P.: Real-time nonintrusive monitoring and prediction of driver fatigue. IEEE Trans Veh Technol 53(4), 1052–1068 (2004)

    Article  Google Scholar 

  33. Ji, Q., Lan, P., Looney, C.: A probabilistic framework for modeling and real-time monitoring human fatigue. Syst Man and Cybern Part A: Syst Humans, IEEE Tran on 36(5), 862–875 (2006)

    Article  Google Scholar 

  34. Bergasa, L., Nuevo, J., Sotelo, M., Barea, R., Lopez, M.: Real-time system for monitoring driver vigilance, intelligent transportation systems, vol. 7, pp. 63 –77 (2006)

  35. Li, L., Werber, K., Calvillo, C.F., Dinh, K.D., Guarde, A., König, A.: Multi-sensor soft-computing system for driver drowsiness detection, Online conference on soft computing in industrial applications, pp. 1–10 (2012)

  36. Angell, L., Auflick, J., Austria, P., Kochhar, D., Tijerina, L., Biever, W., Diptiman, T., J. Hogsett Jr, Kiger, S.: Driver workload metrics project, national highway traffic safety administration. Tech. Rep. HS 810, 635 (2006)

    Google Scholar 

  37. Dinges, D.F., Grace, R.: Perclos: A valid psychophysiological measure of alertness as assessed by psychomotor vigilance, Federal Highway Administration. Office of motor carriers, Tech. Rep. MCRT-98-006 (1998)

  38. Senaratne, R., Hardy, D., Vanderaa, B., Halgamuge, S., Fei, S., Hou, Z., Zhang, H., Sun, C. In: Liu, D. (ed.): Driver fatigue detection by fusing multiple cues, in Advances in Neural Networks - ISNN 2007, ser. Lecture Notes in Computer Science, vol. 4492, pp. 801–809. Springer Berlin Heidelberg (2007)

  39. Smith, P., Shah, M., da Vitoria Lobo, N.: Determining driver visual attention with one camera. Int. Trans. Syst. IEEE Trans. on 4(4), 205–218 (2003)

    Article  Google Scholar 

  40. Damousis, I.G., Tzovaras, D.: Fuzzy fusion of eyelid activity indicators for hypovigilance-related accident prediction. Intell. Trans. Sys. IEEE Trans. on 9(3), 491–500 (2008)

    Article  Google Scholar 

  41. Daza, I., Hernandez, N., Bergasa, L., Parra, I., Yebes, J., Gavilan, M., Quintero, R., Llorca, D., Sotelo, M.: Drowsiness monitoring based on driver and driving data fusion, in Intelligent Transportation Systems (ITSC), 2011 14th International IEEE Conference on, pp. 1199–1204. IEEE (2011)

  42. Pilutti, T., Ulsoy, A. G.: Identification of driver state for lane-keeping tasks: experimental results, in American Control Conference, 1997. Proceedings of the 1997, vol. 5, pp. 3370–3374. IEEE (1997)

  43. Singh, H., Bhatia, J., Kaur, J.: Eye tracking based driver fatigue monitoring and warning system. In: Power Electronics (IICPE), 2010 India International Conference on, pp. 1–6. IEEE (2011)

  44. Yeo, M. V., Li, X., Shen, K., Wilder-Smith, E. P.: Can svm be used for automatic eeg detection of drowsiness during car driving? Saf. Sci. 47(1), 115–124 (2009)

    Article  Google Scholar 

  45. Liu, J., Zhang, C., Zheng, C.: Eeg-based estimation of mental fatigue by using kpca–hmm and complexity parameters. Biomed. Sig. Process. Control 5(2), 124–130 (2010)

    Article  MathSciNet  Google Scholar 

  46. Chua, C.-P., McDarby, G., Heneghan, C.: Combined electrocardiogram and photoplethysmogram measurements as an indicator of objective sleepiness. Physiol. Meas. 29(8), 857 (2008)

    Article  Google Scholar 

  47. Fletcher, L., Loy, G., Barnes, N., Zelinsky, A.: Correlating driver gaze with the road scene for driver assistance systems. Robot. Auton. Syst. 52(1), 71–84 (2005)

    Article  Google Scholar 

  48. Ji, Q., Lan, P., Looney, C.: A probabilistic framework for modeling and real-time monitoring human fatigue. Syst. Man Cybern. Part A: Syst. Hum. IEEE Trans. on 36(5), 862–875 (2006)

    Article  Google Scholar 

  49. Craye, C.: A framework for context-aware driver status assessment systems,Master’s thesis, Electrical and Computer Engineering department, University of Waterloo (2013)

  50. Kinect, M.: Kinect face tracking, http://msdn.microsoft.com/en-us/library/jj130970.aspx (2013)

  51. Tian, Q.-C., Pan, Q., Cheng, Y.-M., Gao, Q.-X.: Fast algorithm and application of hough transform in iris segmentation, in Machine Learning and Cybernetics, 2004. Procee. 2004 Int. Conf. on 7, 3977–3980 (2004)

    Google Scholar 

  52. Kawaguchi, T., Rizon, M.: Iris detection using intensity and edge information. Pattern Recog. 36(2), 549–562 (2003)

    Article  Google Scholar 

  53. Zhang, Y., Sun, N., Gao, Y., Cao, M.: A new eye location method based on ring gabor filter,” in Automation and Logistics, 2008. ICAL 2008. IEEE International Conference on, pp. 301–305. IEEE (2008)

  54. Edwards, G. J., Taylor, C. J., Cootes, T. F.: Interpreting face images using active appearance models,” in Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on, pp. 300–305. IEEE (1998)

  55. Ahlberg, J.: Ahlberg. Candide-3-an updated parameterised face (2001)

  56. Ramirez, J., Gorriz, J. M., Segura, J. C.: Voice activity detection fundamentals and speech recognition system robustness (2007)

  57. Rashwan, A.: Automatic driver fatigue monitoring using hidden markov models and bayesian networks Master’s thesis, University of Waterloo (2013)

  58. Young, S. J., Evermann, G., Gales, M. J. F., Hain, T., Kershaw, D., Moore, G., Odell, J., Ollason, D., Povey, D., Valtchev, V., Woodland, P. C.: The HTK Book, version 3.4. Cambridge UK:Cambridge University Engineering Department (2006)

  59. Zhang, N., Poole, D.: A simple approach to Bayesian network computations. In: Proceedings of the Tenth Canadian Conference on Artificial Intelligence, pp. 171–178 (1994)

  60. car driving, C.: City car driving - car driving simulator, car game, http://citycardriving.com/ (2013)

  61. Chang, C.-C., Lin, C.-J.: LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol. 2, 27:1– 27:27 (2011). software available at, http://www.csie.ntu.edu.tw/cjlin/libsvm

    Article  Google Scholar 

  62. Linde, Y., Buzo, A., Gray, R.: An algorithm for vector quantizer design. Commun. IEEE Trans. on 28 (1), 84–95 (1980)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Electrical and Computer Engineering, University of Waterloo, Ontario, Canada

    Céline Craye, Abdullah Rashwan, Mohamed S. Kamel & Fakhri Karray

Authors
  1. Céline Craye
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abdullah Rashwan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohamed S. Kamel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fakhri Karray
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Céline Craye.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Craye, C., Rashwan, A., Kamel, M.S. et al. A Multi-Modal Driver Fatigue and Distraction Assessment System. Int. J. ITS Res. 14, 173–194 (2016). https://doi.org/10.1007/s13177-015-0112-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13177-015-0112-9

Keywords

Search

Navigation