Skip to main content

Advertisement

Springer Nature Link
Log in

Cell phone usage detection in roadway images: from plate recognition to violation classification

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

    We’re sorry, something doesn't seem to be working properly.

    Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

Distracted driving plays a significant role in road accidents worldwide. Detecting distracted driving in real time is a significant challenge faced by law enforcement officers. Although efforts are continuing to identify this type of infraction in an automated manner, most of the techniques proposed for this problem use images obtained internally from vehicles. Few studies have looked into using roadway images collected in naturalistic situations. We propose in this research a complete and fully automated deep-learning approach that locates vehicles in roadway images, detects and extracts license plate numbers, detects the windshield region, and classifies images into predefined violations. The proposed approach is complete in that it does not rely on roadway license plate systems to localize the vehicle of interest and extract the license plate numbers. The model performance-both overall and in each stage-was evaluated, achieving \(90\%\) overall classification accuracy. The model is trained and tested on a real-world local dataset of 10,000 images. The dataset used in this study is the first dataset acquired from roadway license plate cameras in the Middle Eastern region showing unique variations of plate forms, driving habits, regional attire, and weather conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Data Availability

The dataset used in this study is available from the corresponding authors upon request.

References

  1. Adam B, Zaman F, Yassin I, Abidin HZ, Rizman Z (2017) Performance evaluation of faster R-CNN on GPU for object detection. J Fundam Appl Sci 9(3S):909–923

    Article  Google Scholar 

  2. Alkan B, Balci B, Elihos A, Artan Y (2019) Driver cell phone usage violation detection using license plate recognition camera images. In: Proceedings of the 5th international conference on vehicle technology and intelligent transport systems - VEHITS, pp 468–474. INSTICC, SciTePress. https://doi.org/10.5220/0007725804680474

  3. Alotaibi M, Alotaibi B (2020) Distracted driver classification using deep learning. SIViP 14(3):617–624

    Article  Google Scholar 

  4. Artan Y, Bulan O, Loce RP, Paul P (2014) Driver cell phone usage detection from hov/hot nir images. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, IEEE pp 225–230

  5. Arvin R, Khattak AJ (2020) Driving impairments and duration of distractions: assessing crash risk by harnessing microscopic naturalistic driving data. Accid Anal Prev 146:105733

    Article  Google Scholar 

  6. Balci B, Artan Y, Alkan B, Elihos A (2019) Front-view vehicle damage detection using roadway surveillance camera images. VEHITS 2019 - 5th international conference on vehicle technology and intelligent transport systems, pp 193–198

  7. Chen Y, Fu R, Xu Q, Yuan W (2020) Mobile phone use in a car-following situation: Impact on time headway and effectiveness of driver’s rear-end risk compensation behavior via a driving simulator study. Int J Environ Res Public Health 17(4):1328

    Article  Google Scholar 

  8. Chollet F (2017) Deep Learning with Python. Manning

  9. Database Publishing (2016) VIVA VIVA: Available at https://www.viva.de/en/company/about-viva, Accessed: 2020-03-08

  10. Dutta A, Zisserman A (2019) The via annotation software for images, audio and video. In: Proceedings of the 27th ACM international conference on multimedia, pp 2276–2279

  11. Eraqi HM, Abouelnaga Y, Saad MH, Moustafa MN (2019) Driver distraction identification with an ensemble of convolutional neural networks. J Adv Transp 2019:1–12. https://doi.org/10.1155/2019/4125865

    Article  Google Scholar 

  12. Gumaei A, Al-Rakhami M, Hassan MM, Alamri A, Alhussein M, Razzaque MA, Fortino G (2020) A deep learning-based driver distraction identification framework over edge cloud. Neural Comput Appl, pp 1–16

  13. Hari C, Sankaran P (2021) Driver distraction analysis using face pose cues. Expert Syst Appl 179:115036

    Article  Google Scholar 

  14. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  15. Hoang Ngan Le T, Zheng Y, Zhu C, Luu K, Savvides M (2016) Multiple scale Faster-RCNN approach to driver’s cell-phone usage and hands on steering wheel detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 46–53

  16. Hssayeni MD, Saxena S, Ptucha R, Savakis A (2017) Distracted driver detection: deep learning vs. handcrafted features. Electron Imagin 2017(10):20–26

    Article  Google Scholar 

  17. Huang C, Wang X, Cao J, Wang S, Zhang Y (2020) HCF: a hybrid CNN framework for behavior detection of distracted drivers. IEEE Access 8:109335–109349

    Article  Google Scholar 

  18. Jain DK, Jain R, Lan X, Upadhyay Y, Thareja A (2021) Driver distraction detection using capsule network. Neural Comput Appl 33(11):6183–6196

    Article  Google Scholar 

  19. Jegham I, Khalifa AB, Alouani I, Mahjoub MA (2020) A novel public dataset for multimodal multiview and multispectral driver distraction analysis: 3mdad. Signal Proce: Image Commun 88:115960

    Google Scholar 

  20. Kashevnik A, Lashkov I, Gurtov A (2019) Methodology and mobile application for driver behavior analysis and accident prevention. IEEE Trans Intell Transp Syst 21(6):2427–2436

    Article  Google Scholar 

  21. Koay HV, Chuah JH, Chow CO, Chang YL, Rudrusamy B (2021) Optimally-weighted image-pose approach (owipa) for distracted driver detection and classification. Sensors 21(14):4837

    Article  Google Scholar 

  22. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inform Proce Syst, pp 1097–1105

  23. Li L, Zhong B, Hutmacher C Jr, Liang Y, Horrey WJ, Xu X (2020) Detection of driver manual distraction via image-based hand and ear recognition. Accid Anal Preven 137:105432

    Article  Google Scholar 

  24. Lin T, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: 2017 IEEE international conference on computer vision (ICCV), pp 2999–3007. https://doi.org/10.1109/ICCV.2017.324

  25. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) SSD: single shot multibox detector. Eur Conf Comput Vis, pp 21–37

  26. Lu M, Hu Y, Lu X (2020) Driver action recognition using deformable and dilated faster r-cnn with optimized region proposals. Appl Intell 50(4):1100–1111

    Article  Google Scholar 

  27. Masood S, Rai A, Aggarwal A, Doja MN, Ahmad M (2018) Detecting distraction of drivers using convolutional neural network. Pattern Recognit Lett 139:79–85

    Article  Google Scholar 

  28. Masood S, Rai A, Aggarwal A, Doja MN, Ahmad M (2020) Detecting distraction of drivers using convolutional neural network. Pattern Recogn Lett 139:79–85

    Article  Google Scholar 

  29. MOI (2018) The ministry of interior of Saudi Arabia. Available at https://www.spa.gov.sa/1849802, Accessed: 2019-09-10

  30. NHTSA (2016) National Highway Traffic Safety Administration (NHTSA): Traffic safety facts - distracted driving 2016. https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812517

  31. NHTSA (2017) National Highway Traffic Safety Administration (NHTSA): Distracted driving. https://www.nhtsa.gov/risky-driving/distracted-driving, Accessed: 2019-09-10

  32. NHTSA (2018) National Highway Traffic Safety Administration (NHTSA) U.S. dot and NHTSA kick off 5th annual U drive. U text. U pay. campaign. https://www.nhtsa.gov/press-releases/us-dot-and-nhtsa-kick-5th-annual-u-drive-u-text-u-pay-campaign

  33. NRSC (2019) National Road Safety Center (NRSC) Your response hurts you campaign. https://nrsc.gov.sa

  34. Omerustaoglu F, Sakar CO, Kar G (2020) Distracted driver detection by combining in-vehicle and image data using deep learning. Appl Soft Comput 96:106657

    Article  Google Scholar 

  35. Oppenheim AV, Schafer RW (2004) From frequency to quefrency: a history of the cepstrum. IEEE Signal Process Mag 21(5):95–106

    Article  Google Scholar 

  36. Papantoniou P, Yannis G, Christofa E (2019) Which factors lead to driving errors? A structural equation model analysis through a driving simulator experiment. IATSS Res 43(1):44–50

    Article  Google Scholar 

  37. Redmon J, Farhadi A (2017) YOLO9000: Better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7263–7271

  38. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. Adv Neural Inform Proc Syst 28:91–99

    Google Scholar 

  39. Rezatofighi H, Tsoi N, Gwak J, Sadeghian A, Reid I, Savarese S (2019) Generalized intersection over union: metric and a loss for bounding box regression. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 658–666. https://doi.org/10.1109/CVPR.2019.00075

  40. Shahverdy M, Fathy M, Berangi R, Sabokrou M (2020) Driver behavior detection and classification using deep convolutional neural networks. Expert Syst Appl 149:113240

    Article  Google Scholar 

  41. Sharan S, Reddy R, Reddy P (2021) Multi-level drowsiness detection using multi-contrast convolutional neural networks and single shot detector. In: 2021 international conference on intelligent technologies (CONIT), pp 1–6. IEEE

  42. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Bengio Y, LeCun Y (eds) 3rd International Conference on Learning Representations, ICLR 2015. San Diego, CA, USA

    Google Scholar 

  43. Sundfør HB, Sagberg F, Høye A (2019) Inattention and distraction in fatal road crashes-results from in-depth crash investigations in Norway. Accid Anal Preven 125:152–157

    Article  Google Scholar 

  44. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  45. The Kaggle Co (2017) State farm distracted driver detection. https://www.kaggle.com/c/state-farm-distracted-driver-detection, Accessed: 2020-03-08

  46. Tzutalin D (2015) Labelimg. Git code

  47. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001, https://doi.org/10.1109/CVPR.2001.990517

  48. (VTTI), V.T.T.: Insight data access website. Available at https://insight.shrp2nds.us, Accessed: 2020-03-11

  49. Wagner B, Taffner F, Karaca S, Karge L (2021) Vision based detection of driver cell phone usage and food consumption. IEEE Trans Intell Transp Syst

  50. World Health Organisation (WHO): Road traffic injuries. https://www.who.int/health-topics/road-safety,Accessed: 2019-09-09

  51. Yan C, Coenen F, Zhang B (2016) Driving posture recognition by convolutional neural networks. IET Comput Vision 10(2):103–114

    Article  Google Scholar 

  52. Yasar H (2017) Detection of driver’s mobile phone usage. In: 2017IEEE 9th international conference on humanoid, nanotechnology, information technology, communication and control, environment and management (HNICEM), pp 1–4. IEEE

  53. Zhang Y, Chen Y, Gao C (2021) Deep unsupervised multi-modal fusion network for detecting driver distraction. Neurocomputing 421:26–38

    Article  Google Scholar 

  54. Zhao Z, Xia S, Xu X, Zhang L, Yan H, Xu Y, Zhang Z (2020) Driver distraction detection method based on continuous head pose estimation. Comput Intell Neurosci, 2020

  55. Zhao Z, Zhang Z, Xu X, Xu Y, Yan H, Zhang L (2020) A lightweight object detection network for real-time detection of driver handheld call on embedded devices. Comput Intell Neurosci2020

Download references

Acknowledgements

This work was supported by the Deanship of Scientific Research (DSR), King Abdulaziz University, Jeddah, under grant No. DF-778-165-1441. The authors, therefore, gratefully acknowledge DSR technical and financial support.

Author information

Authors and Affiliations

  1. Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia

    Amal Balabid, Areej Altaban, Maram Albsisi & Areej Alhothali

Authors
  1. Amal Balabid
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Areej Altaban
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Maram Albsisi
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Areej Alhothali
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding authors

Correspondence to Amal Balabid or Areej Alhothali.

Ethics declarations

Conflict of interest

The authors declare that they have no known conflicts of interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Balabid, A., Altaban, A., Albsisi, M. et al. Cell phone usage detection in roadway images: from plate recognition to violation classification. Neural Comput & Applic 35, 4667–4682 (2023). https://doi.org/10.1007/s00521-022-07943-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07943-6

Keywords