Skip to main content

Advertisement

SpringerLink
Log in

Monitoring and surveillance of urban road traffic using low altitude drone images: a deep learning approach

  • 1205: Emerging Technologies for Information Hiding and Forensics in Multimedia Systems
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In the contemporary era, the global explosion of traffic has created many eye-catching concerns for policymakers. This not only enhances pollution but also leads to several road accident fatalities which may be greatly reduced by proper monitoring and surveillance. Further, with the advent of UAV technology and due to the incompatibility of traditional techniques, surveillance has become one of UAVs prominent application domains. However, it requires algorithmic analysis of aerial images which becomes extremely challenging due to multi-scale rotating objects with large aspect ratios, extremely imbalanced categories, cluttered background, and birds-eye view. Therefore, this article presents the novel aerial image traffic monitoring and surveillance algorithms based on the most advanced and popular DL object detection models (Faster-RCNN, SSD, YOLOv3, and YOLOv4) using the AU-AIR dataset. This dataset is exceedingly imbalanced and to resolve this issue, another 500 images have been grabbed by web-mining techniques. The novel contribution of this work is two-fold. First, this article scientifically distinguishes the inappropriateness of ground-view images for aerial object detection. Second, a regress comparison of these algorithms has been made to investigate their effectiveness. Extensive experimental analysis endorses the efficiency of YOLOv4 as it outperforms the other developed models by a minimum mAP margin of 88%. Also, more than 6 times high detection speed and greater adaptability with stronger detection robustness ensure its real-time practical implementation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Abbreviations

CNN:

Convolutional Neural Network

R-CNN:

Region-based Convolutional Neural Network

SPP:

Spatial Pyramid Pooling

DL:

Deep Learning

FPS:

Frames per second

SSD:

Single-shot detector

IoU:

Intersection over Union

SVM:

Support Vector Machine

mAP:

Mean Average Precision

UAV:

Unmanned aerial Vehicles

WHO:

World Health Organization

MAV:

Manned Aerial Vehicle

PANet:

Path Aggregation Network

YOLO:

You Only Look Once

References

  1. Ali N, Jhanjhi NZ, Nawaz S et al (2020) Smart traffic monitoring system using unmanned aerial vehicles ( UAVs ). Comput Commun 157:434–443. https://doi.org/10.1016/j.comcom.2020.04.049

    Article  Google Scholar 

  2. Al-Turjman F (2019) A novel approach for drones positioning in mission critical applications. Trans Emerg Telecommun Technol n/a:e3603. https://doi.org/10.1002/ett.3603

  3. Barmpounakis EN, Vlahogianni EI, Golias JC (2016) Unmanned aerial aircraft systems for transportation engineering: current practice and future challenges. Int J Transp Sci Technol 5:111–122. https://doi.org/10.1016/j.ijtst.2017.02.001

    Article  Google Scholar 

  4. Barnich O, Van Droogenbroeck M (2011) ViBe: a universal background subtraction algorithm for video sequences. IEEE Trans Image Process 20:1709–1724. https://doi.org/10.1109/TIP.2010.2101613

    Article  MathSciNet  MATH  Google Scholar 

  5. Benjdira B, Khursheed T, Koubaa A, et al (2018) Car detection using unmanned aerial vehicles: comparison between faster R-CNN and YOLOv3. 1–6

  6. Bochkovskiy A, Wang C-Y, Liao H-YM (2020) YOLOv4: optimal speed and accuracy of object detection

  7. Bonali FL, Tibaldi A, Marchese F, Fallati L, Russo E, Corselli C, Savini A (2019) UAV-based surveying in volcano-tectonics: an example from the Iceland rift. J Struct Geol 121:46–64. https://doi.org/10.1016/j.jsg.2019.02.004

    Article  Google Scholar 

  8. Bozcan I, Kayacan E (2020) AU-AIR: a multi-modal unmanned aerial vehicle dataset for low altitude traffic surveillance. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). pp 8504–8510

  9. Chang FR, Huang HL, Schwebel DC, Chan AHS, Hu GQ (2020) Global road traffic injury statistics: challenges, mechanisms and solutions. Chinese J Traumatol - English Ed 23:216–218

    Article  Google Scholar 

  10. Choi Y, Kim N, Hwang S, Park K, Yoon JS, An K, Kweon IS (2018) KAIST multi-spectral day/night data set for autonomous and assisted driving. IEEE Trans Intell Transp Syst 19:934–948. https://doi.org/10.1109/TITS.2018.2791533

    Article  Google Scholar 

  11. Chriki A, Touati H, Snoussi H, Kamoun F (2020) Deep learning and handcrafted features for one-class anomaly detection in UAV video. Multimed Tools Appl 80:1–22. https://doi.org/10.1007/s11042-020-09774-w

    Article  Google Scholar 

  12. Christiansen MP, Laursen MS (2017) Designing and testing a UAV mapping system for agricultural field surveying 1:1–19. https://doi.org/10.3390/s17122703

  13. Dai J, Li Y, He K, Sun J (2016) R-FCN: object detection via region-based fully convolutional networks. Advances in Neural Information Processing Systems. Neural information processing systems foundation, In, pp 379–387

    Google Scholar 

  14. Deng J, Dong W, Socher R, et al (2009) ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp 248–255

  15. Everingham M, Van Gool L, Williams CKI et al (2010) The Pascal visual object classes (VOC) challenge. Int J Comput Vis 88:303–338. https://doi.org/10.1007/s11263-009-0275-4

    Article  Google Scholar 

  16. Felzenszwalb PF, Girshick RB, McAllester D, Ramanan D (2010) Object detection with discriminatively trained part-based models. IEEE Trans Pattern Anal Mach Intell 32:1627–1645. https://doi.org/10.1109/TPAMI.2009.167

    Article  Google Scholar 

  17. Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics : the KITTI dataset. 32:1231–1237. https://doi.org/10.1177/0278364913491297

  18. Girshick R, Donahue J, Darrell T, Malik J (2016) Region-based convolutional networks for accurate object detection and segmentation. IEEE Trans Pattern Anal Mach Intell 38:142–158. https://doi.org/10.1109/TPAMI.2015.2437384

    Article  Google Scholar 

  19. Gomaa A, Abdelwahab MM, Abo-Zahhad M (2020) Efficient vehicle detection and tracking strategy in aerial videos by employing morphological operations and feature points motion analysis. Multimed Tools Appl 79:26023–26043. https://doi.org/10.1007/s11042-020-09242-5

    Article  Google Scholar 

  20. Gomez M, Vergara A, Montenegro F et al (2020) Detection of banana plants and their major diseases through aerial images and machine learning methods : a case study in DR Congo and Republic of Benin. ISPRS J Photogramm Remote Sens 169:110–124. https://doi.org/10.1016/j.isprsjprs.2020.08.025

    Article  Google Scholar 

  21. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp 2980–2988

  22. Hendry CR-C (2019) Automatic license plate recognition via sliding-window darknet-YOLO deep learning. Image Vis Comput 87:47–56. https://doi.org/10.1016/j.imavis.2019.04.007

    Article  Google Scholar 

  23. Hildmann H (2019) Review : using unmanned aerial vehicles ( UAVs ) as mobile sensing platforms ( MSPs ) for Disaster response. Civil Security and Public Safety. 13–19. https://doi.org/10.3390/drones3030059, 3

  24. Karthik R, Hariharan M, Anand S et al (2020) Attention embedded residual CNN for disease detection in tomato leaves. Appl Soft Comput 86:105933. https://doi.org/10.1016/j.asoc.2019.105933

    Article  Google Scholar 

  25. Khan NA, Jhanjhi NZ, Brohi SN, Nayyar A (2020) Chapter three - emerging use of UAV’s: secure communication protocol issues and challenges. In: Al-Turjman FBT-D in S-C (ed). Elsevier, pp 37–55

  26. Kumar S, Yadav D, Gupta H, Verma OP, Ansari IA, Ahn CW (2021) A novel yolov3 algorithm-based deep learning approach for waste segregation: towards smart waste management. Electron 10:1–20. https://doi.org/10.3390/electronics10010014

    Article  Google Scholar 

  27. Li Y, Dong H, Li H, Zhang X (2020) Multi-block SSD based on small object detection for UAV railway scene surveillance. Chinese J Aeronaut 33:1747–1755. https://doi.org/10.1016/j.cja.2020.02.024

    Article  Google Scholar 

  28. Lin T-Y, Maire M, Belongie S, et al (2014) Microsoft COCO: common objects in context BT - computer vision – ECCV 2014. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds). Springer International Publishing, Cham, pp. 740–755

  29. Liu W, Anguelov D, Erhan D, et al (2016) SSD: single shot multibox detector. In: lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). Springer Verlag, pp 21–37

  30. Mittal P, Sharma A, Singh R (2020) Deep learning-based object detection in low-altitude UAV datasets: A survey Image Vis Comput 104046. https://doi.org/10.1016/j.imavis.2020.104046,, 104

  31. Nie X, Yang M, Liu RW (2019) Deep neural network-based robust ship detection under different weather conditions. In: 2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019. Institute of Electrical and Electronics Engineers Inc., pp 47–52

  32. Park MW, In Kim J, Lee YJ et al (2017) Vision-based surveillance system for monitoring traffic conditions. Multimed Tools Appl 76:25343–25367. https://doi.org/10.1007/s11042-017-4521-4

    Article  Google Scholar 

  33. Pi Y, Nath ND, Behzadan AH (2020) Convolutional neural networks for object detection in aerial imagery for disaster response and recovery. Adv Eng Informatics 43:101009. https://doi.org/10.1016/j.aei.2019.101009

    Article  Google Scholar 

  34. Rangel JC, Martínez-Gómez J, Romero-González C, García-Varea I, Cazorla M (2018) Semi-supervised 3D object recognition through CNN labeling. Appl Soft Comput 65:603–613. https://doi.org/10.1016/j.asoc.2018.02.005

    Article  Google Scholar 

  35. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, In, pp 779–788

    Google Scholar 

  36. Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39:1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031

    Article  Google Scholar 

  37. Rohan A, Rabah M, Kim S (2019) Convolutional neural network-based real-time object detection and tracking for parrot AR drone 2. IEEE Access 7:69575–69584. https://doi.org/10.1109/ACCESS.2019.2919332

    Article  Google Scholar 

  38. Sadykova D, Pernebayeva D, Bagheri M, James A (2020) IN-YOLO: real-time detection of outdoor high voltage insulators using UAV imaging. IEEE Trans Power Deliv 35:1599–1601. https://doi.org/10.1109/TPWRD.2019.2944741

    Article  Google Scholar 

  39. Saleh M, Jhanjhi N, Abdullah A, Fatima-tuz-Zahra (2020) Proposing a privacy protection model in case of civilian drone. In: 2020 22nd International Conference on Advanced Communication Technology (ICACT). pp 596–602

  40. Shastry AC, Schowengerdt RA (2005) Airborne video registration and traffic-flow parameter estimation. IEEE Trans Intell Transp Syst 6:391–405. https://doi.org/10.1109/TITS.2005.858621

    Article  Google Scholar 

  41. Smitha JA, Rajkumar N (2020) Optimal feed forward neural network based automatic moving vehicle detection system in traffic surveillance system. Multimed Tools Appl 79:18591–18610. https://doi.org/10.1007/s11042-020-08757-1

    Article  Google Scholar 

  42. Tian Y, Yang G, Wang Z, Wang H, Li E, Liang Z (2019) Apple detection during different growth stages in orchards using the improved YOLO-V3 model. Comput Electron Agric 157:417–426. https://doi.org/10.1016/j.compag.2019.01.012

    Article  Google Scholar 

  43. Tzutalin. LabelImg. Git code (2015). https://github.com/tzutalin/labelImg

  44. Wang Z, Liu D, Lei Y, Niu X, Wang S, Shi L (2020) Small target detection based on bird’s visual information processing mechanism. Multimed Tools Appl 79:22083–22105. https://doi.org/10.1007/s11042-020-08807-8

    Article  Google Scholar 

  45. Wu Y, Sui Y, Wang G (2017) Vision-based real-time aerial object localization and tracking for UAV sensing system. IEEE Access 5:23969–23978. https://doi.org/10.1109/ACCESS.2017.2764419

    Article  Google Scholar 

  46. Wu D, Lv S, Jiang M, Song H (2020) Using channel pruning-based YOLO v4 deep learning algorithm for the real-time and accurate detection of apple flowers in natural environments. Comput Electron Agric 178:105742. https://doi.org/10.1016/j.compag.2020.105742

    Article  Google Scholar 

  47. Xu Y, Yu G, Wang Y, Wu X, Ma Y (2017) Car detection from low-altitude UAV imagery with the faster R-CNN. 2017:1–10. https://doi.org/10.1155/2017/2823617

  48. Zhang S, Wen L, Bian X, et al (2018) Single-shot refinement neural network for object detection. In: 2018 IEEE/CVF conference on computer vision and pattern recognition. pp 4203–4212

  49. Zhang J, Liang X, Wang M, Yang L, Zhuo L (2019) Coarse-to-fine object detection in unmanned aerial vehicle imagery using lightweight convolutional neural network and deep motion saliency. Neurocomputing. 398:555–565. https://doi.org/10.1016/j.neucom.2019.03.102

    Article  Google Scholar 

  50. Zhu P, Wen L, Du D et al (2018) Vision meets drones : past. Present and Future:1–20

Download references

Acknowledgements

The first author would like to thank the Ministry of Human Resource Development, New Delhi, India for providing the Research Fellowship for carrying out this work. The authors would also like to thank ISRO, India for providing the support time to time to carry out this area of research.

Author information

Authors and Affiliations

  1. Department of Instrumentation and Control Engineering, Dr B R Ambedkar National Institute of Technology, Jalandhar, India

    Himanshu Gupta & Om Prakash Verma

Authors
  1. Himanshu Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Om Prakash Verma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Om Prakash Verma.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gupta, H., Verma, O.P. Monitoring and surveillance of urban road traffic using low altitude drone images: a deep learning approach. Multimed Tools Appl 81, 19683–19703 (2022). https://doi.org/10.1007/s11042-021-11146-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11146-x

Keywords

Search

Navigation