Skip to main content
SpringerLink
Log in

Jaywalking detection and localization in street scene videos using fine-tuned convolutional neural networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Video Anomaly detection has been the focus of many research studies for a long time and shows potential for endless implementations to detect real-world anomalies. When it comes to road safety, Jaywalking is one such real-life event that can be detected and localized using video anomaly detection techniques. However, this field’s progress depends majorly on the availability of diverse datasets and the kind of anomalies that they depict. The main objective of this work is to propose a video anomaly detection model for jaywalking that can help control the number of road accidents that occur due to jaywalking each year. The proposed model is a novel variation of the InceptionV3 deep CNN model and has been experimenting with one of the latest street scene datasets and its different variations. Our model consists of two separate subsystems based on pre-trained InceptionV3 architecture. The first subsystem, Anomaly Detector, takes a video frame as input and predicts whether it consists of a jaywalking event. The second subsystem, Anomaly Localizer, takes a jaywalking(anomalous) video frame as input and predicts the bounding box labels to locate the Jaywalking object in the frame. To evaluate, we employ the Street Scene Dataset, released in February 2020 that offers several kinds of jaywalking events. In this work, we experiment with different pre-trained CNN models, namely VGG16, ResNet50, and InceptionV3, to evaluate their performance for anomaly detection. Further, we assess the model’s performance for different dataset sizes, having both an even and an uneven number of anomalous and non-anomalous frames. Lastly, we measure the effectiveness of the model on different types of jaywalking episodes. The evaluation shows that our proposed model attains remarkable detection accuracy, in comparison to other state-of-the-art methods proposed previously in this field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Adam A, Rivlin E, Shimshoni I, Reinitz D (2008) Robust real-time unusual event detection using multiple fixed-location monitors. IEEE transactions on pattern analysis and machine intelligence 30(3):555–560

    Article  Google Scholar 

  2. Bansod S, Nandedkar A (2019) Transfer learning for video anomaly detection. Journal of Intelligent & Fuzzy Systems 36(3):1967–1975

    Article  Google Scholar 

  3. Calderara S, Heinemann U, Prati A, Cucchiara R, Tishby N (2011) Detecting anomalies in people’s trajectories using spectral graph analysis. Comput Vis Image Underst 115(8):1099–1111

    Article  Google Scholar 

  4. Highways of Road Transport M (2019) Road accidents in india – 2019. Tech. rep., Government of India, https://morth.nic.in/road-accident-in-india

  5. Jiang X, Wang W, Mao Y, Bengler K, Bubb H (2011) Situational factors of influencing drivers to give precedence to jaywalking pedestrians at signalized crosswalk. International Journal of Computational Intelligence Systems 4(6):1407–1414

    Google Scholar 

  6. John V, Boyali A, Thompson S, Lakshmanan A, Mita S (2020) Visible and thermal camera-based jaywalking estimation using a hierarchical deep learning framework. In: Proceedings of the Asian Conference on Computer Vision

  7. Kaltsa V, Briassouli A, Kompatsiaris I, Strintzis M G (2018) Multiple hierarchical dirichlet processes for anomaly detection in traffic. Comput Vis Image Underst 169:28–39

    Article  Google Scholar 

  8. Li W, Mahadevan V, Vasconcelos N (2013) Anomaly detection and localization in crowded scenes. IEEE transactions on pattern analysis and machine intelligence 36(1):18–32

    Google Scholar 

  9. Liu W, Luo W, Lian D, Gao S (2018) Future frame prediction for anomaly detection–a new baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6536–6545

  10. Loy C C, Xiang T, Gong S (2008) From local temporal correlation to global anomaly detection. In: The 1st International Workshop on Machine Learning for Vision-based Motion Analysis-MLVMA’08

  11. Lu C, Shi J, Jia J (2013) Abnormal event detection at 150 fps in matlab. In: Proceedings of the IEEE international conference on computer vision, pp 2720–2727

  12. Luo W, Liu W, Gao S (2017) A revisit of sparse coding based anomaly detection in stacked rnn framework. In: Proceedings of the IEEE International Conference on Computer Vision, pp 341–349

  13. Medel J R, Savakis A (2016) Anomaly detection in video using predictive convolutional long short-term memory networks. arXiv:161200390

  14. Morris B T, Trivedi M M (2008) A survey of vision-based trajectory learning and analysis for surveillance. IEEE transactions on circuits and systems for video technology 18(8):1114–1127

    Article  Google Scholar 

  15. Park J, Lee Y, Heo J H, Kang S J (2019) Convolutional neural network-based jaywalking data generation and classification. In: 2019 International soc design conference, ISOCC, IEEE, pp 132–133

  16. Ramachandra B, Jones M (2020) Street scene: a new dataset and evaluation protocol for video anomaly detection. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp 2569–2578

  17. Sabokrou M, Fathy M, Hoseini M (2016) Video anomaly detection and localisation based on the sparsity and reconstruction error of auto-encoder. Electron Lett 52(13):1122–1124

    Article  Google Scholar 

  18. Sabokrou M, Fayyaz M, Fathy M, Klette R (2017) Deep-cascade: cascading 3d deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans Image Process 26(4):1992–2004

    Article  MathSciNet  MATH  Google Scholar 

  19. Sabokrou M, Fayyaz M, Fathy M, Moayed Z, Klette R (2018) Deep-anomaly: fully convolutional neural network for fast anomaly detection in crowded scenes. Comput Vis Image Underst 172:88–97

    Article  MATH  Google Scholar 

  20. Sayed T, Zaki M H, Autey J (2013) A Novel approach for diagnosing road safety issues using automated computer vision techniques. In: 16th International Conference Road Safety on Four Continents. Beijing, China (RS4c 2013), pp 15–17, May 2013, Statens väg-ochtransportforskningsinstitut

  21. Sivaraman S, Trivedi M M (2013) Looking at vehicles on the road: a survey of vision-based vehicle detection, tracking, and behavior analysis. IEEE transactions on intelligent transportation systems 14(4):1773–1795

    Article  Google Scholar 

  22. Suarez J J P, Naval Jr P C (2020) A survey on deep learning techniques for video anomaly detection. arXiv:200914146

  23. Sultani W, Chen C, Shah M (2018) Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6479–6488

  24. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  25. Tian B, Yao Q, Gu Y, Wang K, Li Y (2011) Video Processing techniques for traffic flow monitoring: A survey. In: 2011 14Th international IEEE conference on intelligent transportation systems, ITSC, IEEE, pp 1103–1108

  26. Tung F, Zelek J S, Clausi D A (2011) Goal-based trajectory analysis for unusual behaviour detection in intelligent surveillance. Image Vis Comput 29(4):230–240

    Article  Google Scholar 

  27. Vishwakarma S, Agrawal A (2013) A survey on activity recognition and behavior understanding in video surveillance. Vis Comput 29(10):983–1009

    Article  Google Scholar 

  28. WHO (2020) Road traffic injuries. Tech. rep., https://www.who.int/news-room/fact-sheets/detail/road-traffic-injuries

  29. Ye M, Peng X, Gan W, Wu W, Qiao Y (2019) Anopcn: video anomaly detection via deep predictive coding network. In: Proceedings of the 27th ACM International Conference on Multimedia, pp 1805–1813

  30. Yu J, Li X, Zhao Y (2011) Jaywalker detection based on motion path analysis. In: Third International Conference on Transportation Engineering (ICTE) American Society of Civil EngineersChina Communications and Transportation Association

  31. Zaki M H, Sayed T, Tageldin A, Hussein M (2013) Application of computer vision to diagnosis of pedestrian safety issues. Transportation research record 2393(1):75–84

    Article  Google Scholar 

  32. Zhang Y, Lu H, Zhang L, Ruan X (2016) Combining motion and appearance cues for anomaly detection. Pattern Recogn 51:443–452

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge the authors of our base paper [16] to release the dataset comprising Jaywalking that was focus of our work.

Funding

There is no funding source of the work and project.

Author information

Authors and Affiliations

  1. Department of Information Technology, Indira Gandhi Delhi Technical University for Women, New Delhi, 110006, India

    Aarti Bala & Rishabh Kaushal

Authors
  1. Aarti Bala
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rishabh Kaushal
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Aarti Bala performed implementation of models, analyzed the results, and wrote the first verion of this draft. Rishabh Kaushal conceptualization the problem problem, reviewed the methodology, code, results, and write-up in this report.

Corresponding author

Correspondence to Rishabh Kaushal.

Ethics declarations

This is to declare that we shall release the code and customized dataset that we curated from the original dataset on acceptance of our work.

Conflict of Interests

There are no conflicts of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bala, A., Kaushal, R. Jaywalking detection and localization in street scene videos using fine-tuned convolutional neural networks. Multimed Tools Appl 82, 34771–34791 (2023). https://doi.org/10.1007/s11042-023-14922-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14922-z

Keywords

Search

Navigation