Skip to main content
SpringerLink
Log in

Deep learning and handcrafted features for one-class anomaly detection in UAV video

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Visual surveillance systems have recently captured the attention of the research community. Most of the proposed surveillance systems deal with stationary cameras. Nevertheless, these systems may reflect minor applicability in anomaly detection when multiple cameras are required. Lately, under technological progress in electronic and avionics systems, Unmanned Aerial Vehicles (UAVs) are increasingly used in a wide variety of urban missions. Especially, in the surveillance context, UAVs can be used as mobile cameras to overcome weaknesses of stationary cameras. One of the principal advantages that makes UAVs attractive is their ability to provide a new aerial perspective. Despite their numerous advantages, there are many difficulties associated with automatic anomalies detection by an UAV, as there is a lack in the proposed contributions describing anomaly detection in videos recorded by a drone. In this paper, we propose new anomaly detection techniques for assisting UAV based surveillance mission where videos are acquired by a mobile camera. To extract robust features from UAV videos, three different features extraction methods were used, namely a pretrained Convolutional Neural Network (CNN) and two popular handcrafted methods (Histogram of Oriented Gradient (HOG) and HOG3D). One Class Support Vector Machine (OCSVM) has been then applied for the unsupervised classification. Extensive experiments carried on a dataset containing videos taken by an UAV monitoring a car parking, prove the efficiency of the proposed techniques. Specifically, the quantitative results obtained using the challenging Area Under Curve (AUC) evaluation metric show that, despite the variation among them, the proposed methods achieve good results in comparison to the existing technique with an AUC = 0.78 at worst and an AUC = 0.93 at best.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Ballabio D (2015) A MATLAB toolbox for Principal Component Analysis and unsupervised exploration of data structure. Chemometr Intell Lab Syst 149:1–9

    Article  Google Scholar 

  2. Bera A, Kim S, Manocha D (2016) Realtime anomaly detection using trajectory-level crowd behavior learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 50–57

  3. Bonetto M, Korshunov P, Ramponi G, et al. (2015) Privacy in mini-drone based video surveillance. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). IEEE, pp 1–6

  4. Bouindour S, Hu R, Snoussi H (2019) Enhanced Convolutional Neural Network for Abnormal Event Detection in Video Streams. In: 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE). IEEE, pp 172–178

  5. Bradley AP (1997) The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern recognition 30(7):1145–1159

    Article  Google Scholar 

  6. Cai Y, Ma L, Liu G (2019) A Night-time Anomaly Detection System of Hog Activities Based on Passive Infrared Detector. Appl Eng Agric 35 (4):481–493

    Article  Google Scholar 

  7. Carreño A, Inza I, Lozano JA (2020) Analyzing rare event, anomaly, novelty and outlier detection terms under the supervised classification framework. Artif Intell Rev 53:3575–3594. https://doi.org/10.1007/s10462-019-09771-y

    Article  Google Scholar 

  8. Chalapathy R, Chawla S (2019) Deep learning for anomaly detection: A survey. arXiv:1901.03407

  9. Chan Y-T, Wang S-J, Tsai C-H (2017) Extracting foreground ensemble features to detect abnormal crowd behavior in intelligent video-surveillance systems. Journal of Electronic Imaging 26(5):051402

    Article  Google Scholar 

  10. Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM computing surveys (CSUR) 41(3):1–58

    Article  Google Scholar 

  11. Chang C-C, Lin C-J (2011) LIBSVM: A library for support vector machines. ACM transactions on intelligent systems and technology (TIST) 2(3):1–27

    Article  Google Scholar 

  12. Chicco D (2017) Ten quick tips for machine learning in computational biology. BioData mining 10(1):35

    Article  Google Scholar 

  13. Chriki A, Touati H, Snoussi H, Kamoun F (2019) FANET: Communication, mobility models and security issues. Comput Netw 163:106877

    Article  Google Scholar 

  14. Chriki A, Touati H, Snoussi H, Kamoun F (2019) UAV-GCS Centralized Data-Oriented Communication Architecture for Crowd Surveillance Applications. In: 2019 15th International Wireless Communications & Mobile Computing Conference (IWCMC), IEEE, pp 2064–2069

  15. Chriki A, Touati H, Snoussi H, Kamoun F (2019) Centralized Cognitive Radio Based Frequency Allocation for UAVs Communication. In: 2019 15th International Wireless Communications & Mobile Computing Conference (IWCMC), IEEE, pp 1674–1679

  16. Cinelli LP, Thomaz LA, Da Silva AF, et al. (2017) Foreground segmentation for anomaly detection in surveillance videos using deep residual networks. Proceedings of the XXXV simpósio Brasileiro De telecomunicações E Processamento De Sinais, Sao Pedro, Brazil, pp 3–6

  17. Cortes C, Vapnik V (1995) Support-vector networks. Machine learning 20(3):273–297

    MATH  Google Scholar 

  18. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05). IEEE, pp 886–893

  19. Diego F, Evangelidis G, Serrat J, Evangelidis G, Serrat J (2012) “Night-time outdoor surveillance with mobile cameras”

  20. Diego F, Ponsa D, Serrat J, et al. (2010) Video alignment for change detection. IEEE Trans Image Process 20(7):1858–1869

    Article  MathSciNet  Google Scholar 

  21. ExtractHOGFeatures (2020) MathWorks. https://ch.mathworks.com/help/vision/ref/extracthogfeatures.html. Accessed 25 January

  22. Fahn C-S, Ling J, Yeh M-Y, et al. (2019) Abnormal Maritime Activity Detection in Satellite Image Sequences Using Trajectory Features. International Journal of Future Computer and Communication, 8(1)

  23. Farooq MU, Khan NA, Ali MS (2017) Unsupervised video surveillance for anomaly detection of street traffic International Journal of Advanced Computer Science and Applications (IJACSA)

  24. Golda T, Murzyn N, Qu C, et al. (2019) What goes around comes around: Cycle-Consistency-based Short-Term Motion Prediction for Anomaly Detection using Generative Adversarial Networks. In: 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). IEEE, pp 1–8

  25. Googlenet (2020) MathWorks. https://ch.mathworks.com/help/deeplearning/ref/googlenet.html. Accessed 15 January

  26. Gordon G, Tibshirani R (2012) Karush-kuhn-tucker conditions. Optimization 10(725/36):725

    Google Scholar 

  27. Haering N, Venetianer P, Lipton A (2008) The evolution of video surveillance: an overview. Mach Vis Appl 19(5-6):279–290

    Article  Google Scholar 

  28. Henrio J, Nakashima T (2018) Anomaly Detection in Videos Recorded by Drones in a Surveillance Context. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2503–2508

  29. Kawachi Y, Koizumi Y, Murata S, et al. (2019) A two-class hyper-spherical autoencoder for supervised anomaly detection. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp 3047–3051

  30. Kiran BR, Thomas DM, Parakkal R (2018) An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. Journal of Imaging 4(2):36

    Article  Google Scholar 

  31. Klaser A, Marszałek M, Schmid C (2008) A spatio-temporal descriptor based on 3d-gradients

  32. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International journal of computer vision 60(2):91–110

    Article  Google Scholar 

  33. Lu X, Ma C, Ni B, et al. (2018) Deep regression tracking with shrinkage loss. In: Proceedings of the European conference on computer vision (ECCV), pp 353–369

  34. Lu X, Ma C, Ni B, et al. (2019) Adaptive region proposal with channel regularization for robust object tracking IEEE Transactions on Circuits and Systems for Video Technology

  35. Medel JR, Savakis A (2016) Anomaly detection in video using predictive convolutional long short-term memory networks. arXiv:1612.00390

  36. Mishra SR, Mishra TK, Sarkar A, et al. (2020) Detection of anomalies in human action using optical flow and gradient tensor. in : smart intelligent computing and applications. Springer, Singapore, pp 561–570

    Google Scholar 

  37. Motlagh NH, Bagaa M, Taleb T (2017) UAV Based IoT platform: A crowd surveillance use case. IEEE Commun Mag 55(2):128–134

    Article  Google Scholar 

  38. Nawaratne R, Alahakoon D, De Silva D, et al. (2019) Spatiotemporal anomaly detection using deep learning for Real-Time video surveillance. IEEE Transactions on Industrial Informatics 16(1):393–402

    Article  Google Scholar 

  39. Olatunji IE, Cheng C-H (2019) Video analytics for visual surveillance and applications: an overview and survey. In: Machine learning paradigms, Springer, Cham, pp 475–515

  40. Patil N, Biswas PK (2016) Global abnormal events detection in surveillance video—A hierarchical approach. In: 2016 Sixth International Symposium on Embedded Computing and System Design (ISED), IEEE, pp 217–222

  41. Pearson K L III (1901) On lines and planes of closest fit to systems of points in space. The London Edinburgh, and Dublin Philosophical Magazine and Journal of Science 2(11):559–572

    Article  Google Scholar 

  42. Ribeiro M, Lazzaretti AE, Lopes HS (2018) A study of deep convolutional auto-encoders for anomaly detection in videos. Pattern Recogn Lett 105:13–22

    Article  Google Scholar 

  43. Saini R, Ahmed A, Dogra DP, et al. (2017) Surveillance scene segmentation based on trajectory classification using supervised learning. In: Proceedings of International Conference on Computer Vision and Image Processing, Springer, Singapore, pp 261–271

  44. Schölkopf B, Williamson RC, Smola AJ, et al. (2000) Support vector method for novelty detection. In: Advances in neural information processing systems, pp 582–588

  45. Stone M (1974) Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society: Series B (Methodological) 36 (2):111–133

    MathSciNet  MATH  Google Scholar 

  46. Szegedy C, Liu W, Jia Y, et al. (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  47. Thomaz LA, Jardim E, Da Silva AF, et al. (2017) Anomaly detection in moving-camera video sequences using principal subspace analysis. IEEE Transactions on Circuits and Systems I: Regular Papers 65(3):1003–1015

    MathSciNet  Google Scholar 

  48. Wang T, Qiao M, Zhu A, et al. (2018) Abnormal event detection via covariance matrix for optical flow-based feature. Multimedia Tools and Applications 77(13):17375–17395

    Article  Google Scholar 

  49. Wang T, Snoussi H (2014) Detection of abnormal visual events via global optical flow orientation histogram. IEEE Transactions on Information Forensics and Security 9(6):988–998

    Article  Google Scholar 

  50. Wang Y, Yoshihashi R, Kawakami R, et al. (2019) Unsupervised anomaly detection with compact deep features for wind turbine blade images taken by a drone. IPSJ Transactions on Computer Vision and Applications 11(1):1–7

    Article  Google Scholar 

  51. Wang S, Zhu E, Yin J, et al. (2016) Anomaly detection in crowded scenes by SL-HOF descriptor and foreground classification. In: 2016 23rd International Conference on Pattern Recognition (ICPR), IEEE, pp 3398–3403

  52. Wei JY, Zhao JF, Zhao YY, et al. (2018) Unsupervised anomaly detection for traffic surveillance based on background modeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 129–136

  53. Wittek P (2014) Unsupervised Learning. In: Quantum machine learning: what quantum computing means to data mining. Academic Press

  54. Xu D, Yan Y, Ricci E, et al. (2017) Detecting anomalous events in videos by learning deep representations of appearance and motion. Comput Vis Image Underst 156:117–127

    Article  Google Scholar 

  55. Yan W, Yu L (2019) On accurate and reliable anomaly detection for gas turbine combustors: A deep learning approach. arXiv:1908.09238

  56. Zhai X, Liu K, Nash W, et al. (2020) Smart Autopilot Drone System for Surface Surveillance and Anomaly Detection via Customizable Deep Neural Network. In: International Petroleum Technology Conference. International Petroleum Technology Conference

  57. Zhang A, Lipton ZC, Li M, et al. (2019) Dive into Deep Learning. Unpublished draft Retrieved 3:319

    Google Scholar 

  58. Zhou S, Shen W, Zeng D, et al. (2016) Spatial–temporal convolutional neural networks for anomaly detection and localization in crowded scenes. Signal Process Image Commun 47:358–368

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Hatem Bettaher IResCoMath Research Unit, University of Gabés, Gabés, Tunisia

    Amira Chriki & Haifa Touati

  2. National School of Computer Science (ENSI), Manouba, Tunisia

    Amira Chriki & Farouk Kamoun

  3. University of Technology of Troyes (UTT), Troyes, France

    Hichem Snoussi

  4. CRISTAL Laboratory, University of Manouba, Manouba, Tunisia

    Farouk Kamoun

Authors
  1. Amira Chriki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haifa Touati
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hichem Snoussi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Farouk Kamoun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amira Chriki.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chriki, A., Touati, H., Snoussi, H. et al. Deep learning and handcrafted features for one-class anomaly detection in UAV video. Multimed Tools Appl 80, 2599–2620 (2021). https://doi.org/10.1007/s11042-020-09774-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09774-w

Keywords

Search

Navigation