Skip to main content
SpringerLink
Log in

An expert video surveillance system to identify and mitigate shoplifting in megastores

  • 1200: Machine Vision Theory and Applications for Cyber Physical Systems
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Shoplifting has got serious concern because of a steep surge in these types of cases all around. People are found stealing the items from the store without being noticed, either by putting them in bags or hiding objects inside clothes. CCTV cameras are generally installed at any such site, but evidences suggest that these cameras are not very effective unless the video feeds are constantly monitored. Therefore, we intend to build an automated and intelligent surveillance system to catch these shoplifters by identifying their stealing actions. This article proposes a deep neural network-based solution to identify these shoplifting activities. The model proposed uses a dual-stream fusion-based network that effectively binds appearance and motion dynamics in the temporal domain to efficiently identify the shoplifting actions. The deep Inception V3 model is used to extract activity-specific body posture features from video streams through two deep neural network pipelines, one each corresponding to appearance and motion information. Next, a recurrent neural network, namely Long Short Term Memory (LSTM) network, is used to build a temporal relation between features extracted from consecutive frames in order to distinguish human stealing actions accurately. Added to it, this article introduces a shoplifting dataset synthesized in our lab, which contains normal human actions and object stealing actions. The proposed methodology supported with experimental results demonstrates encouraging outcomes with the accuracy achieved up to 91.48%, which outperforms other existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Aggarwal JK, Ryoo MS (2011) Human activity analysis: A review. ACM Computing Surveys (CSUR) 43(3):1–43

    Article  Google Scholar 

  2. Agarwal A, GuptaS, Singh DK (2016) Review of optical flow technique for moving object detection. 2016 2nd International Conference on Contemporary Computing and Informatics (IC3I). IEEE

  3. Ansari MA, Singh DK An Expert Eye for Identifying Shoplifters in Mega Stores. 4TH International Conference on Innovative Computing and Communication (ICICC 2021), Shaheed Sukhdev College of Business Studies, University of Delhi, New Delhi, 20–21st February, 2021.

  4. Arroyo R et al (2015) Expert video-surveillance system for real-time detection of suspicious behaviors in shopping malls. Expert Syst Appl 42(21):7991–8005

    Article  Google Scholar 

  5. Donahue J, et al (2015) Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE conference on computer vision and pattern recognition

  6. Farnebäck G (2003) Two-frame motion estimation based on polynomial expansion. In: Farnebäck G (ed) Scandinavian conference on Image analysis. Springer, Berlin, Heidelberg

    MATH  Google Scholar 

  7. Feichtenhofer C, Pinz A, Zisserman A (2016) Convolutional two-stream network fusion for video action recognition. Proceedings of the IEEE conference on computer vision and pattern recognition

  8. Gholamrezaii M, AlModarresi SMT (2021) A time-efficient convolutional neural network model in human activity recognition. Multimed Tools Appl 80(13):19361–19376

    Article  Google Scholar 

  9. Hochreiter S, Schmidhuber J (1997) LSTM can solve hard long time lag problems. Advances in neural information processing systems 473–479

  10. Ibrahim N, et al (2012) Detection of snatch theft based on temporal differences in motion flow field orientation histograms. Int J Adv Comput Technol 4(12)

  11. Ji S et al (2012) 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell 35(1):221–231

    Article  Google Scholar 

  12. Khan NS, Ghani MS (2021) A Survey of deep learning based models for human activity recognition. Wireless Pers Commun. https://doi.org/10.1007/s11277-021-08525-w

    Article  Google Scholar 

  13. Kumar KPS, Bhavani R (2020) Human activity recognition in egocentric video using HOG, GiST and color features. Multimed Tools Appl 79(5):3543–3559

    Article  Google Scholar 

  14. Kushwaha Arati, Khare Ashish, Khare Manish (2021) Human activity recognition algorithm in video sequences based on integration of magnitude and orientation information of optical flow. Int J Image Grap. https://doi.org/10.1142/S0219467822500097

    Article  Google Scholar 

  15. Ladjailia A et al (2020) Human activity recognition via optical flow: decomposing activities into basic actions. Neural Comput Applic 32(21):16387–16400

    Article  Google Scholar 

  16. Lalapura VS, Amudha J, Satheesh HS (2021) Recurrent neural networks for edge intelligence: A survey. ACM Comput Surv (CSUR) 54(4):1–38

    Article  Google Scholar 

  17. Lingaswamy S, Kumar D (2020) An efficient moving object detection and tracking system based on fractional derivative. Multimed Tools Appl 79(13):8519–8537

    Article  Google Scholar 

  18. Liu L et al (2020) Deep learning for generic object detection: A survey. Int J Comput Vis 128(2):261–318

    Article  Google Scholar 

  19. Martínez-Mascorro GA et al (2021) Criminal intention detection at early stages of shoplifting cases by using 3D convolutional neural networks. Computation 9(2):24

    Article  Google Scholar 

  20. National Retail Federation (2018) National retail security survey

  21. Nguyen TN, LyNQ (2017) Abnormal activity detection based on dense spatial-temporal features and improved one-class learning. Proceedings of the Eighth International Symposium on Information and Communication Technology

  22. Pienaar SW, Malekian R (2019) Human activity recognition using LSTM-RNN deep neural network architecture. 2019 IEEE 2nd Wireless Africa Conference (WAC). IEEE

  23. Rashwan HA et al (2020) Action representation and recognition through temporal co-occurrence of flow fields and convolutional neural networks. Multimed Tools Appl 79(45):34141–34158

    Article  Google Scholar 

  24. Singh DK, Kushwaha DS (2016) Tracking movements of humans in a real-time surveillance scene. In: M Pant, K Deep, JC Bansal, A Nagar, KN Das (eds) Proceedings of fifth international conference on soft computing for problem solving. Springer, Singapore

  25. Singh DK (2018) Human Action Recognition in Video. In: Luhach Ashish Kumar, Singh Dharm, Hsiung Pao-Ann, Hawari Kamarul Bin Ghazali, Lingras Pawan, Singh Pradeep Kumar (eds) International Conference on Advanced Informatics for Computing Research. Springer, Singapore

    Google Scholar 

  26. Singh DK et al (2020) Human crowd detection for city wide surveillance. Procedia Comput Sci 171:350–359

    Article  Google Scholar 

  27. Singh D, Mohan CK (2017) Graph formulation of video activities for abnormal activity recognition. Pattern Recognit 65:265–272

    Article  Google Scholar 

  28. Singh T, Vishwakarma DK (2021) A deeply coupled ConvNet for human activity recognition using dynamic and RGB images. Neural Comput Applic 33(1):469–485

    Article  Google Scholar 

  29. Sultani W, Chen C, Shah M (2018) Real-world anomaly detection in surveillance videos. Proceedings of the IEEE conference on computer vision and pattern recognition

  30. Wang H et al (2013) Dense trajectories and motion boundary descriptors for action recognition. Int J Comput Vis 103(1):60–79

    Article  MathSciNet  Google Scholar 

  31. Wang C et al (2019) Pulmonary image classification based on inception-v3 transfer learning model. IEEE Access 7:146533–146541

    Article  Google Scholar 

  32. Xia K, Huang J, Wang H (2020) LSTM-CNN architecture for human activity recognition. IEEE Access 8:56855–56866

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. CSED, MNNIT Allahabad, Prayagraj, Uttar Pradesh, India

    Mohd. Aquib Ansari & Dushyant Kumar Singh

Authors
  1. Mohd. Aquib Ansari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dushyant Kumar Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohd. Aquib Ansari.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ansari, M.A., Singh, D.K. An expert video surveillance system to identify and mitigate shoplifting in megastores. Multimed Tools Appl 81, 22497–22525 (2022). https://doi.org/10.1007/s11042-021-11438-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11438-2

Keywords

Search

Navigation