Abstract
Shoplifting has got serious concern because of a steep surge in these types of cases all around. People are found stealing the items from the store without being noticed, either by putting them in bags or hiding objects inside clothes. CCTV cameras are generally installed at any such site, but evidences suggest that these cameras are not very effective unless the video feeds are constantly monitored. Therefore, we intend to build an automated and intelligent surveillance system to catch these shoplifters by identifying their stealing actions. This article proposes a deep neural network-based solution to identify these shoplifting activities. The model proposed uses a dual-stream fusion-based network that effectively binds appearance and motion dynamics in the temporal domain to efficiently identify the shoplifting actions. The deep Inception V3 model is used to extract activity-specific body posture features from video streams through two deep neural network pipelines, one each corresponding to appearance and motion information. Next, a recurrent neural network, namely Long Short Term Memory (LSTM) network, is used to build a temporal relation between features extracted from consecutive frames in order to distinguish human stealing actions accurately. Added to it, this article introduces a shoplifting dataset synthesized in our lab, which contains normal human actions and object stealing actions. The proposed methodology supported with experimental results demonstrates encouraging outcomes with the accuracy achieved up to 91.48%, which outperforms other existing methods.
Similar content being viewed by others
References
Aggarwal JK, Ryoo MS (2011) Human activity analysis: A review. ACM Computing Surveys (CSUR) 43(3):1–43
Agarwal A, GuptaS, Singh DK (2016) Review of optical flow technique for moving object detection. 2016 2nd International Conference on Contemporary Computing and Informatics (IC3I). IEEE
Ansari MA, Singh DK An Expert Eye for Identifying Shoplifters in Mega Stores. 4TH International Conference on Innovative Computing and Communication (ICICC 2021), Shaheed Sukhdev College of Business Studies, University of Delhi, New Delhi, 20–21st February, 2021.
Arroyo R et al (2015) Expert video-surveillance system for real-time detection of suspicious behaviors in shopping malls. Expert Syst Appl 42(21):7991–8005
Donahue J, et al (2015) Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE conference on computer vision and pattern recognition
Farnebäck G (2003) Two-frame motion estimation based on polynomial expansion. In: Farnebäck G (ed) Scandinavian conference on Image analysis. Springer, Berlin, Heidelberg
Feichtenhofer C, Pinz A, Zisserman A (2016) Convolutional two-stream network fusion for video action recognition. Proceedings of the IEEE conference on computer vision and pattern recognition
Gholamrezaii M, AlModarresi SMT (2021) A time-efficient convolutional neural network model in human activity recognition. Multimed Tools Appl 80(13):19361–19376
Hochreiter S, Schmidhuber J (1997) LSTM can solve hard long time lag problems. Advances in neural information processing systems 473–479
Ibrahim N, et al (2012) Detection of snatch theft based on temporal differences in motion flow field orientation histograms. Int J Adv Comput Technol 4(12)
Ji S et al (2012) 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell 35(1):221–231
Khan NS, Ghani MS (2021) A Survey of deep learning based models for human activity recognition. Wireless Pers Commun. https://doi.org/10.1007/s11277-021-08525-w
Kumar KPS, Bhavani R (2020) Human activity recognition in egocentric video using HOG, GiST and color features. Multimed Tools Appl 79(5):3543–3559
Kushwaha Arati, Khare Ashish, Khare Manish (2021) Human activity recognition algorithm in video sequences based on integration of magnitude and orientation information of optical flow. Int J Image Grap. https://doi.org/10.1142/S0219467822500097
Ladjailia A et al (2020) Human activity recognition via optical flow: decomposing activities into basic actions. Neural Comput Applic 32(21):16387–16400
Lalapura VS, Amudha J, Satheesh HS (2021) Recurrent neural networks for edge intelligence: A survey. ACM Comput Surv (CSUR) 54(4):1–38
Lingaswamy S, Kumar D (2020) An efficient moving object detection and tracking system based on fractional derivative. Multimed Tools Appl 79(13):8519–8537
Liu L et al (2020) Deep learning for generic object detection: A survey. Int J Comput Vis 128(2):261–318
Martínez-Mascorro GA et al (2021) Criminal intention detection at early stages of shoplifting cases by using 3D convolutional neural networks. Computation 9(2):24
National Retail Federation (2018) National retail security survey
Nguyen TN, LyNQ (2017) Abnormal activity detection based on dense spatial-temporal features and improved one-class learning. Proceedings of the Eighth International Symposium on Information and Communication Technology
Pienaar SW, Malekian R (2019) Human activity recognition using LSTM-RNN deep neural network architecture. 2019 IEEE 2nd Wireless Africa Conference (WAC). IEEE
Rashwan HA et al (2020) Action representation and recognition through temporal co-occurrence of flow fields and convolutional neural networks. Multimed Tools Appl 79(45):34141–34158
Singh DK, Kushwaha DS (2016) Tracking movements of humans in a real-time surveillance scene. In: M Pant, K Deep, JC Bansal, A Nagar, KN Das (eds) Proceedings of fifth international conference on soft computing for problem solving. Springer, Singapore
Singh DK (2018) Human Action Recognition in Video. In: Luhach Ashish Kumar, Singh Dharm, Hsiung Pao-Ann, Hawari Kamarul Bin Ghazali, Lingras Pawan, Singh Pradeep Kumar (eds) International Conference on Advanced Informatics for Computing Research. Springer, Singapore
Singh DK et al (2020) Human crowd detection for city wide surveillance. Procedia Comput Sci 171:350–359
Singh D, Mohan CK (2017) Graph formulation of video activities for abnormal activity recognition. Pattern Recognit 65:265–272
Singh T, Vishwakarma DK (2021) A deeply coupled ConvNet for human activity recognition using dynamic and RGB images. Neural Comput Applic 33(1):469–485
Sultani W, Chen C, Shah M (2018) Real-world anomaly detection in surveillance videos. Proceedings of the IEEE conference on computer vision and pattern recognition
Wang H et al (2013) Dense trajectories and motion boundary descriptors for action recognition. Int J Comput Vis 103(1):60–79
Wang C et al (2019) Pulmonary image classification based on inception-v3 transfer learning model. IEEE Access 7:146533–146541
Xia K, Huang J, Wang H (2020) LSTM-CNN architecture for human activity recognition. IEEE Access 8:56855–56866
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Ansari, M.A., Singh, D.K. An expert video surveillance system to identify and mitigate shoplifting in megastores. Multimed Tools Appl 81, 22497–22525 (2022). https://doi.org/10.1007/s11042-021-11438-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-021-11438-2