Skip to main content
SpringerLink
Log in

Pedestrian identification using motion-controlled deep neural network in real-time visual surveillance

  • Focus
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

In the computer vision applications such as security surveillance and robotics, pedestrian identification shows much attention in the last decade. This is usually achieved by human biometrics. Besides human biometrics, sometimes it is required to identify pedestrians at a distance. This could be accomplished based on a fact of different whole-body appearances. The real-time pedestrian identification is a challenging task due to several factors such as illumination effects, noise, change in viewpoint, and video resolution. The more recent, the deep neural network (DNN) shows a massive performance for various real-world applications. In this article, we present a real-time architecture for pedestrian identification using motion-controlled DNN. In the proposed architecture, the motion vectors are calculating using optical flow and then utilized in the next step, named features extraction. Two types of features, such as HOG and DNN, are computing. The pre-trained VGG19 CNN model is employing and trained through transfer learning. The deep learning features are extracted from two layers—fully connected layers 7 and 8. Also, we proposed a feature selection method named Bayesian modeling along with LSVM. The best selected features of both HOG and DNN are finally fused in one matrix for final identification. The multi-class support vector machine classifier is used for final identification. The videos are recording in the real-time environment for the experimental process and achieve an average accuracy of 98.62%. Overall, identification accuracy shows the effectiveness of the proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Arshad H, Khan MA, Sharif MI, Yasmin M, Tavares JMR, Zhang YD, et al (2020) A multilevel paradigm for deep convolutional neural network features selection with an application to human gait recognition. Expert Syst e12541.

  • Arshad H, Khan MA, Sharif M, Yasmin M, Javed MY (2019) Multi-level features fusion and selection for human gait recognition: an optimized framework of Bayesian model and binomial distribution. Int J Mach Learn Cybern 10:3601–3618

    Article  Google Scholar 

  • Babaee M, Li L, Rigoll GJN (2019) Person identification from partial gait cycle using fully convolutional neural networks. Neurocomputing 338:116–125

    Article  Google Scholar 

  • Bascones JJ, Graña M, Lopez-Guede JM (2019) Robust labeling of human motion markers in the presence of occlusions. Neurocomputing 353:96–105

    Article  Google Scholar 

  • Batchuluun G, Naqvi RA, Kim W, Park KR (2018) Body-movement-based human identification using convolutional neural network. Expert Syst Appl 101:56–77

    Article  Google Scholar 

  • Batool FE, Attique M, Sharif M, Javed K, Nazir M, Abbasi AA, et al (202) Offline signature verification system: a novel technique of fusion of GLCM and geometric features using SVM. Multimed Tools Appl 1–20

  • Cai Z, Saberian MJ, Vasconcelos N (2019) Learning complexity-aware cascades for pedestrian detection. IEEE Trans Pattern Anal Mach Intell 42:2195–2211

    Article  Google Scholar 

  • Chaki J, Dey N, Shi F, Sherratt RSJISJ (2019) Pattern mining approaches used in sensor-based biometric recognition: a review. IEEE Sens J 19:3569–3580

    Article  Google Scholar 

  • Combs TS, Sandt LS, Clamann MP, McDonald NC (2019) Automated vehicles and pedestrian safety: exploring the promise and limits of pedestrian detection. Am J Prev Med 56:1–7

    Article  Google Scholar 

  • Cuntoor N, Kale A, Chellappa R (2003) Combining multiple evidences for gait recognition. In: Proceedings of 2003 IEEE international conference on acoustics, speech, and signal processing (ICASSP'03), pp III-33

  • Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), pp 886–893

  • Dalal N, Triggs B, Schmid C (2006) Human detection using oriented histograms of flow and appearance. In: European conference on computer vision, pp 428–441

  • Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248–255

  • Dey N, Ashour A, Patra PK (2016b) Feature detectors and motion detection in video processing. IGI Global

    Google Scholar 

  • Dey N, Ashour A, Acharjee S (2016a) Applied video processing in surveillance and monitoring systems. IGI Global

    Google Scholar 

  • Dey N, Ashour AS, Hassanien AE (2017) Feature detectors and descriptors generations with numerous images and video applications: a recap. In: Feature detectors and motion detection in video processing. IGI Global, pp 36–65

  • Dutta A, Mondal A, Dey N, Sen S, Moraru L, Hassanien AE (2020) Vision tracking: a survey of the state-of-the-art. SN Comput Sci 1:57

    Article  Google Scholar 

  • Enzweiler M, Eigenstetter A, Schiele B, Gavrila DM. Multi-cue pedestrian classification with partial occlusion handling. In: 2010 IEEE computer society conference on computer vision and pattern recognition, pp 990–997

  • Foster JP, Nixon MS, Prügel-Bennett A (2003) Automatic gait recognition using area-based metrics. Pattern Recognit Lett 24:2489–2497

    Article  Google Scholar 

  • Fouad KM, Hassan BM, Hassan MF (2016) User authentication based on dynamic keystroke recognition. Int J Ambient Comput Intell 7:1–32

    Article  Google Scholar 

  • Hassaballah M, Aly S (2015) Face recognition: challenges, achievements and future directions. IET Comput Vision 9:614–626

    Article  Google Scholar 

  • Hassaballah M, Awad AI (2016) Image feature detectors and descriptors: foundations and applications. Stud Comput Intell 630:11–45

    Article  Google Scholar 

  • Hassaballah M, Hosny KM (2019) Recent advances in computer vision. Studies in computational intelligence 804

  • Hassaballah M, Alshazly H, Ali AA (2020) Robust local oriented patterns for ear recognition. Multimed Tools Appl 79:31183–31204

    Article  Google Scholar 

  • Htun KZ, Zaw SMM (2019) Gait recognition for person identification using statistics of SURF

  • Hu J, Fan XP, Liu S, Huang L (2019) Robust target tracking algorithm based on superpixel visual attention mechanism: robust target tracking algorithm. Int J Ambient Comput Intell (IJACI) 10:1–17

    Article  Google Scholar 

  • Huang P, Hilton A, Starck J (2010) Shape similarity for 3D video sequences of people. Int J Comput Vis 89:362–381

    Article  Google Scholar 

  • Jadhav IS, Gaikwad V, Patil GU (2011) Human Identification using Face and Voice Recognition 1.

  • Jain L (2020) Visual traffic surveillance: a concise survey. Inf Technol Intelligent Transp Syst 323:32

    Google Scholar 

  • Jiang Y, Wang J, Liang Y, Xia J (2019) Combining static and dynamic features for real-time moving pedestrian detection. Multimed Tools Appl 78:3781–3795

    Article  Google Scholar 

  • Kenk MA, Hassaballah M, Brethé J-F (2019) Human-aware robot navigation in logistics warehouses. In: ICINCO (2), pp 371–378

  • Khan MA, Sarfraz MS, Alhaisoni M, Albesher AA, Wang S, Ashraf I (2020e) StomachNet: optimal deep learning features fusion for stomach abnormalities classification. IEEE Access 8:197969–197981

    Article  Google Scholar 

  • Khan MA, Ashraf I, Alhaisoni M, Damaševičius R, Scherer R, Rehman A et al (2020d) Multimodal brain tumor classification using deep learning and robust feature selection: a machine learning application for radiologists. Diagnostics 10:565

    Article  Google Scholar 

  • Khan MA, Kadry S, Alhaisoni M, Nam Y, Zhang Y, Rajinikanth V et al (2020c) Computer-aided gastrointestinal diseases analysis from wireless capsule endoscopy: a framework of best features selection. IEEE Access 8:132850–132859

    Article  Google Scholar 

  • Khan MA, Javed K, Khan SA, Saba T, Habib U, Khan JA et al (2020) Human action recognition using fusion of multiview and deep features: an application to video surveillance. Multimed Tools Appl 1–27

  • Khan MA, Zhang Y-D, Khan SA, Attique M, Rehman A, Seo S (2020) A resource conscious human action recognition framework using 26-layered deep convolutional neural network. Multimed Tools Appl 1–23

  • Lahmyed R, El Ansari M, Ellahyani A (2019) A new thermal infrared and visible spectrum images-based pedestrian detection system. Multimed Tools Appl 78:15861–15885

    Article  Google Scholar 

  • Meinhardt-Llopis E, Pérez JS, Kondermann D (2013) Horn-schunck optical flow with a multi-scale strategy. Image Process on line 2013:151–172

    Article  Google Scholar 

  • Nguyen DT, Zong Z, Ogunbona P, Li W (2010) Object detection using non-redundant local binary patterns. In: 2010 IEEE international conference on image processing, pp 4609–4612

  • Nguyen DT, Ogunbona P, Li W (2011) Human detection with contour-based local motion binary patterns. In: 2011 18th IEEE international conference on image processing, pp 3609–3612

  • Park D, Zitnick CL, Ramanan D, Dollár P (2013) Exploring weak stabilization for motion feature extraction. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2882–2889

  • Rashid M, Khan MA, Sharif M, Raza M, Sarfraz MM, Afza F (2019) Object detection and classification: a joint selection and fusion strategy of deep convolutional neural network and SIFT point features. Multimed Tools Appl 78:15751–15777

    Article  Google Scholar 

  • Rashid M, Khan MA, Alhaisoni M, Wang S-H, Naqvi SR, Rehman A et al (2020) A sustainable deep learning framework for object recognition using multi-layers deep features fusion and selection. Sustainability 12:5037

    Article  Google Scholar 

  • Raza M, Sharif M, Yasmin M, Khan MA, Saba T, Fernandes SL (2018) Appearance based pedestrians’ gender recognition by employing stacked auto encoders in deep learning. Futur Gener Comput Syst 88:28–39

    Article  Google Scholar 

  • Samal A, Iyengar PA (1992) Automatic recognition and analysis of human faces and facial expressions: a survey. Pattern Recognit 25:65–77

    Article  Google Scholar 

  • Sghaier S, Farhat W, Souani C (2018) Novel technique for 3D face recognition using anthropometric methodology. J Ambient Comput Intell 9:60–77

    Article  Google Scholar 

  • Sharif M, Khan MA, Akram T, Javed MY, Saba T, Rehman A (2017) A framework of human detection and action recognition based on uniform segmentation and combination of Euclidean distance and joint entropy-based features selection. EURASIP J Image Video Process 2017:89

    Article  Google Scholar 

  • Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  • Singh T, Vishwakarma DK (2019) A hybrid framework for action recognition in low-quality video sequences. arXiv preprint arXiv:1903.04090

  • Sinha H, Manekar R, Sinha Y, Ajmera PK (2019) Convolutional neural network-based human identification using outer ear images. In: Bansal J, Das K, Nagar A, Deep K, Ojha A (eds) Soft computing for problem solving. Springer, pp 707–719

    Chapter  Google Scholar 

  • Tafazzoli F, Safabakhsh R (2010) Model-based human gait recognition using leg and arm movements. Eng Appl Artif Intell 23:1237–1246

    Article  Google Scholar 

  • Tan C, Sun F, Kong T, Zhang W, Yang C, Liu C (2018) A survey on deep transfer learning. In: International conference on artificial neural networks, pp 270–279

  • Vinayak Kale G, Hemant Patil V (2016) A study of vision based human motion recognition and analysis. arXiv:1608.06761

  • Viola P, Jones MJ, Snow D (2005) Detecting pedestrians using patterns of motion and appearance. Int J Comput Vis 63:153–161

    Article  Google Scholar 

  • Wang L, Tan T, Hu W, Ning H (2003) Automatic gait recognition based on statistical shape analysis. IEEE Trans Image Process 12:1120–1131

    Article  MathSciNet  Google Scholar 

  • Xu Y, Zhou X, Liu P, Xu HJNPL (2019) Rapid pedestrian detection based on deep omega-shape features with partial occlusion handing. Neural Process Lett 49:923–937

    Article  Google Scholar 

  • Xu C, Makihara Y, Li X, Yagi Y, Lu J (2020) Cross-view gait recognition using pairwise spatial transformer networks. IEEE Trans Circuits Syst Video Technol 31:260–274

    Article  Google Scholar 

  • Zhang C, Kim J (2019) Multi-scale pedestrian detection using skip pooling and recurrent convolution. Multimed Tools Appl 78:1719–1736

    Article  Google Scholar 

  • Zhou F, De la Torre F. Spatio-temporal matching for human detection in video. In: European conference on computer vision, pp 62–77

  • Zhou C, Yuan JJPR (2019) Multi-label learning of part detectors for occluded pedestrian detection. Pattern Recognit 86:99–111

    Article  Google Scholar 

Download references

Funding

There is no funding.

Author information

Authors and Affiliations

  1. Department of Computer Science, COMSATS University Islamabad, Wah Campus, Islamabad, Pakistan

    Muhammad Zahid, Faisal Azam & Muhammad Sharif

  2. Department of Computer Science, HITEC University, Museum Road, Taxila, Pakistan

    Muhammad Attique Khan

  3. Faculty of Applied Computing and Technology, Noroff University College, Kristiansand, Norway

    Seifedine Kadry

  4. School of Computer Applications, KIIT Deemed to be University, Bhubaneswar, India

    Jnyana Ranjan Mohanty

Authors
  1. Muhammad Zahid
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Muhammad Attique Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Faisal Azam
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Muhammad Sharif
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Seifedine Kadry
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jnyana Ranjan Mohanty
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Each author has equally contributed in conceptualization, model building, simulation, and writing of the article.

Corresponding author

Correspondence to Jnyana Ranjan Mohanty.

Ethics declarations

Conflict of interest

There is no conflict of interests of any authors.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Authors have taken consents from all concerned of the works in the paper.

Additional information

Communicated by Suresh Chandra Satapathy.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zahid, M., Khan, M.A., Azam, F. et al. Pedestrian identification using motion-controlled deep neural network in real-time visual surveillance. Soft Comput 27, 453–469 (2023). https://doi.org/10.1007/s00500-021-05701-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-021-05701-9

Keywords

Search

Navigation