Skip to main content
Springer Nature Link
Log in

Deep pixel regeneration for occlusion reconstruction in person re-identification

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Person re-identification is very important for monitoring and tracking crowd movement to provide public security. However, re-identification in the presence of occlusion is a challenging area that has not received significant attention yet. In this work, we propose a plausible solution to this problem by developing effective techniques for occlusion detection and reconstruction from RGB images/videos using Deep Neural Networks. Specifically, a CNN-based occlusion detection model is used to detect the occluded frames in an input sequence, following which a Conv-LSTM model or an Autoencoder is employed to reconstruct the pixels corresponding to the occluded regions depending on whether the input frames are sequential or non-sequential. The quality of the reconstructed RGB frames is further refined using a DCGAN. Our method has been evaluated using four public data sets for cumulative rank-based accuracy and Dice score, and the qualitative reconstruction results are indeed appealing. Quantitative evaluation in terms of re-identification accuracy using a Siamese classifier shows a Rank-1 accuracy of over 70% after reconstructing the occlusion present in each of these datasets. A comparative study with popular state-of-the-art approaches also demonstrates the effectiveness of our work for use in real-life surveillance sites.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Ahmed E, Jones M, Marks T K (2015) An improved deep learning architecture for person re-identification. In: Proc. of the Conf. on CVPR, pp 3908–3916

  2. Chung D, Tahboub K, Delp E J (2017) A two stream siamese convolutional neural network for person re-identification. In: Proc. of the ICCV, pp 1983–1991

  3. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: CVPR, 2005. CVPR 2005. IEEE Computer society conf. on, vol 1, pp 886–893

  4. Dalvi C, Rathod M, Patil S, Gite S, Kotecha K (2021) A survey of ai-based facial emotion recognition: Features, ml & dl techniques, age-wise datasets and future directions. IEEE Access 9:165806–165840

    Article  Google Scholar 

  5. De Teyou G K (2020) ConvLSTM for spatio-temporal feature extraction in time-series images. In: Proc. of the NeurIPS: workshop on tackling climate change with machine learning. https://www.climatechange.ai/papers/neurips2020/12/paper.pdf

  6. Ding S, Lin L, Wang G, Chao H (2015) Deep feature learning with relative distance comparison for person re-identification. Pattern Recogn 48(10):2993–3003

    Article  Google Scholar 

  7. Fan X, Luo H, Zhang X, He L, Zhang C, Jiang W (2018) SCPNet: spatial-channel parallelism network for joint holistic and partial person re-identification. In: Proc. of the ACCV, pp 19–34

  8. Fan D-P, Wang W, Cheng M-M, Shen J (2019) Shifting more attention to video salient object detection. In: Proc. of the Conf. on CVPR, pp 8554–8564

  9. Fu J, Zheng H, Mei T (2017) Look closer to see better: recurrent attention convolutional neural network for fine-grained image recognition. In: Proc. of the Conf. on CVPR, pp 4438–4446

  10. Gao S, Wang J, Lu H, Liu Z (2020) Pose-guided visible part matching for occluded person ReID. In: Proc. of the Conf. on CVPR, pp 11744–11752

  11. Ge Y, Li Z, Zhao H, Yin G, Yi S, Wang X, Li H (2018) FD-GAN: pose-guided feature distilling GAN for robust person re-identification. arXiv:1810.02936

  12. Gray D, Tao H (2008) Viewpoint invariant pedestrian recognition with an ensemble of localized features. In: Proc. of the ECCV, pp 262–275

  13. He L, Liang J, Li H, Sun Z (2018) Deep spatial feature reconstruction for partial person re-identification: alignment-free approach. In: Proc. of the Conf. on CVPR, pp 7073–7082

  14. He L, Sun Z, Zhu Y, Wang Y (2018) Recognizing partial biometric patterns. arXiv:1810.07399

  15. He L, Wang Y, Liu W, Zhao H, Sun Z, Feng J (2019) Foreground-aware pyramid reconstruction for alignment-free occluded person re-identification. In: Proc. of the ICCV, pp 8450–8459

  16. He L, Liu W (2020) Guided saliency feature learning for person re-identification in crowded scenes. In: Proc. of the ECCV, pp 357–373

  17. Hermans A, Beyer L, Leibe B (2017) In defense of the triplet loss for person re-identification. arXiv:1703.07737

  18. Hou R, Ma B, Chang H, Gu X, Shan S, Chen X (2019) VRSTC: occlusion-free video person re-identification. In: Proc. of the IEEE/CVF Conf. on CVPR, pp 7183–7192

  19. Javed O, Shafique K, Shah M (2005) Appearance modeling for tracking in multiple non-overlapping cameras. In: Proc. of the Conf. on CVPR, vol 2, pp 26–33

  20. Javed O, Shafique K, Rasheed Z, Shah M (2008) Modeling inter-camera space–time and appearance relationships for tracking across non-overlapping views. Comput Vis Image Underst 109(2):146–162

    Article  Google Scholar 

  21. Jiang K, Zhang T, Zhang Y, Wu F, Rui Y (2020) Self-supervised agent learning for unsupervised cross-domain person re-identification. IEEE Trans Image Process 29:8549–8560

    Article  Google Scholar 

  22. Kalayeh M M, Basaran E, Gökmen M, Kamasak M E, Shah M (2018) Human semantic parsing for person re-identification. In: Proc. of the Conf. on CVPR, pp 1062–1071

  23. Li W, Zhao R, Xiao T, Wang X (2014) DeepReID: deep filter pairing neural network for person re-identification. In: Proc. of the Conf. on CVPR, pp 152–159

  24. Li X, Chen H, Qi X, Dou Q, Fu C-W, Heng P-A (2018) H-DenseUNet: hybrid densely connected Unet for liver and tumor segmentation from CT volumes. IEEE Trans Med Imaging 37(12):2663–2674

    Article  Google Scholar 

  25. Liao S, Hu Y, Zhu X, Li S Z (2015) Person re-identification by local maximal occurrence representation and metric learning. In: Proc. of the Conf. on CVPR, pp 2197–2206

  26. Liu H, Jie Z, Jayashree K, Qi M, Jiang J, Yan S, Feng J (2017) Video-based person re-identification with accumulative motion context. IEEE Trans CSVT 28(10):2788–2802

    Google Scholar 

  27. Liu Y, Yan J, Ouyang W (2017) Quality aware network for set to set recognition. In: Proc. of the Conf. on CVPR, pp 5790–5799

  28. Liu J, Ni B, Yan Y, Zhou P, Cheng S, Hu J (2018) Pose transferrable person re-identification. In: Proc. of the Conf. on CVPR, pp 4099–4108

  29. Liu Y-C, Tan D S, Chen J-C, Cheng W-H, Hua K-L (2019) Segmenting hepatic lesions using residual attention u-net with an adaptive weighted dice loss. In: Proc. of the ICIP, pp 3322–3326

  30. Lu X, Wang W, Ma C, Shen J, Shao L, Porikli F (2019) See more, know more: unsupervised video object segmentation with co-attention siamese networks. In: Proc. of the Conf. on CVPR, pp 3623–3632

  31. McLaughlin N, Martinez del Rincon J, Miller P (2016) Recurrent convolutional network for video-based person re-identification. In: Proc. of the Conf. on CVPR, pp 1325–1334

  32. Miao J, Wu Y, Liu P, Ding Y, Yang Y (2019) Pose-guided feature alignment for occluded person re-identification. In: Proc. of the ICCV, pp 542–551

  33. Miao J, Wu Y, Yang Y (2021) Identifying visible parts via pose estimation for occluded person re-identification. IEEE Transactions on Neural Networks and Learning Systems

  34. Minetto R, Segundo M P, Sarkar S (2019) Hydra: an ensemble of convolutional neural networks for geospatial land classification. IEEE Trans Geosci Remote Sens

  35. Prosser B J, Gong S, Xiang T (2008) Multi-camera matching using bi-directional cumulative brightness transfer functions. In: Proc. of the BMVC, vol 8, 164. Citeseer, p 74

  36. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434

  37. Sarfraz M S, Schumann A, Eberle A, Stiefelhagen R (2018) A pose-sensitive embedding for person re-identification with expanded cross neighborhood re-ranking. In: Proc. of the Conf. on CVPR, pp 420–429

  38. Su C, Li J, Zhang S, Xing J, Gao W, Tian Q (2017) Pose-driven deep convolutional model for person re-identification. In: Proc. of the ICCV, pp 3960–3969

  39. Subramaniam A, Chatterjee M, Mittal A (2016) Deep neural networks with inexact matching for person re-identification. In: Proc. of the advances in NIPS, pp 2667–2675

  40. Sun Y, Zheng L, Yang Y, Tian Q, Wang S (2018) Beyond part models: person retrieval with refined part pooling (and a strong convolutional baseline). In: Proc. of the ECCV, pp 480–496

  41. Sun Y, Xu Q, Li Y, Zhang C, Li Y, Wang S, Sun J (2019) Perceive where to focus: learning visibility-aware part-level features for partial person re-identification. In: Proc. of the Conf. on CVPR, pp 393–402

  42. Tagore N K, Chattopadhyay P (2020) SMSNet: a novel multi-scale siamese model for person re-identification. In: Proc. of the ICETE, pp 103–112

  43. Tagore N K, Chattopadhyay P (2022) A bi-network architecture for occlusion handling in person re-identification. SIViP 16(4):1071–1079

    Article  Google Scholar 

  44. Tagore N K, Chattopadhyay P, Wang L (2020) T-MAN: a neural ensemble approach for person re-identification using spatio-temporal information. Multimed Tools Applic 79(37):28393–28409

    Article  Google Scholar 

  45. Tagore N K, Singh A, Manche S, Chattopadhyay P (2021) Person re-identification from appearance cues and deep siamese features. J Vis Commun Image Represent 75:103029

    Article  Google Scholar 

  46. Vulli A, Srinivasu P N, Sashank M S K, Shafi J, Choi J, Ijaz M F (2022) Fine-tuned densenet-169 for breast cancer metastasis prediction using fastai and 1-cycle policy. Sensors 22(8):2988

    Article  Google Scholar 

  47. Wang X, Doretto G, Sebastian T, Rittscher J, Tu P (2007) Shape and appearance context modeling. In: Proc. of the ICCV, pp 1–8

  48. Wang G, Yang S, Liu H, Wang Z, Yang Y, Wang S, Yu G, Zhou E, Sun J (2020) High-order information matters: learning relation and topology for occluded person re-identification. In: Proc. of the IEEE/CVF Conf. on CVPR, pp 6449–6458

  49. Wang Z, Zhu F, Tang S, Zhao R, He L, Song J (2022) Feature erasing and diffusion network for occluded person re-identification. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4754–4763

  50. Xingjian SHI, Chen Z, Wang H, Yeung D-Y, Wong W-K, Woo W- (2015) Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Proc. of the Advances in NIPS, pp 802–810

  51. Xiong F, Gou M, Camps O, Sznaier M (2014) Person re-identification using kernel-based metric learning methods. In: Proc. of the ECCV, pp 1–16

  52. Xu S, Cheng Y, Gu K, Yang Y, Chang S, Zhou P (2017) Jointly attentive spatial-temporal pooling networks for video-based person re-identification. In: Proc. of the ICCV, pp 4733–4742

  53. Xu J, Zhao R, Zhu F, Wang H, Ouyang W (2018) Attention-aware compositional network for person re-identification. In: Proc. of the Conf. on CVPR, pp 2119–2128

  54. Xu F, Ma B, Chang H, Shan S (2021) PRDP: person reidentification with dirty and poor data. IEEE Transactions on Cybernetics

  55. Yan Y, Ni B, Song Z, Ma C, Yan Y, Yang X (2016) Person re-identification via recurrent feature aggregation. In: Proc. of the ECCV, pp 701–716

  56. Yan C, Pang G, Jiao J, Bai X, Feng X, Shen C (2021) Occluded person re-identification with single-scale global representations. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 11875–11884

  57. Yang Y, Yang J, Yan J, Liao S, Yi D, Li S Z (2014) Salient color names for person re-identification. In: Proc. of the ECCV, pp 536–551

  58. Ye M, Li J, Ma A J, Zheng L, Yuen P C (2019) Dynamic graph co-matching for unsupervised video-based person re-identification. IEEE Trans Image Process 28(6):2976–2990

    Article  MathSciNet  Google Scholar 

  59. Ye M, Lan X, Leng Q, Shen J (2020) Cross-modality person re-identification via modality-aware collaborative ensemble learning, vol 29

  60. Ye M, Shen J, Lin G, Xiang T, Shao L, Hoi SCH (2021) Deep learning for person re-identification: a survey and outlook. IEEE Transactions on PAMI

  61. Zhao L, Li X, Zhuang Y, Wang J (2017) Deeply-learned part-aligned representations for person re-identification. In: Proc. of the ICCV, pp 3219–3228

  62. Zheng W-S, Gong S, Xiang T (2009) Associating groups of people. In: Proc. of the BMVC, vol 2,6, pp 1–11

  63. Zheng W-S, Gong S, Xiang T (2011) Person re-identification by probabilistic relative distance comparison. In: Proc. of the Conf. on CVPR, pp 649–656

  64. Zheng W-S, Li X, Xiang T, Liao S, Lai J, Gong S (2015) Partial person re-identification. In: Proc. of the ICCV, pp 4678–4686

  65. Zheng L, Yang Y, Hauptmann A G (2016) Person re-identification: past, present and future. arXiv:1610.02984

  66. Zheng H, Fu J, Mei T, Luo J (2017) Learning multi-attention convolutional neural network for fine-grained image recognition. In: Proc. of the ICCV, pp 5209–5217

  67. Zhou Z, Huang Y, Wang W, Wang L, Tan T (2017) See the forest for the trees: joint spatial and temporal recurrent neural networks for video-based person re-identification. In: Proc. of the Conf. on CVPR, pp 4747–4756

  68. Zhuo J, Chen Z, Lai J, Wang G (2018) Occluded person re-identification. In: Proc. of the ICME, pp 1–6

  69. Zhuo J, Lai J, Chen P (2019) A novel teacher-student learning framework for occluded person re-identification. arXiv:1907.03253

  70. Zhou S, Wu J, Zhang F, Sehdev P (2020) Depth occlusion perception feature analysis for person re-identification. Pattern Recogn Lett 138:617–623

    Article  Google Scholar 

  71. Zhou K, Yang Y, Cavallaro A, Xiang T (2021) Learning generalisable omni-scale representations for person re-identification. IEEE Transactions on PAMI

Download references

Acknowledgements

The authors would also like to thank SERB, DST for partially supporting this work through project grant (CRG/2020/005465)

Author information

Authors and Affiliations

  1. School of Computer Science Engineering and Technology, Bennett University, Greater Noida, PIN 221310, India

    Nirbhay Kumar Tagore

  2. Department of Data Science and Artificial Intelligence, International Institute of Information Technology, Naya Raipur, Chhattisgarh, PIN 493661, India

    Prathistith Raj Medi

  3. Pattern Recognition Lab, Department of Computer Science and Engineering, Indian Institute of Technology (BHU), Varanasi, Uttar Pradesh, PIN 221005, India

    Pratik Chattopadhyay

Authors
  1. Nirbhay Kumar Tagore
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Prathistith Raj Medi
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Pratik Chattopadhyay
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Nirbhay Kumar Tagore.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tagore, N.K., Medi, P.R. & Chattopadhyay, P. Deep pixel regeneration for occlusion reconstruction in person re-identification. Multimed Tools Appl 83, 4443–4463 (2024). https://doi.org/10.1007/s11042-023-15322-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15322-z

Keywords