Skip to main content
SpringerLink
Log in

Segmentation mask-guided person image generation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Background clutters and pose variation are the key factors which prevents the network from learning a robust Person re-identification (Re-ID) model. To address the problem above, we first introduce the binary segmentation mask to construct the body region served as the input of the generator, then design a segmentation mask-guided person image generation network for the pose transfer. The binary segmentation mask has the capability of removing the background clutters in pixel-level, and contains more details about the edge information, where better shape consistency can be achieved for the generated image with the input image. Compared with the previous methods, the proposed method can dramatically improve the model adaptive ability and deal with the diversity of postures. In addition, we design a lightweight attention mechanism module as a guider module, which can assist the generator to focus on the discriminative features of pedestrians. The experiment results are introduced to demonstrate the effectiveness of the proposed method and the superiority performance over most state-of-the-art methods without over-computing in the design process of the Re-ID model. It is worth mentioning that our ideas can be easily combined with other fields to solve the phenomenon of the current situation with insufficient pose variations in the datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Liu Z, Li D, Ge SS, and Tian F (2019) Small traffic sign detection from large image. Appl Intel pp 1–13

  2. Li X, Zheng WS, Wang X, Xiang T, Gong S (2015) Multi-scale learning for low-resolution person re-identification. In: CVPR, pp. 3765–3773

  3. Tao D, Jin L, Wang Y, Yuan Y, Li X (2013) Person re-identification by regularized smoothing kiss metric learning. IEEE Trans Circ Syst Video Technol 23(10):1675–1685

    Article  Google Scholar 

  4. Zhang R, Lin L, Zhang R, Zuo W, Zhang L (2015) Bit-scalable deep hashing with regularized similarity learning for image retrieval and person re-identification. IEEE Trans Image Process 24(12):4766–4779

    Article  MathSciNet  MATH  Google Scholar 

  5. Zheng WS, Gong S, Xiang T (2012) Reidentification by relative distance comparison. IEEE Trans Pattern Anal Mach Intell 35(3):653–668

    Article  Google Scholar 

  6. Wang T, Gong S, Zhu X, Wang S (2016) Person re-identification by discriminative selection in video ranking. IEEE Trans Pattern Anal Mach Intell 38(12):2501–2514

    Article  Google Scholar 

  7. Chen YC, Zhu X, Zheng WS, Lai JH (2017) Person re-identification by camera correlation aware feature augmentation. IEEE Trans Pattern Anal Mach Intell 40(2):392–408

    Article  Google Scholar 

  8. Protopapadakis E, Voulodimos A, Doulamis A, Doulamis N, Stathaki T (2019) Automatic crack detection for tunnel inspection using deep learning and heuristic image post-processing. Appl Intell 49(7):2793–2806

    Article  Google Scholar 

  9. Song Y, Lee JW, Lee J (2019) A study on novel filtering and relationship between input-features and target-vectors in a deep learning model for stock price prediction. Appl Intell 49(3):897–911

    Article  Google Scholar 

  10. Acharya UR, Fujita H, Oh SL, Hagiwara Y, Tan JH, Adam M, San Tan R (2019) Deep convolutional neural network for the automated diagnosis of congestive heart failure using ecg signals. Appl Intell 49(1):16–27

    Article  Google Scholar 

  11. Ma L, Sun Q, Georgoulis S, Van Gool L, Schiele B, Fritz M (2018) Disentangled person image generation. In CVPR, pp 99–108

  12. Pumarola A, Agudo A, Sanfeliu A, Moreno-Noguer F (2018) Unsupervised person image synthesis in arbitrary poses. In CVPR, pp 8620–8628

  13. Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In CVPR, pp 2414–2423

  14. Liu J, Sun C, Xu X, Xu B, Yu S (2019) A spatial and temporal features mixture model with body parts for video-based person re-identification. Appl Intell 49(9):3436–3446

    Article  Google Scholar 

  15. Li W, Zhu X, Gong S (2018) Harmonious attention network for person re-identification. In CVPR, pp 2285–2294

  16. Chang X, Hospedales TM, Xiang T (2018) Multi-level factorisation net for person re-identification. In CVPR, pp 2109–2118

  17. Li W, Zhao R, Xiao T, Wang X (2014) Deepreid: deep filter pairing neural network for person re-identification. In CVPR, pp 152–159

  18. Zhu Z, Huang T, Shi B, Yu M, Wang B, Bai X (2019) Progressive pose attention transfer for person image generation. In CVPR, pp 2347–2356

  19. Liu J, Ni B, Yan Y, Zhou P, Cheng S, Hu J (2018) Pose transferrable person re-identification. In CVPR, pp 4099–4108

  20. Ma L, Jia X, Sun Q, Schiele B, Tuytelaars T, Van Gool L (2017) Pose guided person image generation. Advances in Neural Information Processing Systems. pp 406–416

  21. Tang H, Zhao Y, Lu H (2019) Unsupervised person re-identification with iterative self-supervised domain adaptation. In CVPR

  22. Su C, Li J, Zhang S, Xing J, Gao W, Tian Q (2017) Pose-driven deep convolutional model for person re-identification. In CVPR, pp 3960–3969

  23. Zheng Z, Yang X, Yu Z, Zheng L, Yang Y, Kautz J (2019) Joint discriminative and generative learning for person re-identification. In CVPR, pp 2138–2147

  24. Siarohin A, Sangineto E, Lathuilière S, Sebe N (2018) Deformable gans for pose-based human image generation. In CVPR, pp 3408–3416

  25. Wang L, Tan T, Ning H, Hu W (2003) Silhouette analysis-based gait recognition for human identification. IEEE Trans Pattern Anal Mach Intell 25(12):1505–1518

    Article  Google Scholar 

  26. Zhong Z, Zheng L, Zheng Z, Li S, Yang Y (2018) Camera style adaptation for person re-identification. In CVPR, pp 5157–5166

  27. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, pp 2223–2232

  28. Wei L, Zhang S, Gao W, Tian Q (2018) Person transfer Gan to bridge domain gap for person re-identification. In CVPR, pp 79–88

  29. Zhong Z, Zheng L, Li S, Yang Y (2018) Generalizing a person retrieval model hetero-and homogeneously. In ECCV, pp 172–188

  30. Choi Y, Choi M, Kim M, Ha J-W, Kim S, Choo J (2018) Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In CVPR, pp 8789–8797

  31. Bak S, Carr P, Lalonde JF (2018) Domain adaptation through synthesis for unsupervised person re-identification. In ECCV, pp 189–205

  32. Song S, Zhang W, Liu J, Mei T (2019) Unsupervised person image generation with semantic parsing transformation. In CVPR, pp 2357–2366

  33. Woo S, Park J, Lee JY, So Kweon I (2018) Cbam: convolutional block attention module. In ECCV, pp 3–19

  34. Hou S, Wang Z (2019) Weighted channel dropout for regularization of deep convolutional neural network. In AAAI, pp 8425–8432

  35. Song C, Huang Y, Ouyang W, Wang L (2018) Mask-guided contrastive attention model for person re-identification. In CVPR, pp 1179–1188

  36. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In CVPR, pp 770–778

  37. Wei SE, Ramakrishna V, Kanade T, Sheikh Y (2016) Convolutional pose machines. In CVPR, pp 4724–4732

  38. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434

  39. Efraimidis PS, Spirakis PG (2006) Weighted random sampling with a reservoir. Inf Process Lett 97(5):181–185

    Article  MathSciNet  MATH  Google Scholar 

  40. Tompson J, Goroshin R, Jain A, LeCun Y, Bregler C (2015) Efficient object localization using convolutional networks. In CVPR, pp 648–656

  41. Liao S, Hu Y, Zhu X, Li SZ (2015) Person re-identification by local maximal occurrence representation and metric learning. In CVPR, pp 2197–2206

  42. Zheng L, Yang Y, Hauptmann AG (2016) Person re-identification: Past, present and future. arXiv:1610.02984

  43. Zheng Z, Zheng L, Yang Y (2017) Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In CVPR, pp 3754–3762

  44. Qian X, Fu Y, Xiang T, W. Wang, J. Qiu, Y. Wu, Y.G. Jiang, and Xue X (2018) Pose-normalized image generation for person re-identification. In ECCV, pp 650–667

  45. Zheng Z, Zheng L, Yang Y (2018) Pedestrian alignment network for large-scale person re-identification. IEEE Trans Circ Syst Video Technol 29(10):3037–3045

    Article  Google Scholar 

  46. Wang G, Yuan Y, Chen X, Li J, Zhou X (2018) Learning discriminative features with multiple granularities for person re-identification. In ACM pp 274–282

  47. Wang Y, Chen Z, Wu F, Wang G (2018) Person re-identification with cascaded pairwise convolutions. In CVPR, pp 1470–1478

  48. Lin Y, Zheng L, Zheng Z, Wu Y, Hu Z, Yan C, Yang Y (2019) Improving person re-identification by attribute and identity learning. 95:151–161

  49. Yang Q, Yu HX, Wu A, Zheng WS (2019) Patch-based discriminative feature learning for unsupervised person re-identification. In CVPR, pp 3633–3642

  50. Zhang C, Wu L, Wang Y (2019) Crossing generative adversarial networks for cross-view person re-identification. Neurocomputing. 340:259–269

    Article  Google Scholar 

  51. Li M, Zhu X, Gong S (2019) Unsupervised tracklet person re-identification. IEEE Trans Pattern Anal Mach Intell 42(7):1770–1782. https://doi.org/10.1109/TPAMI.2019.2903058

  52. Chung D, Delp EJ (2019) Camera-aware image-to-image translation using similarity preserving stargan for person re-identification. In CVPR

  53. Zhong Z, Zheng L, Zheng Z, Li S, Yang Y (2019) Camstyle: a novel data augmentation method for person re-identification. IEEE Trans Image Process 28(3):1176–1190

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The work is supported by National Natural Science Foundation of China (61573114) and Fundamental Research Funds for the Central Universities (HEUCF160415). This work is also supported by College of Intelligent Systems Science and Engineering, Harbin Engineering University. 

Author information

Authors and Affiliations

  1. College of Automation, Harbin Engineering University, Harbin, 150001, China

    Meichen Liu, Xin Yan & Kejun Wang

  2. Department of Statistics, University of California, Los Angeles, 90095-1554, USA

    Chenhui Wang

Authors
  1. Meichen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chenhui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kejun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Chenhui Wang or Kejun Wang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, M., Yan, X., Wang, C. et al. Segmentation mask-guided person image generation. Appl Intell 51, 1161–1176 (2021). https://doi.org/10.1007/s10489-020-01907-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01907-w

Keywords

Search

Navigation