Skip to main content
SpringerLink
Log in

Toward multi-category garments virtual try-on method by coarse to fine TPS deformation

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Virtual try-on facilitates users to evaluate the wearing effect of garments on their bodies. As online clothing shopping develops, the category and style of garments constantly enrich. It is an issue to warp multi-category garments as the user shape without three-dimensional (3d) garment models. To tackle this issue, we propose a novel virtual try-on method toward multi-category garments by coarse to fine thin plate spline (TPS) deformation. To embody the user shape, 3d human body model is reconstructed with the garment pose. With the orientation and width classification criteria, the human body part mask is projected from 3d human body model, then it is adapted to the category and feature of garments. The spatial gradients with various scales are generated by comparing the shape difference between the garment mask and the human body part mask. To eliminate this shape difference, the coarse to fine TPS deformation is utilized to warp garment images from global to local. Ultimately, the warped garment images are worn on the virtual human body to preview the try-on effect. Experiments demonstrated that our method is robust to different human body shapes with different garments. Compared with state-of-the-art VITON methods, our method is superior in preserving the texture details and overall style in the virtual try-on for multi-category garments. The code is available at https://github.com/NerdFNY/MCG-VITON.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Bradley D, Popa T, Sheffer A et al (2008) Markerless garment capture. ACM Trans Gr (TOG) 27(3):1–9

    Article  Google Scholar 

  2. Popa T, Zhou Q, Bradley D, et al (2009) Wrinkling captured garments using space‐time data‐driven deformation. In: Computer Graphics Forum, Blackwell Publishing Ltd, Wiley, vol 28(2), pp 427–435

  3. Yu T, Zheng Z, Zhong Y, et al (2019) Simulcap: Single-view human performance capture with cloth simulation. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, IEEE, pp 5504–5514

  4. Pons-Moll G, Pujades S, Hu S et al (2017) ClothCap: Seamless 4D clothing capture and retargeting. ACM Trans Gr (TOG) 36(4):1–15

    Article  Google Scholar 

  5. Chen X, Zhou B, Lu FX et al (2015) Garment modeling with a depth camera. ACM Trans Gr 34(6):203:1-203:12

    Google Scholar 

  6. Bhatnagar BL, Tiwari G, Theobalt C, et al (2019) Multi-garment net: Learning to dress 3d people from images. In: Proceedings of the IEEE/CVF International Conference on computer vision, IEEE, pp 5420–5430

  7. Natsume R, Saito S, Huang Z, et al (2019) Siclope: Silhouette-based clothed people. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, IEEE, pp 4480–4490

  8. Alldieck T, Magnor M, Bhatnagar BL, et al (2019) Learning to reconstruct people in clothing from a single RGB camera. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, IEEE, pp 1175–1186

  9. Yang S, Pan Z, Amert T et al (2018) Physics-inspired garment recovery from a single-view image. ACM Trans Gr (TOG) 37(5):1–14

    Article  Google Scholar 

  10. Patel C, Liao Z, Pons-Moll G (2020) Tailornet: predicting clothing in 3d as a function of human pose, shape and garment style. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, IEEE, pp 7365–7375

  11. Wang T Y, Ceylan D, Popovic J, et al (2018) Learning a shared shape space for multimodal garment design. arXiv:1806.11335

  12. Provot X (1995) Deformation constraints in a mass-spring model to describe rigid cloth behavior. In: Graphics interface, Canadian Information Processing Society, pp 147–147

  13. Miguel E, Bradley D, Thomaszewski B et al (2012) Data‐driven estimation of cloth simulation models. In: Computer Graphics Forum. Blackwell Publishing Ltd, Wiley, vol 31(2pt2), pp 519–528

  14. Lahner Z, Cremers D, Tung T (2018) Deepwrinkles: accurate and realistic clothing modeling. In: Proceedings of the European Conference on computer vision (ECCV), pp 667–684

  15. Santesteban I, Otaduy M A, Casas D (2019) Learning‐based animation of clothing for virtual try‐on. In: Computer Graphics Forum. John Wiley & Sons Ltd, vol 38(2), pp 355–366

  16. Wang H, Hecht F, Ramamoorthi R et al (2010) Example-based wrinkle synthesis for clothing animation. ACM Trans Gr (TOG) 29(4):1–8

    Google Scholar 

  17. Mir A, Alldieck T, Pons-Moll G (2020) Learning to transfer texture from clothing images to 3d humans, In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, IEEE, pp 7023–7034

  18. Zhu S, Urtasun R, Fidler S, et al (2017) Be your own prada: fashion synthesis with structural coherence. In: Proceedings of the IEEE International Conference on computer vision, IEEE, pp 1680–1688

  19. Han X, Wu Z, Wu Z, et al (2018) Viton: an image-based virtual try-on network. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, IEEE, pp. 7543–7552

  20. Wang B, Zheng H, Liang X, et al (2018) Toward characteristic-preserving image-based virtual try-on network. In: Proceedings of the European Conference on computer vision (ECCV), pp 589–604

  21. Yang H, Zhang R, Guo X, et al (2020) Towards photo-realistic virtual try-on by adaptively generating-preserving image content. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, IEEE, pp 7850–7859

  22. Yu R, Wang X, Xie X (2019) Vtnfp: an image-based virtual try-on network with body and clothing feature preservation. In: Proceedings of the IEEE/CVF International Conference on computer vision, IEEE, pp 10511–10520

  23. Dong H, Liang X, Shen X, et al (2019) Towards multi-pose guided virtual try-on network. In: Proceedings of the IEEE/CVF International Conference on computer vision, IEEE, pp 9026–9035

  24. Mir A, Alldieck T, Pons-Moll G (2020) Learning to transfer texture from clothing images to 3d humans. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, IEEE, pp 7023–7034.

  25. Choi S, Park S, Lee M, et al (2021) VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, IEEE, pp 14131–14140

  26. Liu Z, Luo P, Qiu S, et al (2016) Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, IEEE, pp 1096–1104

  27. Ge Y, Zhang R, Wang X, et al (2019) Deepfashion2: a versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, IEEE, pp 5337–5345

  28. Yu W, Liang X, Gong K, et al (2019) Layout-graph reasoning for fashion landmark detection. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, IEEE, pp 2937–2945

  29. Liu Z, Yan S, Luo P, et al (2016) Fashion landmark detection in the wild. In: European Conference on computer vision. Springer, Cham, IEEE, pp 229–245

  30. Wang W, Xu Y, Shen J, et al (2018) Attentive fashion grammar network for fashion landmark detection and clothing category classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp.4271–4280

  31. Sidnev A, Krapivin A, Trushkov A, et al (2021) Deepmark++: Real-time clothing detection at the edge. In: Proceedings of the IEEE/CVF Winter Conference on applications of computer vision, IEEE, pp 2980–2988

  32. Liu S, Liang X, Liu L et al (2015) Fashion parsing with video context. IEEE Trans Multimed 17(8):1347–1358

    Article  Google Scholar 

  33. He K, Gkioxari G, Dollár P, et al (2017) Mask r-cnn. In: Proceedings of the IEEE International Conference on computer vision, IEEE, pp 2961–2969

  34. Chen Q, Koltun V (2017) Photographic image synthesis with cascaded refinement networks. In: Proceedings of the IEEE international Conference on computer vision, IEEE, pp 1511–1520

  35. Jaderberg M, Simonyan K, Zisserman A (2015) Spatial transformer networks. Adv Neural Inf Process Syst 28:2017–2025

    Google Scholar 

  36. Bookstein FL (1989) Principal warps: Thin-plate splines and the decomposition of deformations. IEEE Trans Pattern Anal Mach Intell 11(6):567–585

    Article  Google Scholar 

  37. Larsson M, Stenborg E, Toft C, et al (2019) Fine-grained segmentation networks: self-supervised segmentation for improved long-term visual localization. In: Proceedings of the IEEE/CVF International Conference on computer vision, IEEE, pp 31–41

  38. Rocco I, Arandjelovic R, Sivic J (2017) Convolutional neural network architecture for geometric matching. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, IEEE, pp 6148–6157

  39. Pavlakos G, Zhu L, Zhou X, et al (2018) Learning to estimate 3D human pose and shape from a single color image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp 459–468

  40. Guler R A, Kokkinos I (2019) Holopose: Holistic 3d human reconstruction in-the-wild. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, IEEE, pp 10884–10894

  41. Liang J, Lin MC (2019) Shape-aware human pose and shape reconstruction using multi-view images. In: Proceedings of the IEEE/CVF International Conference on computer vision and pattern recognition, IEEE, pp 4352–4362

  42. Loper M, Mahmood N, Romero J et al (2015) SMPL: a skinned multi-person linear model. ACM Trans Gr (TOG) 34(6):1–16

    Article  Google Scholar 

  43. Schafer RW (2011) What is a Savitzky-Golay filter?[lecture notes]. IEEE Signal Process Mag 28(4):111–117

    Article  Google Scholar 

  44. Ge C, Song Y, Ge Y, et al (2021) Disentangled cycle consistency for highly-realistic virtual try-on. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, IEEE, pp 16928–16937

  45. He H, Zhang J, Zhang Q, et al (2020) Grapy-ML: graph pyramid mutual learning for cross-dataset human parsing. In: Proceedings of the AAAI Conference on artificial intelligence, AAAI, vol. 34(07), pp. 10949–10956

  46. Wang Z, Bovik AC, Sheikh HR et al (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  47. Zhang R, Isola P, Efros A A, et al (2018) The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, pp 586–595

  48. Salimans T, Goodfellow I, Zaremba W et al (2016) Improved techniques for training gans. Adv Neural Inf Process Syst 29:2234–2242

    Google Scholar 

  49. Dowson DC, Landau BV (1982) The Fréchet distance between multivariate normal distributions. J Multivar Anal 12(3):450–455

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by the National Key R&D Program of China (No. 2018YFB1700700).

Author information

Authors and Affiliations

  1. State Key Laboratory of Fluid Power Transmission & Control, Zhejiang University, Hangzhou, 310027, China

    Naiyu Fang, Lemiao Qiu, Shuyou Zhang, Zili Wang, Kerui Hu & Heng Li

Authors
  1. Naiyu Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lemiao Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuyou Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zili Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kerui Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Heng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lemiao Qiu.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fang, N., Qiu, L., Zhang, S. et al. Toward multi-category garments virtual try-on method by coarse to fine TPS deformation. Neural Comput & Applic 34, 12947–12965 (2022). https://doi.org/10.1007/s00521-022-07173-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07173-w

Keywords

Search

Navigation