Matteo Fincato
Abstract
Virtual try-on has recently emerged in computer vision and multimedia communities with the development of architectures that can generate realistic images of a target person wearing a custom garment. This research interest is motivated by the large role played by e-commerce and online shopping in our society. Indeed, the virtual try-on task can offer many opportunities to improve the efficiency of preparing fashion catalogs and to enhance the online user experience. The problem is far to be solved: current architectures do not reach sufficient accuracy with respect to manually generated images and can only be trained on image pairs with a limited variety. Existing virtual try-on datasets have two main limits: they contain only female models, and all the images are available only in low resolution. This not only affects the generalization capabilities of the trained architectures but makes the deployment to real applications impractical. To overcome these issues, we present Dress Code, a new dataset for virtual try-on that contains high-resolution images of a large variety of upper-body clothes and both male and female models. Leveraging this enriched dataset, we propose a new model for virtual try-on capable of generating high-quality and photo-realistic images using a three-stage pipeline. The first two stages perform two different geometric transformations to warp the desired garment and make it fit into the target person’s body pose and shape. Then, we generate the new image of that same person wearing the try-on garment using a generative network. We test the proposed solution on the most widely used dataset for this task as well as on our newly collected dataset and demonstrate its effectiveness when compared to current state-of-the-art methods. Through extensive analyses on our Dress Code dataset, we show the adaptability of our model, which can generate try-on images even with a higher resolution.
- [1] . 2019. Robust cloth warping via multi-scale patch adversarial loss for virtual try-on framework. In Proceedings of the ICCV Workshops.Google Scholar
Cross Ref
- [2] . 2018. A note on the inception score. In Proceedings of the ICML Workshops.Google Scholar
- [3] . 2020. CLOTH3D: Clothed 3D humans. In Proceedings of the ECCV.Google ScholarDigital Library
- [4] . 2018. Demystifying MMD GANs. In Proceedings of the ICLR.Google Scholar
- [5] . 1989. Principal warps: Thin-plate splines and the decomposition of deformations. IEEE Trans. PAMI 11, 6 (1989), 567–585. Google ScholarDigital Library
- [6] . 2017. Realtime multi-person 2D pose estimation using part affinity fields. In Proceedings of the CVPR.Google ScholarCross Ref
- [7] . 2019. Context-aware visual compatibility prediction. In Proceedings of the CVPR.Google ScholarCross Ref
- [8] . 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the CVPR.Google ScholarCross Ref
- [9] . 2018. Soft-gated warping-GAN for pose-guided person image synthesis. In Proceedings of the NeurIPS. Google ScholarDigital Library
- [10] . 2019. Towards multi-pose guided virtual try-on network. In Proceedings of the ICCV.Google ScholarCross Ref
- [11] . 2019. FW-GAN: Flow-navigated warping GAN for video virtual try-on. In Proceedings of the ICCV.Google ScholarCross Ref
- [12] . 2020. Fashion compatibility modeling through a multi-modal try-on-guided scheme. In Proceedings of the ACM SIGIR. Google ScholarDigital Library
- [13] . 2020. VITON-GT: An image-based virtual try-on model with geometric transformations. In Proceedings of the ICPR.Google Scholar
- [14] . 2019. DeepFashion2: A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images. In Proceedings of the CVPR.Google ScholarCross Ref
- [15] . 2014. Generative adversarial nets. In Proceedings of the NeurIPS. Google ScholarDigital Library
- [16] . 2012. Drape: Dressing any person. ACM Trans. Graph. 31, 4 (2012), 1–10. Google ScholarDigital Library
- [17] . 2018. DensePose: Dense human pose estimation in the wild. In Proceedings of the CVPR.Google ScholarCross Ref
- [18] . 2015. Where to buy it: Matching street clothing photos in online shops. In Proceedings of the ICCV. Google ScholarDigital Library
- [19] . 2014. Subspace clothing simulation using adaptive bases. ACM Trans. Graph. 33, 4 (2014), 1–9. Google ScholarDigital Library
- [20] . 2019. ClothFlow: A flow-based model for clothed person generation. In Proceedings of the ICCV.Google ScholarCross Ref
- [21] . 2018. VITON: An image-based virtual try-on network. In Proceedings of the CVPR.Google ScholarCross Ref
- [22] . 2017. GANs trained by a two time-scale update rule converge to a nash equilibrium. In Proceedings of the NeurIPS. Google ScholarDigital Library
- [23] . 2018. Creating capsule wardrobes from fashion images. In Proceedings of the CVPR.Google ScholarCross Ref
- [24] . 2019. Fit-me: Image-based virtual try-on with arbitrary poses. In Proceedings of the ICIP.Google ScholarCross Ref
- [25] . 2019. FashionOn: Semantic-guided image-based virtual try-on with detailed human and clothing information. In Proceedings of the ACM Multimedia. Google ScholarDigital Library
- [26] . 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the CVPR.Google ScholarCross Ref
- [27] . 2020. Do not mask what you do not need to mask: A parser-free virtual try-on. In Proceedings of the ECCV.Google ScholarDigital Library
- [28] . 2019. LA-VITON: A network for looking-attractive virtual try-on. In Proceedings of the ICCV Workshops.Google ScholarCross Ref
- [29] . 2020. SieveNet: A unified framework for robust image-based virtual try-on. In Proceedings of the WACV.Google ScholarCross Ref
- [30] . 2017. The conditional analogy GAN: Swapping fashion articles on people images. In Proceedings of the ICCV Workshops.Google ScholarCross Ref
- [31] . 2016. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the ECCV.Google ScholarCross Ref
- [32] . 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the CVPR.Google ScholarCross Ref
- [33] . 2020. Analyzing and improving the image quality of StyleGAN. In Proceedings of the CVPR.Google ScholarCross Ref
- [34] . 2015. Adam: A method for stochastic optimization. In Proceedings of the ICLR.Google Scholar
- [35] . 2019. Fashion retrieval via graph reasoning networks on a similarity pyramid. In Proceedings of the ICCV.Google ScholarCross Ref
- [36] . 2021. ShineOn: Illuminating design choices for practical video-based virtual clothing try-on. In Proceedings of the WACV Workshops.Google ScholarCross Ref
- [37] . 2021. VOGUE: Try-on by stylegan interpolation optimization. Retrieved from https://arXiv:2101.02285.Google Scholar
- [38] . 2019. Self-correction for human parsing. Retrieved from https://arXiv:1910.09777.Google Scholar
- [39] . 2015. Deep human parsing with active template regression. IEEE Trans. PAMI 37, 12 (2015), 2402–2414. Google ScholarDigital Library
- [40] . 2016. DeepFashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the CVPR.Google ScholarCross Ref
- [41] . 2019. Unsupervised part-based disentangling of object shape and appearance. In Proceedings of the CVPR.Google ScholarCross Ref
- [42] . 2018. Disentangled person image generation. In Proceedings of the CVPR.Google ScholarCross Ref
- [43] . 2020. Learning to dress 3D people in generative clothing. In Proceedings of the CVPR.Google ScholarCross Ref
- [44] . 2014. A complete system for garment segmentation and color classification. Mach. Vision Appl. 25, 4 (2014), 955–969. Google ScholarDigital Library
- [45] . 2020. CloTH-VTON: Clothing three-dimensional reconstruction for hybrid image-based virtual try-ON. In Proceedings of the ACCV.Google Scholar
- [46] . 2020. CP-VTON+: Clothing shape and texture preserving image-based virtual try-on. In Proceedings of the CVPR Workshops.Google Scholar
- [47] . 2020. Learning to transfer texture from clothing images to 3d humans. In Proceedings of the CVPR.Google ScholarCross Ref
- [48] . 2021. FashionSearch++: Improving consumer-to-shop clothes retrieval with hard negatives. In Proceedings of the Italian Information Retrieval Workshop.Google Scholar
- [49] . 2020. Image based virtual try-on network from unpaired data. In Proceedings of the CVPR.Google ScholarCross Ref
- [50] . 2007. Object retrieval with large vocabularies and fast spatial matching. In Proceedings of the CVPR.Google ScholarCross Ref
- [51] . 2017. ClothCap: Seamless 4D clothing capture and retargeting. ACM Trans. Graph. 36, 4 (2017), 1–15. Google ScholarDigital Library
- [52] . 2020. GarmentGAN: Photo-realistic adversarial fashion transfer. Retrieved from https://arXiv:2003.01894.Google Scholar
- [53] . 2018. SwapNet: Image based garment transfer. In Proceedings of the ECCV.Google ScholarCross Ref
- [54] . 2017. Convolutional neural network architecture for geometric matching. In Proceedings of the CVPR.Google ScholarCross Ref
- [55] . 2015. U-Net: Convolutional networks for biomedical image segmentation. In Proceedings of the MICCAI.Google ScholarCross Ref
- [56] . 2016. Improved techniques for training GANs. In Proceedings of the NeurIPS. Google ScholarDigital Library
- [57] . 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the ICLR.Google Scholar
- [58] . 2019. Compatibility modeling: Data and knowledge applications for clothing matching. Synth. Lect. Info. Conc. Retriev. Serv. 11, 3 (2019), 1–138.Google ScholarCross Ref
- [59] . 2016. Rethinking the inception architecture for computer vision. In Proceedings of the CVPR.Google ScholarCross Ref
- [60] . 2020. SIZER: A dataset and model for parsing 3D clothing and learning size sensitive 3D clothing. In Proceedings of the ECCV.Google ScholarDigital Library
- [61] . 2018. Learning type-aware embeddings for fashion compatibility. In Proceedings of the ECCV.Google ScholarCross Ref
- [62] . 2018. Toward characteristic-preserving image-based virtual try-on network. In Proceedings of the ECCV.Google ScholarCross Ref
- [63] . 2018. High-resolution image synthesis and semantic manipulation with conditional GANs. In Proceedings of the CVPR.Google ScholarCross Ref
- [64] . 2018. Attentive fashion grammar network for fashion landmark detection and clothing category classification. In Proceedings of the CVPR.Google ScholarCross Ref
- [65] . 2019. M2E-try on net: Fashion from model to everyone. In Proceedings of the ACM Multimedia. Google ScholarDigital Library
- [66] . 2017. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. Retrieved from https://arXiv:1708.07747.Google Scholar
- [67] . 2013. Paper doll parsing: Retrieving similar styles to parse clothing items. In Proceedings of the ICCV. Google ScholarDigital Library
- [68] . 2020. Towards photo-realistic virtual try-on by adaptively generating-preserving image content. In Proceedings of the CVPR.Google ScholarCross Ref
- [69] . 2019. Generating high-resolution fashion model images wearing custom outfits. In Proceedings of the ICCV Workshops.Google ScholarCross Ref
- [70] . 2019. VTNFP: An image-based virtual try-on network with body and clothing feature preservation. In Proceedings of the ICCV.Google ScholarCross Ref
- [71] . 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the CVPR.Google ScholarCross Ref
- [72] . 2020. Deep Fashion3D: A dataset and benchmark for 3D garment reconstruction from single images. In Proceedings of the ECCV.Google ScholarDigital Library
Index Terms
- Transform, Warp, and Dress: A New Transformation-guided Model for Virtual Try-on
Recommendations
Per Garment Capture and Synthesis for Real-time Virtual Try-on
UIST '21: The 34th Annual ACM Symposium on User Interface Software and TechnologyVirtual try-on is a promising application of computer graphics and human computer interaction that can have a profound real-world impact especially during this pandemic. Existing image-based works try to synthesize a try-on image from a single image of ...
Dress Code: High-Resolution Multi-category Virtual Try-On
Computer Vision – ECCV 2022AbstractImage-based virtual try-on strives to transfer the appearance of a clothing item onto the image of a target person. Prior work focuses mainly on upper-body clothes (e.g. t-shirts, shirts, and tops) and neglects full-body or lower-body items. This ...
Customized 3D Clothes Modeling for Virtual Try-on System based on Multiple Kinects
GRAPP 2016: Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications: Volume 1: GRAPPMost existing 3D virtual try-on systems put clothes designed in one environment on a human captured in another environment, which cause the mismatching brightness problem. And also typical 3D clothes modeling starts with manually designed 2D patterns, ...
Comments