LGVTON: a landmark guided approach for model to person virtual try-on

Abstract

In this paper, we propose a Landmark Guided Virtual Try-On (LGVTON) method for clothes, which aims to solve the problem of clothing trials on e-commerce websites. Given the images of two people: a person and a model, it generates a rendition of the person wearing the clothes of the model. This is useful considering the fact that on most e-commerce websites images of only clothes are not usually available. We follow a three-stage approach to achieve our objective. In the first stage, LGVTON warps the clothes of the model using a Thin-Plate Spline (TPS) based transformation to fit the person. Unlike previous TPS-based methods, we use the landmarks (of human and clothes) to compute the TPS transformation. This enables the warping to work independently of the complex patterns, such as stripes, florals, and textures, present on the clothes. However, this computed warp may not always be very precise. We, therefore, further refine it in the subsequent stages with the help of a mask generator (Stage 2) and an image synthesizer (Stage 3) modules. The mask generator improves the fit of the warped clothes, and the image synthesizer ensures a realistic output. To tackle the problem of lack of paired training data, we resort to a self-supervised training strategy. Here paired data refers to the image pair of model and person wearing the same cloth. We compare LGVTON with four existing methods on two popular fashion datasets namely MPV and DeepFashion using two performance measures, FID (Fréchet Inception Distance) and SSIM (Structural Similarity Index). The proposed method in most cases outperforms the state-of-the-art methods.

Appendix

In this appendix, first, we discuss Thin-plate spline (TPS) transformation. Second, we give a detailed study on the idea of the correlation layer that has been used in the PGWM of LGVTON. Third, we give a detailed ablation study to clarify in detail the effectiveness of each of the components of LGVTON. Fourth, we provide the implementation details of our network architecture. In addition, we give some results of LGVTON on DeepFashion [23] and MPV dataset [7] in Figs. 24, 25, 26, and 27.

1.1 Thin Plate Spline (TPS)

Given a set of pair of source and target landmarks {($\mathbf {r}_j$, $\mathbf {t}_j$); j = 1, . . . , N}, a polyharmonic smoothing spline fits a function $f(\cdot )$ between the sets {$\mathbf {r}_j$; j = 1, . . . , N} and {$\mathbf {t}_j$; j = 1, . . . , N} that minimizes the following objective functional [36]:

$$\begin{aligned} \tau [f] = \sum _{j=1}^N \Vert f(\mathbf {r}_j)-\mathbf {t}_j\Vert _2^2 + \iint \limits _{\mathbb {R}^2} \Vert \nabla ^m f\Vert _2^2dx\ dy \end{aligned}$$

(7)

where $\nabla ^m f$ is the vector of all m-th order partial derivatives of f that enforces smoothness on f. In our case, $\mathbf {r}_j, \mathbf {t}_j\in \mathbb {N}^{2}$, or in other words, $\mathbf {r}\equiv (r^x, r^y)$ and $\mathbf {t}\equiv (t^x, t^y)$. A special case of polyharmonic smoothing spline where m = 2, called thin-plate spline (TPS), is proposed by Duchon [9]. It minimizes the following objective functional based on eq. (7):

$$\begin{aligned} \tau [f] = \sum _{j=1}^N \Vert f(\mathbf {r}_j)-\mathbf {t}_j\Vert _2^2 + \iint \limits _{\mathbb {R}^2} [f_{xx}^2 + 2f_{xy}^2 + f_{yy}^2]dx\ dy \end{aligned}$$

(8)

The physical analogy to the concept of the thin-plate comes from the minimization of second-order gradients ($f_{xx}$, $f_{xy}$, $f_{yy}$) in its objective, which restricts bending and enforces smoothness in the TPS fit. This is similar to the effect of physical rigidity in the thin metal plate.

A closed form solution of it as proposed in [42] is given by

$$\begin{aligned} \mathbf {t} = f(\mathbf {r}) = \mathbf {a}_0 + \mathbf {a}_1 r^x + \mathbf {a}_2 r^y + \sum _{j=1}^N \mathbf {c}_j\phi (\Vert \mathbf {r} - \mathbf {r}_j\Vert _2), \end{aligned}$$

(9)

where $\mathbf {a}_0$, $\mathbf {a}_1$, $\mathbf {a}_2$, $\mathbf {c}_j$ are parameters with dimension equal to the dimension of landmarks, which is 2 in our case.

The TPS is represented in terms of radial basis function (RBF) $\phi$. Given a set of control points $\mathbf {r}_{j}$, $j=1, ..., N$ a RBF maps a given point $\mathbf {p}$ to a new location $g(\mathbf {p})$, represented by,

$$\begin{aligned} g(\mathbf {p}) = \sum _{j=1}^N \mathbf {c}_j\phi (\Vert \mathbf {p} - \mathbf {r}_j\Vert _2) \end{aligned}$$

(10)

The radial basis kernel used in TPS is $\phi (\mathbf {r})$ = ($\mathbf {r}^2$ ln $\mathbf {r}$).

When the values of $\mathbf {t}_j$ are noisy (due to landmark localization errors), which is a very obvious situation in practice, the interpolation requirement is relaxed by means of regularization. This reduces the problem to an approximation problem [6, 39]. This is obtained by minimizing

$$\begin{aligned} H[f] = \sum _{j=1}^N \Vert f(\mathbf {r}_j)-\mathbf {t}_j\Vert _2^2 + \lambda \iint \limits _{\mathbb {R}^2} [f_{xx}^2 + 2f_{xy}^2 + f_{yy}^2]dx\ dy, \end{aligned}$$

(11)

where $\lambda$ is the regularization parameter, which is a +ve scalar. Here, $\lambda$ determines the relative weight between the approximation behavior and the smoothness of the transformation. In the limiting case, $\lambda$ = 0, we get an interpolation transformation. If the value of $\lambda$ is small we get a good approximation behavior; for a higher value of $\lambda$, we get a very smooth transformation function. But then the local deformations determined by the set of landmarks are maintained poorly.

1.2 Correlation layer

Correlation is a statistical technique that can show whether and how strongly pairs of variables are related. We use the idea of correlation in our Pose Guided Warping Module (PGWM). Here we model the warping of the source clothes as a function of the correlation between the human landmarks of the model and the person, and fashion landmarks of the source clothes. The reason for using the correlation is that when a person wears clothes it gets molded according to his body shape and pose. Now, here we are transferring the clothes from model to person hence, the clothes have to undergo deformation according to the way the body shape and pose changes from model to person. This change is modeled by correlation.

Coming into more detail, given the data $f_a, f_b \in \mathbb {R}^{1\times l}$, the correlation layer produces the correlation map $C_{ab} \in \mathbb {R}^{l \times l}$, where,

$$\begin{aligned} C_{ab} = {f_a}^{T}{f_b}. \end{aligned}$$

(12)

Therefore a correlation map basically contains pairwise similarity of all the values of $f_a$ and $f_b$. This is useful in establishing relationships among different landmarks of model and person to model the warping.

1.3 Ablation study

In this subsection, we conduct an in-depth qualitative study on the significance of each component of LGVTON.

1.3.1 Utility of different loss functions for training ISM

We run a comparative study on the effect of different loss functions used to train ISM. Keeping the other settings same, we train 3 different instances of ISM with different combinations of loss functions as shown in Fig. 19.

Fig. 19

Effectiveness of different losses during training the Image Synthesizer module (ISM). (1) LGVTON (non-GAN, DSSIM loss) - a non-GAN variant of ISM trained with DSSIM loss, (2) LGVTON (cGAN, DSSIM loss) - cGAN with the generator trained with DSSIM loss, (3) LGVTON - similar to (2) but the generator is trained with DSSIM and VGG perceptual loss both. Notice the clarity of output increases from LGVTON (non-GAN, DSSIM loss) to LGVTON

As GAN tries to approximate the data distribution, so it generates better output than its non-GAN variant [11]. This is evident in the results of the non-GAN variant of LGVTON where the ISM is trained with DSSIM loss. We call this variant LGVTON (non-GAN, DSSIM loss) and the corresponding GAN variant LGVTON (cGAN, DSSIM loss). Comparing the results of LGVTON(cGAN, DSSIM loss) and LGVTON (ours) (a cGAN, where the generator is trained with DSSIM and perceptual loss both), it can be observed that the result improves in the presence of VGG perceptual loss.

1.3.2 Significance of fashion landmarks in PGWM

We conduct a study on PGWM when the warping is done with human landmarks only instead of both human and fashion landmarks. Figure 20 shows two cases portraying the effectiveness of fashion landmarks around the collar and hem. Having only human landmarks might serve the purpose but at the cost of an increase in the amount of warping glitches, which is tackled to some extent in ISM with the help of the mask generated by the MGM. The scores reported in Tables 3 and 4 (LGVTON (w/o fashion landmarks)) shows that without fashion landmarks the performance of LGVTON degrades. However, for obvious reasons, the performance degrades even more if the support of the MGM is also removed (observe the score of LGVTON (w/o fashion landmarks, w/o MGM) in Tables 3 and 4).

Fig. 20

Role of different fashion landmarks in predicting target warp of the model clothes in PGWM. The figure shows the utility of fashion landmarks around the collar (left) and hem (right) respectively. For a better understanding of the viewer, instead of showing the generated warped clothes image only, we show the overlay of it on the person image. (1) Person, (2) model, (3) clothes warped using only human landmarks, (4) transformed locations of fashion landmarks of the model clothes, obtained by the corresponding transformation function of (3), (5) clothes warped by PGWM using both human and fashion landmarks, (6) fashion landmarks predicted in PGWM, which is observed to be more accurate than (4). This results in better warping of the model clothes as shown in (5) in comparison to that in (3)

1.3.3 Effectiveness of correlation layer in PGWM

We study the effectiveness of the correlation layer in the fashion landmark predictor network of PGWM (refer to Fig. 21). The presence of the correlation layer establishes the relationship between the human poses of the model and the person, which in turn assists in predicting better estimates of fashion landmarks of the target warp clothes.

Fig. 21

A study on the effectiveness of correlation layer in fashion landmark predictor network $\mathcal {F}$ of PGWP in LGVTON. (a) Person image, (b) Model image, (c, e) predicted locations of fashion landmarks and warp clothes generated by PGWM when $\mathcal {F}$ is trained w/o correlation layer and with correlation layer respectively. (d, f) final VTON result for the warp cloths shown in (c) and (e) respectively. We observe that the results in (f) are better than that in (d), which justifies the potency of the correlation layer in PGWM

1.3.4 Studying the role of target mask as input to ISM

For understanding the effect of the target mask, we train an instance of ISM without providing the target mask as input. Now it is observed that w/o target mask, the network can not identify the warping glitches. This can be verified from Fig. 22h where the effect of the warping glitch is propagated to the output. However, when a target mask is given as input, ISM can identify the areas of warping glitch and takes the necessary action according to the type of the glitch. The reason being the variety of clothing types makes the network confused to distinguish a warping glitch from the design of the clothes, e.g., in the first two rows, the inappropriate estimation of $c'_{flm}$ causes the sleeves to be stretched more outwards. While that in Fig. 22j is not observed as the network removes those areas of extra stretch and replaces them with the background. In the example of the third row Fig. 22h, the effect of warping glitch exposes some body parts near the right neckline of the person (better viewed when zoomed in), while this gets filled with a clothes texture and color in Fig. 22j.

Fig. 22

Effectiveness of target mask generated by our mask generator module (MGM). We notice significant differences (as shown in (f), (g)) between the mask (d) of the predicted warped clothes (c) and the mask (e) predicted by MGM corresponding to the warp clothes (c). (h) shows the final try-on result contains artifacts when ISM is trained without the target mask. To show the effectiveness of the target mask we also show the result where instead of (e) we give (d) as target mask input to ISM; which is shown in (i). As we observe the artifacts still remain in (i). Whereas, when (e) is given to ISM as target mask, the result (j) improves, e.g., the areas with pixel value 1 in (f) are filled with necessary color and texture details, and those in (g) are replaced with background details resulting in a better VTON result. Note that the hole in the warp clothes in row 2 is not due to a warping glitch. This is due to inaccurate human parsing. However, LGVTON handles this also

1.4 Implementation details

1.4.1 Pose Guided Warping Module (PGWM)

The dense neural network in fashion landmark predictor network $\mathcal {F}$ contains 6 consecutive dense layers with 900, 800, 600, 500, 250, 100 nodes respectively, each having activation function tanh. Finally, the output layer has 12 nodes (for 6 fashion landmarks each with 2 coordinate values x and y) and sigmoid as the activation function.

1.4.2 Mask Generator Module (MGM)

The architecture of MGM is that of an hourglass network [30]. Generally, the processing of clothes requires identifying the different parts of them to establishing a semantic understanding of their structure. A well-known network that suits this requirement is the hourglass network [30].

An hourglass network is a CNN (Convolutional Neural Network) that captures features at various scales and is effective for analyzing spatial relationships among different parts of the input. Multiple of these hourglass networks can be stacked together with intermediate supervision for making it deeper. However, for our purpose, the stack size of 1 is found to be sufficient. An overview of the architecture of one hourglass network is given in Fig. 23.

Fig. 23

Overview of an hourglass network (taken from the original paper [30]), where each block represents a residual module

The network is called hourglass due to its top-down, bottom-up architecture, which matches with the shape of an hourglass. It uses convolution and max pooling layers to downscale features to a low resolution. After that, the top-down sequence of upsampling begins, and at each resolution, the corresponding features from downsampling parts are added as skip connections. However, it applies more convolutions on the features of skip connections and then does element-wise addition of the two sets of features.

1.4.3 Image Synthesizer Module (ISM)

The hourglass network in G (the generator network of ISM) has a stack size of 1. The convolution layer generating $I_m$ in G has a kernel of size 1 $\times$ 1 with $L_{1}$ regularization and sigmoid activation function. For training ISM, we alternate between 3 steps of generator training and 1 step of discriminator training. It is trained with the same settings of Adam as our PGWM. For DSSIM loss, the kernel size taken is 3 $\times$ 3. The discriminator D is a patchGAN discriminator [17]. Instead of classifying the whole image as real or fake, it classifies each patch of the image, where the patch size is much smaller than the input image size. Hence pixels separated by more than a patch diameter gets modeled independently. This makes it work like a texture/style loss as discussed in [17], which helps to keep better texture in the final output image. Existing works have shown the efficacy of patchGAN [17, 47] in image-based problems. For human parsing, we used the human parsing network proposed by [10] pretrained on the LIP dataset [10]. The dataset contains 19 part labels, 6 labels for body parts, and 13 for clothing categories. During our quantitative analysis we used the models of CP-VTON, VTNFP, MG-VTON pretrained on MPV [7] dataset. For VITON we used the VITON dataset pretrained model weights provided in its official implementations. Some more results of this work on DeepFashion and MPV datasets are shown in Figs. 24, 25, 26, and 27.

Fig. 24

Our results on DeepFashion [23] dataset. Image at position (i,j) represents the result of VTON when person at $j^{th}$ column wears the clothes of the model at the $i^{th}$ row

Fig. 25

Our results on MPV [7] dataset. Image at position (i,j) represents the result of VTON when person at $j^{th}$ column wears the clothes of the model at the $i^{th}$ row

Fig. 26

Our results on MPV dataset for different model and person combinations

Fig. 27

Our results on MPV dataset for different model and person combinations