Loading [a11y]/accessibility-menu.js
Temporally Consistent Unpaired Multi-domain Video Translation by Contrastive Learning | IEEE Conference Publication | IEEE Xplore

Temporally Consistent Unpaired Multi-domain Video Translation by Contrastive Learning


Abstract:

Unpaired multi-domain video-to-video translation is an attractive solution for diverse video translation, which has to deal with not only unpaired data but also spatio-te...Show More

Abstract:

Unpaired multi-domain video-to-video translation is an attractive solution for diverse video translation, which has to deal with not only unpaired data but also spatio-temporal inconsistency. Most current video-to-video translation models based on cycle consistency introduce optical flow as motion information to achieve spatio-temporal consistency. However, the warping of the optical flow generates a meaningless invisible regions outside the field of view, which is produced by stretching the edge area of the original image and negatively affects the model training. In this work, we propose the Contrastive learning for Multi-domain Video-to-video Translation to replace the cycle consistency, in order to avoid the affects of invisible regions. Specifically, we first introduce synthetic optical flow to maintain spatio-temporal consistency. Then, we use the attention mechanism as the selection principle of positive and negative, and eliminate the features of invisible regions by sorting the feature entropy. Frequency domain information is also used to maintain individual consistency. Experiments on the public datasets Viper and INIT show that our methods is universal across multiple datasets and achieves state-of-the-art performance in generating temporally consistent multi-domain videos.
Date of Conference: 30 June 2024 - 05 July 2024
Date Added to IEEE Xplore: 09 September 2024
ISBN Information:

ISSN Information:

Conference Location: Yokohama, Japan

Funding Agency:


Contact IEEE to Subscribe

References

References is not available for this document.