Skip to main content
SpringerLink
Log in

DLOW: Domain Flow and Applications

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

In this work, we present a domain flow generation (DLOW) model to bridge two different domains by generating a continuous sequence of intermediate domains flowing from one domain to the other. The benefits of our DLOW model are twofold. First, it is able to transfer source images into a domain flow, which consists of images with smoothly changing distributions from the source to the target domain. The domain flow bridges the gap between source and target domains, thus easing the domain adaptation task. Second, when multiple target domains are provided for training, our DLOW model is also able to generate new styles of images that are unseen in the training data. The new images are shown to be able to mimic different artists to produce a natural blend of multiple art styles. Furthermore, for the semantic segmentation in the adverse weather condition, we take advantage of our DLOW model to generate images with gradually changing fog density, which can be readily used for boosting the segmentation performance when combined with a curriculum learning strategy. We demonstrate the effectiveness of our model on benchmark datasets for different applications, including cross-domain semantic segmentation, style generalization, and foggy scene understanding. Our implementation is available at https://github.com/ETHRuiGong/DLOW.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  • Almahairi, A., Rajeswar, S., Sordoni, A., Bachman, P., & Courville, A. (2018). Augmented CycleGAN: Learning many-to-many mappings from unpaired data. In ICML.

  • Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein gan. arXiv:1701.07875

  • Baktashmotlagh, M., Harandi, M. T., Lovell, B. C., & Salzmann, M. (2013). Unsupervised domain adaptation by domain invariant projection. In ICCV.

  • Baktashmotlagh, M., Harandi, M. T., Lovell, B. C., & Salzmann, M. (2014). Domain adaptation on the statistical manifold. In CVPR.

  • Chang, W. L., Wang, H. P., Peng, W. H., & Chiu, W. C. (2019). All about structure: Adapting structural information across domains for boosting semantic segmentation. In IEEE conference on computer vision and pattern recognition (CVPR).

  • Chen, Y., Li, W., & Van Gool, L. (2018). Road: Reality oriented adaptation for semantic segmentation of urban scenes. In CVPR.

  • Chen, Y., Li, W., Chen, X., & Van Gool, L. (2018). Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach. arXiv:1812.05040.

  • Chen, Y., Li, W., Sakaridis, C., Dai, D., & Van Gool, L. (2018). Domain adaptive faster R-CNN for object detection in the wild. In CVPR.

  • Chen, M., Xue, H., & Cai, D. (2019). Domain adaptation for semantic segmentation with maximum squares loss. In ICCV.

  • Chen, L. C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4), 834–848.

    Article  Google Scholar 

  • Choi, Y., Choi, M., Kim, M., Ha, J. W., Kim, S., & Choo, J. (2018). StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In CVPR.

  • Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., et al. (2016). The cityscapes dataset for semantic urban scene understanding. In CVPR.

  • Cui, Z., Li, W., Xu, D., Shan, S., Chen, X., & Li, X. (2014). Flowing on Riemannian manifold: Domain adaptation by shifting covariance. IEEE Transactions on Cybernetics, 44(12), 2264–2273.

    Article  Google Scholar 

  • Dai, D., Sakaridis, C., Hecker, S., & Van Gool, L. (2019). Curriculum model adaptation with synthetic and real data for semantic foggy scene understanding. In International Journal of Computer Vision, pp. 1–23.

  • Du, L., Tan, J., Yang, H., Feng, J., Xue, X., Zheng, Q., et al. (2019). SSF-DAN: Separated semantic feature based domain adaptation network for semantic segmentation. In ICCV.

  • Dundar, A., Liu, M. Y., Wang, T. C., Zedlewski, J., & Kautz, J. (2018). Domain stylization: A strong, simple baseline for synthetic to real image domain adaptation. arXiv:1807.09384

  • Fernando, B., Habrard, A., Sebban, M., & Tuytelaars, T. (2013). Unsupervised visual domain adaptation using subspace alignment. In ICCV.

  • Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016). Virtual worlds as proxy for multi-object tracking analysis. In CVPR.

  • Ganin, Y., & Lempitsky, V. S. (2015). Unsupervised domain adaptation by backpropagation. In ICML.

  • Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? the Kitti vision benchmark suite. In CVPR.

  • Ghifary, M., Bastiaan Kleijn, W., Zhang, M., & Balduzzi, D. (2015). Domain generalization for object recognition with multi-task autoencoders. In ICCV.

  • Gong, R., Li, W., Chen, Y., & Van Gool, L. (2019). DLOW: Domain flow for adaptation and generalization. In CVPR.

  • Gong, B., Shi, Y., Sha, F., & Grauman, K. (2012). Geodesic flow kernel for unsupervised domain adaptation. In CVPR.

  • Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., et al. (2014). Generative adversarial nets. In NIPS.

  • Gopalan, R., Li, R., & Chellappa, R. (2011). Domain adaptation for object recognition: An unsupervised approach. In ICCV.

  • Hahner, M., Dai, D., Sakaridis, C., Zaech, J. N., & Van Gool, L. (2019). Semantic understanding of foggy scenes with purely synthetic data. In ITSC.

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In CVPR.

  • He, Z., Zuo, W., Kan, M., Shan, S., & Chen, X. (2017). Arbitrary facial attribute editing: Only change what you want. arXiv:1711.10678.

  • He, K., Sun, J., & Tang, X. (2010). Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12), 2341–2353.

    Google Scholar 

  • Hoffman, J., Tzeng, E., Park, T., Zhu, J. Y., Isola, P., Saenko, K., et al. (2018). CyCADA: Cycle-consistent adversarial domain adaptation. In ICML.

  • Hoffman, J., Wang, D., Yu, F., & Darrell, T. (2016). FCNs in the wild: Pixel-level adversarial and constraint-based adaptation. arXiv:1612.02649

  • Hong, W., Wang, Z., Yang, M., & Yuan, J. (2018). Conditional generative adversarial network for structured domain adaptation. In CVPR.

  • Huang, X., & Belongie, S. (2017). Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV.

  • Huang, H., Huang, Q., & Krähenbühl, P. (2018). Domain transfer through deep activation matching. In ECCV.

  • Huang, X., Liu, M. Y., Belongie, S., & Kautz, J. (2018). Multimodal unsupervised image-to-image translation. In ECCV.

  • Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In CVPR.

  • Jhuo, I. H., Liu, D., Lee, D., & Chang, S. F. (2012). Robust visual domain adaptation with low-rank reconstruction. In CVPR.

  • Kim, T., Cha, M., Kim, H., Lee, J. K., & Kim, J. (2017). Learning to discover cross-domain relations with generative adversarial networks. In ICML.

  • Kodirov, E., Xiang, T., Fu, Z., & Gong, S. (2015). Unsupervised domain adaptation for zero-shot learning. In ICCV.

  • Lample, G., Zeghidour, N., Usunier, N., Bordes, A., DENOYER, L., et al. (2017). Fader networks: Manipulating images by sliding attributes. In NIPS.

  • Lee, H. Y., Tseng, H. Y., Huang, J. B., Singh, M. K., & Yang, M. H. (2018). Diverse image-to-image translation via disentangled representations. In ECCV.

  • Li, H., Pan, S. J., Wang, S., & Kot, A. C. (2018a). Domain generalization with adversarial feature learning. In CVPR.

  • Li, Y., Tian, X., Gong, M., Liu, Y., Liu, T., Zhang, K., & Tao, D. (2018c). Deep domain generalization via conditional invariant adversarial networks. In ECCV.

  • Lin, G., Milan, A., Shen, C., & Reid, I. (2017). Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In CVPR.

  • Lin, J., Xia, Y., Qin, T., Chen, Z., & Liu, T. Y. (2018). Conditional image-to-image translation. In CVPR.

  • Liu, M. Y., Breuel, T., & Kautz, J. (2017). Unsupervised image-to-image translation networks. In NIPS.

  • Li, W., Xu, Z., Xu, D., Dai, D., & Van Gool, L. (2018b). Domain generalization and adaptation using low rank exemplar SVMs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(5), 1114–1127.

    Article  Google Scholar 

  • Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In CVPR.

  • Lu, Y., Tai, Y.W., & Tang, C.K. (2017). Conditional cyclegan for attribute guided face image generation. arXiv:1705.09966

  • Luo, Y., Zheng, L., Guan, T., Yu, J., & Yang, Y. (2018). Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. arXiv:1809.09478

  • Luo, Y., Zheng, L., Guan, T., Yu, J., & Yang, Y. (2019). Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In CVPR.

  • Motiian, S., Piccirilli, M., Adjeroh, D.A., & Doretto, G. (2017). Unified deep supervised domain adaptation and generalization. In ICCV.

  • Muandet, K., Balduzzi, D., & Schölkopf, B. (2013). Domain generalization via invariant feature representation. In ICML.

  • Murez, Z., Kolouri, S., Kriegman, D., Ramamoorthi, R., & Kim, K. (2017). Image to image translation for domain adaptation. arXiv:1712.00479.

  • Niu, L., Li, W., & Xu, D. (2015). Multi-view domain generalization for visual recognition. In ICCV.

  • Niu, L., Li, W., & Xu, D. (2015). Visual recognition by learning from web data: A weakly supervised domain generalization approach. In CVPR.

  • Pan, X., Luo, P., Shi, J., & Tang, X. (2018). Two at once: Enhancing learning and generalization capacities via ibn-net. In ECCV.

  • Pan, F., Shin, I., Rameau, F., Lee, S., & Kweon, I.S. (2020). Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR).

  • Peng, X., Usman, B., Kaushik, N., Wang, D., Hoffman, J., Saenko, K., et al. (2018). Visda: A synthetic-to-real benchmark for visual domain adaptation. In CVPR Workshops.

  • Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016). Playing for data: Ground truth from computer games. In ECCV.

  • Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016). The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR.

  • Saito, K., Watanabe, K., Ushiku, Y., & Harada, T. (2017). Maximum classifier discrepancy for unsupervised domain adaptation. arXiv:1712.02560.

  • Sakaridis, C., Dai, D., Hecker, S., & Van Gool, L. (2018a). Model adaptation with synthetic and real data for semantic dense foggy scene understanding. In ECCV.

  • Sakaridis, C., Dai, D., & Van Gool, L. (2018b). Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 126(9), 973–992.

    Article  Google Scholar 

  • Sakaridis, C., Dai, D., & Van Gool, L. (2020). Map-guided curriculum domain adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence,. https://doi.org/10.1109/TPAMI.2020.3045882.

    Article  Google Scholar 

  • Saleh, F. S., Aliakbarian, M. S., Salzmann, M., Petersson, L., & Alvarez, J. M. (2018). Effective use of synthetic data for urban scene semantic segmentation. In ECCV.

  • Sankaranarayanan, S., Balaji, Y., Jain, A., Lim, S. N., & Chellappa, R. (2017). Unsupervised domain adaptation for semantic segmentation with GANs. arXiv:1711.06969

  • Sankaranarayanan, S., Balaji, Y., Jain, A., Lim, S. N., & Chellappa, R. (2018). Learning from synthetic data: Addressing domain shift for semantic segmentation. In CVPR.

  • Tsai, Y. H., Hung, W. C., Schulter, S., Sohn, K., Yang, M. H., & Chandraker, M. (2018). Learning to adapt structured output space for semantic segmentation. In CVPR.

  • Tsai, Y. H., Sohn, K., Schulter, S., & Chandraker, M. (2019). Domain adaptation for structured output via discriminative patch representations. In ICCV.

  • Tzeng, E., Hoffman, J., Saenko, K., & Darrell, T. (2017). Adversarial discriminative domain adaptation. In CVPR.

  • Vu, T. H., Jain, H., Bucher, M., Cord, M., & Pérez, P. (2019). Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR.

  • Wang, T. C., Liu, M. Y., Zhu, J. Y., Tao, A., Kautz, J., & Catanzaro, B. (2018). High-resolution image synthesis and semantic manipulation with conditional gans. In CVPR.

  • Wulfmeier, M., Bewley, A., & Posner, I. (2018). Incremental adversarial domain adaptation for continually changing environments. In ICRA.

  • Yang, D., & Sun, J. (2018). Proximal dehaze-net: A prior learning-based deep network for single image dehazing. In ECCV.

  • Yi, Z., Zhang, H. R., Tan, P., & Gong, M. (2017). Dualgan: Unsupervised dual learning for image-to-image translation. In ICCV.

  • Yu, F., Koltun, V., & Funkhouser, T. (2017). Dilated residual networks. In CVPR.

  • Yu, F., Xian, W., Chen, Y., Liu, F., Liao, M., Madhavan, V., & Darrell, T. (2018). Bdd100k: A diverse driving video database with scalable annotation tooling. arXiv:1805.04687

  • Zendel, O., Honauer, K., Murschitz, M., Steininger, D., & Fernandez Dominguez, G. (2018). Wilddash—creating hazard-aware benchmarks. In ECCV.

  • Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018). mixup: Beyond empirical risk minimization. In ICLR.

  • Zhang, Y., David, P., & Gong, B. (2017). Curriculum domain adaptation for semantic segmentation of urban scenes. In ICCV.

  • Zhang, W., Ouyang, W., Li, W., & Xu, D. (2018). Collaborative and adversarial network for unsupervised domain adaptation. In CVPR.

  • Zhang, Y., Qiu, Z., Yao, T., Liu, D., & Mei, T. (2018). Fully convolutional adaptation networks for semantic segmentation. In CVPR.

  • Zhang, Y., Zhang, Y., & Cai, W. (2018). A unified framework for generalizable style transfer: Style and content separation. arXiv:1806.05173.

  • Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017). Pyramid scene parsing network. In CVPR.

  • Zheng, Z., Wu, Y., Han, X., & Shi, J. (2020). Forkgan: Seeing into the rainy night. In The IEEE European Conference on Computer Vision (ECCV).

  • Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV.

  • Zhu, J. Y., Zhang, R., Pathak, D., Darrell, T., Efros, A. A., Wang, O., & Shechtman, E. (2017). Toward multimodal image-to-image translation. In NIPS.

  • Zhu, X., Zhou, H., Yang, C., Shi, J., & Lin, D. (2018). Penalizing top performers: Conservative loss for semantic segmentation adaptation. arXiv:1809.00903.

  • Zou, Y., Yu, Z., Kumar, B. V., & Wang, J. (2018). Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In ECCV.

Download references

Acknowledgements

This work is partially supported by the Major Project for New Generation of AI under Grant Agreement No. 2018AAA0100400 and China Postdoctoral Science Foundation (NO. 2019TQ0051). This research has also received funding from the EU Horizon 2020 research and innovation programme under grant agreement No. 820434. The authors gratefully acknowledge the support by armasuisse, and the support by Toyota via the research project TRACE-Zurich. We also thank Amazon Web Services (AWS) for providing GPU credits for our research.

Author information

Authors and Affiliations

  1. ETH Zurich, Zurich, Switzerland

    Rui Gong, Yuhua Chen, Dengxin Dai & Luc Van Gool

  2. University of Electronic Science and Technology of China, Chengdu, China

    Wen Li

  3. KU Leuven, Leuven, Belgium

    Luc Van Gool

  4. MPI for Informatics, Saarbrucken, Germany

    Dengxin Dai

Authors
  1. Rui Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuhua Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dengxin Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Luc Van Gool
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wen Li.

Additional information

Communicated by Kristen Grauman.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gong, R., Li, W., Chen, Y. et al. DLOW: Domain Flow and Applications. Int J Comput Vis 129, 2865–2888 (2021). https://doi.org/10.1007/s11263-021-01496-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-021-01496-2

Keywords

Search

Navigation