Skip to main content
Springer Nature Link
Log in

Online Domain Adaptation for Semantic Segmentation in Ever-Changing Conditions

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13694))

Included in the following conference series:

Abstract

Unsupervised Domain Adaptation (UDA) aims at reducing the domain gap between training and testing data and is, in most cases, carried out in offline manner. However, domain changes may occur continuously and unpredictably during deployment (e.g. sudden weather changes). In such conditions, deep neural networks witness dramatic drops in accuracy and offline adaptation may not be enough to contrast it. In this paper, we tackle Online Domain Adaptation (OnDA) for semantic segmentation. We design a pipeline that is robust to continuous domain shifts, either gradual or sudden, and we evaluate it in the case of rainy and foggy scenarios. Our experiments show that our framework can effectively adapt to new domains during deployment, while not being affected by catastrophic forgetting of the previous domains.

T. Panagiotakopoulos and L. Härenstam-Nielsen—Part of the work carried out while at Univrses.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, pp. 41–48. Association for Computing Machinery, New York (2009). https://doi.org/10.1145/1553374.1553380

  2. Bobu, A., Hoffman, J., Tzeng, E., Darrell, T.: Adapting to continuously shifting domains. In: ICLR 2018 Workshop Program Chairs (2018). https://openreview.net/forum?id=BJsBjPJvf

  3. Cardace, A., Zama Ramirez, P., Salti, S., Di Stefano, L.: Shallow features guide unsupervised domain adaptation for semantic segmentation at class boundaries. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1160–1170, January 2022

    Google Scholar 

  4. Chang, W.G., You, T., Seo, S., Kwak, S., Han, B.: Domain-specific batch normalization for unsupervised domain adaptation. CoRR (2019). http://arxiv.org/abs/1906.03950

  5. Chen, C., et al.: Progressive feature alignment for unsupervised domain adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 627–636 (2019)

    Google Scholar 

  6. Chen, L.-C., et al.: Naive-student: leveraging semi-supervised learning in video sequences for urban scene segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 695–714. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_40

    Chapter  Google Scholar 

  7. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2017)

    Article  Google Scholar 

  8. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2018). https://doi.org/10.1109/tpami.2017.2699184, http://dx.doi.org/10.1109/TPAMI.2017.2699184

  9. Chen, Y.H., Chen, W.Y., Chen, Y.T., Tsai, B.C., Wang, Y.C.F., Sun, M.: No more discrimination: cross city adaptation of road scene segmenters. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2011–2020. IEEE (2017). https://doi.org/10.1109/ICCV.2017.220, http://ieeexplore.ieee.org/document/8237482/

  10. Cicek, S., Soatto, S.: Unsupervised domain adaptation via regularized conditional alignment. CoRR (2019). http://arxiv.org/abs/1905.10885

  11. Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding (2016)

    Google Scholar 

  12. Dundar, A., Liu, M.Y., Yu, Z., Wang, T.C., Zedlewski, J., Kautz, J.: Domain stylization: a fast covariance matching framework towards domain adaptation. IEEE Trans. Pattern Anal. Mach. Intell. p. 1 (2020). https://doi.org/10.1109/TPAMI.2020.2969421

  13. Furlanello, T., Lipton, Z., Tschannen, M., Itti, L., Anandkumar, A.: Born again neural networks. In: International Conference on Machine Learning, pp. 1607–1616. PMLR (2018)

    Google Scholar 

  14. Ganin, Y., et al.: Domain-adversarial training of neural networks. The J. Mach. Learn. Res. 17(1), 2030–2096 (2016)

    MathSciNet  Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385

  16. Hoffman, J., et al.: CyCADA: cycle-consistent adversarial domain adaptation. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 1989–1998. PMLR, Stockholmsmässan, Stockholm Sweden, 10–15 July 2018

    Google Scholar 

  17. Hoffman, J., Wang, D., Yu, F., Darrell, T.: FCNs in the wild: pixel-level adversarial and constraint-based adaptation. CoRR (2016). http://arxiv.org/abs/1612.02649

  18. Hoyer, L., Dai, D., Van Gool, L.: Daformer: improving network architectures and training strategies for domain-adaptive semantic segmentation. arXiv preprint arXiv:2111.14887 (2021)

  19. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1501–1510 (2017)

    Google Scholar 

  20. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456. PMLR (2015)

    Google Scholar 

  21. Iwasawa, Y., Matsuo, Y.: Test-time classifier adjustment module for model-agnostic domain generalization. In: Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (2021)

    Google Scholar 

  22. Kim, M., Byun, H.: Learning texture invariant representation for domain adaptation of semantic segmentation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. https://doi.org/10.1109/cvpr42600.2020.01299

  23. Klingner, M., Termöhlen, J.A., Ritterbach, J., Fingscheidt, T.: Unsupervised batchnorm adaptation (UBNA): a domain adaptation method for semantic segmentation without using source domain representations. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 210–220 (2022)

    Google Scholar 

  24. Kuznietsov, Y., Proesmans, M., Van Gool, L.: Towards unsupervised online domain adaptation for semantic segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 261–271 (2022)

    Google Scholar 

  25. Lao, Q., Jiang, X., Havaei, M., Bengio, Y.: Continuous domain adaptation with variational domain-agnostic feature replay. arXiv preprint arXiv:2003.04382 (2020)

  26. Lee, D.: Pseudo-label : the simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on Challenges in Representation Learning, ICML (2013)

    Google Scholar 

  27. Li, Y., Yuan, L., Vasconcelos, N.: Bidirectional learning for domain adaptation of semantic segmentation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019. https://doi.org/10.1109/cvpr.2019.00710

  28. Liang, J., Hu, D., Feng, J.: Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. CoRR (2020). http://arxiv.org/abs/2002.08546

  29. Liu, Y., Zhang, W., Wang, J.: Source-free domain adaptation for semantic segmentation (2021). http://arxiv.org/abs/2103.16372

  30. Liu, Y., Kothari, P., van Delft, B.G., Bellot-Gurlet, B., Mordan, T., Alahi, A.: TTT++: when does self-supervised test-time training fail or thrive? In: Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (2021)

    Google Scholar 

  31. Liu, Z., et al.: Open compound domain adaptation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12403–12412. IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.01242, https://ieeexplore.ieee.org/document/9157145/

  32. Mei, K., Zhu, C., Zou, J., Zhang, S.: Instance adaptive self-training for unsupervised domain adaptation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12371, pp. 415–430. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58574-7_25

    Chapter  Google Scholar 

  33. Pan, F., Shin, I., Rameau, F., Lee, S., Kweon, I.S.: Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. https://doi.org/10.1109/cvpr42600.2020.00382

  34. Poularikas, A.D.: The Handbook of Formulas and Tables for Signal Processing. The electrical engineering handbook series, CRC Press; Springer: IEEE Press (1999)

    Google Scholar 

  35. Sakaridis, C., Dai, D., Hecker, S., Van Gool, L.: Model adaptation with synthetic and real data for semantic dense foggy scene understanding (2018). http://arxiv.org/abs/1808.01265

  36. Sohn, K., et al.: Fixmatch: simplifying semi-supervised learning with consistency and confidence. CoRR abs/2001.07685 (2020). https://arxiv.org/abs/2001.07685

  37. Stan, S., Rostami, M.: Unsupervised model adaptation for continual semantic segmentation. In: AAAI (2021)

    Google Scholar 

  38. Su, P., Tang, S., Gao, P., Qiu, D., Zhao, N., Wang, X.: Gradient regularized contrastive learning for continual domain adaptation (2020). http://arxiv.org/abs/2007.12942

  39. Sun, T., et al.: SHIFT: a synthetic driving dataset for continuous multi-task domain adaptation. In: Computer Vision and Pattern Recognition (2022)

    Google Scholar 

  40. Tonioni, A., Tosi, F., Poggi, M., Mattoccia, S., Stefano, L.D.: Real-time self-adaptive deep stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 195–204 (2019)

    Google Scholar 

  41. Tremblay, M., Halder, S.S., de Charette, R., Lalonde, J.-F.: Rain rendering for evaluating and improving robustness to bad weather. Int. J. Comput. Vision 1–20 (2020). https://doi.org/10.1007/s11263-020-01366-3

  42. Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M.: Learning to adapt structured output space for semantic segmentation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018. https://doi.org/10.1109/cvpr.2018.00780

  43. Vu, T.H., Jain, H., Bucher, M., Cord, M., Pérez, P.: Dada: depth-aware domain adaptation in semantic segmentation. In: ICCV (2019)

    Google Scholar 

  44. Vu, T.H., Jain, H., Bucher, M., Cord, M., Perez, P.: Advent: adversarial entropy minimization for domain adaptation in semantic segmentation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. https://doi.org/10.1109/cvpr.2019.00262

  45. Vu, T.H., Jain, H., Bucher, M., Cord, M., Pérez, P.: ADVENT: adversarial entropy minimization for domain adaptation in semantic segmentation (2019) http://arxiv.org/abs/1811.12833

  46. Wang, D., Shelhamer, E., Liu, S., Olshausen, B., Darrell, T.: Tent: fully test-time adaptation by entropy minimization. In: International Conference on Learning Representations (2021)

    Google Scholar 

  47. Wang, H., Shen, T., Zhang, W., Duan, L., Mei, T.: Classes matter: a fine-grained adversarial approach to cross-domain semantic segmentation. In: The European Conference on Computer Vision (ECCV), August 2020

    Google Scholar 

  48. Wang, Y., Ma, X., Chen, Z., Luo, Y., Yi, J., Bailey, J.: Symmetric cross entropy for robust learning with noisy labels (2019). http://arxiv.org/abs/1908.06112

  49. Wu, Z., et al.: DCAN: dual channel-wise alignment networks for unsupervised scene adaptation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11209, pp. 535–552. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01228-1_32

    Chapter  Google Scholar 

  50. Wu, Z., Wang, X., Gonzalez, J., Goldstein, T., Davis, L.: ACE: adapting to changing environments for semantic segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2121–2130. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00221, https://ieeexplore.ieee.org/document/9009823/

  51. Wulfmeier, M., Bewley, A., Posner, I.: Incremental adversarial domain adaptation for continually changing environments (2018). http://arxiv.org/abs/1712.07436

  52. Yang, Y., Soatto, S.: FDA: fourier domain adaptation for semantic segmentation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4084–4094. IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00414, https://ieeexplore.ieee.org/document/9157228/

  53. Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Bisenet: bilateral segmentation network for real-time semantic segmentation. In: Proceedings of the European conference on computer vision (ECCV), pp. 325–341 (2018)

    Google Scholar 

  54. Yuan, Y., Chen, X., Chen, X., Wang, J.: Segmentation transformer: object-contextual representations for semantic segmentation. In: European Conference on Computer Vision (ECCV), vol. 1 (2021)

    Google Scholar 

  55. Zhai, X., Oliver, A., Kolesnikov, A., Beyer, L.: S4L: self-supervised semi-Supervised Learning. arXiv e-prints arXiv:1905.03670, May 2019

  56. Zhang, P., Zhang, B., Zhang, T., Chen, D., Wang, Y., Wen, F.: Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation (2021). http://arxiv.org/abs/2101.10979

  57. Zhang, Q., Zhang, J., Liu, W., Tao, D.: Category anchor-guided unsupervised domain adaptation for semantic segmentation. Adv. Neural Inf. Process. Syst. (2019). http://arxiv.org/abs/1910.13049

  58. Zheng, Z., Yang, Y.: Unsupervised scene adaptation with memory regularization in vivo. In: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, July 2020. https://doi.org/10.24963/ijcai.2020/150, http://dx.doi.org/10.24963/ijcai.2020/150

  59. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), October 2017. https://doi.org/10.1109/iccv.2017.244

  60. Zou, Y., Yu, Z., Kumar, B., Wang, J.: Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 289–305 (2018)

    Google Scholar 

  61. Zou, Y., Yu, Z., Liu, X., Kumar, B.V.K.V., Wang, J.: Confidence regularized self-training. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), October 2019. https://doi.org/10.1109/iccv.2019.00608

  62. Zou, Y., Yu, Z., Liu, X., Kumar, B., Wang, J.: Confidence regularized self-training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5982–5991 (2019)

    Google Scholar 

Download references

Acknowledgement

The authors thank Hossein Azizpour, Hedvig Kjellström and Raoul de Charette for the helpful discussions and guidance.

Author information

Authors and Affiliations

  1. King, Stockholm, Sweden

    Theodoros Panagiotakopoulos

  2. Univrses, Stockholm, Sweden

    Pier Luigi Dovesi

  3. Kudan, Tokyo, Japan

    Linus Härenstam-Nielsen

  4. Technical University of Munich, Munich, Germany

    Linus Härenstam-Nielsen

  5. University of Bologna, Bologna, Italy

    Matteo Poggi

Authors
  1. Theodoros Panagiotakopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pier Luigi Dovesi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linus Härenstam-Nielsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Matteo Poggi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matteo Poggi .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 4841 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Panagiotakopoulos, T., Dovesi, P.L., Härenstam-Nielsen, L., Poggi, M. (2022). Online Domain Adaptation for Semantic Segmentation in Ever-Changing Conditions. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13694. Springer, Cham. https://doi.org/10.1007/978-3-031-19830-4_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19830-4_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19829-8

  • Online ISBN: 978-3-031-19830-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics