Skip to main content

Advertisement

Springer Nature Link
Log in

Kill Two Birds with One Stone: Domain Generalization for Semantic Segmentation via Network Pruning

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Deep models are notoriously known to perform poorly when encountering new domains with different statistics. To alleviate this issue, we present a new domain generalization method based on network pruning, dubbed NPDG. Our core idea is to prune the filters or attention heads that are more sensitive to domain shift while preserving those domain-invariant ones. To this end, we propose a new pruning policy tailored to improve generalization ability, which identifies the filter and head sensibility of domain shift by judging its activation variance among different domains (unary manner) and its correlation to other filters (binary manner). To better reveal those potentially sensitive filters and heads, we present a differentiable style perturbation scheme to imitate the domain variance dynamically. NPDG is trained on a single source domain and can be applied to both CNN- and Transformer-based backbones. To our knowledge, we are among the pioneers in tackling domain generalization in segmentation via network pruning. NPDG not only improves the generalization ability of a segmentation model but also decreases its computation cost. Extensive experiments demonstrate the state-of-the-art generalization performance of NPDG with a lighter-weight structure.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The datasets that support the findings of this study are publicly available. GTA5 Richter et al. (2016) is available at https://download.visinf.tu-darmstadt.de/data/from_games. Synthia Ros et al. (2016) can be downloaded from https://synthia-dataset.net/. BDD100K Yu et al. (2020) is available at https://doc.bdd100k.com/download.html. Cityscapes Cordts et al. (2016) can be available at https://www.cityscapes-dataset.com/ after registration. Mapillary Neuhold et al. (2017) is available at https://www.mapillary.com/datasets after registration.

Notes

  1. We use the notation \(\psi \) for mean and \(\xi \) for standard deviation in D-VAE, in order to avoid confusion to the \(\mu (.)\) and \(\sigma (.)\) in AdaIN.

  2. Unless otherwise noted, we employ \(\textbf{A}\) to denote the original feature maps and \(\textbf{E}\) as the standardized ones. \(\textbf{E}\) is for calculating w only and is not forwarded to the next layer.

References

  • Bottou, Léon. (2012). Stochastic gradient descent tricks. In Neural networks: Tricks of the trade.

  • Cai, J., Zhu, C., Cui, C., Li, H., Wu, T., Zhang, S., & Yang, L. (2021). Generalizing nucleus recognition model in multi-source ki67 immunohistochemistry stained images via domain-specific pruning. In MICCAI, pages 277–287.

  • Cai, R., Li, Z., Wei, P., Qiao, J., Zhang, K., & Hao, Z. (2019). Learning disentangled semantic representation for domain adaptation. In IJCAI, pages 2060–2066.

  • Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. T-PAMI, 40(4), 834–848.

    Article  MATH  Google Scholar 

  • Chen, S., Wang, W., & Pan, S. J. (2019). Cooperative pruning in cross-domain deep neural network compression. In IJCAI, pages 2102–2108.

  • Choi, S., Jung, S., Yun, H., Kim, J. T., Kim, S., & Choo, J. (2021). Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In CVPR, pages 11580–11590.

  • Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016). The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213–3223.

  • Dubey, A., Chatterjee, M., & Ahuja, N. (2018). Coreset-based neural network compression. In ECCV, pages 454–470.

  • Fu, Y., Zhang, M., Xu, X., Cao, X., Ma, C., Ji, Y., Zuo, K., & Lu, H. (2021). Partial feature selection and alignment for multi-source domain adaptation. In CVPR.

  • Gatys, L. A., Ecker, A. S., & Bethge, M. (2016). Image style transfer using convolutional neural networks. In CVPR, pages 2414–2423.

  • Gong, R., Li, W., Chen, Y., Dai, D., & Van Gool, L. (2021). Dlow: Domain flow and applications. International Journal of Computer Vision, 129(10), 2865–2888.

    Article  MATH  Google Scholar 

  • Han, S., Pool, J., Tran, J., & Dally, W. (2015). Learning both weights and connections for efficient neural network. NeurIPS, 28.

  • Hassibi, B., & Stork, D.(1992). Second order derivatives for network pruning: Optimal brain surgeon. In NeurIPS, page 164–171.

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In CVPR, pages 770–778.

  • He, W., Wu, M., Liang, M., & Lam, S. (2021). Cap: Context-aware pruning for semantic segmentation. In WACV, pages 960–969.

  • He, Y., Ding, Y., Liu, P., Zhu, L., Zhang, H. & Yang, Y. (2020). Learning filter pruning criteria for deep convolutional neural networks acceleration. In CVPR, pages 2009–2018.

  • He, Y., Kang, G., Dong, X., Fu, Y., & Yang, Y. (2018). Soft filter pruning for accelerating deep convolutional neural networks. In IJCAI, pages 2234–2240.

  • He, Y., Liu, P., Wang, Z., Hu, Z., & Yang, Yi (2019). Filter pruning via geometric median for deep convolutional neural networks acceleration. In CVPR, pages 4340–4349.

  • He, Y., & Xiao, L. (2023). Structured pruning for deep convolutional neural networks: A survey. TPAMI.

  • Hoyer, L., Dai, D., & Van Gool, L. (2022). Hrda: Context-aware high-resolution domain-adaptive semantic segmentation. In ECCV, pages 372–391.

  • Huang, J., Guan, D., Xiao, A., & Lu, S. (2021). Fsdr: Frequency space domain randomization for domain generalization. In CVPR, pages 6891–6902.

  • Huang, J., Guan, D., Xiao, A., & Lu, S. (2021). Fsdr: Frequency space domain randomization for domain generalization. In CVPR, pages 6891–6902.

  • Huang, X., & Belongie, S. (2017). Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, pages 1501–1510.

  • Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, pages 448–456.

  • LeCun, Y., Denker, J., & Solla, S. (1989). Optimal brain damage. In NeurIPS, page 598–605.

  • Li, D., & Hospedales, T. (2020). Online meta-learning for multi-source and semi-supervised domain adaptation. In ECCV, pages 382–403.

  • Li, D., Yang, Y., Song, Y.-Z., & Hospedales, T. M. (2017). Deeper, broader and artier domain generalization. In ICCV, pages 5542–5550.

  • Li, T., Wen, Z., Li, Y., & Lee, T. S. (2024). Emergence of shape bias in convolutional neural networks through activation sparsity. NeurIPS.

  • Li, X., Li, M., Wang, Y., Ren, C.-X., & Guo, X. (2023). Adaptive texture filtering for single-domain generalized segmentation. arXiv preprint arXiv:2303.02943.

  • Li, Y., Yuan, L., & Vasconcelos, N. (2019). Bidirectional learning for domain adaptation of semantic segmentation. In CVPR, pages 6936–6945.

  • Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017). Learning efficient convolutional networks through network slimming. In ICCV, pages 2736–2744.

  • Long, M., Cao, Y., Wang, J., & Jordan, M. (2015). Learning transferable features with deep adaptation networks. In ICML, pages 97–105.

  • Long, S., Zhou, Q., Ying, C., Ma, L., & Luo, Y. (2023). Rethinking domain generalization: Discriminability and generalizability. arXiv preprint arXiv:2309.16483.

  • Luo, J.-H., Wu, J., & Lin, W. (2017). Thinet: A filter level pruning method for deep neural network compression. In CVPR, pages 5058–5066.

  • Luo, Y., Liu, P., Guan, T., Yu, J., & Yang, Y. (2020). Adversarial style mining for one-shot unsupervised domain adaptation. In NeurIPS, pages 20612–20623.

  • Luo, Y., Liu, P., Zheng, L., Guan, T., Yu, J., & Yang, Y. (2021). Category-level adversarial adaptation for semantic segmentation using purified features. T-PAMI.

  • Luo, Y., Zheng, L., Guan, T., Yu, J., & Yang, Y. (2019). Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In CVPR, pages 2507–2516.

  • Muandet, K., Balduzzi, D., & Schölkopf, B. (2019). Domain generalization via invariant feature representation. In ICML, pages 10–18.

  • Neuhold, G., Ollmann, T., Bulo, S. R. & Kontschieder, P. (2017). The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, pages 4990–4999.

  • Nguyen, B., Moëllic, P.-A., & Blayac, S. (2022). Domain generalization on constrained platforms: On the compatibility with pruning techniques. In Global IoT Summit.

  • Nichol, K. (2016). Painter by numbers, wikiart. https://www.kaggle.com/c/painter-by-numbers.

  • Pan, X., Luo, P., Shi, J., & Tang, X. (2018). Two at once: Enhancing learning and generalization capacities via ibn-net. In ECCV, pages 464–479.

  • Pan, X., Zhan, X., Shi, J., Tang, X., & Luo, P. (2019). Switchable whitening for deep representation learning. In ICCV, pages 1863–1871.

  • Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., & Lerer, A. (2017). Automatic differentiation in pytorch. In NeurIPS.

  • Peng, D., Lei, Y., Liu, L., Zhang, P., & Liu, J. (2021). Global and local texture randomization for synthetic-to-real semantic segmentation. T-IP, 30, 6594–6608.

    MATH  Google Scholar 

  • Peng, X., Huang, Z., Sun, X., & Saenko, K. (2019). Domain agnostic learning with disentangled representations. In ICML, pages 5102–5112.

  • Qian, X., & Klabjan, D. (2021). A probabilistic approach to neural network pruning. In ICML, pages 8640–8649.

  • Qiao, F., & Peng, X. (2021). Uncertainty-guided model generalization to unseen domains. In CVPR, pages 6790–6800.

  • Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016). Playing for data: Ground truth from computer games. In ECCV, pages 102–118.

  • Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016). The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, pages 3234–3243.

  • Rosenfeld, J. S., Frankle, J., Carbin, M., & Shavit, N. (2021). On the predictability of pruning across scales. In ICML, pages 9075–9083.

  • Sehwag, V., Wang, S., Mittal, P., & Jana, S. (2020). Hydra: Pruning adversarially robust neural networks. In NeurIPS, pages 19655–19666.

  • Sun, X. (2023). Pruning for better domain generalizability. arXiv preprint arXiv:2306.13237.

  • Tang, Z., Gao, Y., Zhu, Y., Zhang, Z., Li, M., & Metaxas, D. N. (2021). Selfnorm and crossnorm for out-of-distribution robustness. In ICCV.

  • Tian, C. X., Li, H., Xie, X., Liu, Y., & Wang, S. (2022). Neuron coverage-guided domain generalization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), 1302–1311.

    Article  Google Scholar 

  • Tjio, G., Liu, P., Zhou, J. T., & Goh, R. S. M. (2021). Adversarial semantic hallucination for domain generalized semantic segmentation. CoRR.

  • Tzeng, E., Hoffman, J., Saenko, K., & Darrell, T. (2017). Adversarial discriminative domain adaptation. In CVPR, pages 7167–7176.

  • Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2016). Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022.

  • Wang, J., Lan, C., Liu, C., Ouyang, Y., Zeng, W., & Qin, T. (2021). Generalizing to unseen domains: A survey on domain generalization. arXiv preprint arXiv:2103.03097.

  • Wang, J., & Jiang, J. (2021). Learning across tasks for zero-shot domain adaptation from a single source domain. T-PAMI.

  • Wang, W., Zhong, Z., Wang, W., Chen, X., Ling, C., Wang, B., & Sebe, N. (2023). Dynamically instance-guided adaptation: A backward-free approach for test-time domain adaptive semantic segmentation. In CVPR, pages 24090–24099.

  • Wu, K., Tang, F., Liu, N., Deussen, O., Dong, W., & Lee, T.-Y., et al. (2024). Lighting image/video style transfer methods by iterative channel pruning. In ICASSP, pages 3800–3804. IEEE.

  • Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021). Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS.

  • Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., & Darrell, T. (2020). Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, pages 2636–2645.

  • Yue, X., Zhang, Y., Zhao, S., Sangiovanni-Vincentelli, A., Keutzer, K., & Gong, B. (2019). Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In ICCV, pages 2100–2110.

  • Zhang, P., Zhang, B., Zhang, T., Chen, D., Wang, Y., & Wen, F. (2021). Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In CVPR, pages 12414–12424.

  • Zhang, Q., Zhang, J., Liu, W., & Tao, D. (2019). Category anchor-guided unsupervised domain adaptation for semantic segmentation. In NeurIPS.

  • Zhao, S., Yue, X., Zhang, S., Li, B., Zhao, H., Bichen, W., Krishna, R., Gonzalez, J. E., Sangiovanni-Vincentelli, A. L., Seshia, S. A., & Keutzer, K. (2020). A review of single-source deep unsupervised visual domain adaptation. IEEE Transactions on Neural Networks and Learning Systems, 33(2), 473–493.

  • Zhao, Y., Zhong, Z., Yang, F., Luo, Z., Lin, Y., Li, S., & Sebe, N. (2021). Learning to generalize unseen domains via memory-based multi-source meta-learning for person re-identification. In CVPR, pages 6277–6286.

  • Zhao, Y., Zhong, Z., Zhao, N., Sebe, N., & Lee, G. H. (2022). Style-hallucinated dual consistency learning for domain generalized semantic segmentation. In ECCV, pages 535–552. Springer.

  • Zhao, Y., Zhong, Z., Zhao, N., Sebe, N., & Lee, G. H. (2023). Style-hallucinated dual consistency learning: A unified framework for visual domain generalization. IJCV.

  • Zheng, Z., & Yang, Y. (2021). Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. International Journal of Computer Vision, 129(4), 1106–1120.

    Article  MATH  Google Scholar 

  • Zhong, Z., Zhao, Y., Lee, G. H., & Sebe, N. (2022). Adversarial style augmentation for domain generalized urban-scene segmentation. NeurIPS, 35, 338–350.

    Google Scholar 

  • Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2021). Domain generalization: A survey. arXiv preprint arXiv:2103.02503.

  • Zhou, K., Yang, Y., Qiao, Y., & Xiang, T. (2020). Domain generalization with mixstyle. In International conference on learning representations

  • Zhuang, T., Zhang, Z., Huang, Y., Zeng, X., Shuang, K., & Li, X. (2020). Neuron-level structured pruning using polarization regularizer. In NeurIPS, pages 9865–9877.

  • Zou, Y., Yang, X., Yu, Z., Kumar, B. V. K. V., & Kautz, J. (2020). Joint disentangling and adaptation for cross-domain person re-identification. In ECCV, pages 87–104.

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China ( 62293554, 62206249, U2336212 ), Natural Science Foundation of Zhejiang Province, China ( LZ24F020002 ), Young Elite Scientists Sponsorship Program by CAST (2023QNRC001), and the Fundamental Research Funds for the Central Universities(No. 226-2022-00051).

Author information

Authors and Affiliations

  1. Zhejiang University, Hangzhou, China

    Yawei Luo & Yi Yang

  2. Ping Liu is with the Department of Computer Science and Engineering, University of Nevada, Reno, NV, USA

    Ping Liu

Authors
  1. Yawei Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ping Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yi Yang.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Communicated by Nicu Sebe.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luo, Y., Liu, P. & Yang, Y. Kill Two Birds with One Stone: Domain Generalization for Semantic Segmentation via Network Pruning. Int J Comput Vis 133, 335–352 (2025). https://doi.org/10.1007/s11263-024-02194-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-024-02194-5

Keywords