skip to main content
10.1145/3503161.3547890acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Domain-Specific Conditional Jigsaw Adaptation for Enhancing transferability and Discriminability

Published: 10 October 2022 Publication History

Abstract

Unsupervised Domain Adaptation (UDA) aims to transfer knowledge from a label-rich source domain to a target domain where the label is unavailable. Existing approaches tend to reduce the distribution discrepancy between the source and target domains or assign the pseudo target labels to implement a self-training strategy. However, the transferability or discriminability lackage of the traditional methods results in the limited ability to generalize on the target domain. To remedy this issue, a novel unsupervised domain adaptation framework called Domain-specific Conditional Jigsaw Adaptation Network (DCJAN) is proposed for UDA, which simultaneously encourages the network to extract transferable and discriminative features. To improve the discriminability, a conditional jigsaw module is presented to reconstruct class-aware features of the original images by reconstructing that of corresponding shuffled images. Moreover, in order to enhance the transferability, a domain-specific jigsaw adaptation is proposed to deal with the domain gaps, which utilizes the prior knowledge of jigsaw puzzles to reduce mismatching. It trains conditional jigsaw modules for each domain and updates the shared feature extractor to make the domain-specific conditional jigsaw modules could perform well not only on the corresponding domain but also on the other domain. A consistent conditioning strategy is proposed to ensure the safe training of conditional jigsaw. Experiments conducted on the widely-used Office-31, Office-Home, VisDA-2017, and DomainNet datasets demonstrate the effectiveness of the proposed approach, which outperforms the state-of-the-art methods.

Supplementary Material

MP4 File (MM22-fp0568.mp4)
To enhance the transferability and discriminability of the unsupervised domain adaptation at the same time, we propose the domain-specific conditional jigsaw adaptation network. The network consists of the conditional jigsaw module, the domain-specific jigsaw adaptation module and the prediction consistency conditioning module. We will introduce these in the presentation video.

References

[1]
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer W. Vaughan. 2010. A theory of learning from different domains. Machine learning, Vol. 79, 1 (2010), 151--175.
[2]
Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan M. Liao. 2020. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020).
[3]
Dov Bridger, Dov Danon, and Ayellet Tal. 2020. Solving jigsaw puzzles with eroded boundaries. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3526--3535.
[4]
Silvia Bucci, Antonio D'Innocente, and Tatiana Tommasi. 2019. Tackling partial domain adaptation with self-supervision. In Proceedings of International Conference on Image Analysis and Processing. 70--81.
[5]
Fabio M. Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. 2019. Domain generalization by solving jigsaw puzzles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2229--2238.
[6]
Lin Chen, Huaian Chen, Zhixiang Wei, Xin Jin, Xiao Tan, Yi Jin, and Enhong Chen. 2022. Reusing the Task-specific Classifier as a Discriminator: Discriminator-free Adversarial Domain Adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7181--7190.
[7]
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of European Conference on Computer Vision. 801--818.
[8]
Minghao Chen, Shuai Zhao, Haifeng Liu, and Deng Cai. 2020. Adversarial-learned loss for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 3521--3528.
[9]
Pengguang Chen, Shu Liu, and Jiaya Jia. 2021. Jigsaw clustering for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11526--11535.
[10]
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. 2019. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In Proceedings of International Conference on Machine Learning. 1081--1090.
[11]
Zhi-Qi Cheng, Xiao Wu, Siyu Huang, Jun-Xiu Li, Alexander G. Hauptmann, and Qiang Peng. 2018. Learning to transfer: Generalizable attribute learning with multitask neural model search. In Proceedings of the 26th ACM international conference on Multimedia. 90--98.
[12]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 248--255.
[13]
Weijian Deng, Stephen Gould, and Liang Zheng. 2021. What does rotation prediction tell us about classifier accuracy under varying testing environments?. In Proceedings of International Conference on Machine Learning. 2579--2589.
[14]
Geoffrey French, Michal Mackiewicz, and Mark Fisher. 2018. Self-ensembling for visual domain adaptation. In Proceedings of International Conference on Learning Representations.
[15]
Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In Proceedings of International Conference on Machine Learning. 1180--1189.
[16]
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francc ois Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The journal of machine learning research, Vol. 17, 1 (2016), 2096--2030.
[17]
Xiang Gu, Jian Sun, and Zongben Xu. 2020. Spherical space domain adaptation with robust pseudo-label loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9101--9110.
[18]
Jun-Yan He, Xiao Wu, Zhi-Qi Cheng, Zhaoquan Yuan, and Yu-Gang Jiang. 2021b. DB-LSTM: Densely-connected Bi-directional LSTM for human action recognition. Neurocomputing, Vol. 444 (2021), 319--331.
[19]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision. 2961--2969.
[20]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.
[21]
Qi He, Qi Dai, Xiao Wu, and Jun-Yan He. 2021a. A novel class restriction loss for unsupervised domain adaptation. Neurocomputing, Vol. 461 (2021), 254--265.
[22]
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4700--4708.
[23]
Siyu Huang, Xi Li, Zhi-Qi Cheng, Zhongfei Zhang, and Alexander Hauptmann. 2018. Gnas: A greedy neural architecture search method for multi-attribute learning. In Proceedings of the 26th ACM international conference on Multimedia. 2049--2057.
[24]
Xin Jin, Cuiling Lan, Wenjun Zeng, and Zhibo Chen. 2021. Re-energizing Domain Discriminator with Sample Relabeling for Adversarial Domain Adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9154--9163.
[25]
Guoliang Kang, Jiang Lu, Yi Yang, and Alexander G. Hauptmann. 2019. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4893--4902.
[26]
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of International Conference on Learning Representations.
[27]
Chen-Yu Lee, Tanmay Batra, Mohammad H. Baig, and Daniel Ulbricht. 2019. Sliced wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10285--10295.
[28]
Ru Li, Shuaicheng Liu, Guangfu Wang, Guanghui Liu, and Bing Zeng. 2021b. JigsawGAN: Self-supervised Learning for Solving Jigsaw Puzzles with Generative Adversarial Networks. IEEE Transactions on Image Processing, Vol. 31 (2021), 513--524.
[29]
Shuang Li, Chi Liu, Qiuxia Lin, Binhui Xie, Zhengming Ding, Gao Huang, and Jian Tang. 2020. Domain conditioned adaptation network. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 11386--11393.
[30]
Shuang Li, Mixue Xie, Kaixiong Gong, Chi Harold Liu, Yulin Wang, and Wei Li. 2021c. Transferable semantic augmentation for domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11516--11525.
[31]
Shuang Li, Mixue Xie, Fangrui Lv, Chi Harold Liu, Jian Liang, Chen Qin, and Wei Li. 2021d. Semantic concentration for domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision. 9102--9111.
[32]
Yong Li, Lingjie Lao, Zhen Cui, Shiguang Shan, and Jian Yang. 2021a. Graph jigsaw learning for cartoon face recognition. arXiv preprint arXiv:2107.06532 (2021).
[33]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Proceedings of European Conference on Computer Vision. 740--755.
[34]
Chenxi Liu, Piotr Dollár, Kaiming He, Ross Girshick, Alan Yuille, and Saining Xie. 2020. Are labels necessary for neural architecture search?. In Proceedings of European Conference on Computer Vision. 798--813.
[35]
Hong Liu, Jianmin Wang, and Mingsheng Long. 2021b. Cycle self-training for domain adaptation. In Proceedings of Advances in neural information processing systems.
[36]
Xiaofeng Liu, Site Li, Yubin Ge, Pengyi Ye, Jane You, and Jun Lu. 2021a. Recursively conditional gaussian for ordinal unsupervised domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision. 764--773.
[37]
Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015b. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3431--3440.
[38]
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. 2015a. Learning transferable features with deep adaptation networks. In Proceedings of International Conference on Machine Learning. 97--105.
[39]
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. 2018. Conditional adversarial domain adaptation. In Proceedings of Advances in neural information processing systems. 1647--1657.
[40]
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. 2017. Deep transfer learning with joint adaptation networks. In Proceedings of International Conference on Machine Learning. 2208--2217.
[41]
You-Wei Luo and Chuan-Xian Ren. 2021. Conditional Bures Metric for Domain Adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 13989--13998.
[42]
Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang. 2020. Instance Adaptive Self-training for Unsupervised Domain Adaptation. In Proceedings of European Conference on Computer Vision, Vol. 12371. 415--430.
[43]
Ishan Misra and Laurens van der Maaten. 2020. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6707--6717.
[44]
Jaemin Na, Heechul Jung, Hyung Jin Chang, and Wonjun Hwang. 2021. FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1094--1103.
[45]
Mehdi Noroozi and Paolo Favaro. 2016. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proceedings of European Conference on Computer Vision. 69--84.
[46]
Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong-Wah Ngo, and Tao Mei. 2019. Transferrable prototypical networks for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2239--2247.
[47]
Marie-Morgane Paumard, David Picard, and Hedi Tabia. 2020. Deepzzle: Solving visual jigsaw puzzles with deep learning and shortest path optimization. IEEE Transactions on Image Processing, Vol. 29 (2020), 3569--3581.
[48]
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. 2019. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision. 1406--1415.
[49]
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. 2017. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 (2017).
[50]
Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. 2021. SENTRY: Selective Entropy Optimization via Committee Consistency for Unsupervised Domain Adaptation. In Proceedings of the IEEE International Conference on Computer Vision. 8558--8567.
[51]
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 779--788.
[52]
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proceedings of Advances in neural information processing systems. 91--99.
[53]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of International Conference on Medical image computing and computer-assisted intervention. 234--241.
[54]
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. 2010. Adapting visual category models to new domains. In Proceedings of European Conference on Computer Vision. 213--226.
[55]
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3723--3732.
[56]
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision. 618--626.
[57]
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. 2019. Unsupervised domain adaptation through self-supervision. arXiv preprint arXiv:1909.11825 (2019).
[58]
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. 2020. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of International Conference on Machine Learning. 9229--9248.
[59]
Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of International Conference on Machine Learning. 6105--6114.
[60]
Hui Tang, Ke Chen, and Kui Jia. 2020. Unsupervised domain adaptation via structurally regularized deep clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8725--8735.
[61]
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. 2014. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014).
[62]
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research, Vol. 9, 11 (2008), 2579--2605.
[63]
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. 2017. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5018--5027.
[64]
Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. 2018. Learning semantic representations for unsupervised domain adaptation. In Proceedings of International Conference on Machine Learning. 5423--5432.
[65]
Jianfei Yang, Han Zou, Yuxun Zhou, Zhaoyang Zeng, and Lihua Xie. 2020. Mind the discriminability: Asymmetric adversarial domain adaptation. In Proceedings of European Conference on Computer Vision. 589--606.
[66]
Ji Zhang, Jingkuan Song, Yazhou Yao, and Lianli Gao. 2021a. Curriculum-based meta-learning. In Proceedings of the 29th ACM International Conference on Multimedia. 1838--1846.
[67]
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. 2019. Bridging theory and algorithm for domain adaptation. In Proceedings of International Conference on Machine Learning. 7404--7413.
[68]
Yixin Zhang, Zilei Wang, and Yushi Mao. 2021b. Rpn prototype alignment for domain adaptive object detector. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12425--12434.

Cited By

View all
  • (2024)Source-free domain adaptation with Class Prototype DiscoveryPattern Recognition10.1016/j.patcog.2023.109974145(109974)Online publication date: Jan-2024
  • (2023)Synthesizing Videos from Images for Image-to-Video AdaptationProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611897(8294-8303)Online publication date: 26-Oct-2023
  • (2023)Adaptive Feature Swapping for Unsupervised Domain AdaptationProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611896(7017-7028)Online publication date: 26-Oct-2023

Index Terms

  1. Domain-Specific Conditional Jigsaw Adaptation for Enhancing transferability and Discriminability

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        MM '22: Proceedings of the 30th ACM International Conference on Multimedia
        October 2022
        7537 pages
        ISBN:9781450392037
        DOI:10.1145/3503161
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 10 October 2022

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. neural networks
        2. self-supervised learning
        3. unsupervised domain adaptation
        4. visual classification

        Qualifiers

        • Research-article

        Funding Sources

        Conference

        MM '22
        Sponsor:

        Acceptance Rates

        Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)35
        • Downloads (Last 6 weeks)2
        Reflects downloads up to 16 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)Source-free domain adaptation with Class Prototype DiscoveryPattern Recognition10.1016/j.patcog.2023.109974145(109974)Online publication date: Jan-2024
        • (2023)Synthesizing Videos from Images for Image-to-Video AdaptationProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611897(8294-8303)Online publication date: 26-Oct-2023
        • (2023)Adaptive Feature Swapping for Unsupervised Domain AdaptationProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611896(7017-7028)Online publication date: 26-Oct-2023

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media