Skip to main content
Springer Nature Link
Log in

Learnable Masked Tokens for Improved Transferability of Self-supervised Vision Transformers

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13715))

  • 1257 Accesses

Abstract

Vision transformers have recently shown remarkable performance in various visual recognition tasks specifically for self-supervised representation learning. The key advantage of transformers for self supervised learning, compared to their convolutional counterparts, is the reduced inductive biases that makes transformers amenable to learning rich representations from massive amounts of unlabelled data. On the other hand, this flexibility makes self-supervised vision transformers susceptible to overfitting when fine-tuning them on small labeled target datasets. Therefore, in this work, we make a simple yet effective architectural change by introducing new learnable masked tokens to vision transformers whereby we reduce the effect of overfitting in transfer learning while retaining the desirable flexibility of vision transformers. Through several experiments based on two seminal self-supervised vision transformers, SiT and DINO, and several small target visual recognition tasks, we show consistent and significant improvements in the accuracy of the fine-tuned models across all target tasks.

This work is partially supported by KTH Digital Futures and Wallenberg AI, Autonomous Systems and Software Program (WASP).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    We omit LN and MLP layers in between for convenience.

  2. 2.

    We remove the layer index l, and replace it with the image index k for convenience.

References

  1. Vaswani, A., et al.: Attention is all you need. Advances In: Neural Information Processing Systems, pp. 5998–6008 (2017)

    Google Scholar 

  2. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. ArXiv Preprint ArXiv:2010.11929. (2020)

  3. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A. & Jégou, H. Training data-efficient image transformers & distillation through attention. In: International Conference on Machine Learning, pp. 10347–10357 (2021)

    Google Scholar 

  4. Bahdanau, D., Cho, K. & Bengio, Y.: Neural machine translation by jointly learning to align and translate. ArXiv Preprint ArXiv:1409.0473. (2014)

  5. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. ArXiv Preprint ArXiv:1409.1556. (2014)

  6. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  7. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.: Densely connected convolutional networks. In: Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  8. Atito, S., Awais, M., Kittler, J.: SIT: Self-supervised vision transformer. ArXiv Preprint ArXiv:2104.03602 (2021)

  9. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. ArXiv Preprint ArXiv:2104.14294 (2021)

  10. Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. ArXiv Preprint ArXiv:2003.04297 (2020)

  11. Grill, J., et al.: Bootstrap your own latent: A new approach to self-supervised learning. ArXiv Preprint ArXiv:2006.07733 (2020)

  12. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607 (2020)

    Google Scholar 

  13. Park, T., Efros, A.A., Zhang, R., Zhu, J.-Y.: Contrastive learning for unpaired image-to-image translation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 319–345. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_19

    Chapter  Google Scholar 

  14. Prillo, S., Eisenschlos, J.: SoftSort: a continuous relaxation for the argsort operator. International Conference on Machine Learning, pp. 7793–7802 (2020)

    Google Scholar 

  15. Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. ArXiv Preprint ArXiv:1803.07728 (2018)

  16. Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. ArXiv Preprint ArXiv:2006.09882 (2020)

  17. Krizhevsky, A., Hinton, G., Others Learning multiple layers of features from tiny images. (Citeseer 2009

    Google Scholar 

  18. Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L. ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)

    Google Scholar 

  19. Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 215–223 (2011)

    Google Scholar 

  20. Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-UCSD birds 200 (California Institute of Technology, 2010)

    Google Scholar 

  21. Yu, X., Zhao, Y., Gao, Y., Yuan, X., Xiong, S.: Benchmark platform for ultra-fine-grained visual categorization beyond human performance. In: Proceedings Of The IEEE/CVF International Conference on Computer Vision, pp. 10285–10295 (2021)

    Google Scholar 

  22. Zhao, J., Zhang, Y., He, X., Xie, P.: Covid-CT-dataset: a CT scan dataset about Covid-19. ArXiv Preprint ArXiv:2003.13865 490 (2020)

  23. Brown, T., et al.: Language models are few-shot learners. ArXiv Preprint ArXiv:2005.14165 (2020)

  24. Lepikhin, D., et al.: Gshard: scaling giant models with conditional computation and automatic sharding. ArXiv Preprint ArXiv:2006.16668 (2020)

  25. Pan, S., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22, 1345–1359 (2009)

    Article  Google Scholar 

  26. Noroozi, M., Favaro, P.: unsupervised learning of visual representations by solving jigsaw puzzles. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 69–84. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_5

    Chapter  Google Scholar 

  27. Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40

    Chapter  Google Scholar 

  28. Wu, Z., Xiong, Y., Yu, S., Lin, D.: Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings Of The IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733–3742 (2018)

    Google Scholar 

  29. Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. ArXiv Preprint ArXiv:1807.03748 (2018)

  30. Zheng, H., Fu, J., Mei, T., Luo, J.: Learning multi-attention convolutional neural network for fine-grained image recognition. In: Proceedings Of The IEEE International Conference on Computer Vision, pp. 5209–5217 (2017)

    Google Scholar 

  31. Wei, X., Xie, C., Wu, J., Shen, C.: Mask-CNN: localizing parts and selecting descriptors for fine-grained bird species categorization. Pattern Recogn. 76, 704–714 (2018)

    Article  Google Scholar 

  32. Nawaz, S., Calefati, A., Caraffini, M., Landro, N., Gallo, I.: Are these birds similar: Learning branched networks for fine-grained representations. In: 2019 International Conference on Image and Vision Computing New Zealand (IVCNZ), pp. 1–5 (2019)

    Google Scholar 

  33. Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D., Chao, L.: Learning deep transformer models for machine translation. ArXiv Preprint ArXiv:1906.01787 (2019)

  34. Baevski, A., Auli, M.: Adaptive input representations for neural language modeling. ArXiv Preprint ArXiv:1809.10853 (2018)

  35. Choe, J.., Shim, H.: Attention-based dropout layer for weakly supervised object localization. In: Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition, pp. 2219–2228 (2019)

    Google Scholar 

  36. Yun, S., Han, D., Oh, S., Chun, S., Choe, J., Yoo, Y.: CutMix: regularization strategy to train strong classifiers with localizable features. In: Proceedings Of The IEEE/CVF International Conference On Computer Vision, pp. 6023–6032 (2019)

    Google Scholar 

  37. Dosovitskiy, A., Springenberg, J., Riedmiller, M., Brox, T.: Discriminative unsupervised feature learning with convolutional neural networks. Adv. Neural. Inf. Process. Syst. 27, 766–774 (2014)

    Google Scholar 

  38. Jenni, S., Favaro, P.: Self-supervised feature learning by learning to spot artifacts. In: Proceedings of The IEEE Conference on Computer Vision And Pattern Recognition, pp. 2733–2742 (2018)

    Google Scholar 

  39. Haeusser, P., Plapp, J., Golkov, V., Aljalbout, E., Cremers, D.: Associative deep clustering: training a classification network with no labels. In: German Conference On Pattern Recognition, pp. 18–32 (2018)

    Google Scholar 

  40. Ji, X., Henriques, J., Vedaldi, A.: Invariant information clustering for unsupervised image classification and segmentation. In: Proceedings Of The IEEE/CVF International Conference On Computer Vision, pp. 9865–9874 (2019)

    Google Scholar 

  41. Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 139–156. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_9

    Chapter  Google Scholar 

  42. Hjelm, R., et al.: Learning deep representations by mutual information estimation and maximization. ArXiv Preprint ArXiv:1808.06670 (2018)

  43. Patacchiola, M., Storkey, A.: Self-supervised relational reasoning for representation learning. ArXiv Preprint ArXiv:2006.05849 (2020)

  44. He, X., Yang, X., Zhang, S., Zhao, J., Zhang, Y., Xing, E., Xie, P.: Sample-efficient deep learning for COVID-19 diagnosis based on CT scans. Medrxiv (2020)

    Google Scholar 

  45. Wei, X., Zhang, Y., Gong, Y., Zhang, J., Zheng, N.: Grassmann pooling as compact homogeneous bilinear pooling for fine-grained visual classification. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 365–380. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_22

    Chapter  Google Scholar 

  46. Dubey, A., Gupta, O., Raskar, R., Naik, N.: Maximum-entropy fine-grained classification. ArXiv Preprint ArXiv:1809.05934 (2018)

  47. Wang, Y., Morariu, V., Davis, L.: Learning a discriminative filter bank within a CNN for fine-grained recognition. In: Proceedings Of The IEEE Conference on Computer Vision And Pattern Recognition, ,pp. 4148–4157 (2018)

    Google Scholar 

  48. Gao, Y., Han, X., Wang, X., Huang, W., Scott, M.: Channel interaction networks for fine-grained image categorization. In: Proceedings Of The AAAI Conference On Artificial Intelligence. 34, 10818–10825 (2020)

    Google Scholar 

  49. Chen, Y., Bai, Y., Zhang, W., Mei, T.: Destruction and construction learning for fine-grained image recognition. In: Proceedings Of The IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5157–5166 (2019)

    Google Scholar 

  50. Luo, W., et al,: Cross-X learning for fine-grained visual categorization. In: Proceedings Of The IEEE/CVF International Conference On Computer Vision, pp. 8242–8251 (2019)

    Google Scholar 

  51. He, J., et al.:TransFG: a Transformer Architecture for fine-grained recognition. ArXiv Preprint ArXiv:2103.07976 (2021)

  52. Chen, X., He, K.: Exploring simple Siamese representation learning. In: Proceedings Of The IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750–15758 (2021)

    Google Scholar 

  53. Asano, Y., Rupprecht, C.., Vedaldi, A.: Self-labelling via simultaneous clustering and representation learning. ArXiv Preprint ArXiv:1911.05371 (2019)

  54. Li, J., Zhou, P., Xiong, C., Hoi, S..: Prototypical contrastive learning of unsupervised representations. ArXiv Preprint ArXiv:2005.04966 (2020)

  55. Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., Sutskever, I.: Generative pretraining from pixels. In: International Conference On Machine Learning, pp. 1691–1703 (2020)

    Google Scholar 

  56. Devlin, J., Chang, M., Lee, K.,Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. ArXiv Preprint ArXiv:1810.04805 (2018)

  57. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)

    Google Scholar 

  58. Yuan, L., et al.: Tokens-to-token vit: Training vision transformers from scratch on ImageNet. ArXiv Preprint ArXiv:2101.11986 (2021)

  59. Yuan, K., Guo, S., Liu, Z., Zhou, A., Yu, F., Wu, W.:Incorporating convolution designs into visual transformers. ArXiv Preprint ArXiv:2103.11816 (2021)

  60. Li, Y., Zhang, K., Cao, J., Timofte, R., Van Gool, L.: Localvit: Bringing locality to vision transformers. ArXiv Preprint ArXiv:2104.05707 (2021)

  61. Hudson, D., Zitnick, C.: Generative adversarial transformers. ArXiv Preprint ArXiv:2103.01209 (2021)

  62. Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., Hsieh, C.: Dynamicvit: efficient vision transformers with dynamic token sparsification. Adv. Neural. Inf. Process. Syst. 34, 13937–13949 (2021)

    Google Scholar 

  63. Liang, Y., Ge, C., Tong, Z., Song, Y., Wang, J., Xie, P.: Not all patches are what you need: Expediting vision transformers via token reorganizations. ArXiv Preprint ArXiv:2202.07800 (2022)

  64. Tang, Y., Han, K., Wang, Y., Xu, C., Guo, J., Xu, C., Tao, D.: Patch slimming for efficient vision transformers. In: Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition, pp. 12165–12174 (2022)

    Google Scholar 

Download references

Acknowledgements

The project was partially funded by KTH Digital Futures and the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.

Author information

Authors and Affiliations

  1. KTH Royal Institute of Technology, Stockholm, Sweden

    Hao Hu, Federico Baldassarre & Hossein Azizpour

Authors
  1. Hao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Federico Baldassarre
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hossein Azizpour
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hao Hu .

Editor information

Editors and Affiliations

  1. Grenoble Alpes University, Saint Martin d'Hères, France

    Massih-Reza Amini

  2. INSA Rouen Normandy, Saint Etienne du Rouvray, France

    Stéphane Canu

  3. Ruhr-Universität Bochum, Bochum, Germany

    Asja Fischer

  4. KU Leuven, Leuven, Belgium

    Tias Guns

  5. Central European University, Vienna, Austria

    Petra Kralj Novak

  6. Aristotle University of Thessaloniki, Thessaloniki, Greece

    Grigorios Tsoumakas

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hu, H., Baldassarre, F., Azizpour, H. (2023). Learnable Masked Tokens for Improved Transferability of Self-supervised Vision Transformers. In: Amini, MR., Canu, S., Fischer, A., Guns, T., Kralj Novak, P., Tsoumakas, G. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2022. Lecture Notes in Computer Science(), vol 13715. Springer, Cham. https://doi.org/10.1007/978-3-031-26409-2_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-26409-2_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-26408-5

  • Online ISBN: 978-3-031-26409-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation