Skip to main content
SpringerLink
Log in

Discrete Bidirectional Matrix Factorization Hashing for Zero-Shot Cross-Media Retrieval

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 13020))

Included in the following conference series:

Abstract

Recently, cross-modal retrieval has gained much attention due to ever-increasing multimedia data. However, existing cross-modal algorithms require that the training set and the test set share the same categories, which cannot well search data of newly emerged categories. Therefore, more practical zero-shot cross-modal retrieval (ZSCMR) has become a promising direction, which aims to search unseen classes (new classes) that never present in the training set. It is very challenging that ZSCMR needs to solve not only the inconsistent semantic between seen and unseen classes but also the semantic gap of the heterogeneous multimedia data. To mitigate these problems, a novel discrete bidirectional matrix factorization hashing method is developed for zero-shot cross-modal retrieval (DMZCR). The proposed DMZCR contains three contributions: 1) A bidirectional matrix factorization scheme is proposed in our model, more discriminative low-rank representation can be learned and the redundant information can also be removed. 2) Inspired by zero-shot learning, we build a multi-layer semantic transmission scheme to model the relationships between classes, features and attributes, then the knowledge can be transferred from seen to unseen classes. 3) The hash codes can be learned by a discrete scheme, reducing the large quantization error caused by relaxation. As far as we know, this work first employs matrix factorization scheme to solve ZSCMR task. Experiments on three popular databases illustrate the efficacy of DMZCR compared with several state-of-the-art algorithms for ZSCMR task.

This work was supported by NSFC [Grant 62020106012, U1836218, 61672265].

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Changpinyo, S., Chao, W.L., Gong, B., Sha, F.: Synthesized classifiers for zero-shot learning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    Google Scholar 

  2. Chi, J., Peng, Y.: Dual adversarial networks for zero-shot cross-media retrieval. In: IJCAI, pp. 663–669 (2018)

    Google Scholar 

  3. Chua, T.S., Tang, J., Hong, R., Li, H., Luo, Z., Zheng, Y.: NUS-WIDE: a real-world web image database from National University of Singapore. In: Proceedings of the ACM International Conference on Image and Video Retrieval, pp. 1–9 (2009)

    Google Scholar 

  4. Ding, G., Guo, Y., Zhou, J.: Collective matrix factorization hashing for multimodal data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2075–2082 (2014)

    Google Scholar 

  5. Everingham, M., Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)

    Article  Google Scholar 

  6. Guo, Y., Ding, G., Han, J., Gao, Y.: SitNet: discrete similarity transfer network for zero-shot hashing. In: IJCAI, pp. 1767–1773 (2017)

    Google Scholar 

  7. Hu, M., Yang, Y., Shen, F., Xie, N., Hong, R., Shen, H.T.: Collective reconstructive embeddings for cross-modal hashing. IEEE Trans. Image Process. 28(6), 2770–2784 (2019)

    Article  MathSciNet  Google Scholar 

  8. Lin, Z., Ding, G., Hu, M., Wang, J.: Semantics-preserving hashing for cross-view retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3864–3872 (2015)

    Google Scholar 

  9. Liu, H., Ji, R., Wu, Y., Huang, F., Zhang, B.: Cross-modality binary code learning via fusion similarity hashing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7380–7388 (2017)

    Google Scholar 

  10. Long, Y., Liu, L., Shao, L.: Towards fine-grained open zero-shot learning: inferring unseen visual features from attributes. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 944–952. IEEE (2017)

    Google Scholar 

  11. Pachori, S., Deshpande, A., Raman, S.: Hashing in the zero shot framework with domain adaptation. Neurocomputing 275, 2137–2149 (2018)

    Article  Google Scholar 

  12. Peng, Y., Qi, J., Huang, X., Yuan, Y.: CCL: cross-modal correlation learning with multigrained fusion by hierarchical network. IEEE Trans. Multimedia 20(2), 405–420 (2018)

    Article  Google Scholar 

  13. Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  14. Rasiwasia, N., et al.: A new approach to cross-modal multimedia retrieval. In: Proceedings of the 18th ACM International Conference on Multimedia, pp. 251–260 (2010)

    Google Scholar 

  15. Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: LabelMe: a database and web-based tool for image annotation. Int. J. Comput. Vis. 77(1), 157–173 (2008)

    Article  Google Scholar 

  16. Schönemann, P.H.: A generalized solution of the orthogonal procrustes problem. Psychometrika 31(1), 1–10 (1966)

    Article  MathSciNet  Google Scholar 

  17. Shen, F., Shen, C., Liu, W., Shen, H.T.: Supervised discrete hashing. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 37–45 (2015)

    Google Scholar 

  18. Song, J., Yang, Y., Yang, Y., Huang, Z., Shen, H.T.: Inter-media hashing for large-scale retrieval from heterogeneous data sources. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pp. 785–796 (2013)

    Google Scholar 

  19. Tang, J., Wang, K., Shao, L.: Supervised matrix factorization hashing for cross-modal retrieval. IEEE Trans. Image Process. 25(7), 3157–3166 (2016)

    Article  MathSciNet  Google Scholar 

  20. Wang, B., Yang, Y., Xu, X., Hanjalic, A., Shen, H.T.: Adversarial cross-modal retrieval. In: Proceedings of the 25th ACM International Conference on Multimedia, pp. 154–162 (2017)

    Google Scholar 

  21. Wang, Y., He, S., Xu, X., Yang, Y., Li, J., Shen, H.T.: Self-supervised adversarial learning for cross-modal retrieval. In: Proceedings of the 2nd ACM International Conference on Multimedia in Asia, pp. 1–7 (2021)

    Google Scholar 

  22. Xian, Y., Schiele, B., Akata, Z.: Zero-shot learning-the good, the bad and the ugly. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4582–4591 (2017)

    Google Scholar 

  23. Xu, X., Lu, H., Song, J., Yang, Y., Shen, H.T., Li, X.: Ternary adversarial networks with self-supervision for zero-shot cross-modal retrieval. IEEE Trans. Cybern. 50(6), 2400–2413 (2019)

    Article  Google Scholar 

  24. Xu, X., Shen, F., Yang, Y., Shen, H.T., Li, X.: Learning discriminative binary codes for large-scale cross-modal retrieval. IEEE Trans. Image Process. 26(5), 2494–2507 (2017)

    Article  MathSciNet  Google Scholar 

  25. Xu, Y., Yang, Y., Shen, F., Xu, X., Zhou, Y., Shen, H.T.: Attribute hashing for zero-shot image retrieval. In: 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 133–138. IEEE (2017)

    Google Scholar 

  26. Yang, Y., Luo, Y., Chen, W., Shen, F., Shao, J., Shen, H.T.: Zero-shot hashing via transferring supervised knowledge. In: Proceedings of the 24th ACM International Conference on Multimedia, pp. 1286–1295 (2016)

    Google Scholar 

  27. Yuan, X., Wang, G., Chen, Z., Zhong, F.: CHOP: an orthogonal hashing method for zero-shot cross-modal retrieval. Pattern Recogn. Lett. 145, 247–253 (2021)

    Article  Google Scholar 

  28. Zhang, D., Wu, X.J.: Scalable discrete matrix factorization and semantic autoencoder for cross-media retrieval. IEEE Trans. Cybern. (2020)

    Google Scholar 

  29. Zhang, D., Wu, X.J., Liu, Z., Yu, J., Kitter, J.: Fast discrete cross-modal hashing based on label relaxation and matrix factorization. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 4845–4850. IEEE (2021)

    Google Scholar 

  30. Zhang, D., Li, W.J.: Large-scale supervised multimodal hashing with semantic correlation maximization. In: Twenty-Eighth AAAI Conference on Artificial Intelligence (2014)

    Google Scholar 

  31. Zhang, J., Peng, Y., Yuan, M.: Unsupervised generative adversarial cross-modal hashing. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)

    Google Scholar 

  32. Zhong, F., Chen, Z., Min, G.: An exploration of cross-modal retrieval for unseen concepts. In: Li, G., Yang, J., Gama, J., Natwichai, J., Tong, Y. (eds.) DASFAA 2019. LNCS, vol. 11447, pp. 20–35. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-18579-4_2

    Chapter  Google Scholar 

  33. Zhou, J., Ding, G., Guo, Y.: Latent semantic sparse hashing for cross-modal similarity search. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 415–424 (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China

    Donglin Zhang & Xiao-Jun Wu

  2. College of Computer and Communication Engineering, Zhengzhou University of Light Industry, Zhengzhou, China

    Jun Yu

Authors
  1. Donglin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiao-Jun Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jun Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiao-Jun Wu .

Editor information

Editors and Affiliations

  1. University of Science and Technology Beijing, Beijing, China

    Huimin Ma

  2. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  3. Tsinghua University, Beijing, China

    Changshui Zhang

  4. Zhejiang University, Hangzhou, China

    Fei Wu

  5. Chinese Academy of Sciences, Beijing, China

    Tieniu Tan

  6. Hunan University, Changsha, China

    Yaonan Wang

  7. Sun Yat-Sen University, Guangzhou, Guangdong, China

    Jianhuang Lai

  8. Beijing Jiaotong University, Beijing, China

    Yao Zhao

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, D., Wu, XJ., Yu, J. (2021). Discrete Bidirectional Matrix Factorization Hashing for Zero-Shot Cross-Media Retrieval. In: Ma, H., et al. Pattern Recognition and Computer Vision. PRCV 2021. Lecture Notes in Computer Science(), vol 13020. Springer, Cham. https://doi.org/10.1007/978-3-030-88007-1_43

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-88007-1_43

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-88006-4

  • Online ISBN: 978-3-030-88007-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation