Skip to main content
Springer Nature Link
Log in

Hierarchical multi-granularity classification based on bidirectional knowledge transfer

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Hierarchical multi-granularity classification is the task of classifying objects according to multiple levels or granularities. The class hierarchy is vital side information for hierarchical multi-granularity classification. The existing hierarchical multi-granularity classification research utilizes class hierarchy to classify from coarse to fine or fine to coarse. Although these methods are effective in many cases, there are still two issues: (1) multi-task learning for hierarchical multi-granularity classification leads to decreased classification performance; (2) class hierarchy transfer learning is prone to error propagation. In this paper, we propose a bidirectional knowledge transfer model framework to address these issues. Firstly, we improve classification performance through data augmentation. Specifically, by learning the similarity between the original image and the enhanced image, better learn discriminative features, which is beneficial for subsequent classification. Secondly, using class hierarchy trees, we propose reverse hierarchical knowledge transfer to correct some errors in forward hierarchical propagation and improve hierarchical consistency. In addition, we also construct a hierarchical network that adds features from coarse-grained levels to fine-grained levels. The experimental results on six datasets with different class hierarchies demonstrate the effectiveness and superiority of the proposed model. Especially on the CUB-200-2011 and Cifar-100 datasets, our model improved classification accuracy by 3.61% and 4.17% compared to the suboptimal model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

This research uses well-known data from public repositories that can be shared based on request.

References

  1. McClelland, J.L., Rumelhart, D.E.: Explorations in Parallel Distributed Processing: A Handbook of Models, Programs, and Exercises. MIT Press, Cambridge (1989)

    Google Scholar 

  2. Zhai, J., Zhang, S., Wang, C.: The classification of imbalanced large data sets based on mapreduce and ensemble of elm classifiers. Int. J. Mach. Learn. Cybern. 8, 1009–1017 (2017)

    Article  Google Scholar 

  3. Tenenbaum, J.B., Kemp, C., Griffiths, T.L., Goodman, N.D.: How to grow a mind: statistics, structure, and abstraction. Science 331(6022), 1279–1285 (2011)

    Article  MathSciNet  Google Scholar 

  4. Bellmund, J.L.S., Gärdenfors, P., Moser, E.I., Doeller, C.F.: Navigating cognition: spatial codes for human thinking. Science 362(6415), eaat6766 (2018)

    Article  Google Scholar 

  5. Ye, Q., Shi, W., Kaige, Q., He, H., Zhuang, W., Shen, X.: Joint ran slicing and computation offloading for autonomous vehicular networks: a learning-assisted hierarchical approach. IEEE Open J. Veh. Technol. 2, 272–288 (2021)

    Article  Google Scholar 

  6. Miller, G.A.: Wordnet: a lexical database for English. Commun. ACM 38(11), 39–41 (1995)

    Article  Google Scholar 

  7. Zheng, Yu., Fan, J., Zhang, J., Gao, X.: Exploiting related and unrelated tasks for hierarchical metric learning and image classification. IEEE Trans. Image Process. 29, 883–896 (2019)

    Article  MathSciNet  Google Scholar 

  8. Zhou, Yu., Li, X., Yucan Zhou, Yu., Wang, Q.H., Wang, W.: Deep collaborative multi-task network: a human decision process inspired model for hierarchical image classification. Pattern Recognit. 124, 108449 (2022)

    Article  Google Scholar 

  9. Bertinetto, L., Mueller, R., Tertikas, K., Samangooei, S., Lord N.A.: Making better mistakes: leveraging class hierarchies with deep networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12506–12515 (2020)

  10. Mao, R., He, J., Shao, Z., Yarlagadda, S.K., Zhu, F.: Visual aware hierarchy based food recognition. In: International Conference on Pattern Recognition, pp. 571–598. Springer (2021)

  11. Qiu, Z., Minjie, H., Zhao, H.: Hierarchical classification based on coarse-to fine-grained knowledge transfer. Int. J. Approx. Reason. 149, 61–69 (2022)

    Article  MathSciNet  Google Scholar 

  12. Chen, J., Wang, P., Liu, J., Qian, Y.: Label relation graphs enhanced hierarchical residual network for hierarchical multi-granularity classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4858–4867 (2022)

  13. Roy, D., Panda, P., Roy, K.: Tree-CNN: a hierarchical deep convolutional neural network for incremental learning. Neural Netw. 121, 148–160 (2020)

    Article  Google Scholar 

  14. Zhu, X., Bain, M.: B-CNN: branch convolutional neural network for hierarchical classification. arXiv preprint arXiv:1709.09890 (2017)

  15. Chen, T., Wu, W., Gao, Y., Dong, L., Luo, X., Lin, L.: Fine-grained representation learning and recognition by exploiting hierarchical semantic embedding. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 2023–2031 (2018)

  16. Blondel, V.D., Guillaume, J.-L., Lambiotte, R., Lefebvre, E.: Fast unfolding of communities in large networks. J. Stat. Mech. Theory Exp. 2008(10), P10008 (2008)

    Article  Google Scholar 

  17. Zhang, J., Mei, K., Zheng, Y., Fan, J.: Learning multi-layer coarse-to-fine representations for large-scale image classification. Pattern Recognit. 91, 175–189 (2019)

    Article  Google Scholar 

  18. Zheng, J., Wang, Y., Pei, S., Hu, Q.: Exploring and exploiting hierarchical structures for large-scale classification. Int. J. Mach. Learn. Cybern. 15, 2427–2437 (2023)

    Article  Google Scholar 

  19. Huang, H., Wang, Y., Hu, Q.: Building hierarchical class structures for extreme multi-class learning. Int. J. Mach. Learn. Cybern. 14(7), 2575–2590 (2023)

    Article  Google Scholar 

  20. Zheng, Yu., Fan, J., Zhang, J., Gao, X.: Hierarchical learning of multi-task sparse metrics for large-scale image classification. Pattern Recognit. 67, 97–109 (2017)

    Article  Google Scholar 

  21. Guo, B., Han, S., Han, X., Huang, H., Ting, L.: Label confusion learning to enhance text classification models. In: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, pp. 12929–12936 (2021)

  22. Zheng, Yu., Chen, Q., Fan, J., Gao, X.: Hierarchical convolutional neural network via hierarchical cluster validity based visual tree learning. Neurocomputing 409, 408–419 (2020)

    Article  Google Scholar 

  23. Tan, Z., Wang, Z., Zhang, Y.: Seal: simultaneous label hierarchy exploration and learning. arXiv preprint arXiv:2304.13374 (2023)

  24. Yang, Z., Bastan, M., Zhu, X., Gray, D., Samaras, D.: Hierarchical proxy-based loss for deep metric learning. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1859–1868 (2022)

  25. Barz, B., Denzler, J.: Hierarchy-based image embeddings for semantic image retrieval. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 638–647. IEEE (2019)

  26. Wang, H., Shen, J., Liu, Y., Gao, Y., Gavves, E.: NFormer: robust person re-identification with neighbor transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7297–7307 (2022)

  27. Tang, S., Guo, D., Hong, R., Wang, M.: Graph-based multimodal sequential embedding for sign language translation. IEEE Trans. Multimedia 24, 4433–4445 (2021)

    Article  Google Scholar 

  28. Yang, B., Zou, Y., Liu, F., Zhang, C.: Non-autoregressive coarse-to-fine video captioning. In: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, pp. 3119–3127 (2021)

  29. Yan, Z., Zhang, H., Piramuthu, R., Jagadeesh, V., DeCoste, D., Di, W., Yu, Y.: HD-CNN: hierarchical deep convolutional neural networks for large scale visual recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2740–2748 (2015)

  30. Cerri, R., Barros, R.C., de Carvalho, A.C.P.L.F., Jin, Y.: Reduction strategies for hierarchical multi-label classification in protein function prediction. BMC Bioinform. 17(1), 1–24 (2016)

    Article  Google Scholar 

  31. Wehrmann, J., Cerri, R., Barros, R.: Hierarchical multi-label classification networks. In: International Conference on Machine Learning, pp. 5075–5084. PMLR (2018)

  32. Giunchiglia, E., Lukasiewicz, T.: Coherent hierarchical multi-label classification networks. Adv. Neural Inf. Process. Syst. 33, 9662–9673 (2020)

    Google Scholar 

  33. Chang, D., Pang, K., Zheng, Y., Ma, Z., Song, Y.-Z., Guo, J.: Your “flamingo” is my “bird”: fine-grained, or not. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11476–11485 (2021)

  34. Chopra, S., Hadsell, R., LeCun, Y. Learning a similarity metric discriminatively, with application to face verification. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 1, pp. 539–546. IEEE (2005)

  35. Fuglede, B., Topsoe, F.: Jensen–Shannon divergence and Hilbert space embedding. In: International Symposium on Information Theory, 2004. ISIT 2004. Proceedings, pp. 31. IEEE (2004)

  36. Kullback, S., Leibler, R.A.: On information and sufficiency. Ann. Math. Stat. 22(1), 79–86 (1951)

    Article  MathSciNet  Google Scholar 

  37. Wah, C., Branson, C., Welinder, P., Perona, P., Belongie, S.: The CALTECH-UCSD birds-200-2011 dataset (2011)

  38. Maji, M, Rahtu, E., Kannala, J., Blaschko,M., Vedaldi, A.: Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 (2013)

  39. Krause, J., Stark, M., Deng, J., Fei-Fei, L .: 3D object representations for fine-grained categorization. In: Proceedings of the IEEE international conference on computer vision workshops, pp. 554–561 (2013)

  40. Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

  41. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)

  42. Kiritchenko, S., Matwin, A., Famili, A.F. et al.: Functional annotation of genes using hierarchical text categorization. In: Proceedings of the ACL Workshop on Linking Biological Literature, Ontologies and Databases: Mining Biological Semantics (2005)

  43. Pizarro, I., Ñanculef, R., Valle, C.: An attention-based architecture for hierarchical classification with CNNs. IEEE Access 11, 32972–32995 (2023)

    Article  Google Scholar 

  44. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

Download references

Acknowledgements

This work was supported by the Fujian Province Nature Science Foundation under Grant No. 2020J01813 and No. 2021J011002, the Research Project on Education and Teaching Reform of Undergraduate Colleges and Universities in Fujian Province under Grant No. FBJG20210070 and No. FBJY20230170, and the 2022 Annual Project of the Fourteenth Five-Year Plan for Fujian Educational Science under Grant No. FJJKBK22-173.

Author information

Author notes

  1. Wenjie Zhang and Hongbin Zhang contributed equally to this work.

Authors and Affiliations

  1. School of Computer Science, Minnan Normal University, Zhangzhou, 363000, Fujian, China

    Juan Jiang, Jingmin Yang, Wenjie Zhang & Hongbin Zhang

  2. Key Laboratory of Data Science and Intelligence Application, Fujian Province University, Zhangzhou, 363000, Fujian, China

    Juan Jiang, Jingmin Yang, Wenjie Zhang & Hongbin Zhang

  3. Department of Electronic Engineering, National Taipei University of Technology, Taipei, China

    Jingmin Yang

Authors
  1. Juan Jiang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Jingmin Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Wenjie Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Hongbin Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Jingmin Yang.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Communicated by Xun Yang.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, J., Yang, J., Zhang, W. et al. Hierarchical multi-granularity classification based on bidirectional knowledge transfer. Multimedia Systems 30, 207 (2024). https://doi.org/10.1007/s00530-024-01412-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00530-024-01412-x

Keywords