Skip to main content
SpringerLink
Log in

Few-shot relation classification based on the BERT model, hybrid attention and fusion networks

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Relation classification (RC) is an essential task in information extraction. The distance supervision (DS) method can use many unlabeled data and solve the lack of training data on the RC task. However, the DS method has the problems of long tails and noise. Intuitively, people can solve these problems using few-shot learning (FSL). Our work aims to improve the accuracy and rapidity of convergence on the few-shot RC task. We believe that entity pairs have an essential role in the few-shot RC task. We propose a new context encoder, which is improved based on the bidirectional encoder representations from transformers (BERT) model to fuse entity pairs and their dependence information in instances. At the same time, we design hybrid attention, which includes support instance-level and query instance-level attention. The support instance level dynamically assigns the weight of each instance in the support set. It makes up for the insufficiency of prototypical networks, which distribute weights to sentences equally. Query instance-level attention is dynamically assigned weights to query instances by similarity with the prototype. The ablation study shows the effectiveness of our proposed method. In addition, a fusion network is designed to replace the Euclidean distance method of previous works when class matching is performed, improving the convergence’s rapidity. This makes our model more suitable for industrial applications. The experimental results show that the proposed model’s accuracy is better than that of several other models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

Data openly available in a public repository.

Notes

  1. https://github.com/thunlp/FewRel

References

  1. Wang, L., Cao, Z., De Melo, G., Liu, Z.: Relation classification via multi-level attention cnns. In: 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7, 2016 - August 12, 2016. 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Long Papers, vol. 3, pp. 1298–1307. Association for Computational Linguistics (ACL). 10.18653/v1/p16-1123

  2. Chen T, Wang N, Wang H, Zhan H (2021) Distant supervision for relation extraction with sentence selection and interaction representation. Wireless Communications and Mobile Computing 2021:1–16. https://doi.org/10.1155/2021/8889075

    Article  Google Scholar 

  3. Feng RW, Zheng XS, Gao TX, Chen JT, Wang WZ, Chen DZ, Wu J (2021) Interactive few-shot learning: Limited supervision, better medical image segmentation. IEEE Transactions on Medical Imaging 40(10):2575–2588. https://doi.org/10.1109/tmi.2021.3060551

    Article  Google Scholar 

  4. Ye HJ, Hu HX, Zhan DC (2021) Learning adaptive classifiers synthesis for generalized few-shot learning. International Journal of Computer Vision 129(6):1930–1953. https://doi.org/10.1007/s11263-020-01381-4

    Article  MATH  Google Scholar 

  5. Gao, T., Han, X., Zhu, H., Liu, Z., Li, P., Sun, M., Zhou, J.: Fewrel 2.0: Towards more challenging few-shot relation classification. In: 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, November 3, 2019 - November 7, 2019. EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pp. 6250–6255. Association for Computational Linguistics. 10.18653/v1/D19-1649

  6. Han, X., Zhu, H., Yu, P., Wang, Z., Yao, Y., Liu, Z., Sun, M.: Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In: 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, October 31, 2018 - November 4, 2018. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, pp. 4803–4809. Association for Computational Linguistics. 10.18653/v1/D18-1514

  7. Chen YS, Chiang SW, Wu ML (2022) A few-shot transfer learning approach using text-label embedding with legal attributes for law article prediction. Applied Intelligence 52(3):2884–2902. https://doi.org/10.1007/s10489-021-02516-x

    Article  Google Scholar 

  8. Gao, T., Han, X., Liu, Z., Sun, M.: Hybrid attention-based prototypical networks for noisy few-shot relation classification. In: 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Annual Conference on Innovative Applications of Artificial Intelligence, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, January 27, 2019 - February 1, 2019. 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, pp. 6407–6414. AAAI Press. 10.1609/aaai.v33i01.33016407

  9. Xie Y, Wang H, Yu B, Zhang C (2020) Secure collaborative few-shot learning. Knowledge-Based Systems 203:10. https://doi.org/10.1016/j.knosys.2020.106157

    Article  Google Scholar 

  10. Xu H, Wang JX, Li H, Ouyang DQ, Shao J (2021) Unsupervised meta-learning for few-shot learning. Pattern Recognition 116:10. https://doi.org/10.1016/j.patcog.2021.107951

    Article  Google Scholar 

  11. Li DW, Tian YJ (2018) Survey and experimental study on metric learning methods. Neural Networks 105:447–462. https://doi.org/10.1016/j.neunet.2018.06.003

    Article  Google Scholar 

  12. Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: 31st Annual Conference on Neural Information Processing Systems, NIPS 2017, December 4, 2017 - December 9, 2017. Advances in Neural Information Processing Systems, vol. 2017-December, pp. 4078–4088. Neural information processing systems foundation

  13. Li LQ, Wang JB, Li JC, Ma QL, Wei J (2019) Relation classification via keyword-attentive sentence mechanism and synthetic stimulation loss. IEEE-ACM Transactions on Audio Speech and Language Processing 27(9):1392–1404. https://doi.org/10.1109/taslp.2019.2921726

    Article  Google Scholar 

  14. Sun HY, Grishman R (2022) Lexicalized dependency paths based supervised learning for relation extraction. Computer Systems Science and Engineering 43(3):861–870. https://doi.org/10.32604/csse.2022.030759

    Article  Google Scholar 

  15. Shi Y, Xiao Y, Quan P, Lei ML, Niu LF (2021) Distant supervision relation extraction via adaptive dependency-path and additional knowledge graph supervision. Neural Networks 134:42–53. https://doi.org/10.1016/j.neunet.2020.10.012

    Article  Google Scholar 

  16. Liu Y, Li SJ, Wei FR, Ji H (2016) Relation classification via modeling augmented dependency paths. IEEE-ACM Transactions on Audio Speech and Language Processing 24(9):1589–1598. https://doi.org/10.1109/taslp.2016.2573050

    Article  Google Scholar 

  17. Ma, Y.H., Zhu, J., Liu, J.: Enhanced semantic representation learning for implicit discourse relation classification. Applied Intelligence (2021). 10.1007/s10489-021-02785-6

  18. Runyan Z, Fanrong M, Yong Z, Bing L (2018) Relation classification via recurrent neural network with attention and tensor layers. Big Data Mining and Analytics 1(3):234–244. https://doi.org/10.26599/bdma.2018.9020022

    Article  Google Scholar 

  19. Mintz, M., Bills, S., Snow, R., Jurafsky, D.: Distant supervision for relation extraction without labeled data. In: ACL 2009, Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the AFNLP, 2-7 August 2009, Singapore. 10.3115/1690219.1690287

  20. Zhang, N., Deng, S., Sun, Z., Wang, G., Chen, X., Zhang, W., Chen, H.: Long-tail relation extraction via knowledge graph embeddings and graph convolution networks. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3016–3025. Association for Computational Linguistics. 10.18653/v1/N19-1306

  21. Khan MS, Lohani QMD (2022) Topological analysis of intuitionistic fuzzy distance measures with applications in classification and clustering. Engineering Applications of Artificial Intelligence 116. https://doi.org/10.1016/j.engappai.2022.105415

  22. Hallajian B, Motameni H, Akbari E (2022) Ensemble feature selection using distance-based supervised and unsupervised methods in binary classification. Expert Systems with Applications 200:18. https://doi.org/10.1016/j.eswa.2022.116794

    Article  Google Scholar 

  23. Jiang W, Huang K, Geng J, Deng XY (2021) Multi-scale metric learning for few-shot learning. IEEE Transactions on Circuits and Systems for Video Technology 31(3):1091–1102. https://doi.org/10.1109/tcsvt.2020.2995754

    Article  Google Scholar 

  24. Lake BM, Salakhutdinov R, Tenenbaum JB (2015) Human-level concept learning through probabilistic program induction. Science 350(6266):1332–1338. https://doi.org/10.1126/science.aab3050

    Article  MathSciNet  MATH  Google Scholar 

  25. Li, W.B., Xu, J.L., Huo, J., Wang, L., Gao, Y., Luo, J.B., Aaai: Distribution consistency based covariance metric networks for few-shot learning. In: 33rd AAAI Conference on Artificial Intelligence / 31st Innovative Applications of Artificial Intelligence Conference / 9th AAAI Symposium on Educational Advances in Artificial Intelligence, pp. 8642–8649. Assoc Advancement Artificial Intelligence, PALO ALTO (2019). 10.1609/aaai.v33i01.33018642

  26. Xie, Y.X., Xu, H., Yang, C.C., Gao, K., Assoc Advancement Artificial, I.: Multi-channel convolutional neural networks with adversarial training for few-shot relation classification. In: 34th AAAI Conference on Artificial Intelligence / 32nd Innovative Applications of Artificial Intelligence Conference / 10th AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI Conference on Artificial Intelligence, vol. 34, pp. 13967–13968. Assoc Advancement Artificial Intelligence, PALO ALTO (2020). 10.1609/aaai.v34i10.7256

  27. Xie Y, Xu H, Li J, Yang C, Gao K (2020) Heterogeneous graph neural networks for noisy few-shot relation classification. Knowledge-Based Systems 194. https://doi.org/10.1016/j.knosys.2020.105548

  28. Ye, Z.X., Ling, Z.H.: Multi-level matching and aggregation network for few-shot relation classification. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 10.18653/v1/P19-1277

  29. Gao, T.Y., Han, X., Xie, R.B., Liu, Z.Y., Lin, F., Lin, L.Y., Sun, M.S., Assoc Advancement Artificial, I.: Neural snowball for few-shot relation learning. In: 34th AAAI Conference on Artificial Intelligence / 32nd Innovative Applications of Artificial Intelligence Conference / 10th AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI Conference on Artificial Intelligence, vol. 34, pp. 7772–7779. Assoc Advancement Artificial Intelligence, PALO ALTO (2020). 10.1609/aaai.v34i05.6281

  30. Pang N, Tan Z, Xu H, Xiao WD (2020) Boosting knowledge base automatically via few-shot relation classification. Frontiers in Neurorobotics 14. https://doi.org/10.3389/fnbot.2020.584192

  31. Xiao, Y., Jin, Y., Hao, K.: Adaptive prototypical networks with label words and joint representation learning for few-shot relation classification. IEEE transactions on neural networks and learning systems PP (2021). 10.1109/tnnls.2021.3105377

  32. Wu JY, Zhao ZB, Sun C, Yan RQ, Chen XF (2020) Few-shot transfer learning for intelligent fault diagnosis of machine. Measurement 166. https://doi.org/10.1016/j.measurement.2020.108202

  33. Hou, Y.T., Lai, Y.K., Wu, Y.S., Che, W.X., Liu, T., Assoc Advancement Artificial, I.: Few-shot learning for multi-label intent detection. In: 35th AAAI Conference on Artificial Intelligence / 33rd Conference on Innovative Applications of Artificial Intelligence / 11th Symposium on Educational Advances in Artificial Intelligence. AAAI Conference on Artificial Intelligence, vol. 35, pp. 13036–13044. Assoc Advancement Artificial Intelligence, PALO ALTO (2021)

  34. Bao, Y., Wu, M., Chang, S., Barzilay, R.: Few-shot text classification with distributional signatures. In: International Conference on Learning Representations (2020). https://doi.org/10.1007/978-981-33-4859-2_14

  35. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: 31st Annual Conference on Neural Information Processing Systems, NIPS 2017, December 4, 2017 - December 9, 2017. Advances in Neural Information Processing Systems, vol. 2017-December, pp. 5999–6009. Neural information processing systems foundation

  36. Chen XF, Wang GH, Ren HP, Cai Y, Leung HF, Wang T (2022) Task-adaptive feature fusion for generalized few-shot relation classification in an open world environment. IEEE-ACM Transactions on Audio Speech and Language Processing 30:1003–1015. https://doi.org/10.1109/taslp.2022.3153254

  37. Wang, W., Yan, M., Wu, C.: Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1705–1714. Association for Computational Linguistics. 10.18653/v1/P18-1158

Download references

Acknowledgements

This paper was supported by the Exercise and Health Laboratory of the Institute of Intelligent Machinery, Chinese Academy of Sciences. Thanks to Tsinghua University for developing the FewRel dataset. We would like to thank the anonymous reviewers for their helpful comments.

Funding

This research was supported by the National Key Research and Development Program of China (2022YFC2010200, 2020YFC2005603), the National Natural Science Foundation of China (NSFC) (grant numbers 61701482), the Key projects of the National Natural Science Foundation of universities in Anhui Province (grant number KJ2020A0112),the Major Special Projects of Anhui Province(grant number 202103a07020004), the Natural Science Foundation of Anhui Province, China (grant number 1808085MF191),the Education Research Project of Anhui Province, China(2020jyxm1573) and the High-level Talents Research Start-up Fund of Hefei Normal University (grant number 2020rcjj45). In addition, we would like to thank the anonymous reviewers who have helped to improve the paper.

Author information

Authors and Affiliations

  1. Institute of Intelligent Machines, Institutes of Physical Science, Chinese Academy of Sciences, Hefei, 230031, China

    Yibing Li, Zenghui Ding, Zuchang Ma, Yichen Wu, Yu Wang & Ruiqi Zhang

  2. Science Island Branch of Graduate School, University of Science and Technology of China, Hefei, 230026, China

    Yibing Li, Yichen Wu, Yu Wang & Ruiqi Zhang

  3. School of Computer Science and Technology, Hefei Normal University, Hefei, 230601, China

    Yibing Li, Fei Xie & Xiaoye Ren

Authors
  1. Yibing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zenghui Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zuchang Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yichen Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ruiqi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Fei Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xiaoye Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Zenghui Ding or Zuchang Ma.

Ethics declarations

Conflict of interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work. There is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of the manuscript entitled.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Ding, Z., Ma, Z. et al. Few-shot relation classification based on the BERT model, hybrid attention and fusion networks. Appl Intell 53, 21448–21464 (2023). https://doi.org/10.1007/s10489-023-04634-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04634-0

Keywords

Search

Navigation