Skip to main content
SpringerLink
Log in

A Word-Concept Heterogeneous Graph Convolutional Network for Short Text Classification

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Text classification is an important task in natural language processing. However, most of the existing models focus on long texts, and their performance in short texts is not satisfied due to the problem of data sparsity. To solve this problem, recent studies have introduced the concepts of words to enrich the representation of short texts. However, these methods ignore the interactive information between words and concepts and lead introduced concepts to be noises unsuitable for semantic understanding. In this paper, we propose a new model called word-concept heterogeneous graph convolution network (WC-HGCN) to introduce interactive information between words and concepts for short text classification. WC-HGCN develops words and relevant concepts and adopts graph convolution networks to learn the representation with interactive information. Furthermore, we design an innovative learning strategy, which can make full use of the introduced concept information. Experimental results on seven real short text datasets show that our model outperforms latest baseline methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Availability of data and material

All datasets are available online.

Code Availability

The code of this paper is available online.

Notes

  1. https://concept.research.microsoft.com/Home/API.

  2. https://drive.google.com/file/d/1Li2QyQm8dCVT81jBLmbT0tQm6HNVzTyA.

  3. https://www.cs.york.ac.uk/semeval-2013/task2/.

  4. http://nlp.stanford.edu/data/glove.6B.zip.

References

  1. Alsmadi IM, Gan KH (2019) Review of short-text classification. Int J Web Inf Syst 15(2):155–182. https://doi.org/10.1108/IJWIS-12-2017-0083

    Article  Google Scholar 

  2. Arevian G (2007) Recurrent neural networks for robust real-world text classification. In: 2007 IEEE / WIC/ACM international conference on web intelligence, WI 2007, 2–5 November 2007, Silicon Valley, CA, USA, Main Conference Proceedings, IEEE Computer Society, pp 326–329. https://doi.org/10.1109/WI.2007.126

  3. Batal I, Hauskrecht M (2009) Boosting KNN text classification accuracy by using supervised term weighting schemes. In: Cheung DW, Song I, Chu WW, Hu X, Lin JJ (eds) Proceedings of the 18th ACM conference on information and knowledge management, CIKM 2009, Hong Kong, China, November 2–6, 2009, ACM, pp 2041–2044. https://doi.org/10.1145/1645953.1646296

  4. Chen J, Hu Y, Liu J, Xiao Y, Jiang H (2019) Deep short text classification with knowledge powered attention. In: The thirty-third aaai conference on artificial intelligence, AAAI 2019, the thirty-first innovative applications of artificial intelligence conference, IAAI 2019, The Ninth AAAI symposium on educational advances in artificial intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27–February 1, 2019, AAAI Press, pp 6252–6259. https://doi.org/10.1609/aaai.v33i01.33016252

  5. Dilrukshi I, de Zoysa K (2014) A feature selection method for twitter news classification. Int J Mach Learn Comput 4(4):365

    Article  Google Scholar 

  6. Ding W, Yu S, Wang Q, Yu J, Guo Q (2008) A novel naive bayesian text classifier. In: Yu F, Luo Q (eds) International symposium on information processing, ISIP 2008/international pacific workshop on web mining, and web-based application, WMWA 2008, Moscow, Russia, 23–25 May 2008, IEEE Computer Society, pp 78–82.https://doi.org/10.1109/ISIP.2008.54

  7. Du C, Huang L (2018) Text classification research with attention-based recurrent neural networks. Int J Comput Commun Control 13(1):50–61. https://doi.org/10.15837/ijccc.2018.1.3142

    Article  Google Scholar 

  8. Han E, Karypis G, Kumar V (2001) Text categorization using weight adjusted k-nearest neighbor classification. In: Cheung DW, Williams GJ, Li Q (eds) Knowledge discovery and data mining—PAKDD 2001, 5th Pacific-Asia Conference, Hong Kong, China, April 16–18, 2001, Proceedings, Springer, Lecture Notes in Computer Science, vol 2035, pp 53–65.https://doi.org/10.1007/3-540-45357-1_9

  9. Hindi KME, Aljulaidan RR, AlSalman H (2020) Lazy fine-tuning algorithms for naïve bayesian text classification. Appl Soft Comput 96:106652.https://doi.org/10.1016/j.asoc.2020.106652

  10. Hu L, Yang T, Shi C, Ji H, Li X (2019) Heterogeneous graph attention networks for semi-supervised short text classification. In: Inui K, Jiang J, Ng V, Wan X (eds) Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3–7, 2019, Association for Computational Linguistics, pp 4820–4829.https://doi.org/10.18653/v1/D19-1488

  11. Huang L, Ma D, Li S, Zhang X, Wang H (2019) Text level graph neural network for text classification. In: Inui K, Jiang J, Ng V, Wan X (eds) Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3–7, 2019, Association for Computational Linguistics, pp 3442–3448.https://doi.org/10.18653/v1/D19-1345

  12. Islam MZ, Liu J, Li J, Liu L, Kang W (2019) A semantics aware random forest for text classification. In: Zhu W, Tao D, Cheng X, Cui P, Rundensteiner EA, Carmel D, He Q, Yu JX (eds) Proceedings of the 28th ACM international conference on information and knowledge management, CIKM 2019, Beijing, China, November 3–7, 2019, ACM, pp 1061–1070. https://doi.org/10.1145/3357384.3357891

  13. Joulin A, Grave E, Bojanowski P, Mikolov T (2017) Bag of tricks for efficient text classification. In: Lapata M, Blunsom P, Koller A (eds) Proceedings of the 15th conference of the european chapter of the association for computational linguistics, EACL 2017, Valencia, Spain, April 3–7, 2017, Volume 2: Short Papers, Association for Computational Linguistics, pp 427–431.https://doi.org/10.18653/v1/e17-2068

  14. Keerthi SS (2005) Generalized LARS as an effective feature selection tool for text classification with svms. In: Raedt LD, Wrobel S (eds) Machine learning, proceedings of the twenty-second international conference (ICML 2005), Bonn, Germany, August 7–11, 2005, ACM, ACM International Conference Proceeding Series, vol 119, pp 417–424.https://doi.org/10.1145/1102351.1102404

  15. Kim Y (2014) Convolutional neural networks for sentence classification. In: Moschitti A, Pang B, Daelemans W (eds) Proceedings of the 2014 conference on empirical methods in natural language processing, EMNLP 2014, October 25–29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, ACL, pp 1746–1751.https://doi.org/10.3115/v1/d14-1181

  16. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: Bengio Y, LeCun Y (eds) 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings.http://arxiv.org/abs/1412.6980

  17. Li C, Ouyang J, Li X (2019) Classifying extremely short texts by exploiting semantic centroids in word mover’s distance space. In: Liu L, White RW, Mantrach A, Silvestri F, McAuley JJ, Baeza-Yates R, Zia L (eds) The world wide web conference, WWW 2019, San Francisco, CA, USA, May 13–17, 2019, ACM, pp 939–949.https://doi.org/10.1145/3308558.3313397

  18. Li Y, Liu B (2020) A new vector representation of short texts for classification. Int Arab J Inf Technol 17(2):241–249. https://doi.org/10.34028/iajit/17/2/12

    Article  Google Scholar 

  19. Li Y, Tarlow D, Brockschmidt M, Zemel RS (2016) Gated graph sequence neural networks. In: Bengio Y, LeCun Y (eds) 4th international conference on learning representations, ICLR 2016, San Juan, Puerto Rico, May 2–4, 2016, Conference Track Proceedings.http://arxiv.org/abs/1511.05493

  20. Liu P, Qiu X, Huang X (2016) Recurrent neural network for text classification with multi-task learning. In: Kambhampati S (ed) Proceedings of the twenty-fifth international joint conference on artificial intelligence, IJCAI 2016, New York, NY, USA, 9–15 July 2016, IJCAI/AAAI Press, pp 2873–2879.http://www.ijcai.org/Abstract/16/408

  21. Liu X, You X, Zhang X, Wu J, Lv P (2020) Tensor graph convolutional networks for text classification. In: The thirty-fourth AAAI conference on artificial intelligence, AAAI 2020, the thirty-second innovative applications of artificial intelligence conference, IAAI 2020, the tenth AAAI symposium on educational advances in artificial intelligence, EAAI 2020, New York, NY, USA, February 7–12, 2020, AAAI Press, pp 8409–8416.https://aaai.org/ojs/index.php/AAAI/article/view/6359

  22. Niu G, Xu H, He B, Xiao X, Wu H, Gao S (2019) Enhancing local feature extraction with global representation for neural text classification. In: Inui K, Jiang J, Ng V, Wan X (eds) Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3–7, 2019, Association for Computational Linguistics, pp 496–506.https://doi.org/10.18653/v1/D19-1047

  23. Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation. In: Moschitti A, Pang B, Daelemans W (eds) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, ACL, pp 1532–1543.https://doi.org/10.3115/v1/d14-1162

  24. Salles T, Gonçalves MA, Rodrigues V, da Rocha LC (2018) Improving random forests by neighborhood projection for effective text classification. Inf Syst 77:1–21

    Article  Google Scholar 

  25. Scarselli F, Gori M, Tsoi AC, Hagenbuchner M, Monfardini G (2009) The graph neural network model. IEEE Trans Neural Netw 20(1):61–80. https://doi.org/10.1109/TNN.2008.2005605

    Article  Google Scholar 

  26. Shanahan JG, Roma N (2003) Improving SVM text classification performance through threshold adjustment. In: Lavrac N, Gamberger D, Todorovski L, Blockeel H (eds) Machine learning: ECML 2003, 14th European conference on machine learning, Cavtat-Dubrovnik, Croatia, September 22–26, 2003, Proceedings, Springer, Lecture Notes in Computer Science, vol 2837, pp 361–372.https://doi.org/10.1007/978-3-540-39857-8_33

  27. Song G, Ye Y, Du X, Huang X, Bie S (2014) Short text classification: a survey. J Multim 9(5):635–643. https://doi.org/10.4304/jmm.9.5.635-643

    Article  Google Scholar 

  28. Tang J, Qu M, Mei Q (2015) PTE: predictive text embedding through large-scale heterogeneous text networks. In: Cao L, Zhang C, Joachims T, Webb GI, Margineantu DD, Williams G (eds) Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, Sydney, NSW, Australia, August 10–13, 2015, ACM, pp 1165–1174.https://doi.org/10.1145/2783258.2783307

  29. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Guyon I, von Luxburg U, Bengio S, Wallach HM, Fergus R, Vishwanathan SVN, Garnett R (eds) Advances in neural information processing systems 30: annual conference on neural information processing systems 2017, December 4–9, 2017, Long Beach, CA, USA, pp 5998–6008.http://papers.nips.cc/paper/7181-attention-is-all-you-need

  30. Wang S, Li D, Song X, Wei Y, Li H (2011) A feature selection method based on improved fisher’s discriminant ratio for text sentiment classification. Expert Syst Appl 38(7):8696–8702. https://doi.org/10.1016/j.eswa.2011.01.077

    Article  Google Scholar 

  31. Xu B, Huang JZ, Williams GJ, Li MJ, Ye Y (2012) Hybrid random forests: advantages of mixed trees in classifying text data. In: Tan P, Chawla S, Ho CK, Bailey J (eds) Advances in knowledge discovery and data mining—16th Pacific-Asia conference, PAKDD 2012, Kuala Lumpur, Malaysia, May 29-June 1, 2012, Proceedings, Part I, Springer, Lecture Notes in Computer Science, vol 7301, pp 147–158.https://doi.org/10.1007/978-3-642-30217-6_13

  32. Xu J, Cai Y, Wu X, Lei X, Huang Q, Leung H, Li Q (2020) Incorporating context-relevant concepts into convolutional neural networks for short text classification. Neurocomputing 386:42–53. https://doi.org/10.1016/j.neucom.2019.08.080

    Article  Google Scholar 

  33. Xu S (2018) Bayesian naïve bayes classifiers to text classification. J Inf Sci 44(1):48–59. https://doi.org/10.1177/0165551516677946

    Article  MathSciNet  Google Scholar 

  34. Yang Y, Wang H, Zhu J, Wu Y, Jiang K, Guo W, Shi W (2020) Dataless short text classification based on biterm topic model and word embeddings. In: Bessiere C (ed) Proceedings of the twenty-ninth international joint conference on artificial intelligence, IJCAI 2020, ijcai.org, pp 3969–3975.https://doi.org/10.24963/ijcai.2020/549

  35. Yao L, Mao C, Luo Y (2019) Graph convolutional networks for text classification. In: The thirty-third AAAI conference on artificial intelligence, AAAI 2019, the thirty-first innovative applications of artificial intelligence conference, IAAI 2019, the Ninth AAAI symposium on educational advances in artificial intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27—February 1, 2019, AAAI Press, pp 7370–7377.https://doi.org/10.1609/aaai.v33i01.33017370

  36. Zeng J, Li J, Song Y, Gao C, Lyu MR, King I (2018) Topic memory networks for short text classification. In: Riloff E, Chiang D, Hockenmaier J, Tsujii J (eds) Proceedings of the 2018 conference on empirical methods in natural language processing, Brussels, Belgium, October 31—November 4, 2018, Association for Computational Linguistics, pp 3120–3131.https://doi.org/10.18653/v1/d18-1351

  37. Zhang H, Ni W, Zhao M, Lin Z (2019) Cluster-gated convolutional neural network for short text classification. In: Bansal M, Villavicencio A (eds) Proceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, Hong Kong, China, November 3-4, 2019, Association for Computational Linguistics, pp 1002–1011.https://doi.org/10.18653/v1/K19-1094

  38. Zhang Y, Yu X, Cui Z, Wu S, Wen Z, Wang L (2020) Every document owns its structure: Inductive text classification via graph neural networks. In: Jurafsky D, Chai J, Schluter N, Tetreault JR (eds) Proceedings of the 58th annual meeting of the association for computational linguistics, ACL 2020, Online, July 5–10, 2020, Association for Computational Linguistics, pp 334–339. https://doi.org/10.18653/v1/2020.acl-main.31

Download references

Acknowledgements

We thank the anonymous reviewers for their many innovative comments and suggestions.

Funding

This research was supported in part by the National Key R &D Program of China under Grant 2017YFC1703905, the Natural Science Foundation of Sichuan under Grant 2022NSFSC0958, and the Sichuan Science and Technology Program under Grants 2020YFS0372, 2020YFS0302 and 2020YFS0283.

Author information

Authors and Affiliations

  1. Knowledge and Data Engineering Laboratory of Chinese Medicine, School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu, 610054, People’s Republic of China

    Shigang Yang, Yongguo Liu, Yun Zhang & Jiajing Zhu

Authors
  1. Shigang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongguo Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiajing Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yongguo Liu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, S., Liu, Y., Zhang, Y. et al. A Word-Concept Heterogeneous Graph Convolutional Network for Short Text Classification. Neural Process Lett 55, 735–750 (2023). https://doi.org/10.1007/s11063-022-10906-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-022-10906-6

Keywords

Search

Navigation