Shaokang Wang
Abstract
Structured text with plentiful hierarchical structure information is an important part in real-world complex texts. Structured text classification is attracting more attention in natural language processing due to the increasing complexity of application scenarios. Most existing methods treat structured text from a local hierarchy perspective, focusing on the semantics dependency and the graph structure of the structured text independently. However, structured text has global hierarchical structures with sophisticated dependency when compared to unstructured text. According to the variety of structured texts, it is not appropriate to use the existing methods directly. The function of distinction information within semantics dependency and graph structure for structured text, referred to as meta-information, should be stated more precisely. In this article, we propose HGMETA, a novel meta-information embedding frame network for structured text classification, to obtain the fusion embedding of hierarchical semantics dependency and graph structure in a structured text, and to distill the meta-information from fusion characteristics. To integrate the global hierarchical features with fused structured text information, we design a hierarchical LDA module and a structured text embedding module. Specially, we employ a multi-hop message passing mechanism to explicitly incorporate complex dependency into a meta-graph. The meta-information is constructed from meta-graph via neighborhood-based propagation to distill redundant information. Furthermore, using an attention-based network, we investigate the complementarity of semantics dependency and graph structure based on global hierarchical characteristics and meta-information. Finally, the fusion embedding and the meta-information can be straightforwardly incorporated for structured text classification. Experiments conducted on three real-world datasets show the effectiveness of meta-information and demonstrate the superiority of our method.
- [1] Ion Androutsopoulos, John Koutsias, Konstantinos Chandrinos, Georgios Paliouras, and Constantine D. Spyropoulos. 2000. An evaluation of Naive Bayesian anti-spam filtering. CoRR cs.CL/0006013 (2000). https://arxiv.org/abs/cs/0006013.Google Scholar
- [2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1409.0473.Google Scholar
- [3] Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew M. Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. 2018. Relational inductive biases, deep learning, and graph networks. CoRR abs/1806.01261 (2018). arXiv:1806.01261 http://arxiv.org/abs/1806.01261.Google Scholar
- [4] . 2021. Structural mining for link prediction using various machine learning algorithms. International Journal of Social Ecology and Sustainable Development 12, 3 (2021), 66–78.Google ScholarCross Ref
- [5] . 2006. Correlated topic models. In Proceedings of the 18th International Conference on Neural Information Processing Systems. 147.Google Scholar
- [6] . 2007. Supervised topic models. In Proceedings of the 20th International Conference on Neural Information Processing Systems. 121–128.Google ScholarDigital Library
- [7] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res. 3 (2003), 993–1022. http://jmlr.org/papers/v3/blei03a.html.Google Scholar
- [8] . 2018. A comprehensive survey of graph embedding: Problems, techniques, and applications. IEEE Transactions on Knowledge and Data Engineering 30, 9 (2018), 1616–1637.Google ScholarDigital Library
- [9] . 2017. Aspect-level sentiment classification with heat (hierarchical attention) network. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 97–106.Google ScholarDigital Library
- [10] . 1998. Learning to Extract Symbolic Knowledge from the World Wide Web.
Technical Report . Carnegie-Mellon Univ Pittsburgh PA School of Computer Science.Google Scholar - [11] . 2008. BNS feature scaling: An improved representation over TF-IDF for SVM text classification. In Proceedings of the 17th ACM Conference on Information and Knowledge Management. 263–270.Google ScholarDigital Library
- [12] . 2019. Text level graph neural network for text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 3435–3441.Google ScholarCross Ref
- [13] . 1998. Text categorization with support vector machines: Learning with many relevant features. In Proceedings of the 10th European Conference on Machine Learning. Springer, 137–142.Google ScholarDigital Library
- [14] . 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing.Google Scholar
- [15] . 2014. Adam: A method for stochastic optimization. InProceedings of the International Conference on Learning Representations.Google Scholar
- [16] . 2016. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations.Google Scholar
- [17] . 2018. Analogical reasoning on Chinese morphological and semantic relations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 138–143.Google ScholarCross Ref
- [18] . 2015. Gated graph sequence neural networks. In Proceedings of the International Conference on Learning Representations.Google Scholar
- [19] . 2019. Fi-GNN: Modeling feature interactions via graph neural networks for CTR prediction. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 539–548.Google ScholarDigital Library
- [20] . 2018. Hierarchical attention transfer network for cross-domain sentiment classification. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32.Google ScholarCross Ref
- [21] Tingting Liang, Lei Zheng, Liang Chen, Yao Wan, Philip S. Yu, and Jian Wu. 2020. Multi-view factorization machines for mobile app recommendation based on hierarchical attention. Knowl. Based Syst. 187 (2020), 104821. Google ScholarCross Ref
- [22] . 2017. The application of LDA model in patent text classification. Journal of Modern Information 37, 3 (2017), 35–39.Google Scholar
- [23] . 2017. A structured self-attentive sentence embedding. InProceedings of the International Conference on Learning Representations.Google Scholar
- [24] . 2016. Recurrent neural network for text classification with multi-task learning. In Proceedings of the 25th International Joint Conference on Artificial Intelligence. 2873–2879.Google Scholar
- [25] . 2010. Recurrent neural network based language model. In Proceedings of the 11th Annual Conference of the International Speech Communication Association.Google ScholarCross Ref
- [26] . 2020. Message passing attention networks for document understanding. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34, 8544–8551.Google ScholarCross Ref
- [27] . 2017. Text classification method based on self-training and LDA topic models. Expert Systems with Applications 80, C (2017), 83–93.Google ScholarDigital Library
- [28] . 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 1532–1543.Google ScholarCross Ref
- [29] . 2015. Text categorization as a graph classification problem. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 1702–1712.Google ScholarCross Ref
- [30] Md. Shahriare Satu, Md. Imran Khan, Mufti Mahmud, Shahadat Uddin, Matthew A. Summers, Julian M. W. Quinn, and Mohammad Ali Moni. 2021. TClustVID: A novel machine learning classification model to investigate topics and sentiment in COVID-19 tweets. Knowl. Based Syst. 226 (2021), 107126. Google ScholarCross Ref
- [31] . 2018. Stance detection with hierarchical attention network. In Proceedings of the 27th International Conference on Computational Linguistics. 2399–2409.Google Scholar
- [32] . 2006. An effective refinement strategy for KNN text classifier. Expert Systems with Applications 30, 2 (2006), 290–298.Google ScholarDigital Library
- [33] . 2020. Distinguish confusing law articles for legal judgment prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 3086–3095.Google ScholarCross Ref
- [34] . 2020. Neural message passing for multi-relational ordered and recursive hypergraphs. In Proceedings of the 34th International Conference on Neural Information Processing Systems. 33.Google Scholar
- [35] . 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1480–1489.Google ScholarCross Ref
- [36] . 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33, 7370–7377.Google ScholarDigital Library
- [37] . 2018. Sequential recommender system based on hierarchical attention network. In Proceedings of the 27th International Joint Conference on Artificial Intelligence.Google ScholarDigital Library
- [38] . 2018. Learning structured representation for text classification via reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32.Google ScholarCross Ref
- [39] . 2020. Every document Owns its structure: Inductive text classification via graph neural networks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 334–339.Google ScholarCross Ref
- [40] . 2020. Relational graph neural network with hierarchical attention for knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34, 9612–9619.Google ScholarCross Ref
Index Terms
- Meta-Information Fusion of Hierarchical Semantics Dependency and Graph Structure for Structured Text Classification
Recommendations
A structured documents retrieval method supporting attribute-based structure information
SAC '02: Proceedings of the 2002 ACM symposium on Applied computingThere are many studies on retrieval methods for structured documents but most of the studies are for those whose structure information is expressed by elements. But when elements are used to describe a document structure, the structure becomes static ...
Meta-Information in Conversational Search
The exchange of meta-information has always formed part of information behavior. In this article, we show that this rule also extends to conversational search. Information about the user’s information need, their preferences, and the quality of search ...
Efficiently linking text documents with relevant structured information
VLDB '06: Proceedings of the 32nd international conference on Very large data basesFaced with growing knowledge management needs, enterprises are increasingly realizing the importance of interlinking critical business information distributed across structured and unstructured data sources. We present a novel system, called EROCS, for ...
Comments