Skip Abstract Section
Abstract
We propose a semantic template-based distributed representation for the convolutional neural network called Semantic Template-based Convolutional Neural Network (STCNN) for text categorization that imitates the perceptual behavior of human comprehension. STCNN is a highly automatic approach that learns semantic templates that characterize a domain from raw text and recognizes categories of documents using a semantic-infused convolutional neural network that allows a template to be partially matched through a statistical scoring system. Our experiment results show that STCNN effectively classifies documents in about 140,000 Chinese news articles into predefined categories by capturing the most prominent and expressive patterns and achieves the best performance among all compared methods for Chinese topic classification. Finally, the same knowledge can be directly used to perform a semantic analysis task.
- [1] . 1994. A comparison of two learning algorithms for text categorization. In Proceedings of the 3rd Annual Symposium on Document Analysis and Information Retrieval. 81–93.Google Scholar
- [2] . 2019. Knowledge-Based systems survey. Int. J. Acad. Eng. Res. 3, 7 (2019), 1–22.Google Scholar
- [3] . 2016. Diversity of rule-based approaches: Classic systems and recent applications. Avant: Trends Interdisc. Studies 7, 2 (2016), 104–116.Google Scholar
- [4] . 2000. Rule-based and rule-generating systems. In: (Eds.). Downward Causation. University of Aarhus Press, Aarhus, Denmark, 278–301.Google Scholar
- [5] . 2016. On the Incoherence Objection to Rule-Utilitarianism. Ethical Theory Moral Pract. 19, 4 (2016), 857–876.Google ScholarCross Ref
- [6] . 2013. Rule-based information extraction is dead! Long live rule-based information extraction systems! In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 827–832.Google Scholar
- [7] . 2004. Discovering golden nuggets: Data mining in financial application. IEEE Trans. Syst., Man, Cybernet. C (Appl. Rev.) 34, 4 (2004), 513–522.
DOI: Google ScholarDigital Library
- [8] . 2020. Deep learning for financial applications: A survey. Appl. Soft Comput. 93 (2020), 106384. Google ScholarCross Ref
- [9] . 2015. A systematic review of machine learning techniques for software fault prediction. Appl. Soft Comput. 27 (2015), 504–518.Google ScholarDigital Library
- [10] . 2002. Machine learning in automated text categorization, ACM Comput. Surveys 34, 1 (2002), 1–47.Google ScholarDigital Library
- [11] . 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12 (Aug. 2011), 2493–2537.Google ScholarCross Ref
- [12] . 2021. Computational Intelligence in the hospitality industry: A systematic literature review and a prospect of challenges. Appl. Soft Comput. (2021), 107082. Google ScholarDigital Library
- [13] . 2019. Darpa's explainable artificial intelligence program. AI Mag. 40, 2 (2019), 44–58.Google ScholarDigital Library
- [14] . 2020. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Info. Fusion 58 (2020), 82–115.Google ScholarDigital Library
- [15] . 2001. Applying machine learning toward an automatic classification of it. Lit. Ling. Comput. 16, 1 (2001), 45–58.Google ScholarCross Ref
- [16] . 1962. Automatic document classification. Technical Report TM-771, System Development Corporation, Santa Monica, CA.Google Scholar
- [17] . 2005. Text categorization. In Encyclopedia of Database Technologies and Applications, IGI Global, 683–687.Google ScholarCross Ref
- [18] . 2010. Text clustering based on domain ontology and latent semantic analysis. In Proceedings of the International Conference on Asian Language Processing, IEEE, 219–222.Google ScholarDigital Library
- [19] . 2010. Multi-level structured models for document-level sentiment classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1046–1056.Google Scholar
- [20] . 2009. Template matching. In Pattern Recognition, 4th ed. Sergios Theodoridis, Konstantinos Koutroumbas (Eds.). Academic Press, 481–519. Google ScholarCross Ref
- [21] . 2010. Improving the use of pseudo-words for evaluating selectional preferences. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. 445–453.Google ScholarDigital Library
- [22] . 2013. Discriminative object tracking via sparse representation and online dictionary learning, IEEE Trans. Cybernet. 44, 4 (2013), 539–553.Google ScholarCross Ref
- [23] . 2015. A framework for multi-document abstractive summarization based on semantic role labelling. Appl. Soft Comput. 30 (2015), 737–747.Google ScholarDigital Library
- [24] . 1979. An entity/event data modelling method. Comput. J. 22, 3 (1979), 270–281.Google ScholarCross Ref
- [25] . 2021. Deep learning and multilingual sentiment analysis on social media data: An overview. Appl. Soft Comput. (2021), 107373.Google ScholarCross Ref
- [26] . 1969. New methods in automatic extracting. J. ACM 16, 2 (1969), 264–285.Google ScholarDigital Library
- [27] . 2013. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 8 (2013), 1798–1828.Google ScholarDigital Library
- [28] . 2008. The element of statistical learning: Data mining, inference and prediction. In Springer Series in Statistics (2nd ed.). Springer, 764.Google Scholar
- [29] . 2013. Efficient estimation of word representations in vector space. In Proceedings of International Conference of Learning Representations Workshop.Google Scholar
- [30] . 2014. Distributed representations of sentences and documents. In Proceedings of the International Conference on Machine Learning. 1188–1196.Google Scholar
- [31] . 1992. Class-based n-gram models of natural language. Comput. Linguist. 18, 4 (1992), 467–480.Google ScholarDigital Library
- [32] . 2018. Automated learning of templates for data-to-text generation: comparing rule-based, statistical and neural methods. In Proceedings of the 11thInternational Conference on Natural Language Generation. 35–45.Google ScholarCross Ref
- [33] . 2017. MONPA: Multi-objective named-entity and part-of-speech annotator for Chinese using recurrent neural network. In Proceedings of the 8th International Joint Conference on Natural Language Processing. Asian Federation of Natural Language Processing, 80–85.Google Scholar
- [34] . 1993. Accurate methods for the statistics of surprise and coincidence. Comput. Linguist. 19, 1 (1993), 61–74.Google ScholarDigital Library
- [35] . 2003. An extensive empirical study of feature selection metrics for text classification. J. Mach. Learn. Res. 3 (2003), 1289–1305.Google ScholarCross Ref
- [36] . 2020. Evaluation of feature selection methods for text classification with small datasets using multiple criteria decision-making methods. Appl. Soft Comput. 86 (2020), 105836.Google ScholarDigital Library
- [37] . 2020. Named entity recognition as dependency parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 6470–6476Google ScholarCross Ref
- [38] . 2011. Topic detection and tracking interface with named entities approach. In Proceedings of the International Conference on Semantic Technology and Information Retrieval. IEEE, 215–219.Google ScholarCross Ref
- [39] . 1999. Foundations of Statistical Natural Language Processing. MIT Press.Google ScholarDigital Library
- [40] . 1993. Random walks on graphs. Combinatorics, Paul Erdos is Eighty 2, 1–46 (1993), 4.Google Scholar
- [41] . 2010. Random walks on digraphs, the generalized digraph laplacian and the degree of asymmetry. In Proceedings of the International Workshop on Algorithms and Models for the Web-Graph. 74–85.Google ScholarCross Ref
- [42] . 2011. Modern Information Retrieval: The Concepts and Technology Behind Search. Addison-Wesley Professional, New Jersey.Google Scholar
- [43] . 1991. Contextual correlates of semantic similarity. Lang. Cogn. Process. 6, 1 (1991), 1–28.Google ScholarCross Ref
- [44] . 2014. Semantic frame-based statistical approach for topic detection. In: Proceedings of the 28th Pacific Asia Conference on Language, Information and Computation (PACLIC’14). 75–84.Google Scholar
- [45] . 2008. Introduction to Information Retrieval. Cambridge University Press.Google ScholarCross Ref
- [46] . 2016. FastText.zip: Compressing text classification models. Retrieved from https://arXiv:1612.03651Google Scholar
- [47] . 2014. Convolutional neural networks for sentence classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'14). 1746–1751.Google ScholarCross Ref
- [48] . 2021. Exploring syntactic and semantic features for authorship attribution. Appl. Soft Comput. 111 (2021), 107815.Google ScholarDigital Library
- [49] . 2007. Text simplification for language learners: A corpus analysis. In Proceedings of the Workshop on Speech and Language Technology in Education.Google Scholar
- [50] . 1985. Some effects of the nature and frequency of vocabulary instruction on the knowledge and use of words, Read. Res. Quart. (1985), 522–535.Google ScholarCross Ref
- [51] . 2014. A corpus analysis of grant guidelines: The education and training programme word list (ETPWL). J. Teach. English Spec. Acad. Purp. 2, 3 (2014), 501–514.Google Scholar
- [52] . 2015. Effective use of word order for text categorization with convolutional neural networks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 103–112.
DOI: Google ScholarCross Ref - [53] . 2020. Structural block driven enhanced convolutional neural representation for relation extraction. Appl. Soft Comput. 86 (2020), 105913.Google ScholarDigital Library
- [54] . 1998. A semantic network of English verbs. WordNet: An Electronic Lexical Database 3 (1998), 153–178.Google Scholar
- [55] . 2021. Design knowledge representation with technology semantic network. In Proc. Design Soc. 1 (2021), 1043–1052.Google ScholarCross Ref
- [56] . 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.Google Scholar
- [57] . 1997. Long short-term memory. Neural Comput. 9, 8 (1997), 1735–1780.Google ScholarDigital Library
- [58] . 2014. A Convolutional Neural Network for Modelling Sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.Google ScholarCross Ref
- [59] . 2014. Convolutional neural networks for sentence classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'14).Google ScholarCross Ref
- [60] . 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324. Google ScholarCross Ref
- [61] . 2019. BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 4 (2019), 1234–1240. Google ScholarCross Ref
- [62] . 2021. BertGCN: Transductive text classification by combining GNN and BERT. In Proceedings of the Association for Computational Linguistics (ACL-IJCNLP'21).Google ScholarCross Ref
- [63] . 2021. FinBERT: A pre-trained financial language representation model for financial text mining. In Proceedings of the 29th International Joint Conference on Artificial Intelligence.Google Scholar
- [64] . 2020. BERTweet: A pre-trained language model for English Tweets. In Proceedings of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations.Google ScholarCross Ref
- [65] . 2014. Learning semantic representations using convolutional neural networks for web search. In Proceedings of the 23rd International Conference on World Wide Web. Google ScholarDigital Library
- [66] . 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems.Google ScholarDigital Library
- [67] . 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.Google ScholarCross Ref
- [68] . 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems.Google ScholarDigital Library
- [69] . 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence. 7370–7377. Google ScholarDigital Library
- [70] . 2014. Semantic parsing for single-relation question answering. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.Google ScholarCross Ref
- [71] . 2022. Set-CNN: A text convolutional neural network based on semantic extension for short text classification. Knowl.-Based Syst. 257 (2022), 109948. Google ScholarDigital Library
- [72] . 2015. Linguistic template extraction for recognizing reader-emotion and emotional resonance writing assistance. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. 775–780.Google ScholarCross Ref
- [73] . 2021. Limitations of transformers on clinical text classification. IEEE J. Biomed. Health Inform. 25, 9 (2021), 3596–3607.Google ScholarCross Ref
- [74] . 2022. Multifaceted assessments of traditional chinese word segmentation tool on large corpora. In Proceedings of the 34th Conference on Computational Linguistics and Speech Processing. 193–199.Google Scholar
- [75] . 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. Retrieved from https://arxiv.org/abs/1907.11692Google Scholar
- [76] . 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. Retrieved from https://arxiv.org/abs/1910.01108?context=csGoogle Scholar
Index Terms
- Semantic Template-based Convolutional Neural Network for Text Classification
Recommendations
Text Classification Based on Word2vec and Convolutional Neural Network
Neural Information ProcessingAbstractText representations in text classification usually have high dimensionality and are lack of semantics, resulting in poor classification effect. In this paper, TF-IDF is optimized by using optimization factors, then word2vec with semantic ...
Adversarial Convolutional Neural Network for Text Classification
EITCE '20: Proceedings of the 2020 4th International Conference on Electronic Information Technology and Computer EngineeringText classification is a fundamental task of natural language processing. The convolutional neural network (CNN) has been employed popularly and achieved excellent results on text classification. Nevertheless, the training parameters of CNN is prone to ...
Performance comparison of text-based sentiment analysis using recurrent neural network and convolutional neural network
ICCIP '17: Proceedings of the 3rd International Conference on Communication and Information ProcessingOne biggest challenge in sentiment analysis is that it should include Natural Language Processing (NLP), to make the machine understand the human language. With the current development of Artificial Neural Network (ANN), with its implementation, ...
Comments