Tham Vo
Abstract
In recent years, along with the dramatic developments of deep learning in the natural language processing (NLP) domain, notable multilingual pre-trained language techniques have been proposed. These recent multilingual text analysis and mining models have demonstrated state-of-the-art performance in several primitive NLP tasks, including cross-lingual text classification (CLC). However, these recent multilingual pre-trained language models still suffer limitations regarding their adaptation for specific task-driven fine-tuning in the context of low-resource languages. Moreover, they also encounter problems related to the capability of preserving the global semantic (e.g., topic, etc.) and long-range relationships between words to better fine-tune and effectively handle the cross-lingual text classification task. To meet these challenges, in this article, we propose a novel topic-driven multi-typed text graph attention–based representation learning method for dealing with the cross-lingual text classification problem called TG-CTC. In the proposed TG-CTC model, we utilize a novel fused topic-driven multi-typed text graph representation to jointly learn the rich-schematic structural and global semantic information of texts to effectively handle the CLC task. More specifically, we integrate the heterogeneous text graph attention network with the neural topic modelling approach to enrich the semantic information of learned textual representations in the context of multiple languages. Extensive experiments in benchmark multilingual datasets showed the effectiveness of the proposed TG-CTC model compared with the contemporary state-of-the-art baselines.
- [1] . 2018. Wikipedia-based relatedness measurements for multilingual short text clustering. ACM Trans. Asian Low-Resour. Lang. Inf. Process. 18, 2 (2018), 1–25.Google Scholar
Digital Library
- [2] . 2021. Event graph neural network for opinion target classification of microblog comments. Trans. Asian Low-Resour. Lang. Inf. Process. 21, 1 (2021), 1–13.Google Scholar
- [3] . 2021. A transformer-based approach to multilingual fake news detection in low-resource languages. Trans. Asian Low-Resour. Lang. Inf. Process. 21, 1 (2021), 1–20.Google Scholar
- [4] . Co-training for cross-lingual sentiment classification. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP.Google Scholar
- [5] . 2011. Is machine translation ripe for cross-lingual sentiment classification? In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies.Google ScholarDigital Library
- [6] . 2010. Cross language text classification by model translation and semi-supervised learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.Google Scholar
- [7] . 2015. Cross-lingual text classification using topic-dependent word probabilities. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.Google ScholarCross Ref
- [8] . 2016. Cross-lingual text classification via model translation with limited dictionaries. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management.Google ScholarDigital Library
- [9] . 2018. Deep pivot-based modeling for cross-language cross-domain transfer with minimal guidance. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.Google ScholarCross Ref
- [10] . 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Trans. Assoc. Comput. Ling. 6 (2018), 557–570.Google ScholarCross Ref
- [11] . 2018. Deep contextualized word representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.Google ScholarCross Ref
- [12] . 2018. Improving language understanding by generative pre-training. OpenAI.Google Scholar
- [13] . 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.Google Scholar
- [14] . 2017. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems.Google ScholarDigital Library
- [15] . 2016. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations (ICLR'17). Google Scholar
- [16] . 2018. Graph attention networks. In Proceedings of the International Conference on Learning Representations (ICLR’18).Google Scholar
- [17] . 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence.Google ScholarDigital Library
- [18] . 2020. Tensor graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence.Google ScholarCross Ref
- [19] . 2020. Text graph transformer for document classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP’20).Google ScholarCross Ref
- [20] . 2020. Topic modeling: a comprehensive review. EAI Endors. Trans. Scal. Inf. Syst. 7, 24, (2020).Google Scholar
- [21] . 2020. Copula guided neural topic modelling for short texts. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.Google ScholarDigital Library
- [22] . 2020. Neural topic modeling with bidirectional adversarial training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Google ScholarCross Ref
- [23] . 2020. tBERT: Topic models and BERT joining forces for semantic similarity detection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Google ScholarCross Ref
- [24] . 2021. Classification aware neural topic model for COVID-19 disinformation categorisation. PloS One 16, 2 (2021), e0247086.Google ScholarCross Ref
- [25] . 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the International Conference on Machine Learning (PMLR’17).Google Scholar
- [26] . 2017. Autoencoding variational inference for topic models. In Proceedings of the 5th International Conference on Learning Representations (ICLR).Google Scholar
- [27] . 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Google ScholarCross Ref
- [28] . 2021. Cross-lingual text classification with heterogeneous graph neural network. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 612--620.Google ScholarCross Ref
- [29] . 2020. Cross-lingual unsupervised sentiment classification with multi-view transfer learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Google ScholarCross Ref
- [30] . 2020. Cosda-ml: Multi-lingual code-switching data augmentation for zero-shot cross-lingual nlp. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 3853--3860.Google Scholar
- [31] . 2019. A survey of cross-lingual word embedding models. J. Artif. Intell. Res. 65 (2019) 569–631.Google ScholarDigital Library
- [32] . 2020. Cross-lingual text classification with minimal resources by transferring a sparse teacher. In Findings of the Association for Computational Linguistics: EMNLP 2020. 3604--3622.Google ScholarCross Ref
- [33] . 2020. Exploiting cross-lingual subword similarities in low-resource document classification. In Proceedings of the AAAI Conference on Artificial Intelligence.Google ScholarCross Ref
- [34] . 2020. Interactive refinement of cross-lingual word embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP’20).Google ScholarCross Ref
- [35] . 2020. Why overfitting isn't always bad: Retrofitting cross-lingual word embeddings to dictionaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2214--2220.Google ScholarCross Ref
- [36] . 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems.Google Scholar
- [37] . 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the International Conference on Machine Learning (PMLR).Google Scholar
- [38] . 2009. Polylingual topic models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.Google ScholarCross Ref
- [39] . 2018. Multilingual anchoring: interactive topic modeling and alignment across languages. In Proceedings of the Conference and Workshop on Neural Information Processing Systems (NeurIPS’18), 8667–8677.Google Scholar
- [40] . 2019. A multilingual topic model for learning weighted topic links across corpora with low comparability. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19).Google ScholarCross Ref
- [41] . 2020. Learn to cross-lingual transfer with meta graph learning across heterogeneous languages. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’20).Google ScholarCross Ref
- [42] . 1997. Long short-term memory. Neur. Comput. 9, 8 (1997), 1735–1780.Google ScholarDigital Library
- [43] . 2005. Bidirectional LSTM networks for improved phoneme classification and recognition. In Proceedings of the International Conference on Artificial Neural Networks.Google ScholarCross Ref
- [44] . 2014. The stanford CORENLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations.Google ScholarCross Ref
- [45] . 2019. Heterogeneous graph attention networks for semi-supervised short text classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP’19).Google ScholarCross Ref
Index Terms
- An Integrated Topic Modelling and Graph Neural Network for Improving Cross-lingual Text Classification
Recommendations
Cross-lingual Text Classification via Model Translation with Limited Dictionaries
CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge ManagementCross-lingual text classification (CLTC) refers to the task of classifying documents in different languages into the same taxonomy of categories. An open challenge in CLTC is to classify documents for the languages where labeled training data are not ...
A word embedding-based approach to cross-lingual topic modeling
AbstractThe cross-lingual topic analysis aims at extracting latent topics from corpora of different languages. Early approaches rely on high-cost multilingual resources (e.g., a parallel corpus), which is hard to come by in many real cases. Some works ...
Unsupervised Bilingual Sentiment Word Embeddings for Cross-lingual Sentiment Classification
ICIAI '20: Proceedings of the 2020 the 4th International Conference on Innovation in Artificial IntelligenceIn recent years, bilingual word embeddings have been used to promote sentiment classification task in low-resource languages. However, existing bilingual word embedding methods either require annotated cross-lingual data or fail to capture enough ...
Comments