Abstract
There are advanced neural network architectures, such as: auto-encoding (AE), transformer, etc. have been recently applied and achieved remarkable successes in multiple downstream tasks of natural language processing (NLP) area including text summarization. However, recent transformer-based and graph neural network (GNN) based extractive text summarization still encounters many drawbacks. These drawbacks are related to the capability of preserving the global long-range and syntactical relationships in texts. It supports to achieve better fine-tune for leveraging the performance of extractive summarization task. To deal with these challenges, in this paper we proposed a novel text graph-based neural learning mechanism with attention mechanism for the extractive text summarization, called as TGA4ExSum. In our proposed TGA4ExSum model, we mainly apply the text graph multi-headed attention network (TGA) to effectively learn the representations of sentences upon different types of text graphs at different levels. These learnt rich contextual and structural text representations support to improve the performance of the extractive summary generation process. Moreover, we also integrate pre-trained BERT model in our TGA4ExSum at the initial steps to jointly capture the sequential rich contextual representations of words and sentences in each input text. They are later used to facilitate the TGA-based learning process. Extensive experiments in benchmark datasets demonstrate the effectiveness of our proposed TGA4ExSum model in comparing with contemporary state-of-the-art baselines in extractive text summarization task.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
CNN news platform: https://www.cnn.com/.
Daily-Mail news platform: https://www.dailymail.co.uk/.
CNN/Daily-Mail dataset: https://github.com/abisee/cnn-dailymail.
The New York Times news platform: https://www.nytimes.com/.
NYT dataset: https://catalog.ldc.upenn.edu/LDC2008T19.
CoreNLP library for NLP (Java): https://stanfordnlp.github.io/CoreNLP/.
PyTorch ML framework: https://pytorch.org/.
Pre-trained BERT (large, uncased): https://github.com/google-research/bert.
References
Abed, S.H., Al-Waisy, A.S., Mohammed, H.J., Al-Fahdawi, S.: A modern deep learning framework in robot vision for automated bean leaves diseases detection. Int. J. Intell. Robot. Appl. 5(2), 235–251 (2021)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations, ICLR (2015)
Cheng, J., Lapata, M.: Neural summarization by extracting sentences and words. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (2016)
Cho, S.G., Yoshikawa, M., Ding, M., Takamatsu, J., Ogasawara, T.: Machine-learning-based hand motion recognition system by measuring forearm deformation with a distance sensor array. Int. J. Intell. Robot. Appl. 3(4), 418–429 (2019)
Chopra, S., Auli, M., Rush, A. M.: Abstractive sentence summarization with attentive recurrent neural networks. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2016)
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2019)
Dong, Y., Shen, Y., Crawford, E., van Hoof, H., Cheung, J. C. K.: BanditSum: extractive summarization as a contextual bandit. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (2018)
El-Kassas, W.S., Salama, C.R., Rafea, A.A., Mohamed, H.K.: Automatic text summarization: a comprehensive survey. Expert Syst. Appl. 165, 113679 (2020)
Gambhir, M., Gupta, V.: Recent automatic text summarization techniques: a survey. Artif. Intell. Rev. 47(1), 1–66 (2017)
Glüge, S., Hamid, O.H., Wendemuth, A.: A simple recurrent network for implicit learning of temporal sequences. Cogn. Comput. 2(4), 265–271 (2010)
Hamid, O.H., Wendemuth, A., Braun, J.: Temporal context and conditional associative learning. BMC Neurosci. 11(1), 1–16 (2010)
Hamid, O. H.: The role of temporal statistics in the transfer of experience in context-dependent reinforcement learning. In: 14th International Conference on Hybrid Intelligent Systems (2014)
Hamilton, W. L., Ying, R., Leskovec, J.: Inductive representation learning on large graphs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (2017)
Jia, R., Cao, Y., Tang, H., Fang, F., Cao, C., Wang, S.: Neural extractive summarization with hierarchical attentive heterogeneous graph network. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2020)
Kipf, T. N., Welling, M.: Semi-supervised classification with graph convolutional networks (2016). http://arxiv.org/abs/1609.02907
Li, Y., Li, G., Zhang, X.: Structural constraint deep matrix factorization for sequential data clustering. Int. J. Intell. Robot. Appl. 3(4), 443–456 (2019)
Lin, C. Y.: Rouge: a package for automatic evaluation of summaries. In: Text Summarization Branches Out (2004)
Liu, Y.: Fine-tune BERT for extractive summarization (2019). http://arxiv.org/abs/1903.10318
Liu, Y., Lapata, M.: Text summarization with pretrained encoders. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (2019)
Liu, X., You, X., Zhang, X., Wu, J., Lv, P.: Tensor graph convolutional networks for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence (2020)
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S., McClosky, D.: The Stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations (2014)
Moratanch, N., Chitrakala, S.: A survey on extractive text summarization. In: 2017 International Conference on Computer, Communication and Signal Processing (ICCCSP) (2017)
Nallapati, R., Zhou, B., dos Santos, C., Gu̇lçehre, Ç., Xiang, B.: Abstractive text summarization using sequence-to-sequence RNNs and beyond. In: Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (2016)
Nallapati, R., Zhai, F., Zhou, B.: Summarunner: a recurrent neural network based sequence model for extractive summarization of documents. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (2017)
Narayan, S., Cohen, S. B., Lapata, M.: Ranking sentences for extractive summarization with reinforcement learning. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2018)
Nenkova, A., McKeown, K.: A survey of text summarization techniques. In: Mining Text Data, pp. 43–76 (2012)
Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2018)
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training. In: OpenAI (2018)
Rush, A. M., Chopra, S., Weston, J.: A neural attention model for abstractive sentence summarization. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (2015)
Sutskever, I., Vinyals, O., Le, Q. V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems (2014)
Takase, S., Suzuki, J., Okazaki, N., Hirao, T., Nagata, M.: Neural headline generation on abstract meaning representation. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016)
Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (2017)
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. In: International Conference on Learning Representations (ICLR) (2018)
Vo, T.: GOWSeqStream: an integrated sequential embedding and graph-of-words for short text stream clustering. Neural Comput. Appl. 1–21 (2021a)
Vo, T.: SE4ExSum: an integrated semantic-aware neural approach with graph convolutional network for extractive text summarization. Trans. Asian Low-Resour. Lang. Inf. Process. 20(6), 1–22 (2021b)
Wang, D., Liu, P., Zheng, Y., Qiu, X., Huang, X. J.: Heterogeneous graph neural networks for extractive document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)
Wu, Y., Hu, B.: Learning to extract coherent summary via deep reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence (2018)
Xu, J., Gan, Z., Cheng, Y., Liu, J.: Discourse-aware neural extractive text summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)
Yao, L., Mao, C., Luo, Y.: Graph convolutional networks for text classification. In: Proceedings of the AAAI Conference on Artificial Intelligence (2019)
Zhang, H., Zhang, J.: Text graph transformer for document classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2020)
Zhang, H., Xu, J., Wang, J.: Pretraining-based natural language generation for text summarization. In: Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) (2019)
Zhou, Q., Yang, N., Wei, F., Huang, S., Zhou, M., Zhao, T.: Neural document summarization by jointly learning to score and select sentences. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (2018)
Acknowledgements
This research is funded by Thu Dau Mot University, Binh Duong, Vietnam.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Vo, T. An approach of syntactical text graph representation learning for extractive summarization. Int J Intell Robot Appl 7, 190–204 (2023). https://doi.org/10.1007/s41315-022-00228-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41315-022-00228-0