Abstract
A growing trend of extractive summarization research is to take the document structure into account which has been shown to correlate with the important contents in a text. However, building complex document structures such as Rhetorical Structure Theory (RST) is time-consuming and requires a large effort to prepare labeled training data. Therefore, how to effectively learn a document structure for summarization remains an open question. Recent findings in the language model area show that the syntactic distance of the basic semantic unit could be used to induce the syntactic structure without any extra labeled data. Inspired by these findings, we propose to extend the basic semantic units from words to sentences and extract the syntactic distance of each sentence in the document by building an ON-LSTM (Ordered Neuron LSTM) based model with the document-level language model objective. We then leverage these syntactic distances to evaluate whether a sentence could be extracted to the summary. Our model achieves state-of-the-art performance in terms of Rouge-1 (48.67) and Rouge-2 (26.32) on the CNN/Daily Mail data set.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dong, Y., Shen, Y., Crawford, E., van Hoof, H., Cheung, J.C.K.: Banditsum: extractive summarization as a contextual bandit. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium (2018)
Su, H., et al.: Diversifying dialogue generation with non-conversational text. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)
Marcu, D.: A decision-based approach to rhetorical parsing. SIGIR (1999)
Nallapati, R., Zhai, F., Zhou, B.: SummaRuNNer: a recurrent neural network based sequence model for extractive summarization of documents. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. San Francisco, California USA (2017)
Cheng, J., Lapata, M.: Neural summarization by extracting sentences and words. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany (2016)
Nallapati, R., Zhou, B., Santos, C., Gulcehre, C., Bing, X.: Abstractive text summarization using sequence-to-sequence RNNs and beyond. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany (2016)
Xu, J., Gan, Z., Cheng, Y., Liu, J.: Discourse-aware neural extractive text summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)
Mann, W.C., Thompson, S.A.: Rhetorical structure theory: toward a functional theory of text organization. Text Interdisc. J. Study Discours. 8 (1998)
Carlson, L., Marcu, D., Okurowski, M.E.: Building a discourse-tagged corpus in the framework of rhetorical structure theory. In: van Kuppevelt, J., Smith, R.W. (eds.) Current and New Directions in Discourse and Dialogue. Text, Speech and Language Technology, vol. 22. Springer, Dordrecht (2003). https://doi.org/10.1007/978-94-010-0019-2_5
Prasad, R., et al.: The penn discourse treebank 2.0. In: LREC, Marrakech, Morocco (2008)
Zhao, K., Huang, L.: Joint syntacto discourse parsing and the syntacto-discourse treebank. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark (2017)
Narayan, S., Cohen, S.B., Lapata, M.: Ranking sentences for extractive summarization with reinforcement learning. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, New Orleans, Louisiana USA (2018)
Yang, L., Titov, I., Lapata, M.: Single document summarization as tree induction. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, Minnesota USA (2019)
Wang, D., Liu, P., Zheng, Y., Qiu, X., Huang, X.: Heterogeneous graph neural networks for extractive document summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)
Mihalcea, R., Tarau, P.: Textrank: bringing order into text. In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain (2004)
Brin, S., Page, M.: Anatomy of a large-scale hypertextual Web search engine. In: Proceedings of the 7th Conference on World Wide Web, pp. 107–117 (1999)
Shen, Y., Tan, S., Sordoni, A., Courville, A.: Ordered neurons: integrating tree structures into recurrent neural networks. In: Proceedings of Seventh International Conference on Learning Representations (2019)
Zhou, Q., Yang, N., Wei, F., Huang, S., Zhou, M., Zhao, T.: Neural document summarization by jointly learning to score and select sentences. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia (2018)
Liu, Y., Lapata, M.: Text summarization with pretrained encoders. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China (2019)
Narayan, S., Cohen, S.B., Lapata, M.: Dont give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium (2018)
Chen, D.: Neural reading comprehension and beyond. Stanford University (2018)
Lin, C.-Y.: ROUGE: a package for automatic evaluation of summaries. In: Text summarization branches out: Proceedings of the ACL-04 Workshop, vol. 8 (2004)
See, A., Liu, P.J., Manning, C.D.: Get to the point: summarization with pointer-generator networks. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada (2017)
Zhong, M., Liu, P., Chen, Y., Wang, D., Qiu, X., Huang, X.: Extractive summarization as text matching. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)
Kang, D., Hovy, E.: Linguistic versus latent relations for modeling coherent flow in paragraphs. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China (2019)
Du, W., Lin, Z., Shen, Y., O’Donnell, T.J., Bengio, Y., Zhang, Y.: Exploiting syntactic structure for better language modeling: a syntactic distance approach. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)
Wang, Y., Li, S., Yang, C.-Y., Sun, X., Wang, H.: Tag-enhanced tree-structured neural networks for implicit discourse relation classification. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing, Taipei, Taiwan (2017)
Page, L., Brin, S., Motwani, R., Winograd, T.: The PageRank Citation Ranking: Bringing Order to the Web (1999)
Paulus, R., Xiong, C., Socher, R.: A deep reinforced model for abstractive sum- marization. In: Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada (2018)
Nallapati, R., Zhai, F., Zhou, B.: SummaRuNNer: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the 31st AAAI Conference on Artificial Intelligence (2017)
Carlson, L., Marcu, D., Okurowski, M.E.: RST Discourse Treebank (RST– DT) LDC2002T07. Linguistic Data Consortium, Philadelphia (2002)
Liu, J., Cohen, S.B., Lapata, M.: Discourse representation structure parsing. In: 429 Proceedings of the 56th Annual Meeting ofthe Association for Computational Linguistics (Long Papers), Melbourne, Australia, 15–20 July, pp. 429–439 (2018)
Gerani, S., Mehdad, Y., Carenini, G., Ng, R.T., Nejat, B.: Abstractive summarization of product reviews using discourse structure. In: Proceedings ofthe Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014)
Niculae, V., Martins, A.F.T., Cardie, C.: Towards dynamic computation graphs via sparse latent structure. In: Proceedings of EMNLP (2018)
Hirao, T., Yoshida, Y., Nishino, M., Yasuda, N., Nagata, M.: Single-document summarization as a tree knapsack problem. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) (2013)
Yogatama, D., Blunsom, P., Dyer, C., Grefenstette, E., Ling, W.: Learning to compose words into sentences with reinforcement learning. In Proceedings of ICLR (2017)
Jurafsky, D., Martin, J.H.: Speech and Language Processing, Third Edition draft (2020). https://web.stanford.edu/~jurafsky/slp3/
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Han, X., Wang, Q., Chen, Z., Hu, L., Hu, P. (2021). OnSum: Extractive Single Document Summarization Using Ordered Neuron LSTM. In: Huang, DS., Jo, KH., Li, J., Gribova, V., Hussain, A. (eds) Intelligent Computing Theories and Application. ICIC 2021. Lecture Notes in Computer Science(), vol 12837. Springer, Cham. https://doi.org/10.1007/978-3-030-84529-2_51
Download citation
DOI: https://doi.org/10.1007/978-3-030-84529-2_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-84528-5
Online ISBN: 978-3-030-84529-2
eBook Packages: Computer ScienceComputer Science (R0)