Abstract
Scientific document summarization focusses on condensing scientific literature, research papers, and technical documents into concise summaries while preserving crucial scientific concepts, findings, and conclusions. In this work, we present a novel loss function that incorporates semantic similarity, and use it in the parallel training of extractive and abstractive summarizers, thereby improving the performance of the individual summarizer units. The new loss function is a union of the summarizer cross-entropy losses and the semantic similarity losses among the generated and reference summaries. To validate the effectiveness of the proposed loss function joint with the parallel training, the experiments use a combination of four recently state-of-the-art extractive summarizers and four recently state-of-the-art abstractive summarizers. Results indicate that for all combinations, the extractive and abstractive summarizers both gain significant performance boosts. It is conjectured that the new semantic similarity-induced cross-entropy loss combined with the parallel training will improve any combination of quality extractive and abstractive summarizers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aksenov, D., Schneider, J.M., Bourgonje, P., Schwarzenberg, R., Hennig, L., Rehm, G.: Abstractive text summarization based on language model conditioning and locality modeling. In: Proceedings of the 12th Language Resources and Evaluation Conference, pp. 6680–6689 (2020)
Aralikatte, R., Narayan, S., Maynez, J., Rothe, S., McDonald, R.: Focus attention: promoting faithfulness and diversity in summarization. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6078–6095 (2021)
Beltagy, I., Peters, M.E., Cohan, A.: Longformer: the long-document transformer. arXiv preprint arXiv:2004.05150 (2020)
Cheng, J., Lapata, M.: Neural summarization by extracting sentences and words. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 484–494 (2016)
Cho, S., Song, K., Wang, X., Liu, F., Yu, D.: Toward unifying text segmentation and long document summarization. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 106–118 (2022)
Cohan, A., et al.: A discourse-aware attention model for abstractive summarization of long documents. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 615–621 (2018)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186 (2019)
Fonseca, M., Ziser, Y., Cohen, S.B.: Factorizing content and budget decisions in abstractive summarization of long documents. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6341–6364 (2022)
Fu, X., Wang, J., Zhang, J., Wei, J., Yang, Z.: Document summarization with VHTM: variational hierarchical topic-aware mechanism. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 7740–7747 (2020)
Huang, D., et al.: What have we achieved on text summarization? In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp. 446–469 (2020)
Koo, T., Globerson, A., Carreras Pérez, X., Collins, M.: Structured prediction models via the matrix-tree theorem. In: Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 141–150 (2007)
Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871–7880 (2020)
Liu, Y., Lapata, M.: Text summarization with pretrained encoders. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3730–3740 (2019)
Liu, Y., Titov, I., Lapata, M.: Single document summarization as tree induction. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1745–1755 (2019)
Liu, Y., et al.: Leveraging locality in abstractive text summarization. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6081–6093 (2022)
Mao, Z., et al.: DYLE: dynamic latent extraction for abstractive long-input summarization. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1687–1698 (2022)
Nallapati, R., Zhai, F., Zhou, B.: SummaRuNNer: a recurrent neural network based sequence model for extractive summarization of documents. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI 2017), pp. 3075–3081 (2017)
Narayan, S., Cohen, S.B., Lapata, M.: Ranking sentences for extractive summarization with reinforcement learning. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1747–1759 (2018)
Qiu, Y., Cohen, S.B.: Abstractive summarization guided by latent hierarchical document structure. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5303–5317 (2022)
Ruan, Q., Ostendorff, M., Rehm, G.: HiStruct+: improving extractive text summarization with hierarchical structure information. In: Findings of the Association for Computational Linguistics: ACL 2022, pp. 1292–1308 (2022)
See, A., Liu, P.J., Manning, C.D.: Get to the point: summarization with pointer-generator networks. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073–1083 (2017)
Singha Roy, S., Mercer, R.E.: Generating extractive and abstractive summaries in parallel from scientific articles incorporating citing statements. In: Proceedings of the 4th New Frontiers in Summarization Workshop, pp. 75–86 (2023)
Wang, D., Liu, P., Zheng, Y., Qiu, X., Huang, X.: Heterogeneous graph neural networks for extractive document summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6209–6219 (2020)
Wang, F., et al.: Salience allocation as guidance for abstractive summarization. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6094–6106 (2022)
Wang, Z., et al.: Friendly topic assistant for transformer based abstractive summarization. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp. 485–497 (2020)
Xie, Q., Huang, J., Saha, T., Ananiadou, S.: GRETEL: graph contrastive topic enhanced language model for long document extractive summarization. In: Proceedings of the 29th International Conference on Computational Linguistics, pp. 6259–6269 (2022)
Yu, T., Su, D., Dai, W., Fung, P.: Dimsum @LaySumm 20: BART-based approach for scientific document summarization. In: Proceedings of the First Workshop on Scholarly Document Processing, pp. 303–309 (2020)
Zaheer, M., et al.: Big bird: transformers for longer sequences. In: Advances in Neural Information Processing Systems, vol. 33, pp. 17283–17297 (2020)
Zhang, H., Liu, X., Zhang, J.: HEGEL: hypergraph transformer for long document summarization. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pp. 10167–10176 (2022)
Zhang, X., Wei, F., Zhou, M.: HIBERT: document level pre-training of hierarchical bidirectional transformers for document summarization. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5059–5069 (2019)
Zhong, M., Liu, P., Chen, Y., Wang, D., Qiu, X., Huang, X.J.: Extractive summarization as text matching. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6197–6208 (2020)
Zhou, Q., Yang, N., Wei, F., Huang, S., Zhou, M., Zhao, T.: Neural document summarization by jointly learning to score and select sentences. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 654–663 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Singha Roy, S., Mercer, R.E. (2024). Investigating a Semantic Similarity Loss Function for the Parallel Training of Abstractive and Extractive Scientific Document Summarizers. In: Fred, A., Hadjali, A., Gusikhin, O., Sansone, C. (eds) Deep Learning Theory and Applications. DeLTA 2024. Communications in Computer and Information Science, vol 2172. Springer, Cham. https://doi.org/10.1007/978-3-031-66705-3_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-66705-3_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-66704-6
Online ISBN: 978-3-031-66705-3
eBook Packages: Computer ScienceComputer Science (R0)