Abstract
The automatic summarization of scientific papers, to assist researchers in conducting literature surveys, has garnered significant attention because of the rapid increase in the number of scientific articles published each year. However, whether and how these summaries actually help readers in comprehending scientific papers has not been examined yet. In this work, we study the effectiveness of automatically generated summaries of scientific papers for students who do not have sufficient knowledge in research. We asked six students, enrolled in bachelor’s and master’s programs in Japan, to prepare a presentation on a scientific paper by providing them either the article alone, or the article and its summary generated by an automatic summarization system, after 15 min of reading time. The comprehension of an article was judged by four evaluators based on the participant’s presentation. The experimental results show that the completeness of the comprehension of the four participants was higher overall when the summary of the paper was provided. In addition, four participants, including the two whose completeness score reduced when the summary was provided, mentioned that the summary is helpful to comprehend a research article within a limited time span.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abu-Jbara, A., Radev, D.: Coherent citation-based summarization of scientific papers. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 500–509 (2011)
Bornmann, L., Mutz, R.: Growth rates of modern science: a bibliometric analysis based on the number of publications and cited references. J. Assoc. Inf. Sci. Technol. 66(11), 2215–2222 (2015)
Carney, R.N., Levin, J.R.: Pictorial illustrations still improve students’ learning from text. Educ. Psychol. Rev. 14(1), 5–26 (2002)
Chaganty, A., Mussmann, S., Liang, P.: The price of debiasing automatic metrics in natural language evalaution. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 643–653 (2018)
Cohan, A., et al.: A discourse-aware attention model for abstractive summarization of long documents. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 615–621 (2018)
Cohan, A., Goharian, N.: Scientific article summarization using citation-context and article’s discourse structure. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 390–400 (2015)
Cohan, A., Goharian, N.: Scientific document summarization via citation contextualization and scientific discourse. Int. J. Digit. Libr. 19(2–3), 287–303 (2018)
Elkiss, A., Shen, S., Fader, A., Erkan, G., States, D., Radev, D.R.: Blind men and elephants: what do citation summaries tell us about a research article? J. Am. Soc. Inf. Sci. Technol. 59, 51–62 (2008)
Hermann, K.M., et al.: Teaching machines to read and comprehend. In: Advances in Neural Information Processing Systems, vol. 28, pp. 1693–1701 (2015)
Huang, D., et al.: What have we achieved on text summarization? In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 446–469 (2020)
Kataoka, H., et al.: cvpaper.challenge in 2016: futuristic computer vision through 1,600 papers survey. arXiv preprint arXiv:1707.06436 (2017)
Kryscinski, W., Keskar, N.S., McCann, B., Xiong, C., Socher, R.: Neural text summarization: a critical evaluation. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 540–551 (2019)
Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches out, pp. 74–81 (2004)
Liu, C.W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., Pineau, J.: How NOT to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2122–2132 (2016)
Matsumura, K., et al.: Chi benkyokai 2017: nettowâku renkei shita benkyokai to sono shien shisutemu. [chi study meeting 2017: Networked study meeting and support system.] (in japanese, title translated by the author of this article.). IPSJ SIG Technical Reports 2017-HCI-174(13), pp. 1–8 (2017)
Mei, Q., Zhai, C.: Generating impact-based summaries for scientific literature. In: Proceedings of the ACL-08: HLT, pp. 816–824 (2008)
Mihalcea, R., Tarau, P.: Textrank: bringing order into text. In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 404–411 (2004)
Nanba, H., Kando, N., Okumura, M.: Classification of research papers using citation links and citation types: towards automatic review article generation. Adv. Classif. Res. Online 11(1), 117–134 (2000)
Nanba, H., Okumura, M.: Towards multi-paper summarization reference information. In: Proceedings of the 16th International Joint Conference on Artificial Intelligence, vol. 2, pp. 926–931 (1999)
Novikova, J., Dušek, O., Cercas Curry, A., Rieser, V.: Why we need new evaluation metrics for NLG. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2241–2252 (2017)
Ozuru, Y., Dempsey, K., McNamara, D.S.: Prior knowledge, reading skill, and text cohesion in the comprehension of science texts. Learn. Instr. 19(3), 228–242 (2009)
Qazvinian, V., Radev, D.R.: Scientific paper summarization using citation summary networks. In: Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pp. 689–696 (2008)
Siegel, N., Lourie, N., Power, R., Ammar, W.: Extracting scientific figures with distantly supervised neural networks. In: Proceedings of the 18th ACM/IEEE Joint Conference on Digital Libraries, pp. 223–232 (2018)
Suppe, F.: The structure of a scientific paper. Philos. Sci. 65(3), 381–405 (1998)
Yamamoto, S., Fukuhara, Y., Suzuki, R., Morishima, S., Kataoka, H.: Automatic paper summary generation from visual and textual information. In: Eleventh International Conference on Machine Vision (ICMV 2018), vol. 11041, pp. 214–221 (2019)
Acknowledgment
This work was supported by JST ACCEL (JPMJAC1602). We would like to thank the members of cvpaper.challenge for their help in the user study.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Yamamoto, S., Suzuki, R., Kataoka, H., Morishima, S. (2021). Comprehending Research Article in Minutes: A User Study of Reading Computer Generated Summary for Young Researchers. In: Yamamoto, S., Mori, H. (eds) Human Interface and the Management of Information. Information Presentation and Visualization. HCII 2021. Lecture Notes in Computer Science(), vol 12765. Springer, Cham. https://doi.org/10.1007/978-3-030-78321-1_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-78321-1_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78320-4
Online ISBN: 978-3-030-78321-1
eBook Packages: Computer ScienceComputer Science (R0)