Abstract
We propose a multi-task learning approach to reducing text complexity which combines text summarization and simplification methods. For the purposes of this research, two datasets were used: the Simple English Wikipedia dataset for simplification and the CNN/DailyMail dataset for summarization.
We describe several experiments with reducing text complexity. One experiment consists in first training the model on summarization data, then fine-tuning it on simplification data. Another experiment involves training the model on both datasets simultaneously while augmenting source texts with a task-specific tag that shows the model which task (summarization or simplification) needs to be performed on a given text. Models with a similar architecture were also trained on each dataset separately for comparison. Our experiments have shown that the multi-task learning approach with task-specific tags is more effective than the fine-tuning approach, and the models trained for both tasks simultaneously can perform as good at each of them as the models that were trained only for that specific task.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alva-Manchego, F., Martin, L., Scarton, C., Specia, L.: EASSE: easier automatic sentence simplification evaluation. arXiv preprint arXiv:1908.04567 (2019)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Bott, S., Saggion, H.: An unsupervised alignment algorithm for text simplification corpus construction. In: Proceedings of the Workshop on Monolingual Text-To-Text Generation, pp. 20–26 (2011)
Chandrasekar, R., Srinivas, B.: Automatic induction of rules for text simplification. Knowl.-Based Syst. 10(3), 183–190 (1997)
Gambhir, M., Gupta, V.: Recent automatic text summarization techniques: a survey. Artif. Intell. Rev. 47(1), 1–66 (2016). https://doi.org/10.1007/s10462-016-9475-9
Gehrmann, S., Deng, Y., Rush, A.M.: Bottom-up abstractive summarization. arXiv preprint arXiv:1808.10792 (2018)
Hermann, K.M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., Blunsom, P.: Teaching machines to read and comprehend. In: Advances in Neural Information Processing Systems, pp. 1693–1701 (2015)
Johnson, M., et al.: Enabling zero-shot translation: Google’s multilingual neural machine translation system. Trans. Assoc. Comput. Linguist. 5, 339–351 (2017)
Karpov, N., Sibirtseva, V.: Towards automatic text adaptation in Russian. Higher School of Economics Research Paper No. WP BRP, 16 (2014)
Kauchak, D.: Improving text simplification language modeling using unsimplified text data. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Long papers), vol. 1, pp. 1537–1546, August 2013
Keskisärkkä, R.: Automatic text simplification via synonym replacement (2012)
Klein, G., Kim, Y., Deng, Y., Senellart, J., Rush, A.M.: Opennmt: open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810 (2017)
Kriz, R., et al.: Complexity-weighted loss and diverse reranking for sentence simplification. arXiv preprint arXiv:1904.02767 (2019)
Lal, P., Ruger, S.: Extract-based summarization with simplification. In: Proceedings of the ACL, July 2002
Nallapati, R., Zhou, B., Gulcehre, C., Xiang, B.: Abstractive text summarization using sequence-to-sequence RNNs and beyond. arXiv preprint arXiv:1602.06023 (2016)
Nallapati, R., Zhai, F., Zhou, B.: SummaRuNNer: a recurrent neural network based sequence model for extractive summarization of documents. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019)
Saggion, H.: Automatic text simplification. Synth. Lect. Hum. Lang. Technol. 10(1), 1–137 (2017)
See, A., Liu, P.J., Manning, C.D.: Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368 (2017)
Siddharthan, A.: A survey of research on text simplification. ITL-Int. J. Appl. Linguist. 165(2), 259–298 (2014)
Štajner, S., Saggion, H.: Data-driven text simplification. In: Proceedings of the 27th International Conference on Computational Linguistics: Tutorial Abstracts, pp. 19–23 (2018)
Sulem, E., Abend, O., Rappoport, A.: BLEU is not suitable for the evaluation of text simplification. arXiv preprint arXiv:1810.05995 (2018)
Surya, S., Mishra, A., Laha, A., Jain, P., Sankaranarayanan, K.: Unsupervised Neural Text Simplification. arXiv preprint arXiv:1810.07931 (2018)
Vinyals, O., Fortunato, M., Jaitly, N.: Pointer networks. In: Advances in Neural Information Processing Systems, pp. 2692–2700 (2015)
Wang, T., Chen, P., Rochford, J., Qiang, J.: Text simplification using neural machine translation. In: Thirtieth AAAI Conference on Artificial Intelligence (2016)
Xu, W., Callison-Burch, C., Napoles, C.: Problems in current text simplification research: new data can help. Trans. Assoc. Comput. Linguist. 3, 283–297 (2015)
Zhang, X., Lapata, M.: Sentence simplification with deep reinforcement learning. arXiv preprint arXiv:1703.10931 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Dmitrieva, A., Tiedemann, J. (2021). A Multi-task Learning Approach to Text Simplification. In: van der Aalst, W.M.P., et al. Recent Trends in Analysis of Images, Social Networks and Texts. AIST 2020. Communications in Computer and Information Science, vol 1357. Springer, Cham. https://doi.org/10.1007/978-3-030-71214-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-71214-3_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-71213-6
Online ISBN: 978-3-030-71214-3
eBook Packages: Computer ScienceComputer Science (R0)