Abstract
Easy-to-Read (E2R) is a method of enhancing the accessibility of written text by using clear, direct, and simple language. E2R texts are designed to improve readability and accessibility, especially for individuals with cognitive disabilities. However, there is a significant lack of standardized evaluation methods for these texts. Traditional Automatic Text Simplification (ATS) evaluation methods such as BLEU, SARI or ROUGE present several limitations for E2R evaluation. Readability measures such as Flesch Reading Ease (FRE) and Flesch-Kincaid Reading Grade Level (FKGL) do not take into account all document factors. Manual evaluation methods, such as Likert scales, are resource-intensive and lead to subjective assessments. This paper proposes a threefold evaluation method for E2R texts. The first step is an automatic evaluation to measure quantitative aspects related to text complexity. The second step is a checklist-based manual evaluation that takes into account qualitative aspects. The third step is a user evaluation, focusing on the needs of end-users and the understandability of texts. Our methodology ensures thorough assessments of E2R texts, even when user evaluations are not feasible. This approach aims to bring standardization and reliability to the evaluation process of E2R texts.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
Leichte Sprache is the German version of E2R.
- 4.
- 5.
TTR calculated with respect to the lemmata in first 200 tokens of a document.
- 6.
The value corresponds to the ratio between content words (nouns, proper nouns, verbs, adjectives, adverbs) over the total number of words in a document.
References
Alva-Manchego, F., Scarton, C., Specia, L.: Data-driven sentence simplification: survey and benchmark. Comput. Linguist. 46(1), 135–187 (2020)
Alva-Manchego, F., Scarton, C., Specia, L.: The (un) suitability of automatic evaluation metrics for text simplification. Comput. Linguist. 47(4), 861–889 (2021)
Amstad, T.: Wie verständlich sind unsere Zeitungen? Studenten-Schreib-Service (1978)
Bengoetxea, K., Gonzalez-Dios, I.: Multiaztertest: a multilingual analyzer on multiple levels of language for readability assessment. arXiv preprint arXiv:2109.04870 (2021)
Brunato, D., Cimino, A., Dell’Orletta, F., Venturi, G., Montemagni, S.: Profiling-UD: a tool for linguistic profiling of texts. In: Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 7145–7151 (2020)
Cumbicus-Pineda, O.M., Gonzalez-Dios, I., Soroa, A.: Linguistic capabilities for a checklist-based evaluation in automatic text simplification. In: CTTS@ SEPLN (2021)
Fernández Huerta, J.: Medidas sencillas de lecturabilidad. Consigna 214, 29–32 (1959)
Grabar, N., Saggion, H.: Evaluation of automatic text simplification: where are we now, where should we go from here. In: Traitement Automatique des Langues Naturelles, pp. 453–463. ATALA (2022)
Hansen-Schirra, S., Maaß, C.: Easy language, plain language, easy language plus: perspectives on comprehensibility and stigmatisation. Easy Lang. Res. Text User Perspectives 2, 17 (2020)
Jindal, P., MacDermid, J.C.: Assessing reading levels of health information: uses and limitations of Flesch formula. Educ. Health 30(1), 84–88 (2017)
Lin, C.Y.: Rouge: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004)
Lin, C.Y., Hovy, E.: Automatic evaluation of summaries using n-gram co-occurrence statistics. In: Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pp. 150–157 (2003)
Madina, M., Gonzalez-Dios, I., Siegel, M.: A preliminary study of ChatGPT for Spanish E2R text adaptation. In: LREC-Coling 2024 [forthcoming] (2024)
Morato, J., Campillo, A., Sanchez-Cuadrado, S., Iglesias, A., Berrios, O.: An accessible evaluation tool to detect easy-to-read barriers. In: Proceedings of the 9th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion, pp. 55–60 (2020)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 (2002)
Reichrath, E., Moonen, X.: Assessing the effects of Language for all. Nord. J. Linguist. 45(2), 232–248 (2022)
Suárez-Figueroa, M.C., Ruckhaus, E., López-Guerrero, J., Cano, I., Cervera, Á.: Towards the assessment of easy-to-read guidelines using artificial intelligence techniques. In: Miesenberger, K., Manduchi, R., Covarrubias Rodriguez, M., Peňáz, P. (eds.) ICCHP 2020. LNCS, vol. 12376, pp. 74–82. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58796-3_10
Sulem, E., Abend, O., Rappoport, A.: BLEU is not suitable for the evaluation of text simplification. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 738–744. Association for Computational Linguistics, Brussels, Belgium, Oct-November 2018. https://doi.org/10.18653/v1/D18-1081, https://aclanthology.org/D18-1081
Toborek, V., Busch, M., Boßert, M., Bauckhage, C., Welke, P.: A new aligned simple German corpus. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11393–11412. Association for Computational Linguistics, Toronto, Canada, July 2023. https://doi.org/10.18653/v1/2023.acl-long.638, https://aclanthology.org/2023.acl-long.638
UNE: UNE 153101:2018 EX Lectura fácil. Pautas y Recomendaciones para la Elaboración de Documentos. Madrid: Asociación Española de Normalización (2018). https://www.une.org/encuentra-tu-norma/busca-tu-norma/norma?c=N0060036
Xu, W., Napoles, C., Pavlick, E., Chen, Q., Callison-Burch, C.: Optimizing statistical machine translation for text simplification. Trans. Assoc. Comput. Linguist. 4, 401–415 (2016)
Acknowledgements
We would like to thank the evaluators who performed the manual evaluation. This work has been partially supported by the following projects i) Ixa group A type research group (IT-1805-22) funded by the Basque Government ii) DeepKnowledge (PID2021-127777OB-C21) project funded by MCIN/AEI/10.13039/501100011033 and by FEDERe and iii) AWARE Commonsense for a new generation of natural language understanding applications (TED2021-131617B-I00) funded by MCIN/AEI /10.13039/501100011033 by the European Union NextGenerationEU/ PRTR.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Madina, M., Gonzalez-Dios, I., Siegel, M. (2024). Towards Reliable E2R Texts: A Proposal for Standardized Evaluation Practices. In: Miesenberger, K., Peňáz, P., Kobayashi, M. (eds) Computers Helping People with Special Needs. ICCHP 2024. Lecture Notes in Computer Science, vol 14751. Springer, Cham. https://doi.org/10.1007/978-3-031-62849-8_28
Download citation
DOI: https://doi.org/10.1007/978-3-031-62849-8_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-62848-1
Online ISBN: 978-3-031-62849-8
eBook Packages: Computer ScienceComputer Science (R0)