Oliver Alonzo
Jessica Trussell
Matt Huenerfauth
ABSTRACT
Research has explored using Automatic Text Simplification for reading assistance, with prior work identifying benefits and interests from Deaf and Hard-of-Hearing (DHH) adults. While the evaluation of these technologies remains a crucial aspect of research in the area, researchers lack guidance in terms of how to evaluate text complexity with DHH readers. Thus, in this work we conduct methodological research to evaluate metrics identified from prior work (including reading speed, comprehension questions, and subjective judgements of understandability and readability) in terms of their effectiveness for evaluating texts modified to be at various complexity levels with DHH adults at different literacy levels. Subjective metrics and low-linguistic-complexity comprehension questions distinguished certain text complexity levels with participants with lower literacy. Among participants with higher literacy, only subjective judgements of text readability distinguished certain text complexity levels. For all metrics, participants with higher literacy scored higher or provided more positive subjective judgements overall.
Supplemental Material
Available for Download
- J. Albertini and C. Mayer. 2011. Using Miscue Analysis to Assess Comprehension in Deaf College Readers. Journal of Deaf Studies and Deaf Education 16, 1 (2011), 35–46. https://doi.org/10.1093/deafed/enq017Google Scholar
Cross Ref
- Oliver Alonzo, Lisa Elliot, Becca Dingman, and Matt Huenerfauth. 2020. Reading Experiences and Interest in Reading-Assistance Tools Among Deaf and Hard-of-Hearing Computing Professionals. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery, New York, NY, USA, 13. https://doi.org/10.1145/3373625.3416992Google ScholarDigital Library
- Oliver Alonzo, Matthew Seita, Abraham Glasser, and Matt Huenerfauth. 2020. Automatic Text Simplification Tools for Deaf and Hard of Hearing Adults: Benefits of Lexical Simplification and Providing Users with Autonomy. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376563Google ScholarDigital Library
- Mahmoud Azab, Chris Hokamp, and Rada Mihalcea. 2015. Using Word Semantics To Assist English as a Second Language Learners. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations. Association for Computational Linguistics, Denver, Colorado, 116–120. https://doi.org/10.3115/v1/N15-3024Google ScholarCross Ref
- Larwan Berke, Sushant Kafle, and Matt Huenerfauth. 2018. Methods for Evaluation of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at Different Reading Literacy Levels. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, Article 91, 12 pages. https://doi.org/10.1145/3173574.3173665Google ScholarDigital Library
- Jan De Belder and Marie-Francine Moens. 2010. Text simplification for children. In Prroceedings of the SIGIR workshop on accessible search systems. ACM; New York, 19–26.Google Scholar
- Joshua R De Leeuw. 2015. jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior research methods 47, 1 (2015), 1–12.Google Scholar
- Siobhan Devlin and Gary Unthank. 2006. Helping Aphasic People Process Online Information. In Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Portland, Oregon, USA) (Assets ’06). ACM, New York, NY, USA, 225–226. https://doi.org/10.1145/1168987.1169027Google ScholarDigital Library
- Jack Downey. 2010. Careers in Software: Is There Life after Programming?. In Proceedings of the 2010 Special Interest Group on Management Information System’s 48th Annual Conference on Computer Personnel Research on Computer Personnel Research (Vancouver, BC, Canada) (SIGMIS-CPR ’10). Association for Computing Machinery, New York, NY, USA, 1–7. https://doi.org/10.1145/1796900.1796912Google ScholarDigital Library
- Mary C Dyson and Mark Haselgrove. 2001. The Influence of Reading Speed and Line Length on the Effectiveness of Reading from Screen. Int. J. Hum.-Comput. Stud. 54, 4 (April 2001), 585–612. https://doi.org/10.1006/ijhc.2001.0458Google ScholarDigital Library
- Susan R Easterbrooks and Sharon Baker. 2002. Language Learning in Children Who Are Deaf and Hard of Hearing: Multiple Pathways.ERIC.Google Scholar
- Matt Huenerfauth. 2008. Evaluation of a Psycholinguistically Motivated Timing Model for Animations of American Sign Language. In Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (Halifax, Nova Scotia, Canada) (Assets ’08). Association for Computing Machinery, New York, NY, USA, 129–136. https://doi.org/10.1145/1414471.1414496Google ScholarDigital Library
- Matt Huenerfauth, Lijun Feng, and Noémie Elhadad. 2009. Comparing Evaluation Techniques for Text Readability Software for Adults with Intellectual Disabilities. In Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, Pennsylvania, USA) (Assets ’09). Association for Computing Machinery, New York, NY, USA, 3–10. https://doi.org/10.1145/1639642.1639646Google ScholarDigital Library
- Matt Huenerfauth and Hernisa Kacorri. 2015. Best practices for conducting evaluations of sign language animation. In 30th Annual International Technology and Persons with Disabilities Conference Scientific/Research Proceedings. California State University, Northridge.Google Scholar
- Kentaro Inui, Atsushi Fujita, Tetsuro Takahashi, Ryu Iida, and Tomoya Iwakura. 2003. Text Simplification for Reading Assistance: A Project Note. In Proceedings of the Second International Workshop on Paraphrasing - Volume 16 (Sapporo, Japan) (PARAPHRASE ’03). Association for Computational Linguistics, Stroudsburg, PA, USA, 9–16. https://doi.org/10.3115/1118984.1118986Google ScholarDigital Library
- Marcel Adam Just and Patricia A. Carpenter. 1984. Using Eye Fixations to Study Reading Comprehension. Erlbaum, 151–182.Google Scholar
- Lynda J Katz and Franklin C Brown. 2019. Aptitude and achievement testing. In Handbook of Psychological Assessment. Elsevier, 143–168.Google ScholarCross Ref
- Poorna Kushalnagar, Scott Smith, Melinda Hopper, Claire Ryan, Micah Rinkevich, and Raja Kushalnagar. 2018. Making cancer health text on the Internet easier to read for deaf people who use American Sign Language. Journal of Cancer Education 33, 1 (2018), 134–140.Google ScholarCross Ref
- Walter S. Lasecki, Luz Rello, and Jeffrey P. Bigham. 2015. Measuring Text Simplification with the Crowd. In Proceedings of the 12th Web for All Conference(Florence, Italy) (W4A ’15). Association for Computing Machinery, New York, NY, USA, Article 4, 9 pages. https://doi.org/10.1145/2745555.2746658Google ScholarDigital Library
- Qisheng Li, Meredith Ringel Morris, Adam Fourney, Kevin Larson, and Katherina Reinecke. 2019. The Impact of Web Browser Reader Views on Reading Speed and User Experience. In CHI 2019. ACM. https://www.microsoft.com/en-us/research/publication/the-impact-of-web-browser-reader-views-on-reading-speed-and-user-experience/Google Scholar
- Angrosh Annayappan Mandya, Tadashi Nomoto, and Advaith Siddharthan. 2014. Lexico-syntactic text simplification and compression with typed dependencies. In 25th International Conference on Computational Linguistics.Google Scholar
- Marc Marschark, John A. Albertini, and Harry G. Lang. 2002. Educating deaf students: from research to practice. Oxford University Press.Google Scholar
- Carolyn Morrison, Marc Marschark, Thomastine Sarchet, Carol M. Convertino, Georgianna Borgna, and Richard Dirmyer. 2013. Deaf students’ metacognitive awareness during language comprehension. European Journal of Special Needs Education 28, 1 (2013), 78–90. https://doi.org/10.1080/08856257.2012.749610Google ScholarCross Ref
- Carol Padden, Tom Humphries, and Carol Padden. 2009. Inside deaf culture. Harvard University Press.Google Scholar
- Gustavo Paetzold and Lucia Specia. 2016. Understanding the Lexical Simplification Needs of Non-Native Speakers of English. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, 717–727. https://www.aclweb.org/anthology/C16-1069Google Scholar
- S. J. Parault and H. M. Williams. 2010. Reading Motivation, Reading Amount, and Text Comprehension in Deaf and Hearing Adults. Journal of Deaf Studies and Deaf Education 15, 2 (2010), 120–135. https://doi.org/10.1093/deafed/enp031Google ScholarCross Ref
- LeAdelle Phelps and Barbara Jane Branyan. 1990. Academic achievement and nonverbal intelligence in public school hearing-impaired children. Psychology in the Schools 27, 3 (1990), 210–217.Google ScholarCross Ref
- Luz Rello, Ricardo Baeza-Yates, Stefan Bott, and Horacio Saggion. 2013. Simplify or Help?: Text Simplification Strategies for People with Dyslexia. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (Rio de Janeiro, Brazil) (W4A ’13). ACM, New York, NY, USA, Article 15, 10 pages. https://doi.org/10.1145/2461121.2461126Google ScholarDigital Library
- Luz Rello, Roberto Carlini, Ricardo Baeza-Yates, and Jeffrey P. Bigham. 2015. A Plug-in to Aid Online Reading in Spanish. In Proceedings of the 12th Web for All Conference(Florence, Italy) (W4A ’15). Association for Computing Machinery, New York, NY, USA, Article 7, 4 pages. https://doi.org/10.1145/2745555.2746661Google ScholarDigital Library
- Luz Rello, Martin Pielot, and Mari-Carmen Marcos. 2016. Make It Big!: The Effect of Font Size and Line Spacing on Online Readability. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). ACM, New York, NY, USA, 3637–3648. https://doi.org/10.1145/2858036.2858204Google ScholarDigital Library
- Luz Rello, Horacio Saggion, Ricardo Baeza-Yates, and Eduardo Graells. 2012. Graphical schemes may improve readability but not understandability for people with dyslexia. In Proceedings of the First Workshop on Predicting and Improving Text Readability for target reader populations. 25–32.Google Scholar
- Naomi B Robbins, Richard M Heiberger, 2011. Plotting Likert and other rating scales. In Proceedings of the 2011 Joint Statistical Meeting. 1058–1066.Google Scholar
- Horacio Saggion, Sanja Štajner, Stefan Bott, Simon Mille, Luz Rello, and Biljana Drndarevic. 2015. Making It Simplext: Implementation and Evaluation of a Text Simplification System for Spanish. ACM Trans. Access. Comput. 6, 4, Article 14 (May 2015), 36 pages. https://doi.org/10.1145/2738046Google ScholarDigital Library
- Tenace Setor, Damien Joseph, and Shirish C. Srivastava. 2015. Professional Obsolescence in IT: The Relationships between the Threat of Professional Obsolescence, Coping and Psychological Strain.. In Proceedings of the 2015 ACM SIGMIS Conference on Computers and People Research (Newport Beach, California, USA) (SIGMIS-CPR ’15). Association for Computing Machinery, New York, NY, USA, 117–122. https://doi.org/10.1145/2751957.2751962Google ScholarDigital Library
- Matthew Shardlow. 2014. Out in the Open: Finding and Categorising Errors in the Lexical Simplification Pipeline. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14). European Language Resources Association (ELRA), Reykjavik, Iceland, 1583–1590. http://www.lrec-conf.org/proceedings/lrec2014/pdf/479_Paper.pdfGoogle Scholar
- Matthew Shardlow. 2014. A survey of automated text simplification. International Journal of Advanced Computer Science and Applications 4, 1(2014), 58–70.Google ScholarCross Ref
- Advaith Siddharthan. 2014. A survey of research on text simplification. ITL - International Journal of Applied Linguistics 165, 2(2014), 259–298. https://doi.org/10.1075/itl.165.2.06sidGoogle ScholarCross Ref
- Barbara K Strassman. 1997. Metacognition and reading in children who are deaf: A review of the research. Journal of Deaf Studies and Deaf Education(1997), 140–149.Google Scholar
- C. B. Traxler. 2000. The Stanford Achievement Test, 9th Edition: National Norming and Performance Standards for Deaf and Hard-of-Hearing Students. Journal of Deaf Studies and Deaf Education 5, 4 (Jan 2000), 337–348. https://doi.org/10.1093/deafed/5.4.337Google ScholarCross Ref
- Shari Trewin, Diogo Marques, and Tiago Guerreiro. 2015. Usage of Subjective Scales in Accessibility Research. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (Lisbon, Portugal) (ASSETS ’15). ACM, New York, NY, USA, 59–67. https://doi.org/10.1145/2700648.2809867Google ScholarDigital Library
- Dawn Walton, Georgianna Borgna, Marc Marschark, Kathryn Crowe, and Jessica Trussell. 2019. I am not unskilled and unaware: deaf and hearing learners’ self-assessments of linguistic and nonlinguistic skills. European Journal of Special Needs Education 34, 1 (2019), 20–34. https://doi.org/10.1080/08856257.2018.1435010Google ScholarCross Ref
- Willian Massami Watanabe, Arnaldo Candido Junior, Vinícius Rodriguez Uzêda, Renata Pontin de Mattos Fortes, Thiago Alexandre Salgueiro Pardo, and Sandra Maria Aluísio. 2009. Facilita: Reading Assistance for Low-literacy Readers. In Proceedings of the 27th ACM International Conference on Design of Communication (Bloomington, Indiana, USA) (SIGDOC ’09). ACM, New York, NY, USA, 29–36. https://doi.org/10.1145/1621995.1622002Google ScholarDigital Library
- Gary S. Wilkinson and Gary J. Robertson. 2006. Wide Range Achievement Test 4 professional manual. Psychological Assessment Resources, Inc.Google Scholar
- Kristian Woodsend and Mirella Lapata. 2011. Learning to Simplify Sentences with Quasi-Synchronous Grammar and Integer Programming. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Edinburgh, Scotland, UK., 409–420. https://www.aclweb.org/anthology/D11-1038Google Scholar
- Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in Current Text Simplification Research: New Data Can Help. Transactions of the Association for Computational Linguistics 3 (2015), 283–297. https://doi.org/10.1162/tacl_a_00139Google ScholarCross Ref
- Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing Statistical Machine Translation for Text Simplification. Transactions of the Association for Computational Linguistics 4 (2016), 401–415. https://cocoxu.github.io/publications/tacl2016-smt-simplification.pdfGoogle ScholarCross Ref
- Peixuan Yan. 2018. Effects of Metacognition on English Reading Outcomes for d/Deaf and Hard of Hearing Students. Ph.D. Dissertation. The Ohio State University.Google Scholar
- Chen-Hsiang Yu and Robert C. Miller. 2010. Enhancing Web Page Readability for Non-native Readers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). ACM, New York, NY, USA, 2523–2532. https://doi.org/10.1145/1753326.1753709Google ScholarDigital Library
Index Terms
- Comparison of Methods for Evaluating Complexity of Simplified Texts among Deaf and Hard-of-Hearing Adults at Different Literacy Levels
Index terms have been assigned to the content through auto-classification.
Recommendations
Reading Experiences and Interest in Reading-Assistance Tools Among Deaf and Hard-of-Hearing Computing Professionals
ASSETS '20: Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and AccessibilityAutomatic Text Simplification (ATS) software replaces text with simpler alternatives. While some prior research has explored its use as a reading assistance technology, including some empirical findings suggesting benefits for deploying this technology ...
Methods for Evaluating the Fluency of Automatically Simplified Texts with Deaf and Hard-of-Hearing Adults at Various Literacy Levels
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing SystemsResearch has revealed benefits and interest among Deaf and Hard-of-Hearing (DHH) adults in reading-assistance tools powered by Automatic Text Simplification (ATS), a technology whose development benefits from evaluations by specific user groups. While ...
Methods for Evaluation of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at Different Reading Literacy Levels
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsAs Automatic Speech Recognition (ASR) improves in accuracy, it may become useful for transcribing spoken text in real-time for Deaf and Hard-of-Hearing (DHH) individuals. To quantify users' comprehension and opinion of automatic captions, which ...
Comments