Carla Teixeira Lopes
ABSTRACT
Because of terminology mismatches, health consumers frequently face difficulties while searching the Web for health information. Difficulties arise in query formulation but also in understanding the retrieved documents. In this work we analyze how documents' readability affects users' comprehension and how both affect the retrieval performance, measured in different ways. In addition, we analyze how performance measures relate with each other. For this purpose we have conducted a laboratory user study with 40 participants. We found that readability is essential for a document to be at least partially relevant and that it becomes even more important if the document has medico-scientific terminology. Moreover, the relevance of a document to a specific user highly depends on its comprehension. In lay queries we found the medical accuracy of users' answers is related to the session's relevance assessments. This shows that users can, at least in part, relate their relevance assessments with the medical accuracy of the documents. On the other hand, this relationship does not exist with medico-scientific queries.
- Maristella Agosti, Richard Berendsen, Toine Bogers, Martin Braschler, Paul Buitelaar, Khalid Choukri, Giorgio M. Di Nunzio, Nicola Ferro, Pamela Forner, Allan Hanbury, Karin F. Heppin, Preben Hansen, Anni Jävelin, Birger Larsen, Mihai Lupu, Ivano Masiero, Henning Müller, Simone Peruzzo, Vivien Petras, Florina Piroi, Maarten de Rijke, Giuseppe Santucci, Gianmaria Silvello, and Elaine Toms. 2012. PROMISE Retreat Report: Prospects and Opportunities for Information Access Evaluation. Technical Report. PROMISE network of excellence, grant agreement no. 258191.Google Scholar
- Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, Vol. 70, 4 (1968), 213--220. http://psycnet.apa.org/doiLanding?doi=10.1037%2Fh0026256Google ScholarCross Ref
- Johanna Eerola and Pertti Vakkari. 2008. How a general and a specific thesaurus cover expressions in patients' questions and physicians' answers. Journal of Documentation, Vol. 64, 1 (2008), 131--142.Google ScholarCross Ref
- P. R. Fitzsimmons, B. D. Michael, J. L. Hulley, and G. O. Scott. 2010. A readability assessment of online Parkinson's disease information. The Journal of the Royal College of Physicians of Edinburgh, Vol. 40, 4 (Dec. 2010), 292--296.Google ScholarCross Ref
- Daniela B. Friedman, Laurie Hoffman-Goetz, and Jose F. Arocha. 2004. Readability of cancer information on the internet. Journal of Cancer Education, Vol. 19, 2 (2004), 117--122.Google ScholarCross Ref
- Daniela B. Friedman, Laurie Hoffman-Goetz, and Jose. F. Arocha. 2006. Health literacy and the World Wide Web: comparing the readability of leading incident cancers on the Internet. Medical informatics and the Internet in medicine, Vol. 31, 1 (March 2006), 67--87. http://view.ncbi.nlm.nih.gov/pubmed/16754369Google Scholar
- Ian C. Hoppe. 2010. Readability of patient information regarding breast cancer prevention from the Web site of the National Cancer Institute. Journal of Cancer Education, Vol. 25, 4 (Dec. 2010), 490--492. http://view.ncbi.nlm.nih.gov/pubmed/20238201Google ScholarCross Ref
- J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics, Vol. 33, 1 (1977), 159--174. https://www.jstor.org/stable/2529310Google ScholarCross Ref
- Gondy Leroy, James E. Endicott, David Kauchak, Obay Mouradi, and Melissa Just. 2013. User evaluation of the effects of a text simplification algorithm using term familiarity on perception, understanding, learning, and information retention. Journal of Medical Internet Research, Vol. 15, 7 (2013), e144.Google ScholarCross Ref
- Gondy Leroy, Stephen Helmreich, and James R. Cowie. 2010. The influence of text characteristics on perceived and actual difficulty of health information. International Journal of Medical Informatics, Vol. 79, 6 (June 2010), 438--449.Google ScholarCross Ref
- Gondy Leroy, David Kauchak, and Obay Mouradi. 2014. A user-study measuring the effects of lexical simplification and coherence enhancement on perceived and actual text difficulty. International Journal of Medical Informatics, Vol. 82, 8 (Aug. 2014), 717--730.Google Scholar
- Carla Teixeira Lopes and Cristina Ribeiro. 2013. Measuring the value of health query translation: An analysis by user language proficiency. Journal of the American Society for Information Science and Technology Vol. 64, 5 (May 2013), 951--963.Google ScholarCross Ref
- Carla Teixeira Lopes and Cristina Ribeiro. 2015. Effects of Terminology on Health Queries: An Analysis by User's Health Literacy and Topic Familiarity. Advances in Librarianship, Vol. 39 (2015), 145--184.Google ScholarCross Ref
- Carla Teixeira Lopes and Hugo Sousa. 2019. Assisting Health Consumers While Searching the Web Through Medical Annotations. In Proceedings of the 2019 Conference on Human Information Interaction & Retrieval. ACM, New York, NY, USA. Google ScholarDigital Library
- Wen-Hsiang H. Lu, Ray Shih-Jui, Yi-Che C. Chan, and Kuan-Hsi H. Chen. 2006. Overcoming terminology barrier using Web resources for cross-language medical information retrieval.. In AMIA Annual Symposium proceedings. American Medical Informatics Association (AMIA), Maryland, USA, 519--523. http://view.ncbi.nlm.nih.gov/pubmed/17238395Google Scholar
- Gang Luo. 2009. Design and Evaluation of the iMed Intelligent Medical Search Engine. In Proceedings of the 2009 IEEE International Conference on Data Engineering (ICDE'09). IEEE Computer Society, Washington, DC, USA, 1379--1390. Google ScholarDigital Library
- Gang Luo, Chunqiang Tang, Hao Yang, and Xing Wei. 2008. MedSearch: a specialized search engine for medical information retrieval. In CIKM'08: Proceedings of the 17th ACM Conference on Information and Knowledge Mining. ACM, New York, NY, USA, 143--152. Google ScholarDigital Library
- Gheorghe Muresan, Michael Cole, Catherine L. Smith, Lu Liu, and Nicholas J. Belkin. 2006. Does Familiarity Breed Content? Taking Account of Familiarity with a Topic in Personalizing Information Retrieval. In Proceedings of the 39th Annual Hawaii International Conference on System Sciences, 2006. HICSS'06., Vol. 3. IEEE, New York, NY, USA, 53c. Google ScholarDigital Library
- Joao Palotti, Lorraine Goeuriot, Guido Zuccon, and Allan Hanbury. 2016. Ranking Health Web Pages with Relevance and Understandability. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'16). ACM, New York, NY, USA, 965--968. Google ScholarDigital Library
- Stephen E. Robertson, Evangelos Kanoulas, and Emine Yilmaz. 2010. Extending average precision to graded relevance judgments. In Proceeding of the 33rd international ACM SIGIR conference on Research and development in information retrieval (SIGIR '10). ACM, New York, NY, USA, 603--610. Google ScholarDigital Library
- Tefko Saracevic. 1996. Relevance reconsidered. In Information science: Integration in perspectives. Proceedings of the Second Conference on Conceptions of Library and Information Science. Royal School of Librarianship, Copenhagen, 201--218.Google Scholar
- Chenhao Tan, Evgeniy Gabrilovich, and Bo Pang. 2012. To Each His Own: Personalized Content Selection Based on Text Comprehensibility. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining (WSDM'12). ACM, New York, NY, USA, 233--242. Google ScholarDigital Library
- Kevyn C. Thompson, Paul N. Bennett, Ryen W. White, Sebastian de la Chica, and David Sontag. 2011. Personalizing Web Search Results by Reading Level. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM'11). ACM, New York, NY, USA, 403--412. Google ScholarDigital Library
- Tiffany M. Walsh and Teresa A. Volsko. 2008. Readability assessment of internet-based consumer health information. Respiratory Care, Vol. 53, 10 (Oct. 2008), 1310--1315. http://view.ncbi.nlm.nih.gov/pubmed/18811992Google Scholar
- Q. Zeng, S. Kogan, N. Ash, R. A. Greenes, and A. A. Boxwala. 2002. Characteristics of consumer terminology for health information retrieval. Methods of Information in Medicine, Vol. 41, 4 (2002), 289--298. http://view.ncbi.nlm.nih.gov/pubmed/12425240Google ScholarCross Ref
- Qing T. Zeng, Jonathan Crowell, Robert M. Plovnick, Eunjung Kim, Long Ngo, and Emily Dibble. 2006. Assisting consumer health information retrieval with query recommendations. Journal of the American Medical Informatics Association, Vol. 13, 1 (2006), 80--90.Google ScholarCross Ref
- Guido Zuccon. 2016. Understandability Biased Evaluation for Information Retrieval. In Advances in Information Retrieval, Nicola Ferro, Fabio Crestani, Marie-Francine Moens, Josiane Mothe, Fabrizio Silvestri, Giorgio M.Di Nunzio, Claudia Hauff, and Gianmaria Silvello (Eds.). Lecture Notes in Computer Science, Vol. 9626. Springer International Publishing, Cham, 280--292.Google Scholar
Index Terms
- Interplay of Documents' Readability, Comprehension and Consumer Health Search Performance Across Query Terminology
Recommendations
Assisting Health Consumers While Searching the Web through Medical Annotations
CHIIR '19: Proceedings of the 2019 Conference on Human Information Interaction and RetrievalHealth consumers usually face difficulties on their online searches, mainly because of the differences between terminologies used by laypeople and health professionals. This work presents a tool, HealthTranslator, available as a Google Chrome extension ...
Accessibility and readability compliance in Spanish public hospital websites
DSAI '22: Proceedings of the 10th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusionThe number of e-health websites has increased significantly over the past few years. The Internet is used in many ways, from carrying out searches on a wide variety of topics to accessing online services such as those offered by hospitals. Thus, the ...
Enhancing readability of web documents by text augmentation for deaf people
WIMS '13: Proceedings of the 3rd International Conference on Web Intelligence, Mining and SemanticsDeaf people have particular difficulty in understanding text-based web documents because their mother language, or sign language, is essentially visually oriented. To enhance the readability of text-based web documents for deaf people, we propose a news ...
Comments