skip to main content
10.1145/3380851.3416755acmconferencesArticle/Chapter ViewAbstractPublication PagesdocConference Proceedingsconference-collections
research-article

A Mixed-methods Approach to Evaluating the Usability of Telemedicine Communications

Published: 03 October 2020 Publication History

Abstract

Telemedicine is an alternative healthcare delivery mechanism poised to increase access to healthcare. However, telemedicine has received limited uptake and use by patients. Like other health information technologies (HIT), usability is a critical determinant of telemedicine acceptance and use. Users’ first impression of telemedicine and understanding of the appropriate use of telemedicine stems from their interaction with telemedicine provider websites. Without sufficient telemedicine communications, users will not perceive telemedicine to be a quality healthcare delivery method or understand how to perform a virtual physician visit. Therefore, evaluating the usability of telemedicine provider websites is needed in order to determine how users interact with them and understand the information and instructions provided. This paper describes a mixed-methods usability study involving content analysis, remote usability testing and think aloud usability testing. Combined, these methods offered increasing degrees of validity, reliability, and fidelity, which researchers are constantly evolving for the health and medicine context. The multidimensional usability testing framework described in this paper can be applied in other fields and can be modified for other technologies and contexts.

References

[1]
Ruthe C. Ashley. 2020. Telemedicine: legal, ethical, and liability considerations. Journal of the Academy of Nutrition and Dietetics, 102(2), 267-269.
[2]
Rashid L. Bashshur. 1995. On the definition and evaluation of telemedicine. Telemedicine Journal (continued by Telemedicine Journal and e-health), 1(1), 19-30.
[3]
Lori Uscher-Pines and Ateev Mehrotra. 2014. Analysis Of Teladoc Use Seems To Indicate Expanded Access To Care For Patients Without Prior Connection To A Provider. Health Affairs,33(2), 258-264.
[4]
Jane S. Hwang, Charles M. Lappan, Leonard C. Sperling, and Jon H. Meyerle. 2014. Utilization of Telemedicine in the U.S. Military in a Deployed Setting. Military Medicine, 179(11), 1347-1353.
[5]
Ahmad Alaiad and Lina ZhouAlaiad. 2017. Patients' adoption of WSN-Based smart home healthcare systems: An integrated model of facilitators and barriers. IEEE Transactions On Professional Communication, 60(1), 4-23.
[6]
Yousef Jasemian and Lars Arendt-Nielsen. 2005. Design and implementation of a telemedicine system using Bluetooth protocol and GSM/GPRS network, for real time remote patient monitoring. Technology and Health Care, 13(3), 199-219.
[7]
Juan M. García-Gómez, Isabel de la Torre-Díez, Javier Vicente, Montserrat Robles, Miguel López-Coronado and Joel J. Rodrigues. 2014. Analysis of mobile health applications for a broad spectrum of consumers: A user experience approach. Health Informatics Journal, 20(1), 74-84.
[8]
Kathryn E. Artnak, Richard M. McGraw and Vayden F. Stanley. 2011. Health Care Accessibility for Chronic Illness Management and End-of-Life Care: A View from Rural America. Journal of Law, Medicine & Ethics, 39(2), 140-55.
[9]
Thomas S. Bodenheimer and Mark D. Smith. 2013. Primary Care: Proposed Solutions To The Physician Shortage Without Bodenheimer Training More Physicians. Health Affairs, 32(11), 1881-1886.
[10]
Sheba George, Alison Hamilton and Richard S. Baker. 2012. How Do Low-Income Urban African Americans and Latinos Feel about Telemedicine? A Diffusion of Innovation Analysis. International Journal of Telemedicine and Applications, 715194.
[11]
Lori Uscher-Pines, Robert Rudin and Ateev Mehrotra. 2017. Leveraging Telehealth to Bring Volunteer Physicians Into Underserved Communities. Telemedicine and e-Health, 23(6), 533-535.
[12]
Sheba George, Alison Hamilton and Richard S. Baker. 2012. How Do Low-Income Urban African Americans and Latinos Feel about Telemedicine? A Diffusion of Innovation Analysis. International Journal of Telemedicine and Applications, 715194.
[13]
Matthew R. Gardner, Sarah M. Jenkins and Daniel A. O'Neil, 2015. Perceptions of video-based appointments from the patient's home: a patient survey. Telemedicine Journal and e-health: The official journal of the American Telemedicine Association, 21(4), 281-285.
[14]
William Grigsby, Angela G Brega, and Rachel E. Bennett, R. 2007. The slow pace of interactive video telemedicine adoption: The perspective of telemedicine program administrators on physician participation. Telemedicine Journal and e-health: The official journal of the American Telemedicine Association, 13(6), 645-656. 10.1089/tmj.2007.0090
[15]
Carlos Flavián, Raquel Gurrea and Carlos Orús. 2008. A heuristic evaluation of websites design for achieving the web success. International Journal of Services and Standards, 5(1), 17-41.
[16]
Brandon M. Welch, Jillian Harvey, Nathaniel S. O'Connell and James T. McElligott. 2017. Patient preferences for direct-to-consumer telemedicine services: a nationwide survey. BMC Health Services Research,17, 1–7. https://doi-org.ezproxy.net.ucf.edu/10.1186/s12913-017-2744-8
[17]
Danielle R. Bullock, Richard K. Vehe, Lei Zhang and Colleen K. Correll. 2017. Telemedicine and other care models in pediatric rheumatology: an exploratory study of parents' perceptions of barriers to care and care preferences. Pediatric Rheumatology, 151-158.
[18]
Ramakrishna Dantu and RadhaKanta Mahapatra. 2013. Adoption of Telemedicine – Challenges and Opportunities. In Proceedings of the Nineteenth Americas Conference on Information Systems, August 15 - 17, 2013, Chicago, Illinois.
[19]
Fred D. Davis. 1989. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13(3), 319-340.
[20]
Michael J. Albers. 2003. Multidimensional Audience Analysis for Dynamic Information. Journal of Technical Writing and Communication, 33(3), 263-279.
[21]
Rhea E. Powell, Jeffrey M. Henstenburg and Grace Cooper, 2017. Patient Perceptions of Telehealth Primary Care Video Visits. Annals Of Family Medicine,15(3), 225-229.
[22]
Kirk St.Amant. 2017. The Cultural Context of Care in International Communication Design: A Heuristic for Addressing Usability in International Health and Medical Communication. Communication Design Quarterly, 5(2), 62-70.
[23]
Andre W. Kushniruk. 2001. Analysis of complex decision making processes in health care: cognitive approaches to health informatics. Journal of Biomedical Informatics, 34(5), 365–376. 10.1006/jbin.2001.1021
[24]
Michelle L. Rogers, Emily S. Patterson, Roger James Chapman and Marta Render. 2005. Usability Testing and the Relation of Clinical Information Systems to Patient Safety. In Kerm Henriksen, James B. Battles, Eric S. Marks and David I Lewin K. (Eds.), Advances in Patient Safety: From Research to Implementation, Volume 2: Concepts and Methodology. Rockville, MD: Agency for Healthcare Research and Quality. 365 – 378. Retrieved from https://www.ncbi.nlm.nih.gov/books/NBK20503/
[25]
Jang-Sun Hwang, Sally J. McMillan and Guiohk Lee. 2003. Corporate Web Sites as Advertising. Journal of Interactive Advertising, 3(2), 10-23.
[26]
Susannah Fox and Lee Rainie. 2000. The Online Health Care Revolution: How the Web Helps Americans Take Better Care of Themselves. Pew Internet & American Life: Online life report. The Institute for Health Care Research and Policy, Washington, D.C.
[27]
Bradford W. Hesse, David E. Nelson, and Gary L. Kreps, 2005. Trust and Sources of Health Information: The Impact of the Internet and Its Implications for Health Care Providers: Findings From the First Health Information National Trends Survey. Archives of Internal Medicine, 165, 2618-2624.
[28]
Ryan Sterling and Cynthia Lerouge. 2019. On-Demand Telemedicine as a Disruptive Health Technology: Qualitative Study Exploring Emerging Business Models and Strategies Among Early Adopter Organizations in the United States. Journal of medical Internet research, 21(11), e14304.
[29]
Jeongeun Kim and Hyeoun-Ae Park. 2012. Development of a health information technology acceptance model using consumers' health behavior intention. Journal of medical Internet research, 2012 Oct 1;14(5):e133.
[30]
Vicky R. Breakey, Ashley V. Warias and Danial M Ignas, 2013. The value of usability testing for Internet-based adolescent self-management interventions: "Managing Hemophilia Online". BMC Medical Informatics and Decision Making, 13, 113.
[31]
David R. Kaufman, Vimla L. Patel and Charlyn Hilliman, 2003. Usability in the real world: assessing medical information technologies in patients’ homes. Journal of Biomedical Informatics,36(1-2), 45-60.
[32]
Andrew Kushniruk. 2002. Evaluation in the Design of Health Information Systems: Application of Approaches Emerging from Usability Engineering. Computers in Biology and Medicine, 32, 141-49.
[33]
Anne G. Ekeland, Alison Bowes and Signe Flottorp. 2010. Effectiveness of telemedicine: A systematic review of reviews. International Journal of Medical Informatics, 79(11), 736–771.
[34]
Po-Yin Yen and Suzanne Bakken. 2011. Review of health information technology usability study methodologies. Journal of the American Medical Informatics Association: JAMIA, 19(3), 413-422.
[35]
Lisa Melonçon. 2017. Patient experience design: Expanding usability methodologies for healthcare. Communication Design Quarterly, 5(2), 19-28.
[36]
Jeph Preece, J. (n.d.). Best Telemedicine Services of 2018. (n.d.).10 Top Ten Reviews. Retrieved July 7, 2019 from https://www.toptenreviews.com/health/services/best-telemedicine-services/
[37]
James Roland. 2015. 17 Best Telemedicine Companies. Healthline. (October 2015). Retrieved July 7, 2019 from https://www.healthline.com/health/best-telemedicine-companies#1
[38]
Elizabeth Borycki and Andre Kushniruk. 2005. Identifying and Preventing Technology-Induced Error Using Simulations: Application of Usability Engineering Techniques. Healthcare Quarterly, 8, 99-105.
[39]
Olga C. Damman, Michelle Hendriks, Jany Rademakers, Diana MJ. Delnoij and Peter P. Groenewegen. 2009. How do healthcare consumers process and evaluate comparative healthcare information? A qualitative study using cognitive interviews. BMC Public Health, 9(423).
[40]
Calvin K. L. Or and Ben-Tzion Karsh. 2009. A systematic review of patient acceptance of consumer health information technology. Journal of the American Medical Informatics Association (JAMIA), 16(4), 550–560.
[41]
Paul C. Tang, Joan S. Ash, David W. Bates, J. Marc Overhage and Daniel Z. Sands. 2006. Personal health records: definitions, benefits, and Strategies for overcoming barriers to adoption. Journal of American Medical Informatics Association, 13(2), 121-126.
[42]
International Organization for Standardization (ISO), 9241-11. (2018). Ergonomics of human-system interaction – Part 11: Usability: Definitions and concepts (2nd ed.). Switzerland: ISO/IEC.
[43]
Po-Yin Yen and Suzanne Bakken. 2012. Review of health information technology usability study methodologies. Journal of the American Medical Informatics Association: JAMIA, 19(3), 413-422. 10.1136/amiajnl-2010-000020
[44]
Andre W.Kushniruk and Vimla L.Patel. 2004. Cognitive and usability engineering methods for the evaluation of clinical information systems. Journal of Biomedical Informatics, 37(1), 56-76.
[45]
Andre W. Kushniruk, Elizabeth M. Borycki, Shige Kuwata and Francis Ho. 2008. Emerging Approaches to Evaluating the Usability of Health Information Systems. In Andre W. Kushniruk and Elizabeth M. Borycki (Eds.), Human, Social, and Organizational Aspects of Health Information Systems. Hershey, PA: IGI Global. 1 - 22.
[46]
Jacob Nielsen and and Robert L. Mack. (Eds.). 1994. Usability Inspection Methods. John Wiley & Sons, New York, NY.
[47]
Nicholas Mays and Catherine Pope. 1995. Rigour and qualitative research. BMJ: The British Medical Journal, 311, 109-112. 10.1136/bmj.311.6997.109
[48]
Helen Noble and Joanna Smith. 2015. Issues of validity and reliability in qualitative research. Evidence Based Nursing, 18(2), 34-35.
[49]
Andre Kushniruk, Christian Nøhr, Sofia Jensen and Elizabeth M. Borycki. 2013. From Usability Testing to Clinical Simulations: Bringing Context into the Design and Evaluation of Usable and Safe Health Information Technologies. In IMIA Human Factors Engineering for Healthcare Informatics Working Group (Eds.), IMIA Yearbook of Medical Informatics, 2013, Geneva, Switzerland: IMIA. 78 - 85.
[50]
Klaus Krippendorff. 1980. Content analysis: An Introduction to its Methodology.Sage: Beverly Hills, CA.
[51]
Fiona H. McKay, Christina Cheng and Annemarie Wright, 2018. Evaluating mobile phone applications for health behaviour change: A systematic review. Journal of Telemedicine and Telecare, 24(1), 22–30.
[52]
Kristen Shuyler and Kristin M. Knight. 2008. What are patients seeking when they turn to the Internet? Qualitative content analysis of questions asked by visitors to an orthopaedics website. Journal of Medical Internet Research, 5(4), e24.
[53]
Reem Kayyali, Iman Hesso, Evelyn Ejiko, and Shereen Nabhani. 2017. A qualitative study of Telehealth patient information leaflets (TILs): are we giving patients enough information?. BMC Health Services Research, 17(362).
[54]
Kathryn H. Dansky and Jeffrey Ajello. 2005. Marketing Telehealth to Align with Strategy. Journal of Healthcare Management, 50(1), 23-35.
[55]
Razieh Farrahi, Fatemeh Rangraz Jeddi, Ehsan Nabovati, Monireh Sadeqi Jabali and Reza Khajouei. 2019. The relationship between user interface problems of an admission, discharge and transfer module and usability features: a usability testing method. BMC Medical Informatics and Decision Making, 19, 172.
[56]
Meg Wilson. 2009. Readability and patient education materials used for low-income populations. Clinical Nurse Specialist, 23(1), 33-40. Retrieved from https://www.nursingcenter.com/journalarticle?Article_ID=834940&Journal_ID=54033&Issue_ID=834909
[57]
Barry D. Weiss. 2003. Health literacy: A manual for clinicians. American Medical Association Foundation and American Medical Association: Chicago, IL.
[58]
Sonika Goel, Vijay Lakshmi Sharma, Amarjeet Singh and Sonu Goel. 2016. Evaluation of Quality and Readability of Health Information Websites Identified through India's Major Search Engines. Advances in Preventive Medicine, 2016, 4815285.
[59]
Gabriele Paolacci, Jesse Chandler and Panos Ipeirotis. 2010. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411-419. Retrieved from http://sjdm.org/∼baron/journal/10/10630a/jdm10630a.pdf
[60]
Catherine Henshall, Zoe Davey, Cynthia Jacelon and Clare Martin. 2019. A usability study to test the effectiveness, efficiency and simplicity of a newly developed Internet-based Exercise-focused Health App for Lung cancer survivors (iEXHALE): Protocol paper. Health Informatics Journal.
[61]
Alissa L. Russ, Darrell A. Baker and W. Jeffrey Fahner, 2010. A Rapid Usability Evaluation (RUE) Method for Health Information Technology. In AMIA…Annual Symposium Proceedings (AMIA 2010), 2010, AMIA. 702 - 706.
[62]
U.S. Department of Health and Human Services (HHS). Research-Based Web Design and Usability Guidelines. Retrieved from https://guidelines.usability.gov/
[63]
Helen Monkman and Andre Kushniruk. 2013. Applying usability methods to identify health literacy issues: an example using a Personal Health Record. Studies in health technology and informatics, 183, 179-185.
[64]
K. Anders Ericsson and Herbert A. Simon. 1984. Protocol analysis: Verbal reports as data. The MIT Press: Cambridge, MA.
[65]
Andre Kushniruk, Vimla L. Patel and J. J. Cimino. 1997. usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. In Proceedings of Conference of the American Medical Informatics Association (AMIA Fall Symposium), January 1997, AMIA. 218 - 222.
[66]
Elizabeth Borycki and Andre Kushniruk. 2005. Identifying and Preventing Technology-Induced Error Using Simulations: Application of Usability Engineering Techniques. Healthcare Quarterly, 8, 99-105.
[67]
Andre Kushniruk, Vimla L. Patel and J. J. Cimino. 1997. usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. In Proceedings of Conference of the American Medical Informatics Association (AMIA Fall Symposium), January 1997, AMIA. 218 - 222.
[68]
QSR International Ply Ltd. NVivo 12 qualitative data analysis software. QSR International. Retrieved from https://www.qsrinternational.com/nvivo/home
[69]
Andre Kushniruk, Elizabeth Borycki, Nicole Kitson and Joseph Kannry. 2019. Development of a Video Coding Scheme Focused on Socio-Technical Aspects of Human-Computer Interaction in Healthcare. In Studies in Health Technology and Informatics: Volume 257: Improving Usability, Safety and Patient Outcomes with Health Information Technology.IOS Press Ebooks. 236 – 243. Retrieved from http://ebooks.iospress.nl/publication/51166
[70]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101.
[71]
John Brooke. 1986. SUS - A quick and dirty usability scale. Digital Equipment Corporation Ltd. Reading, United Kingdom.
[72]
Bambang Parmanto, Allen Nelson Lewis, Jr., Kristin M. Graham and Marnie H. Bertolet. 2016. Development of the Telehealth Usability Questionnaire (TUQ). International Journal of Telerehabilitation, 8(1), 3-10. Retrieved from https://telerehab.pitt.edu/ojs/index.php/Telerehab/article/view/6196
[73]
Jeff Sauro. 2011. Measuring Usability with the System Usability Scale (SUS). MeasuringU. (February 2, 2011). Retrieved March 2, 2020 from https://measuringu.com/sus/
[74]
Nahid Golafshani. 2003. Understanding Reliability and Validity in Qualitative Research. The Qualitative Report, 8(4), 597-607. Retrieved from https://nsuworks.nova.edu/tqr/vol8/iss4/6/
[75]
John W. Creswell. 2002. Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research (4th ed.). Pearso: New York, NY.
[76]
Carmel Maher, Mark Hadfield, Maggie Hutchings and Adam de Eyto. 2018. Ensuring Rigor in Qualitative Data Analysis: A Design Research Approach to Coding Combining NVivo With Traditional Material Methods. International Journal of Qualitative Methods, 17(1).

Cited By

View all
  • (2022)Identifying Digital Rhetoric in the Telemedicine User InterfaceJournal of Technical Writing and Communication10.1177/0047281622112518453:2(89-105)Online publication date: 15-Sep-2022
  • (2022)A Comprehensive Analysis of Healthcare Websites Usability Features, Testing Techniques and IssuesIEEE Access10.1109/ACCESS.2022.319337810(97701-97718)Online publication date: 2022
  • (2021)From Therapy to Teletherapy: Relocating Mental Health Services OnlineProceedings of the ACM on Human-Computer Interaction10.1145/34795085:CSCW2(1-30)Online publication date: 18-Oct-2021
  • Show More Cited By
  1. A Mixed-methods Approach to Evaluating the Usability of Telemedicine Communications

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGDOC '20: Proceedings of the 38th ACM International Conference on Design of Communication
    October 2020
    273 pages
    ISBN:9781450375252
    DOI:10.1145/3380851
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 03 October 2020

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. HCI
    2. Telemedicine
    3. qualitative methods
    4. quantitative methods
    5. usability
    6. user interface design

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    SIGDOC '20
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 355 of 582 submissions, 61%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)46
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2022)Identifying Digital Rhetoric in the Telemedicine User InterfaceJournal of Technical Writing and Communication10.1177/0047281622112518453:2(89-105)Online publication date: 15-Sep-2022
    • (2022)A Comprehensive Analysis of Healthcare Websites Usability Features, Testing Techniques and IssuesIEEE Access10.1109/ACCESS.2022.319337810(97701-97718)Online publication date: 2022
    • (2021)From Therapy to Teletherapy: Relocating Mental Health Services OnlineProceedings of the ACM on Human-Computer Interaction10.1145/34795085:CSCW2(1-30)Online publication date: 18-Oct-2021
    • (2021)Towards a pluralistic, dynamic definition of usability2021 IEEE International Professional Communication Conference (ProComm)10.1109/ProComm52174.2021.00013(44-49)Online publication date: 21-Oct-2021

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media