Skip to main content

Advertisement

Log in

A Systematic Review of Voice Assistant Usability: An ISO 9241–11 Approach

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Voice assistants (VA) are an emerging technology that have become an essential tool of the twenty-first century. The VA ease of access and use has resulted in high usability curiosity in voice assistants. Usability is an essential aspect of any emerging technology, with every technology having a standardized usability measure. Despite the high acceptance rate on the use of VA, to the best of our knowledge, not many studies were carried out on voice assistants’ usability. We reviewed studies that used voice assistants for various tasks in this context. Our study highlighted the usability measures currently used for voice assistants. Moreover, our study also highlighted the independent variables used and their context of use. We employed the ISO 9241-11 framework as the measuring tool in our study. We highlighted voice assistant’s usability measures currently used; both within the ISO 9241-11 framework, as well as outside of it to provide a comprehensive view. A range of diverse independent variables are identified that were used to measure usability. We also specified that the independent variables still not used to measure some usability experience. We currently concluded what was carried out on voice assistant usability measurement and what research gaps were present. We also examined if the ISO 9241-11 framework can be used as a standard measurement tool for voice assistants.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Hoy MB. Alexa, Siri, Cortana, and more: an introduction to v oice assistants. Med Ref Serv Quart. 2018;37(1):81–8.

    Article  Google Scholar 

  2. Zwakman DS, Pal D, Arpnikanondt C. Usability evaluation of artificial intelligence-based voice assistants: the case of amazon Alexa. SN Comput Sci. 2021. https://doi.org/10.1007/s42979-020-00424-4.

    Article  Google Scholar 

  3. Segi H, Takou R, Seiyama N, Takagi T, Uematsu Y, Saito H, Ozawa S. An automatic broadcast system for a weather report radio program. IEEE Trans Broadcast. 2013;59(3):548–55.

    Article  Google Scholar 

  4. Noel S. Human computer interaction (HCI) based Smart Voice Email (Vmail) Application—Assistant f or Visually Impaired Users (VIU). In: 2020 third international conference on smart systems and inventive technology (ICSSIT) (pp 895–900). IEEE; 2020.

  5. Sangle-Ferriere M, Voyer BG. Friend or foe? Chat as a double-edged sword to assist customers. J Serv Theory Pract. 2019;29:438–61.

    Article  Google Scholar 

  6. Lugano G. Virtual assistants and self-driving cars. In: 2017 15th International Conf erence on ITS Telecommunications (ITST) (pp 1–5). IEEE; 2017.

  7. Rybinski K, Kopciuszewska E. Will artif icial intelligence rev olutionise the student evaluation of teaching? A big data study of 1.6 million student reviews. Assessment & Evaluation in Higher Education; 2020, pp. 1–13

  8. Tankovska H. Number of digital voice assistants in use worldwide 2019–2024 (in billions), 2020. https://www.statista.com/statistics/973815/worldwide-digital-voice-assistant-in-use/, (accessed 17 Nov 2021)

  9. Pal D, Arpnikanondt C, Funilkul S, Chutimaskul W. The adoption analysis of voice-based smart IoT products. IEEE Internet Things J. 2020;7(11):10852–67.

    Article  Google Scholar 

  10. Zwakman DS et al. Voice usability scale: measuring the user experience with voice assistants. In: 2020 IEEE International Symposium on Smart Electronic Sy stems (iSES)(Formerly iNiS). IEEE; 2020.

  11. Coronado E, Deuff D, Carreno-Medrano P, Tian L, Kulić D, Sumartojo S, et al. Towards a modul ar and distributed end-user dev elopment framework f or human-robot interaction. IEEE Access. 2021;9:12675–92.

    Article  Google Scholar 

  12. Maguire M. Development of a heuristic evaluation tool f or voice user interf aces. In: International conference on human-computer interaction. Cham: Springer; 2019. p. 212–25.

    Google Scholar 

  13. Fulfagar L, Gupta A, Mathur A, Shrivastava A. Development and evaluation of usability heuristics for voice user interfaces. In: International conference on research into design. Singapore: Springer; 2021. p. 375–85.

    Google Scholar 

  14. Nowacki C, Gordeeva A, Lizé AH. Improving the usability of voice user interfaces: a new set of ergonomic criteria. In: International conference on human-computer interaction. Cham: Springer; 2020. p. 117–33.

    Google Scholar 

  15. Pal D, Zhang X, Siyal S. Prohibitive factors to the acceptance of Internet of Things (IoT) technology in society: a smart-home context using a resistive modelling approach. Technol Soc. 2021;66: 101683.

    Article  Google Scholar 

  16. Murad C, Munteanu C. “I don't know what you're talking about, HALexa" the case f or voice user interface guidelines. In: Proceedings of the 1st International Conference on Conversational User Interfaces,2019; pp. 1–3.

  17. Budiu R, Laubheimer P. Intelligent assistants have poor usability: a user study of Alexa, Google assistant, and Siri. Nielsen Norman Group; 2018. Available online at https://www.nngroup.com/articles/intelligentassistant-usability/ (last accessed 4/12/2019).

  18. Murphy CN, Yates J. The international organization f or standardization (ISO): global governance through voluntary consensus. Routledge; 2009.

    Book  Google Scholar 

  19. ISO 9241-11. Ergonomic requirements for office work with visual display terminals (VDTs)—Part II guidance on usability; 1998.

  20. Weichbroth P. Usability attributes revisited: a time-framed knowledge map. In 2018 Federated Conference on Computer Science and Information Systems (FedCSIS) (pp 1005–1008). IEEE; 2018.

  21. Petrock V. Voice assistant and smart speaker users 2020. Insider Intelligence; 2020. Retrieved November 22, 2021, from https://www.emarketer.com/content/voice-assistant-and-smart-speaker-users-2020

  22. Pinelle D, Wong N and Stach T. Heuristic evaluation f or games: usability principles for video game design. In: Proceedings of SIGCHI Conference on Human Factors in Computing Sy stems (2008); 2008, pp. 1453–1462. https://doi.org/10.1145/1357054.1357282.

  23. Sutcliffe A, Gault B. Heuristic evaluation of virtual reality applications. Interact Comput 16. 2004;4:831–49. https://doi.org/10.1016/j.intcom.2004.05.00.

    Article  Google Scholar 

  24. Sharif K, Tenbergen B. Smart home voice assistants: a literature survey of user privacy and security vulnerabilities. Complex Syst Inform Model Quart. 2020;24:15–30.

    Article  Google Scholar 

  25. de Barcelos Silva A, Gomes MM, da Costa CA, da Rosa Righi R, Barbosa JLV, Pessin G, et al. Intelligent personal assistants: a systematic literature review. Expert Syst Appl. 2020;147: 113193.

    Article  Google Scholar 

  26. Gubareva R and Lopes RP. Virtual assistants for learning: a systematic literature review. In: CSEDU (1); 2020, pp. 97–103.

  27. Bérubé C, Schachner T, Keller R, Fleisch E, Wangenheim F, Barata F, Kowatsch T. Voice-based conversational agents for the prevention and management of chronic and mental health conditions: systematic literature review. J Med Internet Res. 2021;23(3): e25933.

    Article  Google Scholar 

  28. Chi OH, Gursoy D and Chi CG. Tourists’ attitudes toward the use of artificially intelligent (AI) devices in tourism service delivery: moderating role of service value seeking. J Travel Res. 2020; 0047287520971054.

  29. Kim S. Exploring how older adults use a smart speaker-based voice assistant in their first interactions: qualitative study. JMIR Mhealth Uhealth. 2021;9(1): e20427.

    Article  Google Scholar 

  30. Coursaris CK, Kim DJ. A meta-analytical review of empirical mobile usability studies. J Usability Stud. 2011;6(3):117–71.

    Google Scholar 

  31. Goh ASY, Wong LL, Yap KYL. Evaluation of COVID-19 information provided by digital voice assistants. Int J Digital Health. 2021;1(1):3.

    Article  Google Scholar 

  32. Rapp A, Curti L, Boldi A. The human side of human-chatbot interaction: a systematic literature review of ten years of research on text-based chatbots. Int J Hum-Comput Stud. 2021;151: 102630.

    Article  Google Scholar 

  33. Seaborn K, Miyake NP, Pennefather P, Otake-Matsuura M. Voice in human-agent interaction: a survey. ACM Comput Surv (CSUR). 2021;54(4):1–43.

    Article  Google Scholar 

  34. Castro JW, Ren R, Acuña ST and Lara JD. Usability of chatbots: a systematic mapping study; 2019.

  35. Bhirud N, Tataale S, Randive S, Nahar S. A literature review on chatbots in healthcare domain. Int J Sci Technol Res. 2019;8(7):225–31.

    Google Scholar 

  36. Ahmad NA, Che MH, Zainal A, Abd Rauf MF, Adnan Z. Review of chatbots design techniques. Int J Comput Appl. 2018;181(8):7–10.

    Google Scholar 

  37. Gentner T, Neitzel T, Schulze J and Buettner R. A Systematic literature review of medical chatbot research from a behavior change perspective. In: 2020 IEEE 44th annual computers, software, and applications conference (COMPSAC). IEEE; 2020, pp. 735–740.

  38. Cunningham-Nelson S, Boles W, Trouton L and Margerison E. A review of chatbots in education: practical steps forward. In: 30th Annual Conference for the Australasian Association for Engineering Education (AAEE 2019): Educators Becoming Agents of Change: Innovate, Integrate, Motivate. Engineers Australia; 2019, pp. 299–306.

  39. Van Pinxteren MM, Pluymaekers M, Lemmink JG. Human-like communication in conversational agents: a literature review and research agenda. J Serv Manag. 2020;31:203–25.

    Article  Google Scholar 

  40. Weichbroth P. Usability attributes revisited: a time-framed knowledge map. In: 2018 Federated Conference on Computer Science and Information Systems (FedCSIS). IEEE; 2018, pp. 1005–1008.

  41. Bevan N, Carter J, Earthy J, Geis T, Harker S. New ISO standards for usability, usability reports and usability measures. In: International conference on human-computer interaction. Cham: Springer; 2016. p. 268–78.

    Google Scholar 

  42. Moumane K, Idri A, Abran A. Usability evaluation of mobile applications using ISO 9241 and ISO 25062 standards. Springerplus. 2016;5(1):1–15.

    Article  Google Scholar 

  43. Yahya H, Razali R. A usability-based framework for electronic government systems development. ARPN J Eng Appl Sci. 2015;10(20):9414–23.

    Google Scholar 

  44. Alva ME, Ch THS, López B. Comparison of methods and existing tools for the measurement of usability in the web. In: International conference on web engineering. Berlin: Springer; 2003. p. 386–9.

    MATH  Google Scholar 

  45. He X, Persson H, Östman A. Geoportal usability evaluation. Int J Spatial Data Infrastruct Res. 2012;7:88–106.

    Google Scholar 

  46. Dietlein CS, Bock OL. Development of a usability scale based on the three ISO 9241–11 categories “effectiveness, ”efficacy” and “satisfaction”: a technical note. Accred Qual Assur. 2019;24(3):181–9.

    Article  Google Scholar 

  47. Nik Ahmad NA and Hasni NS. ISO 9241–11 and SUS measurement for usability assessment of dropshipping sales management application. In: 2021 10th International Conference on Software and Computer Applications. 2021; pp. 70–74.

  48. Kitchenham B. Procedures f or perf orming systematic reviews. Keele Univ. 2004;33(2004):1–26.

    Google Scholar 

  49. Seaborn K, Miyake NP, Pennefather P, Otake-Matsuura M. Voice in human–agent interaction: a survey. ACM Comput Surv (CSUR). 2021;54(4):1–43.

    Article  Google Scholar 

  50. Al-Qaysi N, Mohamad-Nordin N, Al-Emran M. Employing the technology acceptance model in social media: a systematic review. Educ Inf Technol. 2020;25(6):4961–5002.

    Article  Google Scholar 

  51. Kitchenham B and Charters S. Guidelines f or performing systematic literature reviews in software engineering; 2007.

  52. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD CD, The PRISMA, et al. statement: an updated guideline f or reporting sy stematic rev iews. BMJ. 2020;2021(372): n71. https://doi.org/10.1136/bmj.n71.

    Article  Google Scholar 

  53. Martelaro N, Teevan J and Iqbal ST. An exploration of speech-based productivity support in the car. In: Proceedings of the 2019 CHI conference on human factors in computing systems. 2019; pp. 1–12

  54. Jeong Y, Lee J and Kang Y. Exploring effects of conversational fillers on user perception of conversational agents. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19). 2019; 1–6. https://doi.org/10.1145/3290607.3312913.

  55. Yu Q, Nguyen T, Prakkamakul S and Salehi N. “I almost fell in love with a machine”: speaking with computers affects self-disclosure. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19). 2019; pp. 1–6. https://doi.org/10.1145/3290607.3312918

  56. Kiesel J, Bahrami A, Stein B, Anand A, and Hagen M. Clarifying false memories in voice-based search. In: Proceedings of the 2019 Conference on Human Information Interaction and Retrieval (CHIIR ’19). 2019; 331–335. https://doi.org/10.1145/3295750.3298961.

  57. Kontogiorgos D, Pereira A, Andersson O, Koivisto M, Rabal EG, Vartiainen V and Gustafson J. The effects of anthropomorphism and non-verbal social behavior in virtual assistants. In: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents (IVA ’19). 2019; 133–140. https://doi.org/10.1145/3308532.3329466

  58. Hoegen R, Aneja D, McDuff D and Czerwinski M. An end-to-end conversational style matching agent. In: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents (IVA ’19). 2019; 111–118. https://doi.org/10.1145/3308532.3329473

  59. Luo Y, Lee B and Choe EK. TandemTrack: shaping consistent exercise experience by complementing a mobile app with a smart speaker. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). 2020; 1–13. https://doi.org/10.1145/3313831.3376616

  60. Doyle PR, Edwards J, Dumbleton O, Clark L and Cowan BR. Mapping perceptions of humanness in intelligent personal assistant interaction. In: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ’19). 2019. https://doi.org/10.1145/3338286.3340116.

  61. Jaber R, McMillan D, Belenguer JS and Brown B. Patterns of gaze in speech agent interaction. In: Proceedings of the 1st International Conference on Conversational User Interfaces - CUI ’19 (the 1st International Conference). 2019; 1–10. https://doi.org/10.1145/3342775.3342791.

  62. Bortoli M, Furini M, Mirri S, Montangero M and Prandi C. Conversational interfaces for a smart campus: a case study. In: Proceedings of the international conference on advanced visual interfaces (AVI ’20). 2020. https://doi.org/10.1145/3399715.3399914.

  63. Wu Y, Edwards Y, Cooney O, Bleakley A, Doyle PR, Clark L, Rough D and Cowan BR. Mental workload and language production in non-native speaker IPA interaction. In: Proceedings of the 2nd Conference on Conversational User Interfaces (CUI ’20). 2020. https://doi.org/10.1145/3405755.3406118

  64. Brüggemeier B, Breiter M, Kurz M and Schiwy J. User experience of Alexa when controlling music: comparison of face and construct validity of four questionnaires. In: Proceedings of the 2nd conference on conversational user interfaces (CUI ’20). 2020. https://doi.org/10.1145/3405755.3406122

  65. Machine body language: expressing a smart speaker’s activity with intelligible physical motion. 57

  66. Bartneck C, Kulić D, Croft E, Zoghbi S. Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int J Soc Robot. 2009;1(1):71–81. https://doi.org/10.1007/s12369-008-0001-3A.

    Article  Google Scholar 

  67. Braun M, Mainz A, Chadowitz R, Pfleging B and Alt F. At your service: designing voice assistant personalities to improve automotive user interfaces. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 2019;40:1–40:11. https://doi.org/10.1145/3290605.3300270

  68. Burbach L, Halbach P, Plettenberg N, Nakayama J, Ziefle M and Valdez AC. “Hey, Siri”, “Ok, Google”, “Alexa”. Acceptance-relevant factors of virtual voice-assistants. In 2019: IEEE International Professional Communication Conference (ProComm) (ProComm ’19), 2019;101–111. https://doi.org/10.1109/ProComm.2019.00025.

  69. Pal D, Arpnikanondt C, Funilkul S, and Varadarajan V. User experience with smart voice assistants: The accent perspective. In 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT ’19), 2019;1–6. https://doi.org/10.1109/ICCCNT45670.2019.8944754.

  70. Chin H, Molefi L, and Yi Y. Empathy is all you need: How a conversational agent should respond to verbal abuse. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), 2020; 1–13. https://doi.org/10.1145/3313831.3376461.

  71. Crowell CR, Villanoy M, Scheutzz M and Schermerhornz P. Gendered voice and robot entities: Perceptions and reactions of male and female subjects. In: Proceedings of the 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2009), 2009; 3735–3741. https://doi.org/10.1109/IROS.2009.5354204

  72. Lee S, Cho M and Lee S. What if conversational agents became invisible? Comparing users’ mental models according to physical entity of AI speaker. In: Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2020; 4, 3. https://doi.org/10.1145/3411840

  73. Dahlbäck N, Wang QY, Nass C and Alwin J. Similarity is more important than expertise: Accent effects in speech interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07), 2007; 1553–1556. https://doi.org/10.1145/1240624.1240859

  74. Lee EJ, Nass C, and Brave S. Can computer-generated speech have gender?: An experimental test of gender stereotype. In Proceedings of the CHI’00 Extended Abstracts on Human factors in Computing Systems (CHI EA ’00), 2000; 289–290. https://doi.org/10.1145/633292.633461

  75. Nass C, Jonsson I-M, Harris H, Reaves B, Endo J, Brave S and Takayama L. Improving automotive safety by pairing driver emotion and car voice emotion. In: CHI ’05 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’05), 2005;1973–6. https://doi.org/10.1145/1056808.1057070.

  76. Shi Y, Yan X, Ma X, Lou Y and Cao N. Designing emotional expressions of conversational states for voice assistants: Modality and engagement. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), 2018;1–6. https://doi.org/10.1145/3170427.3188560.

  77. Kim S, Goh J, and Jun S. The use of voice input to induce human communication with banking chatbots. In: Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction (HRI Companion ’18), 2018;151–152. https://doi.org/10.1145/3173386.3176970.

  78. Shamekhi A, Liao QV, Wang D, Bellamy RKE and Erickson T. Face value? Exploring the effects of embodiment for a group facilitation agent. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18),2018;391:1–391:13. https://doi.org/10.1145/3173574.3173965

  79. Torre I, Goslin J, White L and Zanatto D. Trust in artificial voices: a “congruency effect” of first impressions and behavioral experience. In Proceedings of the 2018 Technology, Mind, and Society Conference (TechMindSociety ’18), 2018. Article No. 40. https://doi.org/10.1145/3183654.3183691.

  80. Yarosh S, Thompson S, Watson K, Chase A, Senthilkumar A, Yuan Y and Brush AJB. Children asking questions: Speech interface reformulations and personification preferences. In: Proceedings of the 17th ACM Conference on Interaction Design and Children (IDC ’18), 2018;300–12. https://doi.org/10.1145/3202185.3202207.

  81. Stucker BE, Wicker R. Direct digital manufacturing of integrated naval systems using ultrasonic consolidation, support material deposition and direct write technologies. UTAH STATE UNIV LOGAN; 2012.

    Book  Google Scholar 

  82. Kaplan A, Haenlein M. Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations, illustrations, and implications of artificial intelligence. Bus Horiz. 2019;62(1):15–25.

    Article  Google Scholar 

  83. Humphry J and Chesher C. Preparing for smart voice assistants: Cultural histories and media innovations. New media Soc. 2020;1461444820923679

  84. Moar JS. Cov id-19 and the Voice Assistants Market. Juniper Research. Retrieved Nov ember 25, 2021, f rom https://www.juniperresearch.com/blog/august-2021/covid-19-and-the-voice-assistants-market

  85. Vailshery LS. Topic: Smart speakers. Statista. Retrieved November 25, 2021, from https://www.statista.com/topics/4748/smart-speakers/#:~:text=As%20of%202019%20an%20estimated,increase%20to%20around%2075%20percent

  86. Pal D, Vanijja V, Zhang X, Thapliyal H. Exploring the antecedents of consumer electronics IoT devices purchase decision: a mixed methods study. IEEE Trans Consum Electron. 2021;67(4):305–18. https://doi.org/10.1109/TCE.2021.3115847.

    Article  Google Scholar 

  87. Pal D, Arpnikanondt C, Razzaque MA, Funilkul S. To trust or not-trust: privacy issues with voice assistants. IT Professional. 2020;22(5):46–53. https://doi.org/10.1109/MITP.2019.2958914.

    Article  Google Scholar 

Download references

Funding

This study was funded by The Asahi Glass Foundation.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Debajyoti Pal.

Ethics declarations

Conflict of Interest

The author declares that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dutsinma, F.L.I., Pal, D., Funilkul, S. et al. A Systematic Review of Voice Assistant Usability: An ISO 9241–11 Approach. SN COMPUT. SCI. 3, 267 (2022). https://doi.org/10.1007/s42979-022-01172-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-022-01172-3

Keywords