Abstract
Voice assistants (VA) are an emerging technology that have become an essential tool of the twenty-first century. The VA ease of access and use has resulted in high usability curiosity in voice assistants. Usability is an essential aspect of any emerging technology, with every technology having a standardized usability measure. Despite the high acceptance rate on the use of VA, to the best of our knowledge, not many studies were carried out on voice assistants’ usability. We reviewed studies that used voice assistants for various tasks in this context. Our study highlighted the usability measures currently used for voice assistants. Moreover, our study also highlighted the independent variables used and their context of use. We employed the ISO 9241-11 framework as the measuring tool in our study. We highlighted voice assistant’s usability measures currently used; both within the ISO 9241-11 framework, as well as outside of it to provide a comprehensive view. A range of diverse independent variables are identified that were used to measure usability. We also specified that the independent variables still not used to measure some usability experience. We currently concluded what was carried out on voice assistant usability measurement and what research gaps were present. We also examined if the ISO 9241-11 framework can be used as a standard measurement tool for voice assistants.







Similar content being viewed by others
References
Hoy MB. Alexa, Siri, Cortana, and more: an introduction to v oice assistants. Med Ref Serv Quart. 2018;37(1):81–8.
Zwakman DS, Pal D, Arpnikanondt C. Usability evaluation of artificial intelligence-based voice assistants: the case of amazon Alexa. SN Comput Sci. 2021. https://doi.org/10.1007/s42979-020-00424-4.
Segi H, Takou R, Seiyama N, Takagi T, Uematsu Y, Saito H, Ozawa S. An automatic broadcast system for a weather report radio program. IEEE Trans Broadcast. 2013;59(3):548–55.
Noel S. Human computer interaction (HCI) based Smart Voice Email (Vmail) Application—Assistant f or Visually Impaired Users (VIU). In: 2020 third international conference on smart systems and inventive technology (ICSSIT) (pp 895–900). IEEE; 2020.
Sangle-Ferriere M, Voyer BG. Friend or foe? Chat as a double-edged sword to assist customers. J Serv Theory Pract. 2019;29:438–61.
Lugano G. Virtual assistants and self-driving cars. In: 2017 15th International Conf erence on ITS Telecommunications (ITST) (pp 1–5). IEEE; 2017.
Rybinski K, Kopciuszewska E. Will artif icial intelligence rev olutionise the student evaluation of teaching? A big data study of 1.6 million student reviews. Assessment & Evaluation in Higher Education; 2020, pp. 1–13
Tankovska H. Number of digital voice assistants in use worldwide 2019–2024 (in billions), 2020. https://www.statista.com/statistics/973815/worldwide-digital-voice-assistant-in-use/, (accessed 17 Nov 2021)
Pal D, Arpnikanondt C, Funilkul S, Chutimaskul W. The adoption analysis of voice-based smart IoT products. IEEE Internet Things J. 2020;7(11):10852–67.
Zwakman DS et al. Voice usability scale: measuring the user experience with voice assistants. In: 2020 IEEE International Symposium on Smart Electronic Sy stems (iSES)(Formerly iNiS). IEEE; 2020.
Coronado E, Deuff D, Carreno-Medrano P, Tian L, Kulić D, Sumartojo S, et al. Towards a modul ar and distributed end-user dev elopment framework f or human-robot interaction. IEEE Access. 2021;9:12675–92.
Maguire M. Development of a heuristic evaluation tool f or voice user interf aces. In: International conference on human-computer interaction. Cham: Springer; 2019. p. 212–25.
Fulfagar L, Gupta A, Mathur A, Shrivastava A. Development and evaluation of usability heuristics for voice user interfaces. In: International conference on research into design. Singapore: Springer; 2021. p. 375–85.
Nowacki C, Gordeeva A, Lizé AH. Improving the usability of voice user interfaces: a new set of ergonomic criteria. In: International conference on human-computer interaction. Cham: Springer; 2020. p. 117–33.
Pal D, Zhang X, Siyal S. Prohibitive factors to the acceptance of Internet of Things (IoT) technology in society: a smart-home context using a resistive modelling approach. Technol Soc. 2021;66: 101683.
Murad C, Munteanu C. “I don't know what you're talking about, HALexa" the case f or voice user interface guidelines. In: Proceedings of the 1st International Conference on Conversational User Interfaces,2019; pp. 1–3.
Budiu R, Laubheimer P. Intelligent assistants have poor usability: a user study of Alexa, Google assistant, and Siri. Nielsen Norman Group; 2018. Available online at https://www.nngroup.com/articles/intelligentassistant-usability/ (last accessed 4/12/2019).
Murphy CN, Yates J. The international organization f or standardization (ISO): global governance through voluntary consensus. Routledge; 2009.
ISO 9241-11. Ergonomic requirements for office work with visual display terminals (VDTs)—Part II guidance on usability; 1998.
Weichbroth P. Usability attributes revisited: a time-framed knowledge map. In 2018 Federated Conference on Computer Science and Information Systems (FedCSIS) (pp 1005–1008). IEEE; 2018.
Petrock V. Voice assistant and smart speaker users 2020. Insider Intelligence; 2020. Retrieved November 22, 2021, from https://www.emarketer.com/content/voice-assistant-and-smart-speaker-users-2020
Pinelle D, Wong N and Stach T. Heuristic evaluation f or games: usability principles for video game design. In: Proceedings of SIGCHI Conference on Human Factors in Computing Sy stems (2008); 2008, pp. 1453–1462. https://doi.org/10.1145/1357054.1357282.
Sutcliffe A, Gault B. Heuristic evaluation of virtual reality applications. Interact Comput 16. 2004;4:831–49. https://doi.org/10.1016/j.intcom.2004.05.00.
Sharif K, Tenbergen B. Smart home voice assistants: a literature survey of user privacy and security vulnerabilities. Complex Syst Inform Model Quart. 2020;24:15–30.
de Barcelos Silva A, Gomes MM, da Costa CA, da Rosa Righi R, Barbosa JLV, Pessin G, et al. Intelligent personal assistants: a systematic literature review. Expert Syst Appl. 2020;147: 113193.
Gubareva R and Lopes RP. Virtual assistants for learning: a systematic literature review. In: CSEDU (1); 2020, pp. 97–103.
Bérubé C, Schachner T, Keller R, Fleisch E, Wangenheim F, Barata F, Kowatsch T. Voice-based conversational agents for the prevention and management of chronic and mental health conditions: systematic literature review. J Med Internet Res. 2021;23(3): e25933.
Chi OH, Gursoy D and Chi CG. Tourists’ attitudes toward the use of artificially intelligent (AI) devices in tourism service delivery: moderating role of service value seeking. J Travel Res. 2020; 0047287520971054.
Kim S. Exploring how older adults use a smart speaker-based voice assistant in their first interactions: qualitative study. JMIR Mhealth Uhealth. 2021;9(1): e20427.
Coursaris CK, Kim DJ. A meta-analytical review of empirical mobile usability studies. J Usability Stud. 2011;6(3):117–71.
Goh ASY, Wong LL, Yap KYL. Evaluation of COVID-19 information provided by digital voice assistants. Int J Digital Health. 2021;1(1):3.
Rapp A, Curti L, Boldi A. The human side of human-chatbot interaction: a systematic literature review of ten years of research on text-based chatbots. Int J Hum-Comput Stud. 2021;151: 102630.
Seaborn K, Miyake NP, Pennefather P, Otake-Matsuura M. Voice in human-agent interaction: a survey. ACM Comput Surv (CSUR). 2021;54(4):1–43.
Castro JW, Ren R, Acuña ST and Lara JD. Usability of chatbots: a systematic mapping study; 2019.
Bhirud N, Tataale S, Randive S, Nahar S. A literature review on chatbots in healthcare domain. Int J Sci Technol Res. 2019;8(7):225–31.
Ahmad NA, Che MH, Zainal A, Abd Rauf MF, Adnan Z. Review of chatbots design techniques. Int J Comput Appl. 2018;181(8):7–10.
Gentner T, Neitzel T, Schulze J and Buettner R. A Systematic literature review of medical chatbot research from a behavior change perspective. In: 2020 IEEE 44th annual computers, software, and applications conference (COMPSAC). IEEE; 2020, pp. 735–740.
Cunningham-Nelson S, Boles W, Trouton L and Margerison E. A review of chatbots in education: practical steps forward. In: 30th Annual Conference for the Australasian Association for Engineering Education (AAEE 2019): Educators Becoming Agents of Change: Innovate, Integrate, Motivate. Engineers Australia; 2019, pp. 299–306.
Van Pinxteren MM, Pluymaekers M, Lemmink JG. Human-like communication in conversational agents: a literature review and research agenda. J Serv Manag. 2020;31:203–25.
Weichbroth P. Usability attributes revisited: a time-framed knowledge map. In: 2018 Federated Conference on Computer Science and Information Systems (FedCSIS). IEEE; 2018, pp. 1005–1008.
Bevan N, Carter J, Earthy J, Geis T, Harker S. New ISO standards for usability, usability reports and usability measures. In: International conference on human-computer interaction. Cham: Springer; 2016. p. 268–78.
Moumane K, Idri A, Abran A. Usability evaluation of mobile applications using ISO 9241 and ISO 25062 standards. Springerplus. 2016;5(1):1–15.
Yahya H, Razali R. A usability-based framework for electronic government systems development. ARPN J Eng Appl Sci. 2015;10(20):9414–23.
Alva ME, Ch THS, López B. Comparison of methods and existing tools for the measurement of usability in the web. In: International conference on web engineering. Berlin: Springer; 2003. p. 386–9.
He X, Persson H, Östman A. Geoportal usability evaluation. Int J Spatial Data Infrastruct Res. 2012;7:88–106.
Dietlein CS, Bock OL. Development of a usability scale based on the three ISO 9241–11 categories “effectiveness, ”efficacy” and “satisfaction”: a technical note. Accred Qual Assur. 2019;24(3):181–9.
Nik Ahmad NA and Hasni NS. ISO 9241–11 and SUS measurement for usability assessment of dropshipping sales management application. In: 2021 10th International Conference on Software and Computer Applications. 2021; pp. 70–74.
Kitchenham B. Procedures f or perf orming systematic reviews. Keele Univ. 2004;33(2004):1–26.
Seaborn K, Miyake NP, Pennefather P, Otake-Matsuura M. Voice in human–agent interaction: a survey. ACM Comput Surv (CSUR). 2021;54(4):1–43.
Al-Qaysi N, Mohamad-Nordin N, Al-Emran M. Employing the technology acceptance model in social media: a systematic review. Educ Inf Technol. 2020;25(6):4961–5002.
Kitchenham B and Charters S. Guidelines f or performing systematic literature reviews in software engineering; 2007.
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD CD, The PRISMA, et al. statement: an updated guideline f or reporting sy stematic rev iews. BMJ. 2020;2021(372): n71. https://doi.org/10.1136/bmj.n71.
Martelaro N, Teevan J and Iqbal ST. An exploration of speech-based productivity support in the car. In: Proceedings of the 2019 CHI conference on human factors in computing systems. 2019; pp. 1–12
Jeong Y, Lee J and Kang Y. Exploring effects of conversational fillers on user perception of conversational agents. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19). 2019; 1–6. https://doi.org/10.1145/3290607.3312913.
Yu Q, Nguyen T, Prakkamakul S and Salehi N. “I almost fell in love with a machine”: speaking with computers affects self-disclosure. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19). 2019; pp. 1–6. https://doi.org/10.1145/3290607.3312918
Kiesel J, Bahrami A, Stein B, Anand A, and Hagen M. Clarifying false memories in voice-based search. In: Proceedings of the 2019 Conference on Human Information Interaction and Retrieval (CHIIR ’19). 2019; 331–335. https://doi.org/10.1145/3295750.3298961.
Kontogiorgos D, Pereira A, Andersson O, Koivisto M, Rabal EG, Vartiainen V and Gustafson J. The effects of anthropomorphism and non-verbal social behavior in virtual assistants. In: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents (IVA ’19). 2019; 133–140. https://doi.org/10.1145/3308532.3329466
Hoegen R, Aneja D, McDuff D and Czerwinski M. An end-to-end conversational style matching agent. In: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents (IVA ’19). 2019; 111–118. https://doi.org/10.1145/3308532.3329473
Luo Y, Lee B and Choe EK. TandemTrack: shaping consistent exercise experience by complementing a mobile app with a smart speaker. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). 2020; 1–13. https://doi.org/10.1145/3313831.3376616
Doyle PR, Edwards J, Dumbleton O, Clark L and Cowan BR. Mapping perceptions of humanness in intelligent personal assistant interaction. In: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ’19). 2019. https://doi.org/10.1145/3338286.3340116.
Jaber R, McMillan D, Belenguer JS and Brown B. Patterns of gaze in speech agent interaction. In: Proceedings of the 1st International Conference on Conversational User Interfaces - CUI ’19 (the 1st International Conference). 2019; 1–10. https://doi.org/10.1145/3342775.3342791.
Bortoli M, Furini M, Mirri S, Montangero M and Prandi C. Conversational interfaces for a smart campus: a case study. In: Proceedings of the international conference on advanced visual interfaces (AVI ’20). 2020. https://doi.org/10.1145/3399715.3399914.
Wu Y, Edwards Y, Cooney O, Bleakley A, Doyle PR, Clark L, Rough D and Cowan BR. Mental workload and language production in non-native speaker IPA interaction. In: Proceedings of the 2nd Conference on Conversational User Interfaces (CUI ’20). 2020. https://doi.org/10.1145/3405755.3406118
Brüggemeier B, Breiter M, Kurz M and Schiwy J. User experience of Alexa when controlling music: comparison of face and construct validity of four questionnaires. In: Proceedings of the 2nd conference on conversational user interfaces (CUI ’20). 2020. https://doi.org/10.1145/3405755.3406122
Machine body language: expressing a smart speaker’s activity with intelligible physical motion. 57
Bartneck C, Kulić D, Croft E, Zoghbi S. Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int J Soc Robot. 2009;1(1):71–81. https://doi.org/10.1007/s12369-008-0001-3A.
Braun M, Mainz A, Chadowitz R, Pfleging B and Alt F. At your service: designing voice assistant personalities to improve automotive user interfaces. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 2019;40:1–40:11. https://doi.org/10.1145/3290605.3300270
Burbach L, Halbach P, Plettenberg N, Nakayama J, Ziefle M and Valdez AC. “Hey, Siri”, “Ok, Google”, “Alexa”. Acceptance-relevant factors of virtual voice-assistants. In 2019: IEEE International Professional Communication Conference (ProComm) (ProComm ’19), 2019;101–111. https://doi.org/10.1109/ProComm.2019.00025.
Pal D, Arpnikanondt C, Funilkul S, and Varadarajan V. User experience with smart voice assistants: The accent perspective. In 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT ’19), 2019;1–6. https://doi.org/10.1109/ICCCNT45670.2019.8944754.
Chin H, Molefi L, and Yi Y. Empathy is all you need: How a conversational agent should respond to verbal abuse. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), 2020; 1–13. https://doi.org/10.1145/3313831.3376461.
Crowell CR, Villanoy M, Scheutzz M and Schermerhornz P. Gendered voice and robot entities: Perceptions and reactions of male and female subjects. In: Proceedings of the 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2009), 2009; 3735–3741. https://doi.org/10.1109/IROS.2009.5354204
Lee S, Cho M and Lee S. What if conversational agents became invisible? Comparing users’ mental models according to physical entity of AI speaker. In: Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2020; 4, 3. https://doi.org/10.1145/3411840
Dahlbäck N, Wang QY, Nass C and Alwin J. Similarity is more important than expertise: Accent effects in speech interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07), 2007; 1553–1556. https://doi.org/10.1145/1240624.1240859
Lee EJ, Nass C, and Brave S. Can computer-generated speech have gender?: An experimental test of gender stereotype. In Proceedings of the CHI’00 Extended Abstracts on Human factors in Computing Systems (CHI EA ’00), 2000; 289–290. https://doi.org/10.1145/633292.633461
Nass C, Jonsson I-M, Harris H, Reaves B, Endo J, Brave S and Takayama L. Improving automotive safety by pairing driver emotion and car voice emotion. In: CHI ’05 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’05), 2005;1973–6. https://doi.org/10.1145/1056808.1057070.
Shi Y, Yan X, Ma X, Lou Y and Cao N. Designing emotional expressions of conversational states for voice assistants: Modality and engagement. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), 2018;1–6. https://doi.org/10.1145/3170427.3188560.
Kim S, Goh J, and Jun S. The use of voice input to induce human communication with banking chatbots. In: Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction (HRI Companion ’18), 2018;151–152. https://doi.org/10.1145/3173386.3176970.
Shamekhi A, Liao QV, Wang D, Bellamy RKE and Erickson T. Face value? Exploring the effects of embodiment for a group facilitation agent. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18),2018;391:1–391:13. https://doi.org/10.1145/3173574.3173965
Torre I, Goslin J, White L and Zanatto D. Trust in artificial voices: a “congruency effect” of first impressions and behavioral experience. In Proceedings of the 2018 Technology, Mind, and Society Conference (TechMindSociety ’18), 2018. Article No. 40. https://doi.org/10.1145/3183654.3183691.
Yarosh S, Thompson S, Watson K, Chase A, Senthilkumar A, Yuan Y and Brush AJB. Children asking questions: Speech interface reformulations and personification preferences. In: Proceedings of the 17th ACM Conference on Interaction Design and Children (IDC ’18), 2018;300–12. https://doi.org/10.1145/3202185.3202207.
Stucker BE, Wicker R. Direct digital manufacturing of integrated naval systems using ultrasonic consolidation, support material deposition and direct write technologies. UTAH STATE UNIV LOGAN; 2012.
Kaplan A, Haenlein M. Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations, illustrations, and implications of artificial intelligence. Bus Horiz. 2019;62(1):15–25.
Humphry J and Chesher C. Preparing for smart voice assistants: Cultural histories and media innovations. New media Soc. 2020;1461444820923679
Moar JS. Cov id-19 and the Voice Assistants Market. Juniper Research. Retrieved Nov ember 25, 2021, f rom https://www.juniperresearch.com/blog/august-2021/covid-19-and-the-voice-assistants-market
Vailshery LS. Topic: Smart speakers. Statista. Retrieved November 25, 2021, from https://www.statista.com/topics/4748/smart-speakers/#:~:text=As%20of%202019%20an%20estimated,increase%20to%20around%2075%20percent
Pal D, Vanijja V, Zhang X, Thapliyal H. Exploring the antecedents of consumer electronics IoT devices purchase decision: a mixed methods study. IEEE Trans Consum Electron. 2021;67(4):305–18. https://doi.org/10.1109/TCE.2021.3115847.
Pal D, Arpnikanondt C, Razzaque MA, Funilkul S. To trust or not-trust: privacy issues with voice assistants. IT Professional. 2020;22(5):46–53. https://doi.org/10.1109/MITP.2019.2958914.
Funding
This study was funded by The Asahi Glass Foundation.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The author declares that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Dutsinma, F.L.I., Pal, D., Funilkul, S. et al. A Systematic Review of Voice Assistant Usability: An ISO 9241–11 Approach. SN COMPUT. SCI. 3, 267 (2022). https://doi.org/10.1007/s42979-022-01172-3
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s42979-022-01172-3