skip to main content
10.1145/3485279.3485288acmconferencesArticle/Chapter ViewAbstractPublication PagessuiConference Proceedingsconference-collections
research-article
Public Access

Trade-offs in Augmented Reality User Interfaces for Controlling a Smart Environment

Published: 09 November 2021 Publication History

Abstract

Smart devices and Internet of Things (IoT) technologies are replacing or being incorporated into traditional devices at a growing pace. The use of digital interfaces to interact with these devices has become a common occurrence in homes, work spaces, and various industries around the world. The most common interfaces for these connected devices focus on mobile apps or voice control via intelligent virtual assistants. However, with augmented reality (AR) becoming more popular and accessible among consumers, there are new opportunities for spatial user interfaces to seamlessly bridge the gap between digital and physical affordances.
In this paper, we present a human-subject study evaluating and comparing four user interfaces for smart connected environments: gaze input, hand gestures, voice input, and a mobile app. We assessed participants’ user experience, usability, task load, completion time, and preferences. Our results show multiple trade-offs between these interfaces across these measures. In particular, we found that gaze input shows great potential for future use cases, while both gaze input and hand gestures suffer from limited familiarity among users, compared to voice input and mobile apps.

References

[1]
N. Abdi, M. Ramokapane, and J. M. Such. 2019. More than Smart Speakers: Security and Privacy Perceptions of Smart Home Personal Assistants. In Fifteenth Symposium on Usable Privacy and Security (SOUPS 2019). USENIX Association, United States.
[2]
G. Aloi, G. Caliciuri, G. Fortino, R. Gravina, P. Pace, W. Russo, and C. Savaglio. 2017. Enabling IoT interoperability through opportunistic smartphone-based mobile gateways. Journal of Network and Computer Applications 81 (2017), 74–84. https://doi.org/10.1016/j.jnca.2016.10.013
[3]
B. Bittner, I. Aslan, C. T. Dang, and E. André. 2019. Of Smarthomes, IoT Plants, and Implicit Interaction Design. In Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction(Tempe, Arizona, USA) (TEI ’19). Association for Computing Machinery, New York, NY, USA, 145–154. https://doi.org/10.1145/3294109.3295618
[4]
A. Blackler, V. Popovic, and D. Mahar. 2010. Investigating users’ intuitive interaction with complex artefacts. Applied Ergonomics 41, 1 (2010), 72–92. https://doi.org/10.1016/j.apergo.2009.04.010
[5]
J. Brooke. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.
[6]
C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard. 2016. Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age. IEEE Transactions on Robotics 32, 6 (2016), 1309–1332.
[7]
Y. Chen, B. Zhang, C. Tuna, Y. Li, E. A Lee, and B. Hartmann. 2013. A context menu for the real world: Controlling physical appliances through head-worn infrared targeting. Technical Report. UC Berkeley Department of Electrical Engineering and Computer Sciences.
[8]
A. El Ali, J. Kildal, and V. Lantz. 2012. Fishing or a Z? Investigating the Effects of Error on Mimetic and Alphabet Device-Based Gesture Interaction(ICMI ’12). Association for Computing Machinery, New York, NY, USA, 93–100. https://doi.org/10.1145/2388676.2388701
[9]
M. A. El Khaddar and M. Boulmalf. 2017. Smartphone: the ultimate IoT and IoE device. Smartphones from an applied research perspective 137 (2017).
[10]
A. Erickson, N. Norouzi, K. Kim, J. J. LaViola, G. Bruder, and G. F. Welch. 2020. Effects of depth information on visual target identification task performance in shared gaze environments. IEEE transactions on visualization and computer graphics 26, 5(2020), 1934–1944.
[11]
A. Erickson, N. Norouzi, K. Kim, R. Schubert, J. Jules, J. J. LaViola, G. Bruder, and G. F. Welch. 2020. Sharing gaze rays for visual target identification tasks in collaborative augmented reality. Journal on Multimodal User Interfaces 14, 4 (2020), 353–371.
[12]
J. Foehr and C. C. Germelmann. 2020. Alexa, Can I Trust You? Exploring Consumer Paths to Trust in Smart Voice-Interaction Technologies. Journal of the Association for Consumer Research 5, 2 (2020), 181–205. https://doi.org/10.1086/707731 arXiv:https://doi.org/10.1086/707731
[13]
L. P. Fu, J. Landay, M. Nebeling, Y. Xu, and C. Zhao. 2018. Redefining natural user interface. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. 1–3.
[14]
S. G. Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908.
[15]
International Telecommunication Union 2012. Recommendation ITU-T Y.2060: Overview of the Internet of things. International Telecommunication Union.
[16]
S. Jang, W. Stuerzlinger, S. Ambike, and K. Ramani. 2017. Modeling Cumulative Arm Fatigue in Mid-Air Interaction Based on Perceived Exertion and Kinetics of Arm Motion. 3328–3339.
[17]
J. C. Kim, T. H. Laine, and C. Åhlund. 2021. Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review. Applied Sciences 11, 4 (2021). https://doi.org/10.3390/app11041738
[18]
K. Kim, M. Billinghurst, G. Bruder, H. Duh, and G. F. Welch. 2018. Revisiting Trends in Augmented Reality Research: A Review of the 2nd Decade of ISMAR (2008–2017). IEEE Transactions on Visualization and Computer Graphics (TVCG) 24, 11(2018), 2947–2962.
[19]
K. Kim, N. Norouzi, D. Jo, G. Bruder, and G. Welch. 2022. The Augmented Reality Internet of Things: Opportunities of Embodied Interactions in Transreality. In Handbook of Augmented Reality, A. Y. C. Nee and S. K. Ong (Eds.). Springer, Switzerland.
[20]
B. Kinsella. 2018. Over Half of Smartphone Owners Use Voice Assistants, Siri Leads the Pack. https://voicebot.ai/2018/04/03/over-half-of-smartphone-owners-use-voice-assistants-siri-leads-the-pack/
[21]
A. S. Lago, J. P. Dias, and H. S. Ferreira. 2020. Conversational Interface for Managing Non-trivial Internet-of-Things Systems. In Computational Science – ICCS 2020, V. V. Krzhizhanovskaya, G. Závodszky, M. H. Lees, J. J. Dongarra, P. Sloot, S. Brissos, and J. Teixeira (Eds.). Springer International Publishing, Cham, 384–397.
[22]
S. RH Langton. 2000. The mutual influence of gaze and head orientation in the analysis of social attention direction. The Quarterly Journal of Experimental Psychology: Section A 53, 3(2000), 825–845.
[23]
J. Lau, B. Zimmerman, and F. Schaub. 2018. Alexa, Are You Listening? Privacy Perceptions, Concerns and Privacy-Seeking Behaviors with Smart Speakers. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 102 (Nov. 2018), 31 pages. https://doi.org/10.1145/3274371
[24]
D. Ledo, S. Greenberg, N. Marquardt, and S. Boring. 2015. Proxemic-Aware Controls: Designing Remote Controls for Ubiquitous Computing Ecologies. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (Copenhagen, Denmark) (MobileHCI ’15). Association for Computing Machinery, New York, NY, USA, 187–198. https://doi.org/10.1145/2785830.2785871
[25]
G. López, L. Quesada, and L. A. Guerrero. 2017. Alexa vs. Siri vs. Cortana vs. Google Assistant: a comparison of speech-based natural user interfaces. In International Conference on Applied Human Factors and Ergonomics. Springer, 241–250.
[26]
P. Lubos, G. Bruder, O. Ariza, and F. Steinicke. 2016. Touching the Sphere: Leveraging Joint-Centered Kinespheres for Spatial User Interaction. In Proceedings of the ACM Symposium on Spatial User Interaction (SUI). 13–22.
[27]
P. Lubos, G. Bruder, and F. Steinicke. 2015. Influence of Comfort on 3D Selection Task Performance in Immersive Desktop Setups. Journal of Virtual Reality and Broadcasting (JVRB) 12, 2 (2015), 1–14.
[28]
M. Mekni and A. Lemieux. 2014. Augmented reality: Applications, challenges and future trends. Applied Computational Science 20 (2014), 205–214.
[29]
D. Merrill and P. Maes. 2007. Augmenting Looking, Pointing and Reaching Gestures to Enhance the Searching and Browsing of Physical Objects. In Pervasive Computing, A. LaMarca, M. Langheinrich, and K. N. Truong(Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 1–18.
[30]
P. Middleton, T. Tsai, M. Yamaji, A. Gupta, and A. Reub. 2017. Forecast: Internet of Things — Endpoints and Associated Services, Worldwide, 2017. (December 21 2017).
[31]
J. Miranda, N. Mäkitalo, J. Garcia-Alonso, J. Berrocal, T. Mikkonen, C. Canal, and J. M. Murillo. 2015. From the Internet of Things to the Internet of People. IEEE Internet Computing 19, 2 (2015), 40–47. https://doi.org/10.1109/MIC.2015.24
[32]
C. Mircioiu and J. Atkinson. 2017. A comparison of parametric and non-parametric methods applied to a Likert scale. Pharmacy 5, 2 (2017), 26.
[33]
G. Norman. 2010. Likert scales, levels of measurement and the “laws” of statistics. Advances in health sciences education 15, 5 (2010), 625–632.
[34]
N. Norouzi, G. Bruder, B. Belna, S. Mutter, D. Turgut, and G. Welch. 2019. A Systematic Review of the Convergence of Augmented Reality, Intelligent Virtual Agents, and the Internet of Things. Springer International Publishing, Cham, 1–24. https://doi.org/10.1007/978-3-030-04110-6_1
[35]
N. Norouzi, A. Erickson, K. Kim, R. Schubert, J. J. LaViola, G. Bruder, and G. Welch. 2019. Effects of shared gaze parameters on visual target identification task performance in augmented reality. In Symposium on Spatial User Interaction. 1–11.
[36]
J. Outlaw and S. Persky. 2019. Industry review boards are needed to protect VR user privacy. In World Economic Forum, August, Vol. 29.
[37]
D. Pal, C. Arpnikanondt, S. Funilkul, and V. Varadarajan. 2019. User Experience with Smart Voice Assistants: The Accent Perspective. In 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT). 1–6. https://doi.org/10.1109/ICCCNT45670.2019.8944754
[38]
K. K. Patel and S. M. Patel. 2016. Internet of Things-IOT: Definition, Characteristics, Architecture, Enabling Technologies, Application & Future Challenges. International Journal of Engineering Science 6122 (2016).
[39]
S. R. Prathibha, A. Hongal, and M. P. Jyothi. 2017. IOT Based Monitoring System in Smart Agriculture. In 2017 International Conference on Recent Advances in Electronics and Communication Technology (ICRAECT). 81–84. https://doi.org/10.1109/ICRAECT.2017.52
[40]
Z. Rashid, J. Melià-Seguí, R. Pous, and E. Peig. 2017. Using Augmented Reality and Internet of Things to improve accessibility of people with motor disabilities in the context of Smart Cities. Future Generation Computer Systems 76 (2017), 248–261. https://doi.org/10.1016/j.future.2016.11.030
[41]
S. S. Rautaray and A. Agrawal. 2015. Vision based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review 43, 1 (Jan. 2015), 1–54. https://doi.org/10.1007/s10462-012-9356-9
[42]
M. Schrepp, A. Hinderks, and J. Thomaschewski. 2017. Design and Evaluation of a Short Version of the User Experience Questionnaire (UEQ-S). International Journal of Interactive Multimedia and Artificial Intelligence 4, 6 (2017), 103.
[43]
K. A Shatilov, D. Chatzopoulos, L. Lee, and P. Hui. 2019. Emerging natural user interfaces in mobile computing: A bottoms-up survey. arXiv preprint arXiv:1911.04794(2019).
[44]
Y. Sun, A. Armengol-Urpi, S. N. Reddy Kantareddy, J. Siegel, and S. Sarma. 2019. MagicHand: Interact with IoT Devices in Augmented Reality Environment. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 1738–1743. https://doi.org/10.1109/VR.2019.8798053
[45]
C. Swindells, K. M. Inkpen, J. C. Dill, and M. Tory. 2002. That One There! Pointing to Establish Device Identity. In Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology (Paris, France) (UIST ’02). Association for Computing Machinery, New York, NY, USA, 151–160. https://doi.org/10.1145/571985.572007
[46]
G. Welch, G. Bruder, P. Squire, and R. Schubert. 2019. Anticipating Widespread Augmented Reality: Insights from the 2018 AR Visioning Workshop. Technical Report. University of Central Florida and Office of Naval Research.
[47]
P. Wellner, W. Mackay, and R. Gold. 1993. Computer-augmented environments: back to the real world. 36, 7 (Jul 1993), 24+. https://link.gale.com/apps/doc/A14042355/CDB?u=orla57816&sid=bookmark-CDB&xid=c4452318 Article.
[48]
R. Xiao, G. Laput, Y. Zhang, and C. Harrison. 2017. Deus EM Machina: On-Touch Contextual Functionality for Smart IoT Appliances. Association for Computing Machinery, New York, NY, USA, 4000–4008. https://doi.org/10.1145/3025453.3025828
[49]
H. Yang, W. Lee, and H. Lee. 2018. IoT Smart Home Adoption: The Importance of Proper Level Automation. Journal of Sensors 2018 (22 May 2018), 11. https://doi.org/10.1155/2018/6464036
[50]
Z. Yang, Q. Zhou, L. Lei, K. Zheng, and W. Xiang. 2016. An IoT-cloud Based Wearable ECG Monitoring System for Smart Healthcare. Journal of Medical Systems 40, 12 (29 Oct 2016), 286. https://doi.org/10.1007/s10916-016-0644-9
[51]
I. Yaqoob, E. Ahmed, I. A. T. Hashem, A. I. A. Ahmed, A. Gani, M. Imran, and M. Guizani. 2017. Internet of Things Architecture: Recent Advances, Taxonomy, Requirements, and Open Challenges. IEEE Wireless Communications 24, 3 (2017), 10–16. https://doi.org/10.1109/MWC.2017.1600421

Cited By

View all
  • (2025)An Updated Systematic Mapping Study on Usability and User Experience Evaluation of Touchable Holographic SolutionsJournal on Interactive Systems10.5753/jis.2025.469416:1(172-198)Online publication date: 1-Jan-2025
  • (2024)Prototype Biofeedback Augmented Reality Glasses for Workload Detection with Machine Learning: A Preliminary StudyProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/1071181324126194868:1(1188-1193)Online publication date: 29-Aug-2024
  • (2024)Understanding affordances in XR interactions through a design spaceProceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems10.1145/3702038.3702112(1-14)Online publication date: 7-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SUI '21: Proceedings of the 2021 ACM Symposium on Spatial User Interaction
November 2021
206 pages
ISBN:9781450390910
DOI:10.1145/3485279
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 November 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. 3D User Interfaces
  2. Augmented Reality
  3. Connected Devices
  4. Internet of Things
  5. Smart Home
  6. Usability

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

SUI '21
SUI '21: Symposium on Spatial User Interaction
November 9 - 10, 2021
Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 86 of 279 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)218
  • Downloads (Last 6 weeks)26
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)An Updated Systematic Mapping Study on Usability and User Experience Evaluation of Touchable Holographic SolutionsJournal on Interactive Systems10.5753/jis.2025.469416:1(172-198)Online publication date: 1-Jan-2025
  • (2024)Prototype Biofeedback Augmented Reality Glasses for Workload Detection with Machine Learning: A Preliminary StudyProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/1071181324126194868:1(1188-1193)Online publication date: 29-Aug-2024
  • (2024)Understanding affordances in XR interactions through a design spaceProceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems10.1145/3702038.3702112(1-14)Online publication date: 7-Oct-2024
  • (2022)The Power of Augmented Reality for Smart Environments: An Explorative Analysis of the Business Process ManagementMachine Learning for Smart Environments/Cities10.1007/978-3-030-97516-6_4(73-91)Online publication date: 6-Apr-2022

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media