Abstract
We propose a multi-modal approach to manipulating smart home devices in a smart home environment simulated in virtual reality. Our multi-modal approach seeks to determine the user’s intent in the form of the user’s target smart home device and the desired action for that device to perform. We do this by examining information from two main modalities: spoken utterance and spatial information (such as gestures, positions, hand interactions, etc.). Our approach makes use of spoken utterance, spatial information, and additional information such as the device’s state to predict the user’s intent. Since the information contained in the user’s utterance and the spatial information can be disjoint or complementary to one another, we process the two sources of information in parallel using multiple machine learning models to determine intent. The results of these models are ensembled to produce our final prediction results. Aside from the proposed approach, we also discuss our prototype and discuss our initial findings.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
In the absence of a VR headset with eye-tracking features, we approximate this with a simple head gaze based on the headset position and rotation.
- 2.
We use VR controller’s position and rotation to simulate user pointing.
- 3.
The same process can be used on the user’s other body-related information, such as a hand direction.
- 4.
The speech recognition service in Windows 10 is used in Unity.
References
Kolve, E., et al.: AI2-THOR: an interactive 3D environment for visual AI. CoRR abs/1712.05474 (2017). http://arxiv.org/abs/1712.05474
Köse, A., Tepljakov, A., Petlenkov, E.: Dynamic predictive modeling approach of user behavior in virtual reality based application. In: 2019 27th Mediterranean Conference on Control and Automation (MED), pp. 57–62 (2019). https://doi.org/10.1109/MED.2019.8798521
Shridhar, M., et al.: ALFRED: a benchmark for interpreting grounded instructions for everyday tasks. CoRR abs/1912.01734 (2019). http://arxiv.org/abs/1912.01734
Xu, Z., Lympouridis, V.: Virtual control interface: discover and control IoT devices intuitively through AR glasses with multi-model interactions. In: 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 763–764. IEEE (2021)
Xu, Z., Yao, P., Lympouridis, V.: Virtual control interface: a system for exploring AR and IoT multimodal interactions within a simulated virtual environment. In: Stephanidis, C., Antona, M., Ntoa, S. (eds.) HCII 2021. CCIS, vol. 1420, pp. 345–352. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-78642-7_47
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Yao, P., Hou, Y., He, Y., Cheng, D., Hu, H., Zyda, M. (2022). Using Multi-modal Machine Learning for User Behavior Prediction in Simulated Smart Home for Extended Reality. In: Chen, J.Y.C., Fragomeni, G. (eds) Virtual, Augmented and Mixed Reality: Design and Development. HCII 2022. Lecture Notes in Computer Science, vol 13317. Springer, Cham. https://doi.org/10.1007/978-3-031-05939-1_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-05939-1_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-05938-4
Online ISBN: 978-3-031-05939-1
eBook Packages: Computer ScienceComputer Science (R0)