Skip to main content

Multimodal Assistance System for the Care of Individuals in Early Stages of Dependency Using Augmented Reality and Artificial Intelligence

  • Conference paper
  • First Online:
Extended Reality (XR Salento 2024)

Abstract

The present work explores the potential offered by the integration of various technologies such as augmented reality and artificial intelligence in the field of home-based care for dependent individuals. The proposed system focuses on providing assistance for everyday questions posed by individuals with certain levels of dependency or by new caregivers. This will start from the user’s interaction with a query about how to perform a specific action, accompanied by an image of their visual context and a manual detailing the routine, arrangement of the dependant’s belongings, and general information about the dependant. This will return a step-by-step guided response on what should be done, along with a list of objects to use that appear in the image of their environment. These objects will subsequently be highlighted in augmented reality for more comprehensive and understandable information.

This solution aims to improve and prolong the independence at home of individuals in the early stages of dependency or dementia, as well as to enhance the adaptation of caregivers to new home environments, thereby improving the quality of life for patients and the work of caregivers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Achiam, J., et al.: GPT-4 Technical report. arXiv preprint arXiv:2303.08774 (2023)

  2. Cheng, T., Song, L., Ge, Y., Liu, W., Wang, X., Shan, Y.: YOLO-world: real-time open-vocabulary object detection. arXiv preprint arXiv:2401.17270 (1 2024)

  3. Eloy, S., Dias, L., Ourique, L., Dias, M.S.: Home mobility hazards detected via object recognition in augmented reality, pp. 415–422 (2019)

    Google Scholar 

  4. Kasowski, J., Johnson, B.A., Neydavood, R., Akkaraju, A., Beyeler, M.: Furthering visual accessibility with extended reality (XR): a systematic. arXiv preprint arXiv:2109.04995 (2021). https://doi.org/10.1145/nnnnnnn.nnnnnnn

  5. Lewis, P., et al.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474 (2020). https://github.com/huggingface/transformers/blob/master/

  6. Park, Y.J., Ro, H., Lee, N.K., Han, T.D.: Deep-cARe: projection-based home care augmented reality system with deep learning for elderly. Appl. Sci. (Switzerland) 9(18) (2019). https://doi.org/10.3390/app9183897

  7. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016). https://doi.org/10.1109/CVPR.2016.91

  8. Vona, F., et al.: Combining virtual and augmented reality to improve daily autonomy for people with autism spectrum disorder. In: Proceedings of the 2022 International Conference on Advanced Visual Interfaces, pp. 1–3. Association for Computing Machinery (2022). https://doi.org/10.1145/3531073.3534499

  9. Wang, L.L., Jia, L.Q., Chu, F.Q., Li, M.X.: Design of home care system for rural elderly based on artificial intelligence. J. Phys. Conf. Ser. 1757 (2021). https://doi.org/10.1088/1742-6596/1757/1/012057

  10. Watson, J., Mac Aodha, O., Prisacariu, V., Brostow, G., Firman, M.: The temporal opportunist: self-supervised multi-frame monocular depth. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1164–1174 (2021). http://arxiv.org/abs/2104.14540

  11. Zeng, D., et al.: SHECS: a local smart hands-free elderly care support system on smart AR glasses with AI technology. In: 2021 IEEE International Symposium on Multimedia (ISM), pp. 66–74. Institute of Electrical and Electronics Engineers Inc. (2021). https://doi.org/10.1109/ISM52913.2021.00019

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Isabel Ferri-Molla .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ferri-Molla, I., Izquierdo-Domenech, J., Aliaga-Torro, C., Linares-Pellicer, J. (2024). Multimodal Assistance System for the Care of Individuals in Early Stages of Dependency Using Augmented Reality and Artificial Intelligence. In: De Paolis, L.T., Arpaia, P., Sacco, M. (eds) Extended Reality. XR Salento 2024. Lecture Notes in Computer Science, vol 15028. Springer, Cham. https://doi.org/10.1007/978-3-031-71704-8_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-71704-8_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-71703-1

  • Online ISBN: 978-3-031-71704-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics