Skip to main content
SpringerLink
Log in

A Tertiary Study on Quality in Use Evaluation of Smart Environment Applications

  • Conference paper
  • First Online:
Research Challenges in Information Science (RCIS 2024)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 513))

Included in the following conference series:

Abstract

As the population grows older, the need for special assistance increases, and a modern alternative to mitigate the absence of face-to-face caregivers (which is expensive) is to take advantage of technological devices in so called smart environments, which can be an economical and practical solution. Guaranteeing the software quality of applications in these spaces before providing it to end users is essential, especially in situations involving senior citizens or people with motor disabilities. In order to investigate how the quality evaluation of smart environment applications has been performed, we carried out a tertiary study. From a total of 1,028 studies, 21 were carefully selected for analysis. The results confirmed that classical questionnaires and interviews are the techniques that are still used the most for evaluation, but that simulation appears as a new trend to that end.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The detailed process and replication packages are available on [1, 2].

  2. 2.

    The acronym QinU was not included, as we tried it on its own in the search string and no studies came up.

  3. 3.

    https://hal.science/.

References

  1. Angeloni, M.P.C., Duque Medina, R., Marçal de Oliveira, K., Grislin-Le Strugeon, E., Tirnauca, C.: Initial results for a tertiary study on quality in use evaluation of smart environment applications (2024). https://hal.science/hal-04506543

  2. Angeloni, M.P.C., Duque Medina, R., Marçal de Oliveira, K., Grislin-Le Strugeon, E., Tirnauca, C.: Research protocol and results for a tertiary study on quality in use evaluation of smart environment applications (2024). https://hal.science/hal-04507278

  3. Angeloni, M.P.C., Marçal de Oliveira, K., Grislin-Le Strugeon, E., Duque, R., Tirnauca, C.: A review on quality in use evaluation of smart environment applications: what’s next? In: Companion Proceedings of the 2023 ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pp. 9–15 (2023). https://doi.org/10.1145/3596454.3597177

  4. Apraiz, A., Lasa, G., Mazmela, M.: Evaluation of user experience in human-robot interaction: a systematic literature review. Int. J. Soc. Robotics 15(2), 187–210 (2023)

    Article  Google Scholar 

  5. Assila, A., Marçal de Oliveira, K., Ezzedine, H.: Standardized usability questionnaires: features and quality focus. Elec. J. of Comp. Sc. and Inf. Tech. 6(1), 15–31 (2016)

    Google Scholar 

  6. Bastardo, R., Martins, A.I., Pavão, J., Silva, A.G., Rocha, N.P.: Methodological quality of user-centered usability evaluation of ambient assisted living solutions: a systematic literature review. Int. J. Environment. Res. Public Health 18(21), 11507 (2021)

    Google Scholar 

  7. Bastardo, R., Pavão, J., Rocha, N.P.: A scoping review of the inquiry instruments being used to evaluate the usability of ambient assisted living solutions. In: Proceedings of the 15th INSTICC International Joint Conference on Biomedical Engineering Systems and Technology - vol. 5: HEALTHINF, pp. 320–327 (2021)

    Google Scholar 

  8. Carbo, J., Sanchez-Pi, N., Molina, J.M.: Agent-based simulation with netlogo to evaluate ambient intelligence scenarios. J. Simulation 12(1), 42–52 (2018). https://doi.org/10.1057/jos.2016.10

    Article  Google Scholar 

  9. Choukou, M.A., et al.: Evaluating the acceptance of ambient assisted living technology (AALT) in rehabilitation: a scoping review. Int. J. Med. Inform. 150, 104461 (2021)

    Google Scholar 

  10. Cioroaica, E., Buhnova, B., Kuhn, T.: Predictive simulation within the process of building trust. In: 2022 IEEE 19th International Conference on Software Architecture Companion, ICSA-C 2022, pp. 47–48 (2022). https://doi.org/10.1109/ICSA-C54293.2022.00017

  11. Colantonio, S., et al.: Are active and assisted living applications addressing the main acceptance concerns of their beneficiaries? Preliminary insights from a scoping review. In: Proceedings of the 15th ACM International Conference on Pervasive Tech. Related to Assistive Environments, pp. 414–421 (2022)

    Google Scholar 

  12. Costal, D., Farré, C., Franch, X., Quer, C.: How tertiary studies perform quality assessment of secondary studies in software engineering. In: XXIV Iberoamerican Conference on Software Engineering (ESELAW@CIbSE) (2021). https://doi.org/10.48550/arXiv.2110.03820

  13. Demiris, G., Hensel, B.K., Skubic, M., Rantz, M.: Senior residents’ perceived need of and preferences for “smart home” sensor technologies. Int. J. Technol. Assess. Health Care 24(1), 120–124 (2008). https://doi.org/10.1017/S0266462307080154

  14. Díaz-Oreiro, I., López, G., Quesada, L., Guerrero, L.A.: UX evaluation with standardized questionnaires in ubiquitous computing and ambient intelligence: a systematic literature review. Adv. Hum. Comput. Interact. 2021 (2021)

    Google Scholar 

  15. Erazo-Garzon, L., Erraez, J., Cedillo, P., Illescas-Peña, L.: Quality assessment approaches for ambient assisted living systems: a systematic review. In: Botto-Tobar, M., Zambrano Vizuete, M., Torres-Carrión, P., Montes León, S., Pizarro Vásquez, G., Durakovic, B. (eds.) ICAT 2019. CCIS, vol. 1193, pp. 421–439. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-42517-3_32

    Chapter  Google Scholar 

  16. Felber, N.A., Tian, Y.J.A., Pageau, F., Elger, B.S., Wangmo, T.: Mapping ethical issues in the use of smart home health technologies to care for older persons: a systematic review. BMC Med. Ethics 24(1) (2023). https://doi.org/10.1186/s12910-023-00898-w

  17. Filho, G.E.K., Guerino, G.C., Valentim, N.M.C.: A systematic mapping study on usability and user experience evaluation of multi-touch systems. In: Proceedings ACM 21st Brazilian Symposium on Human Factors in Computing Systems (2022)

    Google Scholar 

  18. Gambo, Y., Shakir, M.Z.: Review on self-regulated learning in smart learning environment. Smart Learn. Environ. 8(1), 12 (2021)

    Article  Google Scholar 

  19. Haghi Kashani, M., Madanipour, M., Nikravan, M., Asghari, P., Mahdipour, E.: A systematic review of IoT in healthcare: applications, techniques, and trends. J. Netw. Comput. Appl. 192, 103164 (2021)

    Article  Google Scholar 

  20. Holthe, T., Halvorsrud, L., Karterud, D., Hoel, K.A., Lund, A.: Usability and acceptability of technology for community-dwelling older adults with mild cognitive impairment and dementia: a systematic literature review. Clin. Interv. Aging 13, 863–886 (2018). https://doi.org/10.2147/CIA.S154717

    Article  Google Scholar 

  21. ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDT) s- Part 11 Guidance on usability (1998)

    Google Scholar 

  22. ISO/IEC: 14598-5:1998: information technology—software product evaluation—part 5: Process for evaluators (1998). https://www.iso.org/standard/24906.html

  23. ISO/IEC: 9126. software engineering - product quality (2001). https://www.iso.org/standard/22749.html

  24. ISO/IEC: 25012:2008. software engineering—software product quality requirements and evaluation (square)—data quality model (2008). https://www.iso.org/standard/35736.html

  25. ISO/IEC: 25000: Systems and software quality requirements and evaluation (square)—guide to square (2011). https://www.iso.org/standard/64764.html

  26. ISO/IEC: 25010:2011. systems and software engineering—systems and software quality requirements and evaluation (square)—system and software quality models (2011). https://www.iso.org/standard/35733.html

  27. ISO/IEC: 25022:2016. systems and software engineering—systems and software quality requirements and evaluation (square)—measurement of quality in use (2016). https://www.iso.org/fr/standard/35746.html

  28. Kitchenham, B., Charters, S.: Guidelines for performing structural literature reviews in software engineering (version 2.3). Tech. Report, Keele University and University of Durham (2007)

    Google Scholar 

  29. Lyzara, R., Purwandari, B., Zulfikar, M.F., Santoso, H.B., Solichah, I.: E-government usability evaluation: insights from a systematic literature review. In: Proceedings ACM 2nd International Conference on Software Engineering and Information Management, pp. 249–253 (2019)

    Google Scholar 

  30. Mao, C., Chang, D.: Review of cross-device interaction for facilitating digital transformation in smart home context: a user-centric perspective. Adv. Eng. Inform. 57, 102087 (2023). https://doi.org/10.1016/j.aei.2023.102087

    Article  Google Scholar 

  31. Maramba, I., Chatterjee, A., Newman, C.: Methods of usability testing in the development of ehealth applications: a scoping review. Int. J. Med. Inform. 126, 95–104 (2019). https://doi.org/10.1016/j.ijmedinf.2019.03.018

    Article  Google Scholar 

  32. Marengo, L.L., Barberato-Filho, S.: Involvement of human volunteers in the development and evaluation of wearable devices designed to improve medication adherence: a scoping review. Sensors 23(7), 3597 (2023)

    Article  Google Scholar 

  33. Maresova, P., et al.: Health-related ICT solutions of smart environments for elderly-systematic review. IEEE Access 8, 54574–54600 (2020)

    Article  Google Scholar 

  34. McCallum, C., Rooksby, J., Gray, C.: Evaluating the impact of physical activity apps and wearables: interdisciplinary review. JMIR mHealth uHealth 6, e58 (2018). https://doi.org/10.2196/mhealth.9054

  35. Motta, R.C., de Oliveira, K.M., Travassos, G.H.: A conceptual perspective on interoperability in context-aware software systems. Inf. Softw. Technol. 114, 231–257 (2019). https://doi.org/10.1016/j.infsof.2019.07.001

    Article  Google Scholar 

  36. Moyle, W., Murfield, J., Lion, K.: The effectiveness of smart home technologies to support the health outcomes of community-dwelling older adults living with dementia: a scoping review. Int. J. Med. Inform. 153 (2021). https://doi.org/10.1016/j.ijmedinf.2021.104513

  37. Nasr, V., Zahabi, M.: Usability evaluation methods of indoor navigation apps for people with disabilities: a scoping review. In: Proceedings of the 2022 IEEE International Conference on Human-Machine Systems, pp. 1–6 (2022)

    Google Scholar 

  38. Ntoa, S., Margetis, G., Antona, M., Stephanidis, C.: Uxami observer: an automated user experience evaluation tool for ambient intelligence environments. Adv. Intell. Syst. Comput. 868, 1350–1370 (2018). https://doi.org/10.1007/978-3-030-01054-6_94

    Article  Google Scholar 

  39. Paiva, J.O., Andrade, R.M., Carvalho, R.M.: Evaluation of non-functional requirements for IoT applications. In: International Conference on Enterprise Information Systems, ICEIS - Proceedings. vol. 2, p. 111–119 (2021)

    Google Scholar 

  40. Pavlovic, M., Kotsopoulos, S., Lim, Y., Penman, S., Colombo, S., Casalegno, F.: Determining a framework for the generation and evaluation of ambient intelligent agent system designs. Adv. Intell. Syst. Comput. 1069, 318–333 (2020). https://doi.org/10.1007/978-3-030-32520-6_26

    Article  Google Scholar 

  41. Petersen, K., Gencel, C.: Worldviews, research methods, and their relationship to validity in empirical software engineering research. In: 23rd International Workshop on Software Measurement and the 8th International Conference on Software Process and Product Measurement, pp. 81–89 (2013). https://doi.org/10.1109/IWSM-Mensura.2013.22

  42. Petticrew, M., Roberts, H.: Systematic Reviews in the Social Sciences: A Practical Guide. Blackwell Publishing Ltd, Oxford (2006). https://doi.org/10.1016/B978-0-12-374708-2.00014-0

  43. Saleemi, M., Anjum, M., Rehman, M.: Ubiquitous healthcare: a systematic mapping study. J. Ambient Intell. Humanized Comput. 14(5), 5021–5046 (2023)

    Google Scholar 

  44. Singh, V.K., Singh, P., Karmakar, M., Leta, J., Mayr, P.: The journal coverage of web of science, scopus and dimensions: a comparative analysis. Scientometrics 126(6), 5113–5142 (2021)

    Google Scholar 

  45. Tricco, A., Langlois, E., Straus, S., Alliance for Health Policy and Systems Research SCI, WHO Organization: Rapid reviews to strengthen health policy and systems: a practical guide. World Health Organization (2017)

    Google Scholar 

  46. Tónay, G., Pilissy, T., Tóth, A., Fazekas, G.: Methods to assess the effectiveness and acceptance of information and communication technology-based assistive technology for older adults: a scoping review. Int. J. Rehabil. Res. 46(2), 113–125 (2023). https://doi.org/10.1097/MRR.0000000000000571

    Article  Google Scholar 

Download references

Acknowledgments

The present research work is partially supported by the Hauts-de-France Region and the REUNICE (EUNICE Research) project funded by the European Union and the CNRS. C. Tirnauca’s work is partially supported by the project PID2022-139237NB-I00 financed by MICIU/AEI/10.13039/501100011033 and FEDER, UE.

Author information

Authors and Affiliations

  1. UPHF, CNRS, UMR 8201 - LAMIH, Valenciennes, France

    Maria Paula Corrêa Angeloni, Káthia Marçal de Oliveira & Emmanuelle Grislin-Le Strugeon

  2. Depart. de Matemáticas, Estadística y Computación, Universidad de Cantabria, Santander, Spain

    Rafael Duque & Cristina Tirnauca

  3. INSA Hauts-de-France, Valenciennes, France

    Emmanuelle Grislin-Le Strugeon

Authors
  1. Maria Paula Corrêa Angeloni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rafael Duque
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Káthia Marçal de Oliveira
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Emmanuelle Grislin-Le Strugeon
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Cristina Tirnauca
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Paula Corrêa Angeloni .

Editor information

Editors and Affiliations

  1. NOVA University Lisbon, Caparica, Portugal

    João Araújo

  2. University of Castilla La Mancha, Albacete, Albacete, Spain

    Jose Luis de la Vara

  3. University of Minho, Guimarães, Portugal

    Maribel Yasmina Santos

  4. Institut Mines-Télécom Business School, Evry, France

    Saïd Assar

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Angeloni, M.P.C., Duque, R., de Oliveira, K.M., Strugeon, E.GL., Tirnauca, C. (2024). A Tertiary Study on Quality in Use Evaluation of Smart Environment Applications. In: Araújo, J., de la Vara, J.L., Santos, M.Y., Assar, S. (eds) Research Challenges in Information Science. RCIS 2024. Lecture Notes in Business Information Processing, vol 513. Springer, Cham. https://doi.org/10.1007/978-3-031-59465-6_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-59465-6_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-59464-9

  • Online ISBN: 978-3-031-59465-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation