Skip to main content
Springer Nature Link
Log in

A Framework for Managing Quality Requirements for Machine Learning-Based Software Systems

  • Conference paper
  • First Online:
Quality of Information and Communications Technology (QUATIC 2024)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2178))

  • 524 Accesses

Abstract

Systems containing Machine Learning (ML) are becoming common, and the tasks performed by such systems must meet certain quality thresholds, e.g., desired levels of transparency, safety, and trust. Recent research has identified challenges in defining and measuring the achievement of non-functional requirements (NFRs) for ML systems. Managing NFRs is particularly challenging due to the differing nature and definitions of NFRs for ML systems including non-deterministic behavior, the need to scope over different system components (e.g., data, models, and code), and difficulty in establishing new measurements (e.g., measuring explainability). To address these challenges, we propose a framework for identifying, prioritizing, specifying, and measuring attainment of NFRs for ML systems. We present a preliminary evaluation of the framework via an interview study with practitioners. The framework captures a first step towards enabling practitioners to systematically deliver high-quality ML systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    We will use the term “ML systems” to describe such systems.

  2. 2.

    When discussing models, the term “performance” often is a synonym for accuracy, while for code, “performance” generally refers to execution time.

  3. 3.

    The interview guide and presentation to the interviewees can be found at https://doi.org/10.7910/DVN/QUPD8D. Due to the sensitive nature of the collected practitioner data from the interviews, we can not share the full interview transcripts.

References

  1. Abbasi, W., Mori, P., Saracino, A., Frascolla, V.: Privacy vs accuracy trade-off in privacy aware face recognition in smart systems. In: 2022 IEEE Symposium on Computers and Communications (ISCC), pp. 1–8. IEEE (2022)

    Google Scholar 

  2. Achimugu, P., Selamat, A., Ibrahim, R., Mahrin, M.N.: A systematic literature review of software requirements prioritization research. Inf. Softw. Technol. 56(6), 568–585 (2014)

    Article  Google Scholar 

  3. Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., Grundy, J.: Requirements engineering for artificial intelligence systems: a systematic mapping study. Inf. Softw. Technol. 158, 107176 (2023)

    Article  Google Scholar 

  4. Ameller, D., et al.: Dealing with non-functional requirements in model-driven development: a survey. IEEE Trans. Softw. Eng. (2019)

    Google Scholar 

  5. Aydemir, F.B., Dalpiaz, F.: A roadmap for ethics-aware software engineering. In: Proceedings of the International Workshop on Software Fairness, pp. 15–21 (2018)

    Google Scholar 

  6. Bajraktari, E., Krause, T., Kücherer, C.: Documentation of non-functional requirements for systems with machine learning components. In: 2024 REFSQ Workshops (2024)

    Google Scholar 

  7. Baryannis, G., Dani, S., Antoniou, G.: Predicting supply chain risks using machine learning: the trade-off between performance and interpretability. Futur. Gener. Comput. Syst. 101, 993–1004 (2019)

    Article  Google Scholar 

  8. Bosch, J., Olsson, H.H., Crnkovic, I.: It takes three to tango: requirement, outcome/data, and AI driven development. In: SiBW 2018, Software-intensive Business: Start-ups, Ecosystems and Platforms, Espoo, Finland, 3 December 2018, pp. 177–192. CEUR-WS. org (2018)

    Google Scholar 

  9. Burkart, N., Huber, M.F.: A survey on the explainability of supervised machine learning. J. Artif. Intell. Res. 70, 245–317 (2021)

    Article  MathSciNet  Google Scholar 

  10. Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., Tsaneva-Atanasova, K.: Artificial intelligence, bias and clinical safety. BMJ Qual. Saf. 28(3), 231–237 (2019)

    Article  Google Scholar 

  11. Chazette, L., Brunotte, W., Speith, T.: Explainable software systems: from requirements analysis to system evaluation. Requirements Eng. 1–31 (2022)

    Google Scholar 

  12. Chung, L., Nixon, B.A., Yu, E., Mylopoulos, J.: Non-functional Requirements in Software Engineering, vol. 5. Springer, Heidelberg (2012)

    Google Scholar 

  13. European Commission, Directorate-General for Communications Networks, Content and Technology: EUR-Lex - 52021PC0206 - EN - EUR-Lex. CNECT (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021PC0206

  14. Faria, J.M.: Machine learning safety: an overview. In: Proceedings of the 26th Safety-Critical Systems Symposium, York, UK, pp. 6–8 (2018)

    Google Scholar 

  15. Glinz, M.: On non-functional requirements. In: 15th IEEE International requirements engineering Conference (RE 2007), pp. 21–26. IEEE (2007)

    Google Scholar 

  16. Gondal, M.A., Qureshi, N.A., Mukhtar, H., Ahmed, H.F.: An engineering approach to integrate non-functional requirements (NFR) to achieve high quality software process. In: ICEIS (2), pp. 377–384 (2020)

    Google Scholar 

  17. Gruber, K., Huemer, J., Zimmermann, A., Maschotta, R.: Integrated description of functional and non-functional requirements for automotive systems design using SysML. In: 2017 7th IEEE International Conference on System Engineering and Technology (ICSET), pp. 27–31. IEEE (2017)

    Google Scholar 

  18. Habibullah, K.M., Diaz, J.G., Gay, G., Horkoff, J.: Scoping of non-functional requirements for machine learning systems. In: 2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE (2024)

    Google Scholar 

  19. Habibullah, K.M., Gay, G., Horkoff, J.: Non-functional requirements for machine learning: an exploration of system scope and interest. In: SE4RAI Workshop, 44th International Conference on Software Engineering (ICSE 2022) (2022)

    Google Scholar 

  20. Habibullah, K.M., Gay, G., Horkoff, J.: Non-functional requirements for machine learning: understanding current use and challenges among practitioners. Requirements Eng. 28(2), 283–316 (2023)

    Article  Google Scholar 

  21. Habibullah, K.M., et al.: Requirements engineering for automotive perception systems: an interview study. In: Ferrari, A., Penzenstadler, B. (eds.) REFSQ 2023. LNCS, vol. 13975, pp. 189–205. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-29786-1_13

    Chapter  Google Scholar 

  22. Habibullah, K.M., et al.: Requirements and software engineering for automotive perception systems: an interview study. Requirements Eng. 1–24 (2024)

    Google Scholar 

  23. Habibullah, K.M., Horkoff, J.: Non-functional requirements for machine learning: understanding current use and challenges in industry. In: 2021 IEEE 29th International Requirements Engineering Conference (RE), pp. 13–23. IEEE (2021)

    Google Scholar 

  24. Horkoff, J.: Non-functional requirements for machine learning: challenges and new directions. In: 2019 IEEE 27th International Requirements Engineering Conference (RE), pp. 386–391. IEEE (2019)

    Google Scholar 

  25. Hort, M.: Investigating trade-offs for fair machine learning systems. Ph.D. thesis, UCL (University College London) (2023)

    Google Scholar 

  26. ISO: Road vehicles – Functional safety (2011)

    Google Scholar 

  27. ISO/IEC25010: Systems and software engineering systems and software quality requirements and evaluation (square) (2023)

    Google Scholar 

  28. Kamishima, T., Akaho, S., Sakuma, J.: Fairness-aware learning through regularization approach. In: 2011 IEEE 11th International Conference on Data Mining Workshops, pp. 643–650. IEEE (2011)

    Google Scholar 

  29. Kharchenko, V., Fesenko, H., Illiashenko, O.: Quality models for artificial intelligence systems: characteristic-based approach, development and application. Sensors 22(13), 4865 (2022)

    Article  Google Scholar 

  30. Kopczyńska, S., Ochodek, M., Nawrocki, J.: On importance of non-functional requirements in agile software projects—a survey. In: Jarzabek, S., Poniszewska-Marańda, A., Madeyski, L. (eds.) Integrating Research and Practice in Software Engineering. SCI, vol. 851, pp. 145–158. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-26574-8_11

    Chapter  Google Scholar 

  31. Mairiza, D., Zowghi, D.: Constructing a catalogue of conflicts among non-functional requirements. In: Maciaszek, L.A., Loucopoulos, P. (eds.) ENASE 2010. CCIS, vol. 230, pp. 31–44. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23391-3_3

    Chapter  Google Scholar 

  32. Nalchigar, S., Yu, E., Keshavjee, K.: Modeling machine learning requirements from three perspectives: a case report from the healthcare domain. Requirements Eng. 26, 237–254 (2021)

    Article  Google Scholar 

  33. Pandharipande, A., et al.: Sensing and machine learning for automotive perception: a review. IEEE Sens. J. 23(11), 11097–11115 (2023)

    Article  Google Scholar 

  34. Parekh, D., et al.: A review on autonomous vehicles: progress, methods and challenges. Electronics 11(14), 2162 (2022)

    Article  Google Scholar 

  35. Robertson, S., Robertson, J.: Mastering the Requirements Process: Getting Requirements Right. Addison-Wesley (2012)

    Google Scholar 

  36. Ruhe, G., Eberlein, A., Pfahl, D.: Trade-off analysis for requirements selection. Int. J. Softw. Eng. Knowl. Eng. 13(04), 345–366 (2003)

    Article  Google Scholar 

  37. Sculley, D., et al.: Hidden technical debt in machine learning systems. In: Advances in Neural Information Processing Systems, vol. 28 (2015)

    Google Scholar 

  38. Sherif, E., Helmy, W., Galal-Edeen, G.H.: Proposed framework to manage non-functional requirements in agile. IEEE Access (2023)

    Google Scholar 

  39. Shreda, Q.A., Hanani, A.A.: Identifying non-functional requirements from unconstrained documents using natural language processing and machine learning approaches. IEEE Access (2021)

    Google Scholar 

  40. Siau, K., Wang, W.: Building trust in artificial intelligence, machine learning, and robotics. Cutter Bus. Technol. J. 31(2), 47–53 (2018)

    Google Scholar 

  41. Siebert, J., et al.: Construction of a quality model for machine learning systems. Softw. Qual. J. 30(2), 307–335 (2022)

    Article  Google Scholar 

  42. Smola, A., Vishwanathan, S.: Introduction to machine learning. Camb. Univ. UK 32(34), 2008 (2008)

    Google Scholar 

  43. Supakkul, S., Hill, T., Chung, L., Tun, T.T., do Prado Leite, J.C.S.: An NFR pattern approach to dealing with NFRs. In: 2010 18th IEEE International Requirements Engineering Conference, pp. 179–188. IEEE (2010)

    Google Scholar 

  44. Villamizar, H., Escovedo, T., Kalinowski, M.: Requirements engineering for machine learning: a systematic mapping study. In: 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 29–36. IEEE (2021)

    Google Scholar 

  45. Vogelsang, A., Borg, M.: Requirements engineering for machine learning: perspectives from data scientists. In: IEEE 27th International Requirements Engineering Conference Workshops (REW), pp. 245–251. IEEE (2019)

    Google Scholar 

  46. Wan, Z., Xia, X., Lo, D., Murphy, G.C.: How does machine learning change software development practices? IEEE Trans. Softw. Eng. 47(9), 1857–1871 (2019)

    Google Scholar 

  47. Werner, C., Li, Z.S., Lowlind, D., Elazhary, O., Ernst, N.A., Damian, D.: Continuously managing NFRs: opportunities and challenges in practice. IEEE Trans. Softw. Eng. (2021)

    Google Scholar 

Download references

Acknowledgement

This research is supported by a Swedish Research Council (VR) Project: Non-Functional Requirements for Machine Learning: Facilitating Continuous Quality Awareness (iNFoRM).

Author information

Authors and Affiliations

  1. Chalmers and University of Gothenburg, Gothenburg, Sweden

    Khan Mohammad Habibullah, Gregory Gay & Jennifer Horkoff

Authors
  1. Khan Mohammad Habibullah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gregory Gay
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jennifer Horkoff
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Khan Mohammad Habibullah .

Editor information

Editors and Affiliations

  1. Istituto di Scienza e Tecnologie, Consiglio Nazionale delle Ricerche, Pisa, Italy

    Antonia Bertolino

  2. Faculty of Engineering, University of Porto, Porto, Portugal

    João Pascoal Faria

  3. Dept of Computer Science, Vrije Universiteit Amsterdam, Amsterdam, Noord-Holland, The Netherlands

    Patricia Lago

  4. Dipartimento di Informatica, Università di Pisa, Pisa, Italy

    Laura Semini

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Habibullah, K.M., Gay, G., Horkoff, J. (2024). A Framework for Managing Quality Requirements for Machine Learning-Based Software Systems. In: Bertolino, A., Pascoal Faria, J., Lago, P., Semini, L. (eds) Quality of Information and Communications Technology. QUATIC 2024. Communications in Computer and Information Science, vol 2178. Springer, Cham. https://doi.org/10.1007/978-3-031-70245-7_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-70245-7_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-70244-0

  • Online ISBN: 978-3-031-70245-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics