Skip to main content
Springer Nature Link
Log in

Quality and Reliability Metrics for IoT Systems: A Consolidated View

  • Conference paper
  • First Online:
Science and Technologies for Smart Cities (SmartCity360° 2020)

Included in the following conference series:

Abstract

Quality and reliability metrics play an important role in the evaluation of the state of a system during the development and testing phases, and serve as tools to optimize the testing process or to define the exit or acceptance criteria of the system. This study provides a consolidated view on the available quality and reliability metrics applicable to Internet of Things (IoT) systems, as no comprehensive study has provided such a view specific to these systems. The quality and reliability metrics categorized and discussed in this paper are divided into three categories: metrics assessing the quality of an IoT system or service, metrics for assessing the effectiveness of the testing process, and metrics that can be universally applied in both cases. In the discussion, recommendations of proper usage of discussed metrics in a testing process are then given.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.tmap.net/wiki/quality-characteristics.

  2. 2.

    https://www.tmap.net/wiki/quality-characterstics-iot-environment.

References

  1. van der Aalst, L., Roodenrijs, E., Vink, J., Baarda, R.: TMap NEXT: business driven test management. Uitgeverij kleine Uil (2013)

    Google Scholar 

  2. Ahmed, B.S., Bures, M., Frajtak, K., Cerny, T.: Aspects of quality in Internet of Things (IoT) solutions: a systematic mapping study. IEEE Access 7, 13758–13780 (2019)

    Article  Google Scholar 

  3. Ammann, P., Offutt, J.: Introduction to Software Testing. Cambridge University Press, Cambridge (2016)

    Book  Google Scholar 

  4. Ammann, P., Offutt, J., Xu, W.: Coverage criteria for state based specifications. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing. LNCS, vol. 4949, pp. 118–156. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78917-8_4

    Chapter  Google Scholar 

  5. Aráuz, J., Fynn-Cudjoe, T.: Actuator quality in the Internet of Things. In: 2013 IEEE International Workshop of Internet-of-Things Networking and Control (IoT-NC), pp. 34–42 (2013)

    Google Scholar 

  6. Baggen, R., Correia, J.P., Schill, K., Visser, J.: Standardized code quality benchmarking for improving software maintainability. Softw. Qual. J. 20(2), 287–307 (2012)

    Article  Google Scholar 

  7. Bonilla, R.I., Crow, J.J., Basantes, L.S., Cruz, L.G.: A metric for measuring IoT devices security levels. In: 2017 IEEE 15th International Conference on Dependable, Autonomic and Secure Computing, 15th International Conference on Pervasive Intelligence and Computing, 3rd International Conference on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/DataCom/CyberSciTech), pp. 704–709 (2017)

    Google Scholar 

  8. Bures, M.: Framework for assessment of web application automated testability. In: Proceedings of the 2015 Conference on Research in Adaptive and Convergent Systems, pp. 512–514 (2015)

    Google Scholar 

  9. Bures, M.: Metrics for automated testability of web applications. In: Proceedings of the 16th International Conference on Computer Systems and Technologies, pp. 83–89 (2015)

    Google Scholar 

  10. Bures, M., Bellekens, X., Frajtak, K., Ahmed, B.S.: A comprehensive view on quality characteristics of the IoT solutions. In: José, R., Van Laerhoven, K., Rodrigues, H. (eds.) Urb-IoT 2018. EICC, pp. 59–69. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-28925-6_6

    Chapter  Google Scholar 

  11. Chaparro, O., Bavota, G., Marcus, A., Di Penta, M.: On the impact of refactoring operations on code quality metrics. In: 2014 IEEE International Conference on Software Maintenance and Evolution, pp. 456–460. IEEE (2014)

    Google Scholar 

  12. Chawla, M.K., Chhabra, I.: A quantitative framework for integrated software quality measurement in multi-versions systems. In: 2016 International Conference on Internet of Things and Applications (IOTA), pp. 310–315. IEEE (2016)

    Google Scholar 

  13. Chen, Y., Probert, R.L., Robeson, K.: Effective test metrics for test strategy evolution. In: Proceedings of the 2004 Conference of the Centre for Advanced Studies on Collaborative Research, pp. 111–123 (2004)

    Google Scholar 

  14. Chidamber, S.R., Kemerer, C.F.: A metrics suite for object oriented design. IEEE Trans. Softw. Eng. 20(6), 476–493 (1994)

    Article  Google Scholar 

  15. Conte, S.D., Dunsmore, H.E., Shen, Y.: Software Engineering Metrics and Models. Benjamin-Cummings Publishing Co., Inc., San Francisco (1986)

    Google Scholar 

  16. Dias Neto, A.C., Subramanyan, R., Vieira, M., Travassos, G.H.: A survey on model-based testing approaches: a systematic review. In: Proceedings of the 1st ACM International Workshop on Empirical Assessment of Software Engineering Languages and Technologies: Held in Conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 31–36 (2007)

    Google Scholar 

  17. Dromey, R.G.: A model for software product quality. IEEE Trans. Softw. Eng. 21(2), 146–162 (1995)

    Article  Google Scholar 

  18. Heitlager, I., Kuipers, T., Visser, J.: A practical model for measuring maintainability. In: 6th International Conference on the Quality of Information and Communications Technology (QUATIC 2007), pp. 30–39. IEEE (2007)

    Google Scholar 

  19. Hindy, H., et al.: A taxonomy of network threats and the effect of current datasets on intrusion detection systems. IEEE Access 8, 104650–104675 (2020)

    Article  Google Scholar 

  20. Jiang, Y., Cuki, B., Menzies, T., Bartlow, N.: Comparing design and code metrics for software quality prediction. In: Proceedings of the 4th International Workshop on Predictor Models in Software Engineering, pp. 11–18 (2008)

    Google Scholar 

  21. Jung, H.W., Kim, S.G., Chung, C.S.: Measuring software product quality: a survey of ISO/IEC 9126. IEEE Softw. 21(5), 88–92 (2004)

    Article  Google Scholar 

  22. Kan, S.H.: Metrics and Models in Software Quality Engineering. Addison-Wesley Longman Publishing Co., Inc., Boston (2002)

    MATH  Google Scholar 

  23. Kim, M.: A quality model for evaluating IoT applications. Int. J. Comput. Electr. Eng. 8(1), 66 (2016)

    Article  Google Scholar 

  24. Kim, M., Park, J.H., Lee, N.Y.: A quality model for IoT service. In: Park, J.J.J.H., Pan, Y., Yi, G., Loia, V. (eds.) CSA/CUTE/UCAWSN 2016. LNEE, vol. 421, pp. 497–504. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-3023-9_77

    Chapter  Google Scholar 

  25. Kiruthika, J., Khaddaj, S.: Software quality issues and challenges of internet of things. In: 2015 14th International Symposium on Distributed Computing and Applications for Business Engineering and Science (DCABES), pp. 176–179. IEEE (2015)

    Google Scholar 

  26. Koomen, T., Broekman, B., van der Aalst, L., Vroon, M.: TMap next: for result-driven testing. Uitgeverij kleine Uil (2013)

    Google Scholar 

  27. Li, F., Nastic, S., Dustdar, S.: Data quality observation in pervasive environments. In: 2012 IEEE 15th International Conference on Computational Science and Engineering, pp. 602–609. IEEE (2012)

    Google Scholar 

  28. Marinissen, E.J., et al.: IoT: source of test challenges. In: 2016 21th IEEE European Test Symposium (ETS), pp. 1–10. IEEE (2016)

    Google Scholar 

  29. Mell, P., Scarfone, K., Romanosky, S.: Common vulnerability scoring system. IEEE Secur. Priv. 4(6), 85–89 (2006)

    Article  Google Scholar 

  30. Ming, Z., Yan, M.: A modeling and computational method for QoS in IoT. In: 2012 IEEE International Conference on Computer Science and Automation Engineering, pp. 275–279. IEEE (2012)

    Google Scholar 

  31. Nirpal, P.B., Kale, K.: A brief overview of software testing metrics. Int. J. Comput. Sci. Eng. 3(1), 204–2011 (2011)

    Google Scholar 

  32. Pantiuchina, J., Lanza, M., Bavota, G.: Improving code: the (mis) perception of quality metrics. In: 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 80–91. IEEE (2018)

    Google Scholar 

  33. Pezzè, M., Young, M.: Software Testing and Analysis: Process, Principles, and Techniques. Wiley, Hoboken (2008)

    MATH  Google Scholar 

  34. Singh, M., Baranwal, G.: Quality of Service (QoS) in Internet of Things. In: 2018 3rd International Conference On Internet of Things: Smart Innovation and Usages (IoT-SIU), pp. 1–6 (2018)

    Google Scholar 

  35. Snigdh, I., Gupta, N.: Quality of service metrics in wireless sensor networks: a survey. J. Inst. Eng. (India): Ser. B 97(1), 91–96 (2016)

    Google Scholar 

  36. Sollie, R.S.: Security and usability assessment of several authentication technologies. Master’s thesis (2005)

    Google Scholar 

  37. Staron, M., Meding, W.: A portfolio of internal quality metrics for software architects. In: Winkler, D., Biffl, S., Bergsmann, J. (eds.) SWQD 2017. LNBIP, vol. 269, pp. 57–69. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-49421-0_5

    Chapter  Google Scholar 

  38. Staron, M., Meding, W., Karlsson, G., Nilsson, C.: Developing measurement systems: an industrial case study. J. Softw. Maint. Evol. Res. Pract. 23(2), 89–107 (2011)

    Article  Google Scholar 

  39. Van de Ven, T., Bloem, J., Duniau, J.P.: IoTMap: testing in an IoT environment. Uitgeverij kleine Uil (2016)

    Google Scholar 

  40. White, G., Nallur, V., Clarke, S.: Quality of service approaches in IoT: a systematic mapping. J. Syst. Softw. 132, 186–203 (2017)

    Article  Google Scholar 

  41. Zheng, X., Martin, P., Brohman, K., Da Xu, L.: CLOUDQUAL: a quality model for cloud services. IEEE Trans. Ind. Inform. 10(2), 1527–1536 (2014)

    Article  Google Scholar 

Download references

Acknowledgements

This research is conducted as a part of the project TACR TH02010296 Quality Assurance System for the Internet of Things Technology. The authors acknowledge the support of the OP VVV funded project CZ.02.1.01/0.0/0.0/16_019/0000765 “Research Center for Informatics”. Bestoun S. Ahmed has been supported by the Knowledge Foundation of Sweden (KKS) through the Synergi Project AIDA - A Holistic AI-driven Networking and Processing Framework for Industrial IoT (Rek:20200067).

Author information

Authors and Affiliations

  1. Department of Computer Science, FEE, Czech Technical University in Prague, Prague, Czechia

    Matej Klima, Vaclav Rechtberger, Miroslav Bures & Bestoun S. Ahmed

  2. Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow, UK

    Xavier Bellekens

  3. Division of Cyber Security, Abertay University, Dundee, UK

    Hanan Hindy

  4. Department of Mathematics and Computer Science, Karlstad University, Karlstad, Sweden

    Bestoun S. Ahmed

Authors
  1. Matej Klima
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vaclav Rechtberger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Miroslav Bures
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xavier Bellekens
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hanan Hindy
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bestoun S. Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miroslav Bures .

Editor information

Editors and Affiliations

  1. School of Technology and Management, Escola Superior de Tecnologia e Gestão, VIANA DO CASTELO, Portugal

    Sara Paiva

  2. Polytechnic Institute of Viana do Castel, Viana do Castelo, Portugal

    Sérgio Ivan Lopes

  3. SIC Laboratory, INSEEC U, ECE Paris Graduate School of Engineering, PARIS, France

    Rafik Zitouni

  4. Vaagdevi College of Engineering, Telangana, India

    Nishu Gupta

  5. University of Minho, Guimarães, Portugal

    Sérgio F. Lopes

  6. Nagoya University, Nagoya, Japan

    Takuro Yonezawa

Rights and permissions

Reprints and permissions

Copyright information

© 2021 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Klima, M., Rechtberger, V., Bures, M., Bellekens, X., Hindy, H., Ahmed, B.S. (2021). Quality and Reliability Metrics for IoT Systems: A Consolidated View. In: Paiva, S., Lopes, S.I., Zitouni, R., Gupta, N., Lopes, S.F., Yonezawa, T. (eds) Science and Technologies for Smart Cities. SmartCity360° 2020. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 372. Springer, Cham. https://doi.org/10.1007/978-3-030-76063-2_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-76063-2_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-76062-5

  • Online ISBN: 978-3-030-76063-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics