Skip to main content
Springer Nature Link
Log in

Continuously Testing Distributed IoT Systems: An Overview of the State of the Art

  • Conference paper
  • First Online:
Service-Oriented Computing – ICSOC 2021 Workshops (ICSOC 2021)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13236))

Included in the following conference series:

Abstract

The continuous testing of small changes to systems has proven to be useful and is widely adopted in the development of software systems. For this, software is tested in environments that are as close as possible to the production environments. When testing IoT systems, this approach is met with unique challenges that stem from the typically large scale of the deployments, heterogeneity of nodes, challenging network characteristics, and tight integration with the environment among others. IoT test environments present a possible solution to these challenges by emulating the nodes, networks, and possibly domain environments in which IoT applications can be executed. This paper gives an overview of the state of the art in IoT testing. We derive desirable characteristics of IoT test environments, compare 18 tools that can be used in this respect, and give a research outlook of future trends in this area.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://iotify.io/.

  2. 2.

    https://aws.amazon.com/solutions/implementations/iot-device-simulator/.

References

  1. Adjih, C., et al.: FIT IoT-LAB: a large scale open experimental IoT testbed. In: 2015 IEEE 2nd World Forum on Internet of Things (WF-IoT). IEEE (2015)

    Google Scholar 

  2. Behnke, I., Thamsen, L., Kao, O.: HéCtor: a framework for testing IoT applications across heterogeneous DGE and cloud testbeds. In: 12th International Conference on Utility and Cloud Computing Companion. ACM (2019)

    Google Scholar 

  3. Beilharz, J., et al.: Towards a staging environment for the Internet of Things. In: 2021 IEEE International Conference on Pervasive Computing and Communications (PerCom Workshops). IEEE (2021)

    Google Scholar 

  4. Bures, M., et al.: Interoperability and integration testing methods for IoT systems: a systematic mapping study. In: de Boer, F., Cerone, A. (eds.) SEFM 2020. LNCS, vol. 12310, pp. 93–112. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58768-0_6

    Chapter  Google Scholar 

  5. Chernyshev, M., Baig, Z., Bello, O., Zeadally, S.: Internet of things (IoT): research, simulators, and testbeds. IEEE Internet Things J. 5, 1637–1647 (2017)

    Article  Google Scholar 

  6. Coutinho, A., Greve, F., Prazeres, C., Cardoso, J.: Fogbed: a rapid-prototyping emulation environment for fog computing. In: 2018 IEEE International Conference on Communications (ICC). IEEE (2018)

    Google Scholar 

  7. Dastjerdi, A.V., Buyya, R.: Fog computing: helping the Internet of Things realize its potential. Computer 49, 112–116 (2016)

    Article  Google Scholar 

  8. Dias, J.P., Couto, F., Paiva, A.C., Ferreira, H.S.: A brief overview of existing tools for testing the Internet-of-Things. In: 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE (2018)

    Google Scholar 

  9. Geldenhuys, M.K., Will, J., Pfister, B., Haug, M., Scharmann, A., Thamsen, L.: Dependable IoT data stream processing for monitoring and control of urban infrastructures. In: IEEE International Conference on Cloud Engineering. IEEE (2021)

    Google Scholar 

  10. Gontarska, K., Wrazen, W., Beilharz, J., Schmid, R., Thamsen, L., Polze, A.: Predicting medical interventions from vital parameters: towards a decision support system for remote patient monitoring. In: Tucker, A., Henriques Abreu, P., Cardoso, J., Pereira Rodrigues, P., Riaño, D. (eds.) AIME 2021. LNCS (LNAI), vol. 12721, pp. 293–297. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77211-6_33

    Chapter  Google Scholar 

  11. Gupta, H., Vahid Dastjerdi, A., Ghosh, S.K., Buyya, R.: iFogSim: a toolkit for modeling and simulation of resource management techniques in the Internet of Things, Edge and Fog computing environments. Pract. Exp. Softw. 47, 1275–1296 (2017)

    Article  Google Scholar 

  12. Hasenburg, J., Grambow, M., Bermbach, D.: Mockfog 2.0: automated execution of fog application experiments in the cloud. IEEE Trans. Cloud Comput. (2021)

    Google Scholar 

  13. Kang, H.S., et al.: Smart manufacturing: Past research, present findings, and future directions. Int. J. Precis. Eng. Manuf.-Green Technol. (2016)

    Google Scholar 

  14. Keahey, K., et al.: Lessons learned from the Chameleon testbed. In: 2020 USENIX Annual Technical Conference (USENIX ATC 2020) (2020)

    Google Scholar 

  15. Kim, H., Ahmad, A., Hwang, J., Baqa, H., Le Gall, F., Ortega, M., Song, J.: IoT-TaaS: towards a prospective IoT testing framework. IEEE Access (2018)

    Google Scholar 

  16. Ly-Trong, N., Dang-Le-Bao, C., Huynh-Van, D., Le-Trung, Q.: UiTiOt v3: a hybrid testbed for evaluation of large-scale IoT networks. In: 9th International Symposium on Information and Communication Technology. ACM (2018)

    Google Scholar 

  17. Mahadevan, P., Rodriguez, A., Becker, D., Vahdat, A.: MobiNet: a scalable emulation infrastructure for ad hoc and wireless networks. ACM SIGMOBILE Mobile Comput. Commun. Rev. 10, 26–37 (2006)

    Article  Google Scholar 

  18. Malasinghe, L.P., Ramzan, N., Dahal, K.: Remote patient monitoring: a comprehensive study. J. Amb. Intell. Human. Comput. 10, 57–76 (2019)

    Article  Google Scholar 

  19. Masek, P., et al.: A harmonized perspective on transportation management in smart cities: the novel IoT-driven environment for road traffic modeling. Sensors (2016)

    Google Scholar 

  20. Mayer, R., Graser, L., Gupta, H., Saurez, E., Ramachandran, U.: EmuFog: extensible and scalable emulation of large-scale fog computing infrastructures. In: 2017 IEEE Fog World Congress (FWC). IEEE (2017)

    Google Scholar 

  21. Miyachi, T., Chinen, K.i., Shinoda, Y.: StarBED and SpringOS: large-scale general purpose network testbed and supporting software. In: 1st International Conference on Performance Evaluation Methodolgies and Tools. ACM (2006)

    Google Scholar 

  22. Mohammadi, M., Al-Fuqaha, A.: Enabling cognitive smart cities using big data and machine learning: approaches and challenges. IEEE Commun. Mag. 56, 94–101 (2018)

    Article  Google Scholar 

  23. Mäkinen, A., Jiménez, J., Morabito, R.: ELIoT: design of an emulated IoT platform. In: 2017 IEEE 28th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC). IEEE (2017)

    Google Scholar 

  24. Nikolaidis, F., Marazakis, M., Bilas, A.: IOTier: a virtual testbed to evaluate systems for IoT environments. In: 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid). IEEE (2021)

    Google Scholar 

  25. Patel, N.D., Mehtre, B.M., Wankar, R.: Simulators, emulators, and test-beds for internet of things: A comparison. In: 2019 Third International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud). IEEE (2019)

    Google Scholar 

  26. Petersen, E., Cotto, G., To, M.A.: Dockemu 2.0: evolution of a network emulation tool. In: 2019 IEEE 39th Central America and Panama Convention. IEEE (2019)

    Google Scholar 

  27. Ramprasad, B., Fokaefs, M., Mukherjee, J., Litoiu, M.: EMU-IoT - a virtual Internet of Things lab. In: 2019 IEEE International Conference on Autonomic Computing (ICAC). IEEE (2019)

    Google Scholar 

  28. Sarzyniec, L., Buchert, T., Jeanvoine, E., Nussbaum, L.: Design and evaluation of a virtual experimental environment for distributed systems. In: 21st Euromicro International Conference on Parallel, Distributed, and Network-Based Processing. IEEE (2013)

    Google Scholar 

  29. Sendorek, J., Szydlo, T., Brzoza-Woch, R.: Software-defined virtual testbed for IoT systems. Wireless Commun. Mobile Comput. 2018, 1–11 (2018)

    Article  Google Scholar 

  30. Symeonides, M., Georgiou, Z., Trihinas, D., Pallis, G., Dikaiakos, M.D.: Fogify: a fog computing emulation framework. In: 2020 IEEE/ACM Symposium on Edge Computing (SEC). IEEE (2020)

    Google Scholar 

  31. Tonneau, A.S., Mitton, N., Vandaele, J.: How to choose an experimentation platform for wireless sensor networks? A survey on static and mobile wireless sensor network experimentation facilities. Ad Hoc Networks (2015)

    Google Scholar 

  32. Wiesner, P., Thamsen, L.: LEAF: Simulating large energy-aware fog computing environments. In: 2021 IEEE 5th International Conference on Fog and Edge Computing (ICFEC). IEEE (2021)

    Google Scholar 

  33. Worstcase: Blockade (2021). https://github.com/worstcase/blockade

  34. Zeng, X., Garg, S.K., Strazdins, P., Jayaraman, P.P., Georgakopoulos, D., Ranjan, R.: IOTSim: a simulator for analysing IoT applications. J. Syst. Architect. 72, 93–107 (2017)

    Article  Google Scholar 

  35. Zeng, Y., Chao, M., Stoleru, R.: EmuEdge: a hybrid emulator for reproducible and realistic edge computing experiments. In: 2019 IEEE International Conference on Fog Computing (ICFC). IEEE (2019)

    Google Scholar 

  36. Zhao, Y., et al.: Continuous monitoring of train parameters using IoT sensor and edge computing. IEEE Sens. J. 21, 15458–15488 (2021)

    Article  Google Scholar 

  37. Zhou, J., Ji, Z., Bagrodia, R.L.: TWINE: a hybrid emulation testbed for wireless networks and applications. In: INFOCOM, vol. 6. Citeseer (2006)

    Google Scholar 

  38. Zhou, J., Ji, Z., Varshney, M., Xu, Z., Yang, Y., Marina, M., Bagrodia, R.: WHYNET: a hybrid testbed for large-scale, heterogeneous and adaptive wireless networks. In: 1st International Workshop on Wireless Network Testbeds, Experimental Evaluation & Characterization. ACM (2006)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Hasso Plattner Institute, University of Potsdam, Potsdam, Germany

    Jossekin Beilharz, Arne Boockmeyer, Lukas Pirl, Dirk Friedenberger & Andreas Polze

  2. Technische Universität Berlin, Berlin, Germany

    Philipp Wiesner, Florian Brokhausen, Ilja Behnke & Lauritz Thamsen

  3. Humboldt-Universität zu Berlin, Berlin, Germany

    Lauritz Thamsen

Authors
  1. Jossekin Beilharz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Philipp Wiesner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arne Boockmeyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lukas Pirl
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dirk Friedenberger
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Florian Brokhausen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ilja Behnke
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Andreas Polze
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Lauritz Thamsen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jossekin Beilharz .

Editor information

Editors and Affiliations

  1. Zayed University, Dubai, United Arab Emirates

    Hakim Hacid

  2. Zayed University, Abu Dhabi, United Arab Emirates

    Monther Aldwairi

  3. Deakin University, Waurn Ponds, VIC, Australia

    Mohamed Reda Bouadjenek

  4. National Research Council C.N.R., Pisa, Italy

    Marinella Petrocchi

  5. University of Lyon, Villeurbanne, France

    Noura Faci

  6. Zayed University, Dubai, United Arab Emirates

    Fatma Outay

  7. Macquarie University, Sydney, NSW, Australia

    Amin Beheshti

  8. Technical University of Berlin, Berlin, Germany

    Lauritz Thamsen

  9. Royal Melbourne Institute of Technology, Melbourne, VIC, Australia

    Hai Dong

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Beilharz, J. et al. (2022). Continuously Testing Distributed IoT Systems: An Overview of the State of the Art. In: Hacid, H., et al. Service-Oriented Computing – ICSOC 2021 Workshops. ICSOC 2021. Lecture Notes in Computer Science, vol 13236. Springer, Cham. https://doi.org/10.1007/978-3-031-14135-5_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-14135-5_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-14134-8

  • Online ISBN: 978-3-031-14135-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics