Skip to main content
SpringerLink
Log in

CT-IoT: a combinatorial testing-based path selection framework for effective IoT testing

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Testing Internet of Things (IoT) systems is challenging. This is not only because of the various aspects of IoT systems, such as software, hardware, and network that need to be tested, but also because of the unexpected issues caused by a large number of heterogeneous devices brought together by IoT systems. When an IoT system has hundreds, or even thousands, of heterogeneous devices, which devices should be tested to detect more faults? How can we systematically test an IoT system and its numerous devices in a cost-effective way? Are there any coverage criteria for testers to evaluate the thoroughness of the testing against IoT systems? In this paper, we present a combinatorial testing path selection framework for IoT systems, called CT-IoT, that systematically identifies and recommends testing paths in IoT systems for effective testing. We also propose four coverage criteria that can help testers evaluate the testing thoroughness for IoT systems. We conducted an empirical study of CT-IoT on two real-world IoT systems and evaluated the effectiveness of CT-IoT in terms of coverage achievements. The results show the superiority of CT-IoT over a random approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

Not applicable.

References

  • Gubbi J, Buyya R, Marusic S, Palaniswami M (2013) Internet of things (IoT): a vision, architectural elements, and future directions. Future Generation Comput Syst 29(7):1645–1660

    Article  Google Scholar 

  • Atzori L, Iera A, Morabito G (2010) The internet of things: a survey. Comput Netw 54(15):2787–2805

    Article  Google Scholar 

  • “Fast Innovation requires Fast IT”, https://www.cisco.com/c/dam/global/en_ph/assets/ciscoconnect/pdf/bigdata/jim_green_cisco_connect.pdf [Online; accessed 06-11-2021]

  • Voas J (2016) “Networks of ‘things’,” NIST Special Publication 800, no. 183

  • Burnstein I (2006) Practical software testing: a process-oriented approach. Springer Science & Business Media

    MATH  Google Scholar 

  • Mathur A (2013) “Foundations of software testing”, Pearson India, 2nd edn

    Google Scholar 

  • Fang L, Li G (2015) “Test Selection with Equivalence Class Partitioning.” In 2015 2nd International Symposium on Dependable Computing and Internet of Things (DCIT), pp. 40–49. IEEE

  • “Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2017–2022” https://s3.amazonaws.com/media.mediapost.com/uploads/CiscoForecast.pdf [Online; accessed 06-12-2021]

  • Hu L, Wong WE, Kuhn DR, Kacker RN (2020) How does combinatorial testing perform in the real world: an empirical study. Empir Softw Eng 25:2661–2693

    Article  Google Scholar 

  • Lei Y, Kacker R, Kuhn DR, Okun V, Lawrence J (2008a) Ipog/ipog-d: efficient test generation for multi-way combinatorial testing. Software Testing, Verification and Reliability 18(3):125–148

    Article  Google Scholar 

  • Grindal M, Offutt J, Andler SF (2005) Combination testing strategies: a survey. Software Testing, Verification and Reliability 15(3):167–199

    Article  Google Scholar 

  • Yu L, Lei Y, Kacker RN, Kuhn DR (2013) “Acts: A combinatorial test generation tool,” in 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, pp. 370–375

  • “EBNF: A Notation to Describe Syntax”, “https://www.ics.uci.edu/~pattis/misc/ebnf2.pdf”, [Online; accessed on July 02, 2020]

  • Kuhn R, Wallace DR, Gallo AM (2004a) Software fault interactions and implications for software testing. IEEE Trans Softw Eng 30(6):418–421

    Article  Google Scholar 

  • Carlos-Mancilla, Miriam, Ernesto López-Mellado, and Mario Siller (2016) “Wireless sensor networks formation: approaches and techniques.” Journal of Sensors 2016

  • Guy C (2006) “Wireless sensor networks.” In Sixth international symposium on instrumentation and control technology: Signal analysis, measurement theory, photo-electronic technology, and artificial intelligence, vol. 6357, p. 63571I. International Society for Optics and Photonics

  • Gabriel KR, Sokal RR (1969) A new statistical approach to geographic variation analysis. Syst Zool 18(3):259–278

    Article  Google Scholar 

  • Wang K, Wang L, Cai S, Song Q (2009) “An energy-saving algorithm of WSN based on gabriel graph,” In 2009 5th International Conference on Wireless Communications, Networking and Mobile Computing, pp. 1–4. IEEE

  • Hossain MZ, Wahid MA, Hasan M, and Amin MA (2015) “Spatial Subdivision of Gabriel Graph.” In International Conference in Swarm Intelligence, pp. 321–328. Springer, Cham

  • Kalosha H, Nayak A, Ruhrup S, and Stojmenovic I (2008) “Select-and-protest-based beaconless georouting with guaranteed delivery in wireless sensor networks.” In IEEE INFOCOM 2008-The 27th Conference on Computer Communications, pp. 346–350. IEEE

  • Cohen M, Dalal SR, Parelius J, Patton GC (1996) The combinatorial design approach to automatic test generation. IEEE Softw 13(5):83–88

    Article  Google Scholar 

  • Cohen M, Dalal SR, Fredman ML, Patton GC (1997) The aetg system: an approach to testing based on combinatorial design. IEEE Trans Softw Eng 23(7):437–444

    Article  Google Scholar 

  • Kuhn R and Reilly MJ (2002) “An investigation of the applicability of design of experiments to software testing,” in Software Engineering Workshop, 2002. Proceedings. 27th Annual NASA Goddard/IEEE, pp. 91–95, IEEE

  • Kuhn R, Wallace DR, Gallo AM (2004b) Software fault interactions and implications for software testing. IEEE Trans Softw Eng 30(6):418–421

    Article  Google Scholar 

  • Lei Y, Kacker R, Kuhn DR, Okun V, Lawrence J (2008b) Ipog/ipog-d: efficient test generation for multi-way combinatorial testing. Software Testing, Verification and Reliability 18(3):125–148

    Article  Google Scholar 

  • Garvin BJ, Cohen MB, Dwyer MB (2011) Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empir Softw Eng 16(1):61–102

    Article  Google Scholar 

  • Automated combinatorial testing for software — csrc.” https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software. [Online; accessed 02-27-2021]

  • Wu H, Nie C, Petke J, Jia Y, Harman M (2019) “A survey of constrained combinatorial testing.” arXiv preprint arXiv:1908.02480

  • Wang W, Sampath S, Yu L, Kacker R (2008) “An interaction-based test sequence generation approach for testing web applications.” In 2008 11th IEEE High Assurance Systems Engineering Symposium, pp. 209–218. IEEE

  • Yu L, Yu L, Kacker RN, Kuhn DR, and Lawrence J (2012) “Efficient algorithms for t-way test sequence generation.” In 2012 IEEE 17th International Conference on Engineering of Complex Computer Systems, pp. 220–229. IEEE

  • Patil AH (2019) “Design and implementation of combinatorial testing based test suites for operating systems used for internet of things.” Independently published

  • Patil AH, Goveas N, Rangarajan K (2015) Test suite design methodology using combinatorial approach for internet of things operating systems. J Softw Eng Appl 8(07):303

    Article  Google Scholar 

  • Voas J, Kuhn R, Laplante P (2018) “Testing IoT Systems.” In 2018 IEEE Symposium on Service-Oriented System Engineering (SOSE), pp. 48–52. IEEE

  • Chen M, Miao Y, Hao Y, Hwang K (2017) Narrow band internet of things. IEEE access 5:20557–20577

    Article  Google Scholar 

  • Adhikary A, Lin X, and Wang YE (2016) “Performance evaluation of NB-IoT coverage.” in proceedings of 2016 IEEE 84th Vehicular Technology Conference (VTC-Fall), pp. 1–5

  • Zheng Y, Davanian A, Yin H, Song C, Zhu H, Sun L (2019) “FIRM-AFL: high-throughput greybox fuzzing of iot firmware via augmented process emulation.” In 28th {USENIX} Security Symposium ({USENIX} Security 19), pp. 1099–1114

  • Liu X, Cui B, Junsong F, Ma J (2020) HFuzz: towards automatic fuzzing testing of NB-IoT core network protocols implementations. Futur Gener Comput Syst 108:390–400

    Article  Google Scholar 

  • Alhawi OMK, Akinbi A, and Dehghantanha A (2019a) “Evaluation and application of two fuzzing approaches for security testing of IoT applications.” In Handbook of Big Data and IoT Security, pp. 301–327. Springer, Cham

  • Reed JL (2019) “Testbed Design for Autonomous Security Testing of IoT Devices.” PhD diss., The University of Texas at San Antonio

  • Murad G, Badarneh A, Qusef A, and Almasalha F (2018) “Software testing techniques in iot.” In 2018 8th International Conference on Computer Science and Information Technology (CSIT), pp. 17–21. IEEE

  • Jeannotte B, Tekeoglu A (2019) “Artorias: IoT Security Testing Framework.” In 2019 26th International Conference on Telecommunications (ICT), pp. 233–237. IEEE

  • Orjala L-M (2019) Unit testing methods for internet of things Mbed OS operating system. University of Oulu, PhD diss

    Google Scholar 

  • Alam T (2019) “IoT-Fog: A communication framework using blockchain in the internet of things.”, International Journal of Recent Technology and Engineering (IJRTE) 7, no. 6

  • Jacob PM, Mani P (2018a) A Reference Model for Testing Internet of Things Based Applications. Journal of Engineering Science and Technology 13(8):2504–2519

    Google Scholar 

  • Padmanabhan M (2018) “Test Path Identification for Internet of Things using Transaction Based Specification.” In 2018 International Conference on Current Trends towards Converging Technologies (ICCTCT), pp. 1–6. IEEE

  • Ahmad A (2018) Model-based testing for IoT systems: methods and tools. PhD diss, Bourgogne Franche-Comté

    Google Scholar 

  • Rechtberger V, Bures M, and Ahmed BS (2020a) “Alternative Effort-optimal Model-based Strategy for State Machine Testing of IoT Systems.” In Proceedings of the 2020 The 2nd World Symposium on Software Engineering, pp. 141–145

  • Nguyen NMT, Bannour B, Lapitre A, and Gall PL (2019) “Behavioral models and scenario selection for testing IoT Trickle-based lossy multicast networks.” In 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 168–175. IEEE

  • Krichen M, Lahami M, Cheikhrouhou O, Alroobaea R, and Maâlej AJ (2020) “Security testing of internet of things for smart city applications: A formal approach.” In Smart Infrastructure and Applications, pp. 629–653. Springer, Cham

  • Krichen M (2019) Improving formal verification and testing techniques for internet of things and smart cities. Mobile Networks and Applications:1–12

  • Ahmad A, Bouquet F, Fourneret E, Le Gall F, Legeard B (2016a) “Model-based testing as a service for iot platforms.” In International Symposium on Leveraging Applications of Formal Methods, pp. 727–742. Springer, Cham

  • Pering T, Farrington K, Dahm T (2018) Taming the IoT: operationalized testing to secure connected devices. Computer 51(6):90–94

    Article  Google Scholar 

  • Reetz ES (2016) “Service testing for the internet of things.” PhD diss., University of Surrey

  • Kanstrén T, Mäkelä J, Karhula P (2018a) “Architectures and experiences in testing IoT communications.” In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 98–103. IEEE

  • Chen W, Liu CH, Liang WWY, Tsai MY (2018a) “ICAT: An IoT device compatibility testing tool.” In 2018 25th Asia-Pacific Software Engineering Conference (APSEC), pp. 668–672. IEEE

  • Medhat N, Moussa S, Badr N, and Tolba MF (2019a) “Testing techniques in IoT-based systems.” In 2019 Ninth International Conference on Intelligent Computing and Information Systems (ICICIS), pp. 394–401. IEEE

  • Giménez P, Molína B, Palau CE, Esteve M (2013) “SWE Simulation and Testing for the IoT.” In 2013 IEEE International Conference on Systems, Man, and Cybernetics, pp. 356–361. IEEE

  • Bosmans S, Mercelis S, Denil J, Hellinckx P (2019) Testing IoT systems using a hybrid simulation based testing approach. Computing 101(7):857–872

    Article  MathSciNet  Google Scholar 

  • Belchior BFSA (2019) “Testing in IoT Systems: From Simulation to Visual-Based Testing.”

  • Satheesh N, Udayakumar P, and Sengan S (2019) “Testing for IoT Devices and Software's and Effects of New Features on Security and Privacy by using Test Simulation.”

  • Dias JP, Ferreira HS, Sousa TB (2019) “Testing and deployment patterns for the internet-of-things.” In Proceedings of the 24th European Conference on Pattern Languages of Programs, pp. 1–8

  • Georgiev I, and Georgiev I (2019) “Simulation-Based Self-Testing in IoT-Enabled Manufacturing.” In 2019 International Conference on Information Technologies (InfoTech), pp. 1–4. IEEE

  • Sand B (2015) “IoT testing-the big challenge why, what and how.” In International Internet of Things Summit, pp. 70–76. Springer, Cham

  • Bures M, Cerny T, Ahmed BS (2018) “Internet of things: Current challenges in the quality assurance and testing methods.” In International Conference on Information Science and Applications, pp. 625–634. Springer, Singapore

  • Medhat N, Moussa S, Badr N, Tolba MF (2019b) “Testing techniques in IoT-based systems.” In 2019 Ninth International Conference on Intelligent Computing and Information Systems (ICICIS), pp. 394–401. IEEE

  • Antao L, Pinto R, Reis J, and Gonçalves G (2018) “Requirements for testing and validating the industrial internet of things.” In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 110–115. IEEE

  • Gomez AK, Bajaj S (2019) “Challenges of testing complex Internet of Things (IoT) devices and systems.” In 2019 11th International Conference on Knowledge and Systems Engineering (KSE), pp. 1–4. IEEE

  • Ahmad A, Bouquet F, Fourneret E, Le Gall F, Legeard B (2016b) “Model-based testing as a service for iot platforms.” In International Symposium on Leveraging Applications of Formal Methods, pp. 727–742. Springer, Cham, 2016

  • Tappler M, Aichernig BK, Bloem R (2017) “Model-based testing IoT communication via active automata learning.” In 2017 IEEE International conference on software testing, verification and validation (ICST), pp. 276–287. IEEE

  • Truong H, Berardinelli L, Pavkovic I, Copil G (2017) “Modeling and provisioning iot cloud systems for testing uncertainties.” In Proceedings of the 14th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services, pp. 96–105

  • İnçki K, Ari I (2017) “Observing interoperability of IoT systems through model-based testing.” In Interoperability, Safety and Security in IoT, pp. 60–66. Springer, Cham

  • Popereshnyak S, Suprun O, Suprun O, and Wieckowski T (2018) “IoT application testing features based on the modelling network.” In 2018 XIV-th International Conference on Perspective Technologies and Methods in MEMS Design (MEMSTECH), pp. 127–131. IEEE

  • Rechtberger V, Bures M, Ahmed BS (2020b) “Alternative Effort-optimal Model-based Strategy for State Machine Testing of IoT Systems.” In Proceedings of the 2020 The 2nd World Symposium on Software Engineering, pp. 141–145

  • Jacob PM, Mani P (2018b) A Reference Model for Testing Internet of Things Based Applications. Journal of Engineering Science and Technology 13(8):2504–2519

    Google Scholar 

  • Chandan AR, Khairnar VD (2018) “Security Testing Methodology of IoT.” In 2018 International Conference on Inventive Research in Computing Applications (ICIRCA), pp. 1431–1435. IEEE

  • Chen C-K, Zhang Z-K, Lee S-H, Shieh S (2018b) Penetration testing in the iot age. Computer 51(4):82–85

    Article  Google Scholar 

  • Lally G, Sgandurra D (2018) “Towards a framework for testing the security of IoT devices consistently.” In International workshop on emerging technologies for authorization and authentication, pp. 88–102. Springer, Cham

  • Mahmoodi Y, Reiter S, Viehl A, Bringmann O, Rosenstiel W (2018) “Attack surface modeling and assessment for penetration testing of IoT system designs.” In 2018 21st Euromicro Conference on Digital System Design (DSD), pp. 177–181. IEEE

  • Matheu-García SN, Hernández-Ramos JL, Skarmeta AF, Baldini G (2019) Risk-based automated assessment and testing for the cybersecurity certification and labelling of IoT devices. Computer Standards & Interfaces 62:64–83

    Article  Google Scholar 

  • Alhawi OMK, Akinbi A, and Dehghantanha A (2019b) “Evaluation and application of two fuzzing approaches for security testing of IoT applications.” In Handbook of Big Data and IoT Security, pp. 301–327. Springer, Cham

  • Krichen M, Alroobaea R (2019) “A new model-based framework for testing security of iot systems in smart cities using attack trees and price timed automata.” In 14th International Conference on Evaluation of Novel Approaches to Software Engineering, pp. 570–577. SCITEPRESS-Science and Technology Publications

  • Bures M (2017) “Framework for integration testing of iot solutions.” In 2017 International Conference on Computational Science and Computational Intelligence (CSCI), pp. 1838-1839. IEEE

  • Bures M, Klima M, Rechtberger V, Bellekens X, Tachtatzis C, Atkinson R, Ahmed BS (20200 “Interoperability and integration testing methods for IoT systems: A systematic mapping study.” In International Conference on Software Engineering and Formal Methods, pp. 93–112. Springer, Cham

  • Medhat N, Moussa SM, Badr NL, Tolba MF (2020) A framework for continuous regression and integration testing in IoT systems based on deep learning and search-based techniques. IEEE Access 8:215716–215726

    Article  Google Scholar 

  • Kanstrén T, Mäkelä J, Karhula P (2018b) “Architectures and experiences in testing IoT communications.” In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 98–103. IEEE

  • Sri JMK, Narendra VG, Pai V (2019) “Implementing and Testing of Internet of Things (IoT) Technology in Agriculture and Compare the Application Layer Protocols: Message Queuing Telemetry Transport (MQTT) and Hyper Text Transport Protocol (HTTP).” In International Conference on Advanced Informatics for Computing Research, pp. 320–333. Springer, Singapore

  • Shmatkov V, Spynu S, Pimenov V (2019) “The platform prototype for testing heterogeneous networks in IoT environment.” In CEUR Workshop Proceedings

  • Esquiagola J, de Paula Costa LC, Calcina P, Fedrecheski G, Zuffo M (2017) “Performance Testing of an Internet of Things Platform.” In IoTBDS, pp. 309–314

  • Ismail AA, Hamza HS, Kotb AM (2018) “Performance evaluation of open source iot platforms.” In 2018 IEEE Global Conference on Internet of Things (GCIoT), pp. 1–5. IEEE

  • Han R, Gramoli V, Xiwei X (2018) “Evaluating blockchains for IoT.” In 2018 9Th IFIP international conference on new technologies, mobility and security (NTMS), pp. 1–5. IEEE

  • Pontes PM, Lima B, Faria JP (2018a) “Izinto: a pattern-based IoT testing framework.” In Companion Proceedings for the ISSTA/ECOOP 2018 Workshops, pp. 125–131

  • Pontes PM, Lima B, and Faria JP (2018b) “Test patterns for IoT.” In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation, pp. 63–66

  • Chen WK, Liu CH, Liang WWY, Tsai MY (2018c) “ICAT: An IoT device compatibility testing tool.” In 2018 25th Asia-Pacific Software Engineering Conference (APSEC), pp. 668–672. IEEE

  • Abdullah A, Schmidt H, Spichkova M, and Liu H (2018) “Monitoring Informed Testing for IoT.” In 2018 25th Australasian Software Engineering Conference (ASWEC), pp. 91–95. IEEE

  • Celik ZB, McDaniel P, Tan G (2018) “SOTERIA: automated IoT safety and security analysis.” In Proceedings of the 2018 USENIX conference on USENIX annual technical conference (USENIX ATC '18). USENIX Association, USA, 147–158

  • Berkay Celik Z, Tan G, and McDaniel PD (2019) “IoTGuard: Dynamic Enforcement of Security and Safety Policy in Commodity IoT.” in Proceedings of the Network and Distributed System Security Symposium (NDSS)

  • Jia YJ, Chen QA, Wang S, Rahmati A, Fernandes E, Mao ZM, Prakash A (2017) “ContexloT: Towards Providing Contextual Integrity to Appified IoT Platforms.” In NDSS, vol. 2, no. 2, pp. 2–2

  • Ding W, Hongxin H (2018) “On the safety of iot device physical interaction control.” In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 832–846

Download references

Funding

This work was funded by the National Institute of Standards and Technology (Grant No. 60NANB17D322) and the U.S. National Science Foundation (Grant No. 1757828 and 1822137).

Author information

Authors and Affiliations

  1. Computer Science Department, University of Texas at Dallas, 800 W Campbell Rd, Richardson, TX, 75080, USA

    Linghuan Hu, W. Eric Wong & Shuo Li

  2. National Institute of Standards and Technology, Gaithersburg, MD, USA

    D. Richard Kuhn & Raghu N. Kacker

Authors
  1. Linghuan Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. W. Eric Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. D. Richard Kuhn
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raghu N. Kacker
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shuo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to W. Eric Wong.

Ethics declarations

Disclaimer

Any mention of commercial products in this paper is for information only; it does not imply recommendation or endorsement by the National Institute of Standards and Technology (NIST).

Conflicts of interest/competing interests

Not applicable.

Code availability

Not applicable.

Additional information

Publisher’s note

Springer Nature remains neutral with regard tojurisdictional claims in published maps and institutionalaffiliations.

Communicated by: Per Runeson

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, L., Wong, W.E., Kuhn, D.R. et al. CT-IoT: a combinatorial testing-based path selection framework for effective IoT testing. Empir Software Eng 27, 32 (2022). https://doi.org/10.1007/s10664-021-10017-1

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-021-10017-1

Keywords

Search

Navigation