Skip to main content
SpringerLink
Log in

Automatic generation of test system instances for configurable cyber-physical systems

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Cyber-physical systems (CPSs) are ubiquitous systems that integrate digital technologies with physical processes. These systems are becoming configurable to respond to the different needs that users demand. As a consequence, their variability is increasing, and they can be configured in many system variants. To ensure a systematic test execution of CPSs, a test system must be elaborated encapsulating several sources such as test cases or test oracles. Manually building a test system for each configuration is a non-systematic, time-consuming, and error-prone process. To overcome these problems, we designed a test system for testing CPSs and we analyzed the variability that it needed to test different configurations. Based on this analysis, we propose a methodology supported by a tool named ASTERYSCO that automatically generates simulation-based test system instances to test individual configurations of CPSs. To evaluate the proposed methodology, we selected different configurations of a configurable Unmanned Aerial Vehicle, and measured the time required to generate their test systems. On average, around 119 s were needed by our tool to generate the test system for 38 configurations. In addition, we compared the process of generating test system instances between the method we propose and a manual approach. Based on this comparison, we believe that the proposed tool allows a systematic method of generating test system instances. We believe that our approach permits an important step toward the full automation of testing in the field of configurable CPSs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Notes

  1. With variability we refer to configurability, i.e., variability in product space (Thiel and Hein 2002).

  2. We denominate this term as system variant, configuration or system configuration, across the whole article.

  3. For presentation purposes of the article, two independent feature models have been developed.

  4. For presentation reasons of the article, we could not allocate every single feature of the test system.

  5. This step is not mandatory for the manual configuration.

References

  • Ali, S., & Yue, T. (2015). U-test: Evolving, modelling and testing realistic uncertain behaviors of cyber-physical systems. 8th IEEE international conference on software testing, verification and validation, ICST 2015, Graz, Austria, April 13–17, 2015, pp. 1–2.

  • Arcuri, A., & Briand, L. (2011). A practical guide for using statistical tests to assess randomized algorithms in software engineering. 2011 33rd International conference on software engineering (ICSE). IEEE, pp. 1–10.

  • Arrieta, A., Sagardui, G., & Etxeberria, L. (2014). A configurable test architecture for the automatic validation of variability-intensive cyber-physical systems. Valid 2014: The sixth international conference on advances in system testing and validation lifecycle, pp. 79–83.

  • Arrieta, A., Sagardui, G., & Etxeberria, L. (2015). Test control algorithms for the validation of cyber-physical systems product lines. In Proceedings of the 19th international conference on software product line. SPLC ’15. ACM, New York, pp. 273–282.

  • Bak, K., Diskin, Z., Antkiewicz, M., Czarnecki, K., & Wasowski, A. (2015). Clafer: Unifying class and feature modeling. Software & Systems Modeling, 15(3), 811–845.

    Google Scholar 

  • Behjati, R., Yue, T., Briand, L., & Selic, B. (2013). SIMPL: A product-line modeling methodology for families of integrated control systems. Information and Software Technology, 55(3), 607–629. (special issue on software reuse and product lines).

    Article  Google Scholar 

  • Benavides, D., Segura, S., & Ruiz-Cors, A. (2010). Automated analysis of feature models 20 years later: A literature review. Information Systems, 35(6), 615–636.

    Article  Google Scholar 

  • Berger, T., Rublack, R., Nair, D., Atlee, J.M., Becker, M., Czarnecki, K., & Wasowski, A. (2013). A survey of variability modeling in industrial practice. In Proceedings of the seventh international workshop on variability modelling of software-intensive systems (pp. 7:1–7:8). New York, NY: ACM.

  • Briand, L., Nejati, S., Sabetzadeh, M., & Bianculli, D. (2016). Testing the untestable: Model testing of complex software-intensive systems. In Proceedings of the 38th international conference on software engineering companion. ICSE ’16. ACM, New York, pp. 789–792.

  • Bringmann, E., & Kramer, A. (2008). Model-based testing of automotive systems. In Proceedings of the 1st international conference on software testing, verification and validation, pp. 485 – 493.

  • Bristeau, P. J., Callou, F., Vissière, D., & Petit, N. (2011). The navigation and control technology inside the AR.Drone micro UAV. IFAC Proceedings Volumes, 44(1), 1477–1484.

    Article  Google Scholar 

  • Chvatal, V. (1979). A greedy heuristic for the set-covering problem. Mathematics of operations research, 4(3), 233–235.

    Article  MathSciNet  MATH  Google Scholar 

  • Derler, P., Lee, E. A., & Sangiovanni-Vincentelli, A. (2011). Modeling cyber-physical systems. Proceedings of the ieee (special issue on cps), 100(1), 13–28. (special issue on cps)

    Article  Google Scholar 

  • Dukaczewski, M., Schaefer, I., Lachmann, R., & Lochau, M. (2013). Requirements-based delta-oriented SPL testing. 4th International workshop on product line approaches in software engineering, please 2013, pp. 49–52.

  • Dziobek, C., Loew, J., Przystas, W., & Weiland, J. (2008). Functional variants handling in simulink models. Mathworks, Technical report.

  • Eidson, J. C., Lee, E. A., & Matic, S. (2011). Distributed real-time software for cyber-physical systems. Proceedings of the IEEE, 100, 45–59.

    Article  Google Scholar 

  • FreeRTOS. (2014). FreeRTOS. http://www.freertos.org

  • Garvin, B. J., Cohen, M. B., & Dwyer, M. B. (2011). Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Software Engineering, 16(1), 61–102.

    Article  Google Scholar 

  • Jensen, Jeff C., Chang, Danica, & Lee, Edward A. (2011) (July). A model-based design methodology for cyber-physical systems. 2011 7th International wireless communications and mobile computing conference (IWCMC), pp. 1666 – 1671.

  • Johansen, M. F., Haugen, Ø., & Fleurey, F. (2012). An algorithm for generating t-wise covering arrays from large feature models. In Proceedings of the 16th international software product line conference, vol. 1. SPLC ’12. ACM, New York, NY, pp. 46–55.

  • Kane, A., Fuhrman, T. E., & Koopman, P. (2014). Monitor based oracles for cyber-physical system testing: Practical experience report. 44th annual IEEE/IFIP international conference on dependable systems and networks, DSN 2014, Atlanta, June 23–26, 2014, pp. 148–155 .

  • Khakpour, N., & Mousavi, M. R. (2015). Notions of conformance testing for cyber-physical systems: Overview and roadmap (invited paper). In L. Aceto & de Frutos Escrig, D. (Eds.), Leibniz international proceedings in 26th international conference on concurrency theory (concur 2015). Informatics (LIPIcs) (vol. 42). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, pp. 18–40.

  • Kuhn, R., Kacker, R., Lei, Y., & Hunter, J. (2009). Combinatorial software testing. Computer, 42, 94–96.

    Article  Google Scholar 

  • Lee E. A.(2007). Computing foundations and practice for cyber-physical systems: A preliminary report. Berkeley: University of California.

    Google Scholar 

  • Lee, E. A. (2008). Cyber physical systems: Design challenges. 2008 11th IEEE international symposium on object oriented real-time distributed computing (ISORC). IEEE, pp. 363–369.

  • Lee, E. A., & Seshia, S. A. (2015). Introduction to embedded systems, a cyber-physical systems approach (2 ed.). http://LeeSeshia.org

  • Lity, S., Lochau, M., Schaefer, I., & Goltz, U. (2012). Delta-oriented model-based SPL regression testing. 3rd International workshop on product line approaches in software engineering, please 2012. pp. 53–56.

  • Lopez-Herrejon, R. E., Linsbauer, L., & Egyed, A. (2015). A systematic mapping study of search-based software engineering for software product lines. Information and software technology, 61, 33–51.

    Article  Google Scholar 

  • Lu, H., Yue, T., Ali, S., & Zhang, L. (2016a). Integrating search and constraint solving for nonconformity resolving recommendations of system product line configuration. IEEE international conference on software testing, verification and validation (ICST).

  • Lu, H., Yue, T., Ali, S., & Zhang, L. (2016). Model-based incremental conformance checking to enable interactive product configuration. Information and Software Technology, 72(04/2016), 68–89.

    Article  Google Scholar 

  • Lu, Z., Bezemer, M. M., & Broenink, J. F. (2015). Model-driven design of simulation support for the terra robot software tool suite. Communicating process architectures 2015, pp. 143–158.

  • Manz, C., Schulze, M., & Reichert, M. (2014). An approach to detect the origin and distribution of software defects in an evolving cyber-physical system. In Workshop on emerging ideas and trends in engineering of cyber-physical systems (EITEC ’14).

  • MathWorks. (2016, March). http://es.mathworks.com/discovery/cyber-physical-systems.html.

  • Matinnejad, R., Nejati, S., Briand, L. C., & Bruckmann, T. (2016). Automated test suite generation for time-continuous simulink models. In Proceedings of the 38th international conference on software engineering. ICSE ’16. ACM, New York, pp. 595–606.

  • Mjeda, A. (2013). Standard-compliant testing for safety-related automotive software. Ph.D. thesis, University of Limeric.

  • Mohaqeqi, M., Mousavi, M. R., & Taha, W. (2014). Conformance testing of cyber-physical systems: A comparative study. ECEASST, 70, 1–16.

    Google Scholar 

  • Mosterman, P. J., & Zander, J. (2016). Cyber-physical systems challenges: A needs analysis for collaborating embedded software systems. Software & Systems Modeling, 15, 5–16. ISSN:1619-1366.

  • Mosterman, P. J., Sanabria, D. E., Bilgin, E., Zhang, K., & Zander, J. (2014). Automating humanitarian missions with a heterogeneous fleet of vehicles. Annual Reviews in Control, 38(2), 259–270.

    Article  Google Scholar 

  • Mueller, W., Becker, M., Elfeky, A., & DiPasquale, A. (2012). Virtual prototyping of cyber-physical systems. Asia and south pacific design automation conference, pp. 219–226.

  • Nie, K., Yue, T., Ali, S., Zhang, L., & Fan, Z. (2013). Constraints: The core of supporting automated product configuration of cyber-physical systems. Acm/IEEE 16th international conference on model driven engineering languages and systems, pp. 370–387.

  • Pérez, B., Polo, M., & García, I. (2009a). Model-driven testing in software product lines. In Proceedings of the 2009 IEEE international conference on software maintenance (ICSM 2009), pp. 511–514.

  • Pérez, B., Polo, M., & Piattini, M. (2009b). Towards an automated testing framework to manage variability using the UML testing profile. Ast, pp. 10–17.

  • Perrouin, G., Sen, S., Klein, J., Baudry, B., & Le Traon, Y. (2010). Automated and scalable t-wise test case generation strategies for software product lines. 2010 Third international conference on software testing, verification and validation. IEEE, pp. 459–468.

  • Piketec. (2015, July). TPT—model-based testing of embedded control systems.

  • Polzer, A., Merschen, D., Botterweck, G., Pleuss, A., Thomas, J., Hedenetz, B., et al. (2012). Managing complexity and variability of a model-based embedded software product line. Innovations and Systems and Software Engineering, 8(1), 35–49.

    Article  Google Scholar 

  • Pure-Systems. (2014, July). pure::variants. http://www.pure-systems.com.

  • Robinson, W. (2010). A roadmap for comprehensive requirements modeling. Computer, 43(5), 64–72.

    Article  Google Scholar 

  • Sánchez, A. B., Segura, S., & Ruiz-Cortés, A. (2014). A comparison of test case prioritization criteria for software product lines. IEEE international conference on software testing, verification, and validation, pp. 41–50.

  • Shi, J., Wan, J., Yan, He., & Suo, H. (2011). A survey of cyber-physical systems. In Proceedings of the international conference on wireless communications and signal processing, pp. 1–6.

  • Shokry, H., & Hinchey, M. (2009). Model-based verification of embedded software. Computer, 42(4), 53–59.

    Article  Google Scholar 

  • Thiel, S., & Hein, A. (2002). Systematic integration of variability into product line architecture design. Splc, pp. 130–153.

  • Thuem, T., Kastner, C., Benduhn, F., Meinicke, J., Saake, G., & Leich, T. (2014). Featureide: An extensible framework for feature-oriented software development. Science of Computer Programming, 79, 70–85.

    Article  Google Scholar 

  • Tidwell, T., Gao, X., Huang, H.-M., Lu, C., Dyke, S., & Gill, C. (2009). Towards configurable real-time hybrid structural testing: A cyber-physical systems approach. ISORC, pp. 37–44.

  • Wan, J., Suo, H., Yan, H., & Liu, J. (2011). A general test platform for cyber-physical systems: Unmanned vehicle with wireless sensor network navigation. In Procedia engineering of international conference on advances in engineering 2011 (vol. 24). pp. 123– 127.

  • Wang, S., Gotlieb, A., Ali, S., & Liaaen, M. (2013a). Automated test case selection using feature model: An industrial case study. Models, pp. 237–253.

  • Wang, S., Ali, S., & Gotlieb, A. (2013b). Minimizing test suites in software product lines using weight-based genetic algorithms. In Proceedings of the 2013 genetic and evolutionary computation conference, pp. 1493–1500.

  • Wang, S., Buchmann, D., Ali, S., Gotlieb, A., Pradhan, D., & Liaaen, M. (2014). Multi-objective test prioritization in software product line testing: An industrial case study. In Proceedings of the 18th international software product line conference (vol. 1). SPLC ’14. ACM, New York, pp. 32–41.

  • Wang, S., Ali, S., Gotlieb, A., & Liaaen, M. (2016). A systematic test case selection methodology for product lines: Results and insights from an industrial case study. Empirical Software Engineering, 21(4), 1586–1622.

    Article  Google Scholar 

  • Wang, S., Ali, S., & Gotlieb, A. (2015). Cost-effective test suite minimization in product lines using search techniques. Journal of Systems and Software, 103, 370–391.

    Article  Google Scholar 

  • Wetter, M., & Haves, P. (2008). A modular building controls virtual test bed for the integration of heterogeneous systems. In Proceedings of the 3rd simbuild conference, pp. 69–76.

  • Zander-Nowicka, J. (2007). Reactive testing and test control of hybrid embedded software. In Proceedings of the 5th workshop on system testing and validation, pp. 45–62.

  • Zander-Nowicka, J. (2008). Model-based testing of real-time embedded systems in the automotive domain. Ph.D. thesis. Berlin: Technical University.

    Google Scholar 

  • Zhang, M., Selic, B., Ali, S., Yue, T., Okariz, O., & Norgren, R. (2016). Understanding uncertainty in cyber-physical systems: A conceptual model. Cham: Springer.

    Google Scholar 

Download references

Acknowledgments

This work was developed by the embedded systems group of Mondragon Goi Eskola Politektnikoa, supported by the Department of Education, Universities and Research of the Basque Government.

Author information

Authors and Affiliations

  1. Goiru 2, 20500, Arrasate, Spain

    Aitor Arrieta, Goiuria Sagardui, Leire Etxeberria & Justyna Zander

Authors
  1. Aitor Arrieta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Goiuria Sagardui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leire Etxeberria
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Justyna Zander
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aitor Arrieta.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Arrieta, A., Sagardui, G., Etxeberria, L. et al. Automatic generation of test system instances for configurable cyber-physical systems. Software Qual J 25, 1041–1083 (2017). https://doi.org/10.1007/s11219-016-9341-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-016-9341-7

Keywords

Search

Navigation