Abstract
Cyber-physical systems (CPSs) are ubiquitous systems that integrate digital technologies with physical processes. These systems are becoming configurable to respond to the different needs that users demand. As a consequence, their variability is increasing, and they can be configured in many system variants. To ensure a systematic test execution of CPSs, a test system must be elaborated encapsulating several sources such as test cases or test oracles. Manually building a test system for each configuration is a non-systematic, time-consuming, and error-prone process. To overcome these problems, we designed a test system for testing CPSs and we analyzed the variability that it needed to test different configurations. Based on this analysis, we propose a methodology supported by a tool named ASTERYSCO that automatically generates simulation-based test system instances to test individual configurations of CPSs. To evaluate the proposed methodology, we selected different configurations of a configurable Unmanned Aerial Vehicle, and measured the time required to generate their test systems. On average, around 119 s were needed by our tool to generate the test system for 38 configurations. In addition, we compared the process of generating test system instances between the method we propose and a manual approach. Based on this comparison, we believe that the proposed tool allows a systematic method of generating test system instances. We believe that our approach permits an important step toward the full automation of testing in the field of configurable CPSs.




















Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
With variability we refer to configurability, i.e., variability in product space (Thiel and Hein 2002).
We denominate this term as system variant, configuration or system configuration, across the whole article.
For presentation purposes of the article, two independent feature models have been developed.
For presentation reasons of the article, we could not allocate every single feature of the test system.
This step is not mandatory for the manual configuration.
References
Ali, S., & Yue, T. (2015). U-test: Evolving, modelling and testing realistic uncertain behaviors of cyber-physical systems. 8th IEEE international conference on software testing, verification and validation, ICST 2015, Graz, Austria, April 13–17, 2015, pp. 1–2.
Arcuri, A., & Briand, L. (2011). A practical guide for using statistical tests to assess randomized algorithms in software engineering. 2011 33rd International conference on software engineering (ICSE). IEEE, pp. 1–10.
Arrieta, A., Sagardui, G., & Etxeberria, L. (2014). A configurable test architecture for the automatic validation of variability-intensive cyber-physical systems. Valid 2014: The sixth international conference on advances in system testing and validation lifecycle, pp. 79–83.
Arrieta, A., Sagardui, G., & Etxeberria, L. (2015). Test control algorithms for the validation of cyber-physical systems product lines. In Proceedings of the 19th international conference on software product line. SPLC ’15. ACM, New York, pp. 273–282.
Bak, K., Diskin, Z., Antkiewicz, M., Czarnecki, K., & Wasowski, A. (2015). Clafer: Unifying class and feature modeling. Software & Systems Modeling, 15(3), 811–845.
Behjati, R., Yue, T., Briand, L., & Selic, B. (2013). SIMPL: A product-line modeling methodology for families of integrated control systems. Information and Software Technology, 55(3), 607–629. (special issue on software reuse and product lines).
Benavides, D., Segura, S., & Ruiz-Cors, A. (2010). Automated analysis of feature models 20 years later: A literature review. Information Systems, 35(6), 615–636.
Berger, T., Rublack, R., Nair, D., Atlee, J.M., Becker, M., Czarnecki, K., & Wasowski, A. (2013). A survey of variability modeling in industrial practice. In Proceedings of the seventh international workshop on variability modelling of software-intensive systems (pp. 7:1–7:8). New York, NY: ACM.
Briand, L., Nejati, S., Sabetzadeh, M., & Bianculli, D. (2016). Testing the untestable: Model testing of complex software-intensive systems. In Proceedings of the 38th international conference on software engineering companion. ICSE ’16. ACM, New York, pp. 789–792.
Bringmann, E., & Kramer, A. (2008). Model-based testing of automotive systems. In Proceedings of the 1st international conference on software testing, verification and validation, pp. 485 – 493.
Bristeau, P. J., Callou, F., Vissière, D., & Petit, N. (2011). The navigation and control technology inside the AR.Drone micro UAV. IFAC Proceedings Volumes, 44(1), 1477–1484.
Chvatal, V. (1979). A greedy heuristic for the set-covering problem. Mathematics of operations research, 4(3), 233–235.
Derler, P., Lee, E. A., & Sangiovanni-Vincentelli, A. (2011). Modeling cyber-physical systems. Proceedings of the ieee (special issue on cps), 100(1), 13–28. (special issue on cps)
Dukaczewski, M., Schaefer, I., Lachmann, R., & Lochau, M. (2013). Requirements-based delta-oriented SPL testing. 4th International workshop on product line approaches in software engineering, please 2013, pp. 49–52.
Dziobek, C., Loew, J., Przystas, W., & Weiland, J. (2008). Functional variants handling in simulink models. Mathworks, Technical report.
Eidson, J. C., Lee, E. A., & Matic, S. (2011). Distributed real-time software for cyber-physical systems. Proceedings of the IEEE, 100, 45–59.
FreeRTOS. (2014). FreeRTOS. http://www.freertos.org
Garvin, B. J., Cohen, M. B., & Dwyer, M. B. (2011). Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Software Engineering, 16(1), 61–102.
Jensen, Jeff C., Chang, Danica, & Lee, Edward A. (2011) (July). A model-based design methodology for cyber-physical systems. 2011 7th International wireless communications and mobile computing conference (IWCMC), pp. 1666 – 1671.
Johansen, M. F., Haugen, Ø., & Fleurey, F. (2012). An algorithm for generating t-wise covering arrays from large feature models. In Proceedings of the 16th international software product line conference, vol. 1. SPLC ’12. ACM, New York, NY, pp. 46–55.
Kane, A., Fuhrman, T. E., & Koopman, P. (2014). Monitor based oracles for cyber-physical system testing: Practical experience report. 44th annual IEEE/IFIP international conference on dependable systems and networks, DSN 2014, Atlanta, June 23–26, 2014, pp. 148–155 .
Khakpour, N., & Mousavi, M. R. (2015). Notions of conformance testing for cyber-physical systems: Overview and roadmap (invited paper). In L. Aceto & de Frutos Escrig, D. (Eds.), Leibniz international proceedings in 26th international conference on concurrency theory (concur 2015). Informatics (LIPIcs) (vol. 42). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, pp. 18–40.
Kuhn, R., Kacker, R., Lei, Y., & Hunter, J. (2009). Combinatorial software testing. Computer, 42, 94–96.
Lee E. A.(2007). Computing foundations and practice for cyber-physical systems: A preliminary report. Berkeley: University of California.
Lee, E. A. (2008). Cyber physical systems: Design challenges. 2008 11th IEEE international symposium on object oriented real-time distributed computing (ISORC). IEEE, pp. 363–369.
Lee, E. A., & Seshia, S. A. (2015). Introduction to embedded systems, a cyber-physical systems approach (2 ed.). http://LeeSeshia.org
Lity, S., Lochau, M., Schaefer, I., & Goltz, U. (2012). Delta-oriented model-based SPL regression testing. 3rd International workshop on product line approaches in software engineering, please 2012. pp. 53–56.
Lopez-Herrejon, R. E., Linsbauer, L., & Egyed, A. (2015). A systematic mapping study of search-based software engineering for software product lines. Information and software technology, 61, 33–51.
Lu, H., Yue, T., Ali, S., & Zhang, L. (2016a). Integrating search and constraint solving for nonconformity resolving recommendations of system product line configuration. IEEE international conference on software testing, verification and validation (ICST).
Lu, H., Yue, T., Ali, S., & Zhang, L. (2016). Model-based incremental conformance checking to enable interactive product configuration. Information and Software Technology, 72(04/2016), 68–89.
Lu, Z., Bezemer, M. M., & Broenink, J. F. (2015). Model-driven design of simulation support for the terra robot software tool suite. Communicating process architectures 2015, pp. 143–158.
Manz, C., Schulze, M., & Reichert, M. (2014). An approach to detect the origin and distribution of software defects in an evolving cyber-physical system. In Workshop on emerging ideas and trends in engineering of cyber-physical systems (EITEC ’14).
MathWorks. (2016, March). http://es.mathworks.com/discovery/cyber-physical-systems.html.
Matinnejad, R., Nejati, S., Briand, L. C., & Bruckmann, T. (2016). Automated test suite generation for time-continuous simulink models. In Proceedings of the 38th international conference on software engineering. ICSE ’16. ACM, New York, pp. 595–606.
Mjeda, A. (2013). Standard-compliant testing for safety-related automotive software. Ph.D. thesis, University of Limeric.
Mohaqeqi, M., Mousavi, M. R., & Taha, W. (2014). Conformance testing of cyber-physical systems: A comparative study. ECEASST, 70, 1–16.
Mosterman, P. J., & Zander, J. (2016). Cyber-physical systems challenges: A needs analysis for collaborating embedded software systems. Software & Systems Modeling, 15, 5–16. ISSN:1619-1366.
Mosterman, P. J., Sanabria, D. E., Bilgin, E., Zhang, K., & Zander, J. (2014). Automating humanitarian missions with a heterogeneous fleet of vehicles. Annual Reviews in Control, 38(2), 259–270.
Mueller, W., Becker, M., Elfeky, A., & DiPasquale, A. (2012). Virtual prototyping of cyber-physical systems. Asia and south pacific design automation conference, pp. 219–226.
Nie, K., Yue, T., Ali, S., Zhang, L., & Fan, Z. (2013). Constraints: The core of supporting automated product configuration of cyber-physical systems. Acm/IEEE 16th international conference on model driven engineering languages and systems, pp. 370–387.
Pérez, B., Polo, M., & García, I. (2009a). Model-driven testing in software product lines. In Proceedings of the 2009 IEEE international conference on software maintenance (ICSM 2009), pp. 511–514.
Pérez, B., Polo, M., & Piattini, M. (2009b). Towards an automated testing framework to manage variability using the UML testing profile. Ast, pp. 10–17.
Perrouin, G., Sen, S., Klein, J., Baudry, B., & Le Traon, Y. (2010). Automated and scalable t-wise test case generation strategies for software product lines. 2010 Third international conference on software testing, verification and validation. IEEE, pp. 459–468.
Piketec. (2015, July). TPT—model-based testing of embedded control systems.
Polzer, A., Merschen, D., Botterweck, G., Pleuss, A., Thomas, J., Hedenetz, B., et al. (2012). Managing complexity and variability of a model-based embedded software product line. Innovations and Systems and Software Engineering, 8(1), 35–49.
Pure-Systems. (2014, July). pure::variants. http://www.pure-systems.com.
Robinson, W. (2010). A roadmap for comprehensive requirements modeling. Computer, 43(5), 64–72.
Sánchez, A. B., Segura, S., & Ruiz-Cortés, A. (2014). A comparison of test case prioritization criteria for software product lines. IEEE international conference on software testing, verification, and validation, pp. 41–50.
Shi, J., Wan, J., Yan, He., & Suo, H. (2011). A survey of cyber-physical systems. In Proceedings of the international conference on wireless communications and signal processing, pp. 1–6.
Shokry, H., & Hinchey, M. (2009). Model-based verification of embedded software. Computer, 42(4), 53–59.
Thiel, S., & Hein, A. (2002). Systematic integration of variability into product line architecture design. Splc, pp. 130–153.
Thuem, T., Kastner, C., Benduhn, F., Meinicke, J., Saake, G., & Leich, T. (2014). Featureide: An extensible framework for feature-oriented software development. Science of Computer Programming, 79, 70–85.
Tidwell, T., Gao, X., Huang, H.-M., Lu, C., Dyke, S., & Gill, C. (2009). Towards configurable real-time hybrid structural testing: A cyber-physical systems approach. ISORC, pp. 37–44.
Wan, J., Suo, H., Yan, H., & Liu, J. (2011). A general test platform for cyber-physical systems: Unmanned vehicle with wireless sensor network navigation. In Procedia engineering of international conference on advances in engineering 2011 (vol. 24). pp. 123– 127.
Wang, S., Gotlieb, A., Ali, S., & Liaaen, M. (2013a). Automated test case selection using feature model: An industrial case study. Models, pp. 237–253.
Wang, S., Ali, S., & Gotlieb, A. (2013b). Minimizing test suites in software product lines using weight-based genetic algorithms. In Proceedings of the 2013 genetic and evolutionary computation conference, pp. 1493–1500.
Wang, S., Buchmann, D., Ali, S., Gotlieb, A., Pradhan, D., & Liaaen, M. (2014). Multi-objective test prioritization in software product line testing: An industrial case study. In Proceedings of the 18th international software product line conference (vol. 1). SPLC ’14. ACM, New York, pp. 32–41.
Wang, S., Ali, S., Gotlieb, A., & Liaaen, M. (2016). A systematic test case selection methodology for product lines: Results and insights from an industrial case study. Empirical Software Engineering, 21(4), 1586–1622.
Wang, S., Ali, S., & Gotlieb, A. (2015). Cost-effective test suite minimization in product lines using search techniques. Journal of Systems and Software, 103, 370–391.
Wetter, M., & Haves, P. (2008). A modular building controls virtual test bed for the integration of heterogeneous systems. In Proceedings of the 3rd simbuild conference, pp. 69–76.
Zander-Nowicka, J. (2007). Reactive testing and test control of hybrid embedded software. In Proceedings of the 5th workshop on system testing and validation, pp. 45–62.
Zander-Nowicka, J. (2008). Model-based testing of real-time embedded systems in the automotive domain. Ph.D. thesis. Berlin: Technical University.
Zhang, M., Selic, B., Ali, S., Yue, T., Okariz, O., & Norgren, R. (2016). Understanding uncertainty in cyber-physical systems: A conceptual model. Cham: Springer.
Acknowledgments
This work was developed by the embedded systems group of Mondragon Goi Eskola Politektnikoa, supported by the Department of Education, Universities and Research of the Basque Government.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Arrieta, A., Sagardui, G., Etxeberria, L. et al. Automatic generation of test system instances for configurable cyber-physical systems. Software Qual J 25, 1041–1083 (2017). https://doi.org/10.1007/s11219-016-9341-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-016-9341-7