Abstract
This article addresses the need of applying testing activities with the purpose of checking the consistency and the correctness of distributed and dynamically adaptable systems whose structure and behavior evolve dynamically at runtime. We identified in the literature a number of studies that used some traditional testing techniques (e.g., regression testing) applied at development time and runtime verification techniques (e.g., runtime monitoring) to ensure the trustworthiness of software systems. Since traditional testing techniques are no longer adequate for dynamically adaptable systems because some behavioral/structural evolutions are unknown at development time, the runtime testing was introduced (Brenner et al., 2007). It is defined as any testing method that has to be carried out on the final execution environment by performing tests during deployment or in-service time. However, there is a lack of recent comprehensive survey on this emerging verification an validation technique. To fill this gap, we conduct a survey to investigate research done in this area. A list of 43 research studies are deeply discussed to address answers of eight research questions. In particular, this work gives the runtime testing taxonomy, challenges and introduces several approaches dealing with such technique and discusses challenges as well as recent advances in this area. Moreover, it also points out some future directions.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
A fault is a physical imperfection, mistake, or defect which happens in hardware or software.
An error is a part of the system’s state which could lead to failures.
A failure occurs when an error affects the service interface and the delivered service diverges from its nominal defined tasks.
AOP is one of the most suitable solutions that distinguish cross-cutting issues (i.e., including logging, tracking, checking, persistence, etc.) from a software system’s functional code.
Online MBT means that tests are generated at the same time as they are executed on the SUT.
Similar to the MAPE-K loop (Kephart and Chess et al., 2003) which manages the conception and the execution of autonomic systems, MAPE-T allows to monitor, analyze, plan and execute runtime test adaptations.
References
Akour, M., Jaidev, A., & King, T. M. (2011). Towards Change Propagating Test Models in Autonomic and Adaptive Systems. In Proceedings of the 18th IEEE International Conference and Workshops on Engineering of Computer-Based Systems (ECBS’11), pp. 89–96.
Alkawaz, M. H., & Silvarajoo, A. (2019). A survey on test case prioritization and optimization techniques in software regression testing. 2019 IEEE 7th Conference on Systems, Process and Control (ICSPC), 59–64.
Angelis, F. D., Berardini, M. R. D., Muccini, H., & Polini, A. (2014). CASSANDRA : An Online Failure Prediction Strategy for Dynamically Evolving Systems. In Proceedings of the 16th International Conference on Formal Engineering Methods (ICFEM’14), pp. 107–122.
Bai, X., Dai, G., Xu, D., & Tsai, W. T. (2006). A Multi-Agent Based Framework for Collaborative Testing on Web Services. In Proceedings of the 4th IEEE Workshop on Software Technologies for Future Embedded and Ubiquitous Systems, and the 2nd International Workshop on Collaborative Computing, Integration, and Assurance (SEUS-WCCIA’06), pp. 205–210.
Bai, X., Xu, D., Dai, G., Tsai, W. T., & Chen, Y. (2007). Dynamic Reconfigurable Testing of Service-Oriented Architecture. In Proceeding of the 31st Annual International Computer Software and Applications Conference (COMPSAC’07), pp. 368–378.
Basson, H., Bouneffa, M., Matsuda, M., Ahmad, A., Chung, D., & Arai, E. (2019). Manufacturing software units: Iso 16300-3 main guidelines for interoperability verification and validation. In Enterprise Interoperability VIII. Springer, pp. 167–176.
Beese, J., Haki, M. K., Aier, S., & Winter, R. (2019). Simulation-based research in information systems. Business & Information Systems Engineering, 61(4), 503–521.
Behrmann, G., David, A., & Larsen, K. G. (2004). A Tutorial on UPPAAL. In Proceeding of the International School on Formal Methods for the Design of Computer, Communication, and Software Systems (SFM-RT’04), vol. 3185, pp. 200–237.
Bekesiene, S., Hubacekf, M., & Bures, M. (2019). Modelling possibilities of the vehicle movement on communication network for defense and crisis management. In 2019 International Conference on Military Technologies (ICMT), pp. 1–6.
Bershadsky, A., Bozhday, A., Evseeva, Y., & Gudkov, A. (2019). Methods of software self-adaptation based on monitoring the information environment. In Conference on Creativity in Intelligent Technologies and Data Science, Springer, pp. 122–131.
Bertolino, A., & Inverardi, P. (2019). Changing software in a changing world: How to test in presence of variability, adaptation and evolution? In From Software Engineering to Formal Methods and Tools, and Back. Springer, pp. 56–66.
Beszedes, A., Gergely, T., Schrettner, L., Jasz, J., Lango, L., & Gyimothy, T. (2012) Code Coverage-Based Regression Test Selection and Prioritization in WebKit. In 28th IEEE International Conference on Software Maintenance (ICSM’12), pp. 46–55.
Biswas, S., Mall, R., Satpathy, M., & Sukumaran, S. (2011). Regression test selection techniques: A survey. Informatica (Ljubljana) 35.
Borky, J. M., & Bradley, T. H. (2019). Using prototypes, verification, and validation to evaluate and enhance system architecture. In Effective Model-Based Systems Engineering. Springer, pp. 405–435.
Bradatsch, C., Ungerer, T., Zalman, R., & Lajtkep, A. (2011). Towards runtime testing in automotive embedded systems. In 2011 6th IEEE International Symposium on Industrial and Embedded Systems (2011), pp. 55–58.
Brenner, D., Atkinson, C., Malaka, R., Merdes, M., Paech, B., & Suliman, D. (2007). Reducing Verification Effort in Component-based Software Engineering Through Built-In Testing. Information Systems Frontiers, 9(2–3), 151–162.
Briand, L. C., Labiche, Y., & He, S. (2009). Automating Regression Test Selection Based on UML Designs. Information & Software Technology, 51(1), 16–30.
Bruneton, E., Coupaye, T., Leclercq, M., Quéma, V., & Stefani, J. B. (2004). An Open Component Model and Its Support in Java. In Proceeding of the 7th International International ACM Sigsoft Symposium on Component-Based Software Engineering (CBSE’04), pp. 7–22.
Buranapanichkit, D., Jindapetch, N., Thongpull, K., Thongnoo, K., Chetpattananondh, K., Duangsoithong, R., & Sengchuai, K. (2019). A patient monitoring system for multiple iot rehabilitation devices. In 16th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology, pp. 834–837.
Chaari, T., & Fakhfakh, K. (2012). Semantic Modeling and Reasoning at Runtime for Autonomous Systems Engineering. In Proceeding of the 9th International Conference on Ubiquitous Intelligence Computing and Autonomic Trusted Computing (UIC/ATC’12), pp. 415–422.
Chakraborty, J., Pandit, D., Xia, J. C., & Chan, F. (2019). A protocol for simulation modeling of ridesourcing services: Optimisation of fleet size in an urban environment. International Journal of Intelligent Transportation Systems Research, 1–10.
Chen, Y., Probert, R. L., and Ural, H. Model-based Regression Test Suite Generation Using Dependence Analysis. In Proceedings of the 3rd International Workshop on Advances in Model-based Testing (A-MOST’07) (2007), pp. 54–62.
Cheng, B., Eder, K., Gogolla, M., Grunske, L., Litoiu, M., Müller, H., Pelliccione, P., Perini, A., Qureshi, N., Rumpe, B., Schneider, D., Trollmann, F., & Villegas, N. (2014). Using Models at Runtime to Address Assurance for Self-Adaptive Systems. In Models@run.time, pp. 101–136.
Chu, L., Shen, K., Tang, H., Yang, T., & Zhou, J. (2005). Dependency Isolation for Thread-based Multi-tier Internet Services. In Proceeding of the 24th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM’05), pp. 796–806.
Camara, J., de Lemos, R., Laranjeiro, N., Ventura, R., & Vieira, M. (2017). Robustness-driven resilience evaluation of self-adaptive software systems. IEEE Transactions on Dependable and Secure Computing, 14(1), 50–64. https://doi.org/10.1109/TDSC.2015.2429128.
Camara, J., Lemos, R., Laranjeiro, N., Ventura, R., & Vieira, M. (2014). Testing the robustness of controllers for self-adaptive systems. Journal of the Brazilian Computer Society 20. https://doi.org/10.1186/1678-4804-20-1
de Sousa Santos, I., de Castro Andrade, R. M., Rocha, L. S., Matalonga, S., de Oliveira, K. M., & Travassos, G. H. (2017). Test case design for context-aware applications: Are we there yet? Information and Software Technology, 88, 1–16.
de Sousa Santos, I., de Jesus Souza, M. L., Luciano Carvalho, M. L., Alves Oliveira, T., de Almeida, E. S., & de Castro Andrade, R. M. (2017). Dynamically adaptable software is all about modeling contextual variability and avoiding failures. IEEE Software, 34(6), 72–77.
Deussen, P. H., Din, G., & Schieferdecker, I. (2003). A TTCN-3 Based Online Test and Validation Platform for Internet Services. In Proceedings of the 6th International Symposium on Autonomous Decentralized Systems (ISADS’03).
Din, G., Tolea, S., & Schieferdecker, I. (2006). Distributed Load Tests with TTCN-3. In Proceedings of the 18th IFIP TC6/WG6.1 International Conference for Testing of Communicating Systems (TestCom’06), pp. 177–196.
dos Santos, E. B., Andrade, R. M., & de Sousa Santos, I. (2020). Runtime testing of context-aware variability in adaptive systems. Information and Software Technology, 106482. http://www.sciencedirect.com/science/article/pii/S0950584920301002
Eberhardinger, B. (2015). Testing Self-organizing, Adaptive Systems. In Proceedings of the IEEE International Conference on Self-Adaptive and Self-Organizing Systems Workshops (SASOW’15), pp. 140–145.
Eberhardinger, B., Habermaier, A., & Reif, W. (2017). Toward adaptive, self-aware test automation. In 2017 IEEE/ACM 12th International Workshop on Automation of Software Testing (AST), pp. 34–37.
Eberhardinger, B., Ponsar, H., Klumpp, D., & Reif, W. (2017). Test Case Selection Strategy for Self-Organization Mechanisms, 10.
Elqortobi, M., Bentahar, J., and Dssouli, R. (2018). Framework for dynamic web services composition guided by live testing. In Emerging Technologies for Developing Countries (Cham), F. Belqasmi, H. Harroud, M. Agueh, R. Dssouli, and F. Kamoun, Eds., Springer International Publishing, pp. 129–139.
Engström, E., & Runeson, P. A. (2010). qualitative survey of regression testing practices. In Product-Focused Software Process Improvement (Berlin, Heidelberg), M. Ali Babar, M. Vierimaa, and M. Oivo, Eds., Springer Berlin Heidelberg, pp. 3–16.
ETSI. (2005). Methods for Testing and Specification (MTS), The Testing and Test Control Notation version 3, Part 5: TTCN-3 Runtime Interface (TRI).
ETSI. (2005). Methods for Testing and Specification (MTS), The Testing and Test Control Notation version 3, Part 6: TTCN-3 Control Interface (TCI).
Farahmandi, F., Huang, Y., & Mishra, P. (2020) The future of security validation and verification. In System-on-Chip Security. Springer, pp. 273–279.
Felderer, M., & Garousi, V. (2020). Together we are stronger: Evidence-based reflections on industry-academia collaboration in software testing. In International Conference on Software Quality, Springer, pp. 3–12.
Ferrari, F., Cafeo, B., Noppen, J., Chitchyan, R., & Rashid, A. (2011). Investigating testing approaches for dynamically adaptive systems – work in progress.
Firouzi, F., & Farahani, B. (2020). Architecting iot cloud. In Intelligent Internet of Things. Springer, pp. 173–241.
Firouzi, F., Farahani, B., Weinberger, M., DePace, G., & Aliee, F. S. (2020). Iot fundamentals: Definitions, architectures, challenges, and promises. In Intelligent Internet of Things. Springer, pp. 3–50.
Fourneret, E., Bouquet, F., Dadeau, F., & Debricon, S. (2011). Selective Test Generation Method for Evolving Critical Systems. In Proceedings of the 2011 IEEE 4th International Conference on Software Testing, Verification and Validation Workshops (ICSTW’11), pp. 125–134.
Fredericks, E. M., & Cheng, B. H. C. (2015). Automated generation of adaptive test plans for self-adaptive systems. In 2015 IEEE/ACM 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, pp. 157–167. https://doi.org/10.1109/SEAMS.2015.15
Fredericks, E. M., DeVries, B., & Cheng, B. H. C. (2014). Towards Run-time Adaptation of Test Cases for Self-adaptive Systems in the Face of Uncertainty. In Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’14), pp. 17–26.
Fredericks, E. M., Ramirez, A. J., & Cheng, B. H. C. (2013). Towards Run-time Testing of Dynamic Adaptive Systems. In Proceedings of the 8th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’13), pp. 169–174.
Gao, L., Lu, M., Li, L., & Pan, C. (2017). A survey of software runtime monitoring. In 2017 8th IEEE International Conference on Software Engineering and Service Science (ICSESS), pp. 308–313. https://doi.org/10.1109/ICSESS.2017.8342921
González-Sanchez, A., Piel, É., & Gross, H. G. (2008). Architecture Support for Runtime Integration and Verification of Component-based Systems of Systems. In Proceeding of the Automated Software Engineering - Workshops, (ASE Workshops’08), pp. 41–48.
Granja, I., & Jino, M. (1999). Techniques for Regression Testing: Selecting Test Case Sets Tailored to Possibly Modified Functionalities. In Proceedings of the 3rd European Conference on Software Maintenance and Reengineering (CSMR’99), pp. 2–22.
Graves, T. L., Harrold, M. J., Kim, J.-M., Porter, A., & Rothermel, G. (2001). An Empirical Study of Regression Test Selection Techniques. ACM Transactions on Software Engineering and Methodology (TOSEM), 10(2), 184–208.
Greiler, M., Gross, H. G., & van Deursen, A. (2010). Evaluation of Online Testing for Services: A Case Study. In Proceeding of the 2nd International Workshop on Principles of Engineering Service-Oriented System, pp. 36–42.
Hänsel, J., Vogel, T., & Giese, H. (2015). A testing scheme for self-adaptive software systems with architectural runtime models. CoRR abs/1805.07354.
Harrold, M. J. (1998). Architecture-Based Regression Testing of Evolving Systems. In Proceeding of the International Workshop on the Role of Software Architecture in Testing and Analysis (ROSATEA’98), pp. 73–77.
Harrold, M. J. (2000). Testing: a roadmap. In Proceedings of the 16th IEEE Conference on The Future of Software Engineering (ICSE’00), pp. 61–72.
Hielscher, J., Kazhamiakin, R., Metzger, A., & Pistore, M. (2008). A Framework for Proactive Self-adaptation of Service-Based Applications Based on Online Testing. In Proceedings of the 1st European Conference on Towards a Service-Based Internet (ServiceWave’08), pp. 122–133.
Hansel, J., & Giese, H. (2017). Towards collective online and offline testing for dynamic software product line systems. In 2017 IEEE/ACM 2nd International Workshop on Variability and Complexity in Software Design (VACE), pp. 9–12. https://doi.org/10.1109/VACE.2017.5
Huang, S., Yang, S., Hui, Z., Yao, Y., Chen, L., Liu, J., & Chen, Q. (2019). Runtime-environment testing method for android applications. In 2019 IEEE 19th International Conference on Software Quality, Reliability and Security Companion (QRS-C), IEEE, pp. 534–535.
IEEE. (1990) IEEE Standard Glossary of Software Engineering Terminology.
Inverardi, P., & Mori, M. (2011). Model Checking Requirements at Run-time in Adaptive Systems. In Proceedings of the 8th Workshop on Assurances for Self-adaptive Systems (ASAS’11), pp. 5–9.
Jansen, K. (2009). Parametrized Approximation Scheme for The Multiple Knapsack Problem. In Proceedings of the 20th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA’09), pp. 665–674.
Jmal Malej, A., Krichen, M., & Jmaiel, M. (2015). A comparative evaluation of state-of-the-art load and stress testing approaches. International Journal of Computer Applications in Technology 51, 283. https://doi.org/10.1504/IJCAT.2015.070491
Johnsen, E. B., and Yu, I. C. (2018). Dynamic software updates and context adaptation for distributed active objects. In Principled Software Development. Springer, pp. 147–164.
Jussien, N., Rochart, G., and Lorca, X. (2008). Choco: an Open Source Java Constraint Programming Library. In Proceeding of the Workshop on Open-Source Software for Integer and Contraint Programming (OSSICP’08), pp. 1–10.
Kejstová, K., Rockai, P., and Barnat, J. (2018). From model checking to runtime verification and back. http://arxiv.org/abs/1805.12428
Kephart, J. O., & Chess, D. M. (2003). The Vision of Autonomic Computing. Computer, 36(1), 41–50.
Ketfi, A., Belkhatir, N., & yves Cunin, P. Dynamic Updating of Component-Based Applications. In Proceeding of the International Conference on Software Engineering Research and Practice (SERP’02) (2002).
Kim, J. U., & Kishore, R. (2019). Do we fully understand information systems failure? an exploratory study of the cognitive schema of is professionals. Information Systems Frontiers, 21(6), 1385–1419.
King, T. M., Allen, A. A., Cruz, R., & Clarke, P. J. (2011). Safe Runtime Validation of Behavioral Adaptations in Autonomic Software. In Proceedings of the 8th International Conference on Autonomic and Trusted Computing (ATC’11), pp. 31–46.
King, T. M., Allen, A. A., Wu, Y., Clarke, P. J., & Ramirez, A. E. (2011). A comparative case study on the engineering of self-testable autonomic software. In 2011 Eighth IEEE International Conference and Workshops on Engineering of Autonomic and Autonomous Systems, pp. 59–68.
King, T. M., Ramirez, A. E., Cruz, R., & Clarke, P. (2007). An integrated self-testing framework for autonomic computing systems. Journal on Computation, 2, 37–49. https://doi.org/10.1109/EASe.2011.16.
Korel, B., & Al-Yami, A. M. (1998). Automated Regression Test Generation. ACM SIGSOFT Software Engineering Notes, 23(2), 143–152.
Korel, B., Tahat, L., & Vaysburg, B. (2002). Model Based Regression Test Reduction Using Dependence Analysis. In Proceedings of the 18th IEEE International Conference on Software Maintenance (ICSM’02), pp. 214–223.
Kramer, J., & Magee, J. (1985). Dynamic Configuration for Distributed Systems. IEEE Transactions on Software Engineering (TSE), 11(4), 424–436.
Krichen, M. (2019). Improving formal verification and testing techniques for internet of things and smart cities. Mobile Networks and Applications, 1–12.
Krichen, M., & Alroobaea, R. (2019). A new model-based framework for testing security of iot systems in smart cities using attack trees and price timed automata. In 14th International Conference on Evaluation of Novel Approaches to Software Engineering - ENASE 2019.
Krichen, M., Alroobaea, R., & Lahami, M. (2019). Towards a runtime standard-based testing framework for dynamic distributed information systems. In 21st International Conference on Enterprise Information Systems - ICEIS 2019, vol. 1.
Krichen, M., & Lahami, M. (2020). Towards a runtime testing framework for dynamically adaptable internet of things networks in smart cities (pp. 589–607). In Smart Infrastructure and Applications. Springer: Cham.
Krichen, M., Lahami, M., Cheikhrouhou, O., Alroobaea, R., & Maâlej, A. J. (2020). Security testing of internet of things for smart city applications: A formal approach (pp. 629–653). In Smart Infrastructure and Applications. Springer: Cham.
Krichen, M., Maâlej, A. J., & Lahami, M. (2018). A model-based approach to combine conformance and load tests: an ehealth case study. International Journal of Critical Computer-Based Systems, 8(3–4), 282–310.
Krichen, M., Maâlej, A. J., Lahami, M., & Jmaiel, M. (2018). A resource-aware model-based framework for load testing of ws-bpel compositions. In International Conference on Enterprise Information Systems, Springer, Cham, pp. 130–157.
Krupitzer, C., Roth, F. M., VanSyckel, S., Schiele, G., & Becker, C. (2015). A survey on engineering approaches for self-adaptive systems. Pervasive and Mobile Computing 17, 184 – 206. 10 years of Pervasive Computing’ In Honor of Chatschik Bisdikian. https://doi.org/10.1016/j.pmcj.2014.09.009
Kuru, K., & Ansell, D. (2020). Tcitysmartf: A comprehensive systematic framework for transforming cities into smart cities. IEEE Access.
Lahami, M., Fakhfakh, F., Krichen, M., & Jmaïel, M. (2012). Towards a TTCN-3 Test System for Runtime Testing of Adaptable and Distributed Systems. In Proceedings of the 24th IFIP WG 6.1 International Conference Testing Software and Systems (ICTSS’12), pp. 71–86.
Lahami, M., & Krichen, M. (2013). Test Isolation Policy for Safe Runtime Validation of Evolvable Software Systems. In Proceedings of the 22nd IEEE International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE’13), pp. 377–382.
Lahami, M., Krichen, M., & Alroobaea, R. (2018). Towards a test execution platform as-a-service: Application in the e-health domain. In 2018 International Conference on Control, Automation and Diagnosis (ICCAD), IEEE, pp. 1–6.
Lahami, M., Krichen, M., & Alroobaea, R. (2019). Tepaas: test execution platform as-a-service applied in the context of e-health. International Journal of Autonomous and Adaptive Communications Systems, 12(3), 264–283.
Lahami, M., Krichen, M., Barhoumi, H., & Jmaïel, M. (2015). Selective Test Generation Approach for Testing Dynamic Behavioral Adaptations. In Proceedings of the 27th IFIP WG 6.1 International Conference on Testing Software and Systems (ICTSS’15), pp. 224–239.
Lahami, M., Krichen, M., Bouchakwa, M., & Jmaïel, M. (2012). Using Knapsack Problem Model to Design a Resource Aware Test Architecture for Adaptable and Distributed Systems. In Proceedings of the 24th IFIP WG 6.1 International Conference Testing Software and Systems (ICTSS’12), pp. 103–118.
Lahami, M., Krichen, M., & Jmaïel, M. (2016). Safe and Efficient Runtime Testing Framework Applied in Dynamic and Distributed Systems. Science of Computer Programming (SCP) 122, C, 1–28.
Laprie, J. C., Avizienis, A., & Kopetz, H. (Eds.). (1992). Dependability: Basic Concepts and Terminology. New York Inc: Springer-Verlag.
Leal, L., Ceccarelli, A., & Martins, E. (2019). The samba approach for self-adaptive model-based online testing of services orchestrations. In 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), vol. 1, pp. 495–500. https://doi.org/10.1109/COMPSAC.2019.00076
Leung, H., & White, L. (1989). Insights into regression testing [software testing]. In Proceedings of the International Conference on Software Maintenance (ICSM’89), pp. 60–69.
Li, W., Le Gall, F., & Spaseski, N. (2017). A survey on model-based testing tools for test case generation. pp. 77–89. https://doi.org/10.1007/978-3-319-71734-07
López, J. A., Herrera, V. I., Camblong, H., Milo, A., & Gaztañaga, H. (2019). Energy management improvement based on fleet digitalization data exploitation for hybrid electric buses. In Computational Intelligence and Optimization Methods for Control Engineering. Springer, pp. 321–355.
Lou, Y., Chen, J., Zhang, L., & Hao, D. (2019). Chapter one - a survey on regression test-case prioritization. vol. 113 of Advances in Computers. Elsevier, pp. 1 – 46. http://www.sciencedirect.com/science/article/pii/S0065245818300615
Maâlej, A. J., Lahami, M., Krichen, M., & Jmaïel, M. (2018). Distributed and resource-aware load testing of ws-bpel compositions. In ICEIS (2), pp. 29–38.
Macias-Escriva, F. D., Haber, R., del Toro, R., & Hernandez, V. (2013). Self-adaptive systems: A survey of current approaches, research challenges and applications. Expert Systems with Applications, 40(18), 7267–7279. http://www.sciencedirect.com/science/article/pii/S0957417413005125
Mangharam, R. (2019). Fly-by-logic: A tool for unmanned aircraft system fleet planning using temporal logic. NASA Formal Methods. Springer, 355.
Mao, C. (2007). AOP-based Testability Improvement for Component-based Software. In Proceedings of the 31st Annual International Computer Software and Applications Conference (COMPSAC ’07), pp. 547–552.
Marathe, S., Zeeshan, D., Thomas, T., & Vidhya, S. (2019). A wireless patient monitoring system using integrated ecg module, pulse oximeter, blood pressure and temperature sensor. In 2019 International Conference on Vision Towards Emerging Trends in Communication and Networking (ViTECoN), pp. 1–4.
Matalonga, S., Rodrigues, F., & Travassos, G. (2015). Challenges in testing context aware software systems. https://doi.org/10.13140/RG.2.1.3361.6080
Mendoza, I., Kalinowski, M., Souza, U., & Felderer, M. (2019). Relating verification and validation methods to software product quality characteristics: results of an expert survey. In International Conference on Software Quality, Springer, pp. 33–44.
Mendoza, I., Souza, U., Kalinowski, M., Interian, R., & Murta, L. G. P. (2019). An efficient algorithm for combining verification and validation methods. In International Conference on Current Trends in Theory and Practice of Informatics, Springer, pp. 324–340.
Merdes, M., Malaka, R., Suliman, D., Paech, B., Brenner, D., & Atkinson, C. (2006). Ubiquitous RATs: How Resource-Aware Run-Time Tests Can Improve Ubiquitous Software Systems. In Proceedings of the 6th International Workshop on Software Engineering and Middleware (SEM’06), pp. 55–62.
Mezghani, E., & Halima, R. B. (2012). DRF4SOA: A Dynamic Reconfigurable Framework for Designing Autonomic Application Based on SOA. In Proceeding of the 21st IEEE International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE’12), pp. 95–97.
Moreno, G. A., Cámara, J., Garlan, D., & Schmerl, B. (2015). Proactive Self-adaptation Under Uncertainty: A Probabilistic Model Checking Approach. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering, pp. 1–12.
Morin, B., Barais, O., Jezequel, J.-M., Fleurey, F., & Solberg, A. (2009). Models@Runtime to Support Dynamic Adaptation. Computer, 42(10), 44–51.
Muccini, H., Dias, M. S., & Richardson, D. J. (2006). Software Architecture-Based Regression Testing. Journal of Systems and Software, 79(10), 1379–1396.
Murphy, C., Kaiser, G., Vo, I., & Chu, M. (2009). Quality Assurance of Software Applications Using the In Vivo Testing Approach. In Proceedings of the 2nd International Conference on Software Testing Verification and Validation (ICST’09), pp. 111–120.
Neidhardt, E., Bahr, S., Nippold, R., & Dalaff, C. (2019). Web-based transport management tool-suite (keepoperational) for crisis management. In 2019 15th International Conference on Telecommunications (ConTEL), pp. 1–5.
Neto, A., Subramanyan, R., Vieira, M., & Travassos, G. (2007). A survey on model-based testing approaches: a systematic review. pp. 31–36. https://doi.org/10.1145/1353673.1353681
Nguyen, T. H., Pontelli, E., & Son, T. C. (2020). On repairing web services workflows. In International Symposium on Practical Aspects of Declarative Languages, Springer, pp. 37–53.
Niebuhr, D., & Rausch, A. (2007) A concept for dynamic wiring of components: Correctness in dynamic adaptive systems. SAVCBS ’07, Association for Computing Machinery, p. 101–102. https://doi.org/10.1145/1292316.1292330
Niebuhr, D., & Rausch, A. (2009). Guaranteeing Correctness of Component Bindings in Dynamic Adaptive Systems Based on Runtime Testing. In Proceedings of the 4th International Workshop on Services Integration in Pervasive Environments (SIPE’09), pp. 7–12.
Niebuhr, D., Rausch, A., Klein, C., Reichmann, J., & Schmid, R. (2009). Achieving dependable component bindings in dynamic adaptive systems - a runtime testing approach. In Proceedings of the 2009 Third IEEE International Conference on Self-Adaptive and Self-Organizing Systems, SASO ’09, IEEE Computer Society, p. 186–197. https://doi.org/10.1109/SASO.2009.40
Nimară, S., Ploştinaru, V. S., Bogdan, R., & Crişan-Vida, M. (2019). Reference images generation for automotive regression testing. In 2019 23rd International Conference on System Theory, Control and Computing (ICSTCC), IEEE, pp. 774–778.
Okika, J. C., Ravn, A. P., Liu, Z., & Siddalingaiah, L. (2006). Developing a TTCN-3 Test Harness for Legacy Software. In Proceedings of the International Workshop on Automation of Software Test, pp. 104–110.
O’Regan, G. (2019). Concise guide to software testing. Springer.
Panasiuk, O., Holzknecht, O., Şimşek, U., Kärle, E., & Fensel, D. (2019). Verification and validation of semantic annotations. In International Andrei Ershov Memorial Conference on Perspectives of System Informatics, Springer, pp. 223–231.
Piel, É., & González-Sanchez, A. (2009). Data-flow Integration Testing Adapted to Runtime Evolution in Component-Based Systems. In Proceedings of the ESEC/FSE Workshop on Software Integration and Evolution@runtime, pp. 3–10.
Piel, É., González-Sanchez, A., & Groß, H. G. (2010). Automating Integration Testing of Large-Scale Publish/Subscribe Systems. In Principles and Applications of Distributed Event-Based Systems, pp. 140–163.
Piel, E., Gonzalez-Sanchez, A., & Gross, H. G. (2010). Built-in Data-Flow Integration Testing in Large-Scale Component-Based Systems. In Proceedings of the 22nd IFIP WG 6.1 International Conference on Testing Software and Systems (ICTSS’10), pp. 79–94.
Pilskalns, O., Uyan, G., & Andrews, A. (2006). Regression Testing UML Designs. In Proceedings of the 22nd IEEE International Conference on Software Maintenance (ICSM’06), pp. 254–264.
Pohl, D., Bouchachia, A., & Hellwagner, H. (2019). Active online learning for social media analysis to support crisis management. IEEE Transactions on Knowledge and Data Engineering, 1–1.
Powell, B., & Chandran, S. (2019). Improving fleet management strategy and operational intelligence with predictive analytics. In Aligning Business Strategies and Analytics. Springer, pp. 51–66.
Prabhakar, N., Singhal, A., Bansal, A., & Bhatia, V. (2019). A literature survey of applications of meta-heuristic techniques in software testing. In Software Engineering. Springer, pp. 497–505.
Rahman, A., Rahman, T., Ghani, N. H., Hossain, S., & Uddin, J. (2019). Iot based patient monitoring system using ecg sensor. In 2019 International Conference on Robotics,Electrical and Signal Processing Techniques (ICREST), pp. 378–382.
Ramirez, A. E., Morales, B., & King, T. M. (2008). A Self-Testing Autonomic Job Scheduler. In Proceedings of the 46th Annual Southeast Regional Conference on XX (ACM-SE’08), pp. 304–309.
Rentea, C., Schieferdecker, I., & Cristea, V. (2009). Ensuring Quality of Web Applications by Client-side Testing Using TTCN-3. In Proceeding of the 21th IFIP International Conference on Testing of Communicating Systems joint with 9th International Workshop on Formal Approaches to Testing of Software (TestCom/Fates’09).
Ruicka, J., Navrtilov K., & Tich, T. (2019). Crisis traffic management in the city using traffic lights. In Modern Safety Technologies in Transportation, pp. 134–139.
Rothermel, G., & Harrold, M. J. (1997). A Safe, Efficient Regression Test Selection Technique. ACM Transactions on Software Engineering and Methodology, 6, 173–210.
Sanchez, C., Schneider, G., Ahrendt, W., Bartocci, E., Bianculli, D., Colombo, C., et al. (2019). A survey of challenges for runtime verification from advanced application domains (beyond software). Formal Methods in System Design, 54(3), 279–335. https://doi.org/10.1007/s10703-019-00337-w.
Schieferdecker, I., Din, G., & Apostolidis, D. (2005). Distributed Functional and Load Tests for Web Services. International Journal on Software Tools for Technology Transfer (STTT), 7, 351–360.
Schieferdecker, I., & Vassiliou-Gioles, T. (2003). Realizing Distributed TTCN-3 Test Systems With TCI. In Proceedings of the 15th IFIP International Conference on Testing of Communicating Systems (TestCom’03).
Schulz, S., & Vassiliou-Gioles, T. (2002). Implementation of TTCN-3 Test Systems using the TRI. In Proceedings of the IFIP 14th International Conference on Testing Communicating Systems (TestCom’02), pp. 425–442.
Serbanescu, D. A., Molovata, V., Din, G., Schieferdecker, I., & Radusch, I. (2008). Real-Time Testing with TTCN-3. In Proceeding of the 20th IFIP International Conference on Testing of Communicating Systems joint with 8th International Workshop on Formal Approaches to Testing of Software (TestCom/Fates’08), pp. 283–301.
Shevtsov, S., Weyns, D., & Maggio, M. (2019). Self-adaptation of software using automatically generated control-theoretical solutions. In Engineering Adaptive Software Systems. Springer, pp. 35–55.
Siqueira, B. R., Ferrari, F. C., Serikawa, M. A., Menotti, R., & de Camargo, V. V. (2016). Characterisation of challenges for testing of adaptive systems. In Proceedings of the 1st Brazilian Symposium on Systematic and Automated Software Testing (New York, NY, USA), SAST, Association for Computing Machinery. https://doi.org/10.1145/2993288.2993294
Sotomayor, J. P., Allala, S. C., Alt, P., Phillips, J., King, T. M., & Clarke, P. J. (2019). Comparison of runtime testing tools for microservices. In 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), vol. 2, IEEE, pp. 356–361.
Stepien, B., Peyton, L., & Xiong, P. (2008). Framework Testing of Web Applications Using TTCN-3. International Journal on Software Tools for Technology Transfer (STTT), 10(4), 371–381.
Strandberg, P. E., Enoiu, E. P., Afzal, W., Sundmark, D., & Feldt, R. (2019). Information flow in software testing-an interview study with embedded software engineering practitioners. IEEE Access, 7, 46434–46453.
Suleiman, D., Alian, M., & Hudaib, A. (2017). A survey on prioritization regression testing test case. In 2017 8th International Conference on Information Technology (ICIT), pp. 854–862. https://doi.org/10.1109/ICITECH.2017.8079958
Suliman, D., Paech, B., Borner, L., Atkinson, C., Brenner, D., Merdes, M., & Malaka, R. (2006). The MORABIT Approach to Runtime Component Testing. In 30th Annual International Computer Software and Applications Conference, pp. 171–176.
Tabei, F., Gresham, J. M., Askarian, B., Jung, K., & Chong, J. W. (2020). Cuff-less blood pressure monitoring system using smartphones. IEEE Access, 8, 11534–11545.
Tamura, G., Villegas, N., Müller, H., Sousa, J. a. P., Becker, B., Karsai, G., Mankovskii, S., Pezzè, M., Schäfer, W., Tahvildari, L., & Wong, K. (2013). Towards Practical Runtime Verification and Validation of Self-Adaptive Software Systems. In Software Engineering for Self-Adaptive Systems II, pp. 108–132.
Tebes, G., Peppino, D., Becker, P., Matturro, G., Solari, M., & Olsina, L. (2019). A systematic review on software testing ontologies. In International Conference on the Quality of Information and Communications Technology, Springer, pp. 144–160.
Torjusen, A. B., Abie, H., Paintsil, E., Trcek, D., & Skomedal, A. (2014). Towards Run-Time Verification of Adaptive Security for IoT in eHealth. In Proceedings of the 8th European Conference on Software Architecture Workshops (ECSAW’14), pp. 1–8.
Turk, E. C., Altiner, E. P., zdemir, M., & Erogul, O. (2019). Wireless transmission of heart rate and blood pressure measurements for remote patient monitoring. In 2019 Medical Technologies Congress (TIPTEKNO), pp. 1–4.
Utting, M. (2006). and Legeard, B. Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc.
Vincent, J., King, G., Lay, P., & Kinghorn, J. (2002). Principles of Built-In-Test for Run-Time-Testability in Component-Based Software Systems. Software Quality Control, 10(2), 115–133.
Wang, Y., King, G., Patel, D., Patel, S., & Dorling, A. (1999). On Coping With Real-time Software Dynamic Inconsistency by Built-In Tests. Annals of Software Engineering, 7(1–4), 283–296.
Ying Li, Q. L. (2011). Research on Web Application Software Load Test Using Technology of TTCN-3. American Journal of Engineering and Technology Research, 11, 3686–3690.
Yoo, S., & Harman, M. (2012). Regression testing minimization, selection and prioritization: A survey. Software Testing Verification and Reliability. 22, 2, 67–120.
Zarrad, A. (2015). A systematic review on regression testing for web-based applications. Journal of Software 10, 971–990.
Zhou, G., Xie, D., Zhao, X., & Lu, C. (2020). Collaborative optimization of vehicle and charging scheduling for a bus fleet mixed with electric and traditional buses. IEEE Access, 8, 8056–8072.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Lahami, M., Krichen, M. A survey on runtime testing of dynamically adaptable and distributed systems. Software Qual J 29, 555–593 (2021). https://doi.org/10.1007/s11219-021-09558-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-021-09558-x