Abstract
Systematic and rigorous robustness testing is very critical for embedded systems, as for example communication and control systems. Robustness testing aims at testing the behavior of a system in the presence of faulty situations in its operating environment (e.g., sensors and actuators). In such situations, the system should gracefully degrade its performance instead of abruptly stopping execution. To systematically perform robustness testing, one option is to resort to model-based robustness testing (MBRT), based for example on UML/MARTE models. However, to successfully apply MBRT in industrial contexts, new technology needs to be developed to scale to the complexity of real industrial systems. In this paper, we report on our experience of performing MBRT on video conferencing systems developed by Cisco Systems, Norway. We discuss how we developed and integrated various techniques and tools to achieve a fully automated MBRT that is able to detect previously uncaught software faults in those systems. We provide an overview of how we achieved scalable modeling of robustness behavior using aspect-oriented modeling, test case generation using search algorithms, and environment emulation for test case execution. Our experience and lessons learned identify challenges and open research questions for the industrial application of MBRT.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
IEEE Standard Glossary of Software Engineering Terminology. IEEE, IEEE Std 610.12-1990, p. 1 (1990)
IEEE Standard Dictionary of Measures of the Software Aspects of Dependability. IEEE Std 982.1-2005 (Revision of IEEE Std 982.1-1988), pp. 1–34 (2006)
Standard for Software Quality Characteristics. International Organization for Standardization, ISO-9126-3 (2003)
Software Assurance Standard. NASA Technical Standard, NASA-STD-8739.8 (2005)
Shafique, M., Labiche, Y.: A Systematic Review of Model Based Testing Tools. Carleton University, Department of Systems and Computer Engineering, Technical Report, SCE-10-04 (2010)
Ali, S., Hemmati, H., Holt, N.E., Arisholm, E., Briand, L.C.: Model Transformations as a Strategy to Automate Model-Based Testing - A Tool and Industrial Case Studies. Simula Research Laboratory, Technical Report (2010-01) (2010)
Ali, S., Briand, L.C., Hemmati, H.: Modeling Robustness Behavior Using Aspect-Oriented Modeling to Support Robustness Testing of Industrial Systems. Simula Research Laboratory, Technical Report (2010-03) (2010)
Modeling and Analysis of Real-time and Embedded systems, MARTE (2010), http://www.omgmarte.org/
Binder, R.V.: Testing object-oriented systems: models, patterns, and tools. Addison-Wesley Longman Publishing Co., Inc., Amsterdam (1999)
Yedduladoddi, R.: Aspect Oriented Software Development: An Approach to Composing UML Design Models. VDM Verlag Dr. Müller (2009)
Kermeta - Breathe Life into Your Metamodels (2010), http://www.kermeta.org/
Ali, S., Iqbal, M.Z., Arcuri, A., Briand, L.C.: A Search-based OCL Constraint Solver for Model-based Test Data Generation. In: Proceedings of the 11th International Conference On Quality Software, QSIC 2011 (2011)
Egea, M.: EyeOCL Software (2010), http://maude.sip.ucm.es/eos/
netem (2011), http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
Ali, S., Yue, T., Briand, L.C., Malik, Z.I.: Does Aspect-Oriented Modeling Help Improve the Readability of UML State Machines? (2010)
Query/View/Transformation QVT (2011), http://www.omg.org/spec/QVT/1.0/
Cabot, J., Claris, R., Riera, D.: Verification of UML/OCL Class Diagrams using Constraint Programming. In: Proceedings of the 2008 IEEE International Conference on Software Testing Verification and Validation Workshop. IEEE Computer Society, Los Alamitos (2008)
Krieger, M., Knapp, A.: Executing Underspecified OCL Operation Contracts with a SAT Solver. In: 8th International Workshop on OCL Concepts and Tools. ECEASST, vol. 15 (2008)
Hemmati, H., Briand, L., Arcuri, A., Ali, S.: An Enhanced Test Case Selection Approach for Model-Based Testing: An Industrial Case Study. In: 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). ACM, New York (2010)
Pintér, G., Majzik, I.: Modeling and Analysis of Exception Handling by Using UML Statecharts. Scientific Engineering of Distributed Java Applications, 58–67 (2005)
Lei, B., Liu, Z., Morisset, C., Li, X.: State Based Robustness Testing for Components. Electronic Notes of Theoratical Computer Science 260, 173–188
Nebut, C., Fleurey, F., Traon, Y.L., Jezequel, J.-M.: Automatic Test Generation: A Use Case Driven Approach. IEEE Transactions of Software Engineering 32, 140–155 (2006)
Afzal, W., Torkar, R., Feldt, R.: A Systematic Review of Search-based Testing for Non-functional System Properties. Information and Software Technology 51, 957–976 (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ali, S., Briand, L.C., Arcuri, A., Walawege, S. (2011). An Industrial Application of Robustness Testing Using Aspect-Oriented Modeling, UML/MARTE, and Search Algorithms. In: Whittle, J., Clark, T., Kühne, T. (eds) Model Driven Engineering Languages and Systems. MODELS 2011. Lecture Notes in Computer Science, vol 6981. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-24485-8_9
Download citation
DOI: https://doi.org/10.1007/978-3-642-24485-8_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-24484-1
Online ISBN: 978-3-642-24485-8
eBook Packages: Computer ScienceComputer Science (R0)