skip to main content
10.1145/2647908.2655968acmotherconferencesArticle/Chapter ViewAbstractPublication PagessplcConference Proceedingsconference-collections
research-article

Towards the assessment of software product line tests: a mutation system for variable systems

Published: 15 September 2014 Publication History

Abstract

Software product line engineering is an emerging methodology for the development of variant-rich software systems. As software product lines are viable for this purpose, testing them is complicated in contrast to non-variable systems, as there is an increasingly amount of possible products due to the number of features. There exist many methods proposed for testing software product lines, but seldom the quality of the resulting tests was assessed. For assessing test quality mutation analysis is a well-known technique and is frequently applied to non-variable software systems. However, mutation analysis cannot be applied straight-forward onto software product lines.
We present a mutation system for assessing the quality of software product line tests by means of fault detection capability. Our mutation system comprises model-based mutation operators, test case adaption, automated model and test execution, and automated mutation analysis. So far, we developed several mutation operators for feature models, UML state machines, and mapping models. We evaluated the mutation operators against tests that were generated from the specifications and applied them for three case studies. From the results we draw conclusions about the effectiveness of the individual mutation operators.

References

[1]
B. K. Aichernig, F. Lorber, and D. Ničković. Time for Mutants --- Model-Based Mutation Testing with Timed Automata. In D. Hutchison, T. Kanade, J. Kittler, et al., editors, Tests and Proofs, volume 7942 of Lecture Notes in Computer Science, pages 20--38. Springer Berlin Heidelberg, Berlin and Heidelberg, 2013.
[2]
F. Belli, C. J. Budnik, and L. White. Event-based modelling, analysis and testing of user interactions: approach and case study. Software Testing, Verification and Reliability, 16(1):3--32, 2006.
[3]
F. Belli and A. Hollmann. Test generation and minimization with basic statecharts. In E. J. Delp and P. W. Wong, editors, the 2008 ACM symposium, volume vol. 5681, page 718, Bellingham and Wash and Springfield and Va, 2008. SPIE and IS&T.
[4]
F. Belli, A. Hollmann, and S. Padberg. Model-Based Integration Testing with Communication Sequence Graphs. In J. Zander, I. Schieferdecker, and P. J. Mosterman, editors, Model-based testing for embedded systems, Computational analysis, synthesis, and design of dynamic systems. CRC Press, Boca Raton, 2011.
[5]
R. V. Binder. Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley Longman Publishing Co., Inc., Boston and MA and USA, 1999.
[6]
P. E. Black, V. Okun, and Y. Yesha. Mutation operators for specifications. In ASE 2000 15th IEEE International Automated Software Engineering Conference, pages 81--88, 2000.
[7]
H. Cichos and T. S. Heinze. Efficient Reduction of Model-Based Generated Test Suites Through Test Case Pair Prioritization. In Proceedings of the 7th International Workshop on Model-Driven Engineering, Verification and Validation (MoDeVVa 10), pages 37--42, Los Alamitos, 2011. IEEE Computer Society Press.
[8]
H. Cichos, M. Lochau, S. Oster, and A. Schürr. Reduktion von Testsuiten für Software-Produktlinien. In S. Jähnichen, A. Küpper, and S. Albayrak, editors, Software Engineering 2012: Fachtagung des GI-Fachbereichs Softwaretechnik, 27. Februar - 2. März 2012 in Berlin, volume 198 of LNI, pages 143--154. GI, 2012.
[9]
P. Clements and L. Northrop. Software product lines: Practices and patterns. The SEI series in software engineering. Addison-Wesley, Boston Mass. u.a, 7. print edition, 2009.
[10]
Conformiq Qtronic. Semantics and Algorithms for Test Generation: a Conformiq Software Whitepaper, 2008.
[11]
K. Czarnecki and M. Antkiewicz. Mapping Features to Models: A Template Approach Based on Superimposed Variants. In R. Glück, editor, Generative programming and component engineering, volume 3676 of Lecture Notes in Computer Science, pages 422--437. Springer, Berlin {u.a.}, 2005.
[12]
R. A. DeMillo. Mutation analysis as a tool for software quality assurance. Technical report, DTIC Document, 1980.
[13]
E. Engström and P. Runeson. Software product line testing -- A systematic mapping study. Information and Software Technology, 53(1):2--13, 2011.
[14]
S. C. P. F. Fabbri, M. E. Delamaro, J. C. Maldonado, and P. C. Masiero. Mutation analysis testing for finite state machines. In 1994 IEEE International Symposium on Software Reliability Engineering, pages 220--229, 1994.
[15]
H. Grönniger, H. Krahn, C. Pinkernell, and B. Rumpe. Modeling Variants of Automotive Systems using Views. In T. Kühne, W. Reisig, and F. Steimann, editors, Tagungsband zur Modellierung 2008 (Berlin-Adlershof, Deutschland, 12--14. März 2008), LNI, Bonn, 2008. Gesellschaft für Informatik.
[16]
C. Henard, M. Papadakis, G. Perrouin, J. Klein, and Y. Le Traon. Assessing Software Product Line Testing Via Model-Based Mutation: An Application to Similarity Testing. In ICSTW '13: IEEE 6th International Conference On Software Testing, Verification and Validation Workshops 2013, pages 188--197, 2013.
[17]
Y. Jia and M. Harman. Higher order mutation testing. Information and Software Technology, 51(10):1379--1393, 2009.
[18]
Y. Jia and M. Harman. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering, 37(5):649--678, 2011.
[19]
K. C. Kang, S. G. Cohen, J. A. Hess, W. E. Novak, and A. S. Peterson. Feature-Oriented Domain Analysis (FODA) Feasibility Study, 1990.
[20]
S. Kim, J. A. Clark, and J. A. Mcdermid. Class Mutation: Mutation Testing for Object-Oriented Programs. In FMES, 2000.
[21]
H. Lackner, M. Thomas, F. Wartenberg, and S. Weißleder. Model-Based Test Design of Product Lines: Raising Test Design to the Product Line Level. In ICST' 14: International Conference on Software Testing, Verification, and Validation, pages 51--60. 2014.
[22]
M. Lochau, I. Schaefer, J. Kamischke, and S. Lity. Incremental Model-Based Testing of Delta-Oriented Software Product Lines. In D. Hutchison, T. Kanade, J. Kittler, et al., editors, Tests and Proofs, volume 7305 of Lecture Notes in Computer Science, pages 67--82. Springer Berlin Heidelberg, Berlin and Heidelberg, 2012.
[23]
Object Management Group (OMG). UML 2.4.1 Superstructure Specification, 2011.
[24]
A. J. Offutt and R. H. Untch. Mutation 2000: Uniting The Orthogonal, 1971.
[25]
J. Offutt, S. Liu, A. Abdurazik, and P. Ammann. Generating test data from state-based specifications. The Journal of Software Testing, Verification and Reliability, 13:25--53, 2003.
[26]
E. M. Olimpiew and H. Gomaa. Model-Based Testing for Applications Derived from Software Product Lines. ACM SIGSOFT Software Engineering Notes, 30(4):1--7, 2005.
[27]
S. Oster, I. Zorcic, F. Markert, and M. Lochau. MoSo-PoLiTe: tool support for pairwise and model-based software product line testing. In VaMoS '11, pages 79--82, 2011.
[28]
K. Pohl, G. Böckle, and Linden, Frank J. van der. Software Product Line Engineering: Foundations, Principles and Techniques. Springer-Verlag New York, Inc., Secaucus and NJ and USA, 2005.
[29]
Z. Stephenson, Y. Zhan, J. Clark, and J. McDermid. Test Data Generation for Product Lines - A Mutation Testing Approach. In B. Geppert, C. Krueger, and J. Li, editors, SPLiT '04: Proceedings of the International Workshop on Software Product Line Testing, pages 13--18, Boston and MA, 2004.
[30]
M. Utting and B. Legeard. Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc., San Francisco and CA and USA, 1 edition, 2006.
[31]
S. Weißleder. Influencing Factors in Model-Based Testing with UML State Machines: Report on an Industrial Cooperation. In D. Hutchison, T. Kanade, J. Kittler, et al., editors, Model Driven Engineering Languages and Systems, volume 5795 of Lecture Notes in Computer Science, pages 211--225. Springer Berlin Heidelberg, Berlin and Heidelberg, 2009.
[32]
M. R. Woodward. Errors in algebraic specifications and an experimental mutation testing tool. Software Engineering Journal, 8(4):211, 1993.

Cited By

View all
  • (2021)Dynamic test prioritization of product lines: An application on configurable simulation modelsSoftware Quality Journal10.1007/s11219-021-09571-0Online publication date: 20-Oct-2021
  • (2020)Measuring effectiveness of sample-based product-line testingACM SIGPLAN Notices10.1145/3393934.327813053:9(119-133)Online publication date: 7-Apr-2020
  • (2018)Measuring effectiveness of sample-based product-line testingProceedings of the 17th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/3278122.3278130(119-133)Online publication date: 5-Nov-2018
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
SPLC '14: Proceedings of the 18th International Software Product Line Conference: Companion Volume for Workshops, Demonstrations and Tools - Volume 2
September 2014
151 pages
ISBN:9781450327398
DOI:10.1145/2647908
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

  • University of Florence: University of Florence
  • CNR: Istituto di Scienza e Tecnologie dell Informazione

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 September 2014

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. mutation analysis
  2. software product lines
  3. software testing
  4. test quality

Qualifiers

  • Research-article

Funding Sources

Conference

SPLC '14
Sponsor:
  • University of Florence
  • CNR

Acceptance Rates

Overall Acceptance Rate 167 of 463 submissions, 36%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)5
  • Downloads (Last 6 weeks)0
Reflects downloads up to 26 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2021)Dynamic test prioritization of product lines: An application on configurable simulation modelsSoftware Quality Journal10.1007/s11219-021-09571-0Online publication date: 20-Oct-2021
  • (2020)Measuring effectiveness of sample-based product-line testingACM SIGPLAN Notices10.1145/3393934.327813053:9(119-133)Online publication date: 7-Apr-2020
  • (2018)Measuring effectiveness of sample-based product-line testingProceedings of the 17th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/3278122.3278130(119-133)Online publication date: 5-Nov-2018
  • (2018)Equivalent Mutants in Configurable SystemsProceedings of the 12th International Workshop on Variability Modelling of Software-Intensive Systems10.1145/3168365.3168379(11-18)Online publication date: 7-Feb-2018
  • (2018)Mutation operators for feature‐oriented software product linesSoftware Testing, Verification and Reliability10.1002/stvr.167629:1-2Online publication date: 26-Jul-2018
  • (2017)Efficient mutation testing in configurable systemsProceedings of the 2nd International Workshop on Variability and Complexity in Software Design10.5555/3106050.3106052(2-8)Online publication date: 20-May-2017
  • (2017)Automatic Generation of Search-Based Algorithms Applied to the Feature Testing of Software Product LinesProceedings of the XXXI Brazilian Symposium on Software Engineering10.1145/3131151.3131152(114-123)Online publication date: 20-Sep-2017
  • (2017)Efficient Mutation Testing in Configurable Systems2017 IEEE/ACM 2nd International Workshop on Variability and Complexity in Software Design (VACE)10.1109/VACE.2017.3(2-8)Online publication date: May-2017
  • (2017)A domain-specific language for model mutation and its application to the automated generation of exercisesComputer Languages, Systems and Structures10.1016/j.cl.2016.11.00149:C(152-173)Online publication date: 1-Sep-2017
  • (2016)Featured model-based mutation analysisProceedings of the 38th International Conference on Software Engineering10.1145/2884781.2884821(655-666)Online publication date: 14-May-2016
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media