research-article

Mutation Operators for Code Annotations

Authors:

Pedro Pinheiro,

José Carlos Viana,

Leonardo Fernandes,

Márcio Ribeiro,

Fabiano Ferrari,

Baldoino Fonseca,

Rohit GheyiAuthors Info & Claims

SAST '18: Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing

Pages 77 - 86

https://doi.org/10.1145/3266003.3266006

Published: 17 September 2018 Publication History

Abstract

Mutation testing is a technique that injects artificial faults to check whether the existing test suite can detect them. To inject the faults, mutation testing tools rely on mutation operators, such as Arithmetic Operator Replacement, Relational Operator Replacement, Statement Deletion, and Overriding Method Rename. Most of the proposed mutation operators have focused on imperative language constructs and object-oriented constructs. However, many projects make use of meta-programming through code annotations, and mutation operators to simulate annotation-related faults are missing. To minimize this problem, in this paper we propose a set of mutation operators capable of simulating annotation-related faults. In particular, we define nine operators. To evaluate our operators, we mined Java projects that make heavy use of annotations to identify annotation-related faults. We analyzed 100 faults and categorized them as "misuse", when the developer does not know how to use the annotations in a proper way; and "wrong annotation parsing", when the developer wrongly parses annotation code (by using refection, for example). Our operators are able to simulate 95 out of the 100 mined faults. In particular, three operators are able to simulate up to 82% of the analyzed faults. As implications for practice, our operators can help developers to improve their test suite and the code responsible for parsing annotated code. Also, they can help developers to avoid annotation-related faults.

References

[1]

M. Al-Hajjaji, F. Benduhn, T. Thüm, T. Leich, and G. Saake. 2016. Mutation Operators for Preprocessor-Based Variability. Proceedings of the 10th International Workshop on Variability Modelling of Software-intensive Systems (VaMos), 81--88.

Digital Library

[2]

P. Ammann, M. E. Delamaro, and J. Offutt. 2014. Establishing Theoretical Minimal Sets of Mutants. In Proceedings of the International Conference on Software Testing, Verification, and Validation (ICST). 21--30.

Digital Library

[3]

C. Bauer and G. King. 2006. Java Persistence with Hibernate. Manning Publications Co., Shelter Island, NY, USA.

Digital Library

[4]

Luiz Carvalho, Marcio Augusto Guimarães, Márcio Ribeiro, Leonardo Fernandes, Mustafa Al-Hajjaji, Rohit Gheyi, and Thomas Thüm. 2018. Equivalent Mutants in Configurable Systems: An Empirical Study. In Proceedings of the 12th International Workshop on Variability Modelling of Software-Intensive Systems (VaMoS). 11--18.

Digital Library

[5]

T. T. Chekam, M. Papadakis, Y. Le Traon, and M. Harman. 2017. An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption. In Proceedings of the 39th International Conference on Software Engineering (ICSE). 597--608.

Digital Library

[6]

P. Chevalley. 2001. Applying Mutation Analysis for Object-Oriented Programs Using a Reflective Approach. In Proceedings of the 8th Asia-Pacific Software Engineering Conference (APSEC). 267--270.

Digital Library

[7]

H. Coles. 2018. PITest Mutation Testing Tool for Java. (2018). http://pitest.org/ Online; accessed 19 June 2018.

[8]

I. Córdoba-Sánchez and J. de Lara. 2016. Ann: A Domain-specific Language for the Effective Design and Validation of Java Annotations. Computer Languages, Systems & Structures 45 (2016), 164--190.

Digital Library

[9]

I. Darwin. 2010. AnnaBot: A Static Verifier for Java Annotation Usage. Advances in Software Engineering (2010), 1--7.

Digital Library

[10]

C. A. de Araújo, M. E. Delamaro, J. C. Maldonado, and A. M. R. Vincenzi. 2016. Correlating automatic static analysis and mutation testing: towards incremental strategies. Journal of Software Engineering Research & Development (JSERD) 4 (2016).

[11]

P. Delgado-Pérez, I. Medina-Bulo, F. Palomo-Lozano, A. García-Domínguez, and J. Domínguez-Jiménez. 2017. Assessment of class mutation operators for C++ with the MuCPP mutation system. Information and Software Technology 81 (2017), 169--184.

Digital Library

[12]

R. DeMillo, R. Lipton, and F. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11, 4 (1978), 34--41.

Digital Library

[13]

L. Deng, N. Mirzaei, P. Ammann, and J. Offutt. 2015. Towards mutation analysis of Android apps. In Proceedings of the 8th International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 1--10.

[14]

A. Derezinska and K. Kowalski. 2011. Object-Oriented Mutation Applied in Common Intermediate Language Programs Originated from C#. In Proceedings of the 4th International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 342--350.

Digital Library

[15]

A. Estero-Botaro, F. Palomo-Lozano, and I. Medina-Bulo. 2010. Quantitative Evaluation of Mutation Operators for WS-BPEL Compositions. In Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation Workshops (ICSTW). 142--150.

Digital Library

[16]

L. Fernandes, M. Ribeiro, L. Carvalho, R. Gheyi, M. Mongiovi, A. Santos, A. Cavalcanti, F. Ferrari, and J. Maldonado. 2017. Avoiding Useless Mutants. In Proceedings of the 16th International Conference on Generative Programming: Concepts & Experience (GPCE). 187--198.

Digital Library

[17]

F. C. Ferrari, J. C. Maldonado, and A. Rashid. 2008. Mutation Testing for Aspect-Oriented Programs. In Proceedings of the 1st International Conference on Software Testing, Verification, and Validation (ICST). 52--61.

Digital Library

[18]

GitHub Inc. 2018. GitHub Search REST API v3. https://developer.github.com/v3/search/. (2018). Online; accessed 19 June 2018.

[19]

Y. Jia and M. Harman. 2009. Higher Order Mutation Testing. Information and Software Technology Journal 51, 10 (2009), 1379--1393.

Digital Library

[20]

Y. Jia and M. Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering (TSE) 37, 5 (2011), 649--678.

Digital Library

[21]

R. Johnson, J. Hoeller, A. Arendsen, C. Sampaleanu, R. Harrop, T. Risberg, D. Davison, D. Kopylenko, M. Pollack, T. Templier, E. Vervaet, P. Tung, B. Hale, A. Colyer, J. Lewis, C. Leau, and R. Evans. 2006. Spring - Java/J2EE Application Framework. Reference Manual Version 2.0.2. Interface21 Ltd.

[22]

R. Just. 2014. The Major Mutation Framework: Efficient and Scalable Mutation Analysis for Java. In Proceedings of the 23rd International Symposium on Software Testing and Analysis (ISSTA). ACM, 433--436.

Digital Library

[23]

R. Just, D. Jalali, L. Inozemtseva, M. Ernst, R. Holmes, and G. Fraser. 2014. Are Mutants a Valid Substitute for Real Faults in Software Testing?. In Proceedings of the 22nd SIGSOFT International Symposium on Foundations of Software Engineering (FSE). 654--665.

Digital Library

[24]

Y-S. Ma, Y-R. Kwon, and A. J. Offutt. 2002. Inter-class Mutation Operators for Java. In Proceedings of the 13th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 352--366.

Digital Library

[25]

Y-S. Ma, A. J. Offutt, and Y-R. Kwon. 2005. MuJava: An Automated Class Mutation System. Software Testing, Verification and Reliability 15, 2 (2005), 97--133.

[26]

L. Madeyski, W. Orzeszyna, R. Torkar, and M. Jozala. 2014. Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Transactions on Software Engineering Journal (TSE) 40, 1 (2014), 23--42.

Digital Library

[27]

C. Noguera and R. Pawlak. 2007. AVal: An extensible attribute-oriented programming validator for Java. Journal of Software Maintenance and Evolution 19, 4 (2007), 253--275.

Digital Library

[28]

A. J. Offutt, Y-S. Ma, and Y-R. Kwon. 2006. The Class-Level Mutants of MuJava. In Proceedings of the International Workshop on Automation of Software Test (AST). ACM, Shanghai, China, 78--84.

Digital Library

[29]

A. J. Offutt and R. H. Untch. 2000. Mutation 2000: Uniting the Orthogonal. In Proceedings of the Mutation Symposium. 34--44.

[30]

M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. le Traon, and M. Harman. 2018. Mutation Testing Advances: An Analysis and Survey. Joint Technical Report. Luxembourg University; University College London; Peking University, Luxembourg City, Luxembourg; London, UK; Beijing Shi, China.

[31]

G. Petrovic, M. Ivankovic, P. Ammann, B. Kurtz, and R. Just. 2018. An Industrial Application of Mutation Testing: Lessons, Challenges, and Research Directions. In Proceedings of the 13th International Workshop on Mutation Analysis (Mutation). Västerås, Sweden.

[32]

H. Rocha and M. T. Valente. 2011. How Annotations are Used in Java: An Empirical Study. In Proceedings of the 23rd International Conference on Software Engineering and Knowledge Engineering (SEKE). 426--431.

[33]

D. Schuler and A. Zeller. 2013. Covering and Uncovering Equivalent Mutants. Software Testing, Verification and Reliability 23, 5 (2013), 353--374.

[34]

P. Tahchiev, F. Leme, V. Massol, and G. Gregory. 2010. JUnit in Action, Second Edition. Manning Publications Co., Shelter Island, NY, USA.

Digital Library

[35]

C. Wohlin, P. Runeson, M. Host, M. C. Ohlsson, B. Regnell, and A. Wesslén. 2000. Experimentation in software engineering: an introduction. Kluwer Academic Publishers.

Digital Library

Cited By

Kong XChen HYu MZhang L(2022)Boosting Code Search with Structural Code AnnotationElectronics10.3390/electronics1119305311:19(3053)Online publication date: 25-Sep-2022
https://doi.org/10.3390/electronics11193053
Ramírez FMera-Gómez CBahsoon RZhang Y(2022)Mining the Limits of Granularity for Microservice AnnotationsService-Oriented Computing10.1007/978-3-031-20984-0_19(273-281)Online publication date: 22-Nov-2022
https://doi.org/10.1007/978-3-031-20984-0_19
Ramírez FMera-Gómez CChen SBahsoon RZhang Y(2022)Semantics-Driven Learning for Microservice AnnotationsService-Oriented Computing10.1007/978-3-031-20984-0_17(255-263)Online publication date: 22-Nov-2022
https://doi.org/10.1007/978-3-031-20984-0_17
Show More Cited By

Index Terms

Mutation Operators for Code Annotations
1. General and reference
  1. Cross-computing tools and techniques
    1. Verification
2. Software and its engineering
  1. Software creation and management
    1. Software development process management
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software organization and properties
    1. Software functional properties
      1. Formal methods
        Software verification

Recommendations

Mutation Operators for Java Streams
SAST '22: Proceedings of the 7th Brazilian Symposium on Systematic and Automated Software Testing

Mutation testing analyzes test suites to verify their capability to detect artificially injected faults. Mutation testing tools rely on mutation operators to simulate those faults by modifying language constructs. The popularization of Streaming APIs, ...
Mutating code annotations: An empirical evaluation on Java and C# programs
Highlights
- 100 new bugs regarding C# code annotations listed.
- Evaluation on mutation ...
Abstract
Mutation testing injects code changes to check whether tests can detect them. Mutation testing tools use mutation operators that modify program elements such as operators, names, and entire statements. Most existing mutation operators ...
Mutation Operators for Actor Systems
ICSTW '10: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops

Mutation testing is a well known technique for estimating and improving the quality of test suites. Given a test suite T for a system S, mutation testing systematically creates mutants of S and executes T to measure how many mutants T detects. If T does ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SAST '18: Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing

September 2018

107 pages

ISBN:9781450365550

DOI:10.1145/3266003

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SBC: Brazilian Computer Society
UFSCar: Federal University of São Carlos
IFSP: Federal Institute of São Paulo

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 September 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Conference

SAST '18

SAST '18: III Brazilian Symposium on Systematic and Automated Software Testing

September 17 - 21, 2018

SAO CARLOS, Brazil

Acceptance Rates

SAST '18 Paper Acceptance Rate 10 of 20 submissions, 50%;

Overall Acceptance Rate 45 of 92 submissions, 49%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
173
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kong XChen HYu MZhang L(2022)Boosting Code Search with Structural Code AnnotationElectronics10.3390/electronics1119305311:19(3053)Online publication date: 25-Sep-2022
https://doi.org/10.3390/electronics11193053
Ramírez FMera-Gómez CBahsoon RZhang Y(2022)Mining the Limits of Granularity for Microservice AnnotationsService-Oriented Computing10.1007/978-3-031-20984-0_19(273-281)Online publication date: 22-Nov-2022
https://doi.org/10.1007/978-3-031-20984-0_19
Ramírez FMera-Gómez CChen SBahsoon RZhang Y(2022)Semantics-Driven Learning for Microservice AnnotationsService-Oriented Computing10.1007/978-3-031-20984-0_17(255-263)Online publication date: 22-Nov-2022
https://doi.org/10.1007/978-3-031-20984-0_17
Pinheiro PViana JRibeiro MFernandes LFerrari FGheyi RFonseca B(2020)Mutating Code Annotations: An Empirical Evaluation on Java and C# ProgramsScience of Computer Programming10.1016/j.scico.2020.102418(102418)Online publication date: Feb-2020
https://doi.org/10.1016/j.scico.2020.102418

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten