Abstract
Testing refactoring engines is a challenging problem that has gained recent attention in research. Several techniques were proposed to automate generation of programs used as test inputs and to help developers in inspecting test failures. However, these techniques can require substantial effort for writing test generators or finding unique bugs, and do not provide an estimate of how reliable refactoring engines are for refactoring tasks on real software projects.
This paper evaluates an end-to-end approach for testing refactoring engines and estimating their reliability by (1) systematically applying refactorings at a large number of places in well-known, open-source projects and collecting failures during refactoring or while trying to compile the refactored projects, (2) clustering failures into a small, manageable number of failure groups, and (3) inspecting failures to identify non-duplicate bugs. By using this approach on the Eclipse refactoring engines for Java and C, we already found and reported 77 new bugs for Java and 43 for C. Despite the seemingly large numbers of bugs, we found these refactoring engines to be relatively reliable, with only 1.4% of refactoring tasks failing for Java and 7.5% for C.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bettenburg, N., Premraj, R., Zimmermann, T., Kim, S.: Duplicate bug reports considered harmful... really? In: ICSM, pp. 337–345 (2008)
Boyapati, C., Khurshid, S., Marinov, D.: Korat: Automated testing based on Java predicates. In: ISSTA, pp. 123–133 (2002)
CDT home page, http://www.eclipse.org/cdt
Cinnéide, M.Ó., Tratt, L., Harman, M., Counsell, S., Moghadam, I.H.: Experimental assessment of software metrics using automated refactoring. In: ESEM, pp. 49–58 (2012)
Coker, Z., Hafiz, M.: Program transformations to fix C integers. In: ICSE, pp. 792–801 (2013)
Counsell, S., Hassoun, Y., Loizou, G., Najjar, R.: Common refactorings, a dependency graph and some code smells: an empirical study of Java OSS. In: ISESE, pp. 288–296 (2006)
Daniel, B., Dig, D., Garcia, K., Marinov, D.: Automated testing of refactoring engines. In: FSE, pp. 185–194 (2007)
Dig, D., Johnson, R.: How do APIs evolve? A story of refactoring. J. Softw. Maint. Evol. 18(2), 83–107 (2006)
Dig, D., Marrero, J., Ernst, M.D.: Refactoring sequential Java code for concurrency via concurrent libraries. In: ICSE, pp. 397–407 (2009)
Ebraert, P., D’Hondt, T.: Dynamic refactorings: Improving the program structure at run-time. In: RAM-SE, pp. 101–110 (2006)
Fowler, M.: Refactoring: Improving the Design of Existing Code. Addison-Wesley, Boston (1999)
Gligoric, M., Gvero, T., Jagannath, V., Khurshid, S., Kuncak, V., Marinov, D.: Test generation through programming in UDITA. In: ICSE, pp. 225–234 (2010)
GMP home page, http://gmplib.org
Godefroid, P., Kiezun, A., Levin, M.Y.: Grammar-based whitebox fuzzing. In: PLDI, pp. 206–215 (2008)
Hafiz, M., Overbey, J.: OpenRefactory/C: An infrastructure for developing program transformations for C programs. In: SPLASH, pp. 27–28 (2012)
Han, J.: Data Mining: Concepts and Techniques. Morgan Kaufmann Publishers Inc. (2005)
Jagannath, V., Lee, Y.Y., Daniel, B., Marinov, D.: Reducing the costs of bounded-exhaustive testing. In: Chechik, M., Wirsing, M. (eds.) FASE 2009. LNCS, vol. 5503, pp. 171–185. Springer, Heidelberg (2009)
JDT home page, http://www.eclipse.org/jdt
JPF home page, http://babelfish.arc.nasa.gov/trac/jpf
JUnit home page, http://junit.org
Eclipse Juno home page, http://www.eclipse.org/juno
libpng home page, http://www.libpng.org/pub/png/libpng.html
Apache log4j home page, http://logging.apache.org/log4j/2.x
Lucene home page, http://lucene.apache.org
Marinov, D., Khurshid, S.: TestEra: A novel framework for testing Java programs. In: ASE, pp. 22–31 (2001)
Commons Math home page, http://commons.apache.org/math
Misherghi, G., Su, Z.: HDD: Hierarchical delta debugging. In: ICSE, pp. 142–151 (2006)
Murphy-Hill, E., Parnin, C., Black, A.P.: How we refactor, and how we know it. In: ICSE, pp. 287–297 (2009)
Negara, S., Chen, N., Vakilian, M., Johnson, R.E., Dig, D.: A comparative study of manual and automated refactorings. In: Castagna, G. (ed.) ECOOP 2013. LNCS, vol. 7920, pp. 552–576. Springer, Heidelberg (2013)
OpenRefactory/C - A refactoring infrastructure for C, http://openrefactory.org
Overbey, J.L.: A Toolkit For Constructing Refactoring Engines. PhD thesis, University of Illinois at Urbana Champaign (2011)
Refactoring actions home page, http://help.eclipse.org/juno/topic/org.eclipse.jdt.doc.user/concepts/concept-refactoring.htm
Regehr, J., Chen, Y., Cuoq, P., Eide, E., Ellison, C., Yang, X.: Test-case reduction for C compiler bugs. In: PLDI, pp. 335–346 (2012)
Schäfer, M.: Specification, Implementation and Verification of Refactorings. PhD thesis, Oxford University Computing Laboratory (2010)
Schäfer, M., Ekman, T., de Moor, O.: Sound and extensible renaming for Java. In: OOPSLA, pp. 277–294 (2008)
Schäfer, M., Sridharan, M., Dolby, J., Tip, F.: Refactoring Java programs for flexible locking. In: ICSE, pp. 71–80 (2011)
Schäfer, M., Thies, A., Steimann, F., Tip, F.: A comprehensive approach to naming and accessibility in refactoring Java programs. IEEE Trans. Soft. Eng. 38(6), 1233–1257 (2012)
Soares, G.: Making program refactoring safer. In: ICSE, pp. 521–522 (2010)
Soares, G., Catao, B., Varjao, C., Aguiar, S., Gheyi, R., Massoni, T.: Analyzing refactorings on software repositories. In: SBES, pp. 164–173 (2011)
Song, Y., Wang, X., Xie, T., Zhang, L., Mei, H.: JDF: Detecting duplicate bug reports in Jazz. In: ICSE, pp. 315–316 (2010)
Spinellis, D.: CScout: A refactoring browser for C. Sci. of Comp. Prog. 75(4), 216–231 (2010)
Steimann, F., Kollee, C., von Pilgrim, J.: A refactoring constraint language and its application to eiffel. In: Mezini, M. (ed.) ECOOP 2011. LNCS, vol. 6813, pp. 255–280. Springer, Heidelberg (2011)
Sun, B., Shu, G., Podgurski, A., Ray, S.: CARIAL: Cost-aware software reliability improvement with active learning. In: ICST, pp. 360–369 (2012)
Thies, A., Steimann, F.: Systematic testing of refactoring tools. In: AST (poster) (2010), http://www.fernuni-hagen.de/ps/prjs/rtt/rtt_poster.pdf
Vakilian, M., Chen, N., Negara, S., Rajkumar, B.A., Bailey, B.P., Johnson, R.E.: Use, disuse, and misuse of automated refactorings. In: ICSE, pp. 233–243 (2012)
Visual Assist X home page, http://www.wholetomato.com
Wang, X., Zhang, L., Xie, T., Anvik, J., Sun, J.: An approach to detecting duplicate bug reports using natural language and execution information. In: ICSE, pp. 461–470 (2008)
Wloka, J., Hirschfeld, R., Hänsel, J.: Tool-supported refactoring of aspect-oriented programs. In: AOSD, pp. 132–143 (2008)
XRefactory home page, http://www.xref.sk/xrefactory/main.html
Zeller, A., Hildebrandt, R.: Simplifying and isolating failure-inducing input. IEEE Trans. Soft. Eng. 28(2), 183–200 (2002)
zlib home page, http://www.zlib.net
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Gligoric, M., Behrang, F., Li, Y., Overbey, J., Hafiz, M., Marinov, D. (2013). Systematic Testing of Refactoring Engines on Real Software Projects. In: Castagna, G. (eds) ECOOP 2013 – Object-Oriented Programming. ECOOP 2013. Lecture Notes in Computer Science, vol 7920. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39038-8_26
Download citation
DOI: https://doi.org/10.1007/978-3-642-39038-8_26
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39037-1
Online ISBN: 978-3-642-39038-8
eBook Packages: Computer ScienceComputer Science (R0)