research-article

The strength of random search on automated program repair

Authors:

Chengsong WangAuthors Info & Claims

ICSE 2014: Proceedings of the 36th International Conference on Software Engineering

Pages 254 - 265

https://doi.org/10.1145/2568225.2568254

Published: 31 May 2014 Publication History

Abstract

Automated program repair recently received considerable attentions, and many techniques on this research area have been proposed. Among them, two genetic-programming-based techniques, GenProg and Par, have shown the promising results. In particular, GenProg has been used as the baseline technique to check the repair effectiveness of new techniques in much literature. Although GenProg and Par have shown their strong ability of fixing real-life bugs in nontrivial programs, to what extent GenProg and Par can benefit from genetic programming, used by them to guide the patch search process, is still unknown.

To address the question, we present a new automated repair technique using random search, which is commonly considered much simpler than genetic programming, and implement a prototype tool called RSRepair. Experiment on 7 programs with 24 versions shipping with real-life bugs suggests that RSRepair, in most cases (23/24), outperforms GenProg in terms of both repair effectiveness (requiring fewer patch trials) and efficiency (requiring fewer test case executions), justifying the stronger strength of random search over genetic programming. According to experimental results, we suggest that every proposed technique using optimization algorithm should check its effectiveness by comparing it with random search.

References

[1]

T. Ackling, B. Alexander, and I. Grunert. Evolving patches for software repair. In Genetic and Evolutionary Computation (GECCO), pages 1427–1434, 2011.

Digital Library

[2]

A. Arcuri. On the automation of fixing software bugs. In International Conference on Software Engineering (ICSE), pages 1003–1006, 2008.

Digital Library

[3]

A. Arcuri. Evolutionary repair of faulty software. Applied Soft Computing, 11(4):3494 – 3514, 2011.

Digital Library

[4]

A. Arcuri and L. Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In International Conference on Software Engineering (ICSE), pages 1–10, 2011.

Digital Library

[5]

A. Arcuri and X. Yao. A novel co-evolutionary approach to automatic software bug fixing. In IEEE Congress on Evolutionary Computation (CEC), pages 162–168, 2008.

[6]

V. Dallmeier, A. Zeller, and B. Meyer. Generating fixes from object behavior anomalies. In International Conference on Automated Software Engineering (ASE), pages 550–554, 2009.

Digital Library

[7]

V. Debroy and W. Wong. Using mutation to automatically suggest fixes for faulty programs. In International Conference on Software Testing, Verification and Validation (ICST), pages 65 –74, 2010.

Digital Library

[8]

H. Do, G. Rothermel, and A. Kinneer. Empirical studies of test case prioritization in a junit testing environment. In International Symposium on Software Reliability Engineering (ISSRE), pages 113 – 124, 2004.

Digital Library

[9]

S. Elbaum, A. Malishevsky, and G. Rothermel. Test case prioritization: a family of empirical studies. IEEE Transactions on Software Engineering (TSE), 28(2):159 –182, feb 2002.

Digital Library

[10]

E. Fast, C. Le Goues, S. Forrest, and W. Weimer. Designing better fitness functions for automated program repair. In Genetic and Evolutionary Computation (GECCO), pages 965–972, 2010.

Digital Library

[11]

Z. P. Fry, B. Landau, and W. Weimer. A human study of patch maintainability. In International Symposium on Software Testing and Analysis (ISSTA), pages 177–187, 2012.

Digital Library

[12]

A. Griesmayer, R. Bloem, and B. Cook. Repair of boolean programs with an application to c. In Computer Aided Verification (CAV), pages 358–371, 2006.

Digital Library

[13]

M. Harman. Automated patching techniques: the fix is in: technical perspective. Communications of the ACM, 53(5):108–108, 2010.

Digital Library

[14]

M. Harman, S. A. Mansouri, and Y. Zhang. Search-based software engineering: Trends, techniques and applications. ACM Computing Surveys, 45(1):1–61, Dec. 2012.

Digital Library

[15]

M. Harman, P. McMinn, J. Souza, and S. Yoo. Search based software engineering: Techniques, taxonomy, tutorial. Empirical Software Engineering and Verification, 7007:1–59, 2012.

Digital Library

[16]

G. Jin, L. Song, W. Zhang, S. Lu, and B. Liblit. Automated atomicity-violation fixing. In Programming Language Design and Implementation (PLDI), pages 389–400, 2011.

Digital Library

[17]

W. Jin and A. Orso. F3: fault localization for field failures. In International Symposium on Software Testing and Analysis (ISSTA), pages 213–223, 2013.

Digital Library

[18]

D. Kim, J. Nam, J. Song, and S. Kim. Automatic patch generation learned from human-written patches. In International Conference on Software Engineering (ICSE), pages 802–811, 2013.

Digital Library

[19]

B. Korel, G. Koutsogiannakis, and L. Tahat. Application of system models in regression test suite prioritization. In International Conference on Software Maintenance (ICSM), pages 247 –256, 2008.

[20]

C. Le Goues. Automatic program repair using genetic programming. PhD thesis, University of Virginia, 2013.

[21]

C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer. A systematic study of automated program repair: fixing 55 out of 105 bugs for $8 each. In International Conference on Software Engineering (ICSE), pages 3–13, 2012.

Digital Library

[22]

C. Le Goues, S. Forrest, and W. Weimer. Current challenges in automatic software repair. Software Quality Journal, 21(3):421–443, 2013.

Digital Library

[23]

C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer. GenProg: a generic method for automatic software repair. IEEE Transactions on Software Engineering (TSE), 38(1):54 –72, 2012.

Digital Library

[24]

C. Le Goues, W. Weimer, and S. Forrest. Representations and operators for improving evolutionary software repair. In Genetic and Evolutionary Computation (GECCO), 2012.

Digital Library

[25]

P. Liu and C. Zhang. Axis: automatically fixing atomicity violations through solving control constraints. In International Conference on Software Engineering (ICSE), pages 299–309, 2012.

Digital Library

[26]

M. Monperrus. A critical review of “automatic patch generation learned from human-written patches”: an essay on the problem statement and the evaluation of automatic software repair. In International Conference on Software Engineering (ICSE), 2014 (To appear).

Digital Library

[27]

H. D. T. Nguyen, D. Qi, A. Roychoudhury, and S. Chandra. SemFix: program repair via semantic analysis. In International Conference on Software Engineering (ICSE), pages 772–781, 2013.

Digital Library

[28]

Y. Pei, Y. Wei, C. Furia, M. Nordio, and B. Meyer. Code-based automated program fixing. In International Conference on Automated Software Engineering (ASE), pages 392 –395, 2011.

Digital Library

[29]

S. Poulding and J. A. Clark. Eﬃcient software verification: Statistical testing using automated search. IEEE Transactions on Software Engineering (TSE), 36(6):763–777, Nov. 2010.

Digital Library

[30]

Y. Qi, X. Mao, and Y. Lei. Eﬃcient automated program repair through fault-recorded testing prioritization. In International Conference on Software Maintenance (ICSM), pages 180–189, 2013.

Digital Library

[31]

Y. Qi, X. Mao, Y. Lei, Z. Dai, and C. Wang. Does genetic programming work well on automated program repair? In International Conference on Computational and Information Sciences (ICCIS), pages 1875–1878, 2013.

Digital Library

[32]

Y. Qi, X. Mao, Y. Lei, and C. Wang. Using automated program repair for evaluating the effectiveness of fault localization techniques. In International Symposium on Software Testing and Analysis (ISSTA), pages 191–201, 2013.

Digital Library

[33]

Y. Qi, X. Mao, Y. Wen, Z. Dai, and B. Gu. More eﬃcient automatic repair of large-scale programs using weak recompilation. Science China Information Sciences, 55(12):2785–2799, 2012.

[34]

G. Rothermel, R. Untch, C. Chu, and M. Harrold. Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering (TSE), 27(10):929 –948, oct 2001.

Digital Library

[35]

H. Samimi, M. Schäfer, S. Artzi, T. Millstein, F. Tip, and L. Hendren. Automated repair of HTML generation errors in php applications using string constraint solving. In International Conference on Software Engineering (ICSE), pages 277–287, 2012.

Digital Library

[36]

A. Vargha and H. D. Delaney. A critique and improvement of the CL common language effect size statistics of mcgraw and wong. Journal of Educational and Behavioral Statistics, 25(2):101–132, 2000.

[37]

Y. Wei, Y. Pei, C. A. Furia, L. S. Silva, S. Buchholz, B. Meyer, and A. Zeller. Automated fixing of programs with contracts. In International Symposium on Software Testing and Analysis (ISSTA), pages 61–72, 2010.

Digital Library

[38]

W. Weimer, S. Forrest, C. Le Goues, and T. Nguyen. Automatic program repair with evolutionary computation. Communications of the ACM, 53(5):109–116, 2010.

Digital Library

[39]

W. Weimer, Z. P. Fry, and S. Forrest. Leveraging program equivalence for adaptive program repair: models and first results. In International Conference on Automated Software Engineering (ASE), pages 356–366, 2013.

Digital Library

[40]

W. Weimer, T. Nguyen, C. Le Goues, and S. Forrest. Automatically finding patches using genetic programming. In International Conference on Software Engineering (ICSE), pages 364–374, 2009.

Digital Library

[41]

F. Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80 – 83, 1945.

[42]

X. Xie, T. Y. Chen, F.-c. Kuo, and B. Xu. A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization. ACM Transactions on Software Engineering and Methodology (TOSEM), 22(4):Article 31, 2013.

Digital Library

[43]

S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab, 22(4):67–120, 2012.

Digital Library

[44]

L. Zhang, S.-S. Hou, C. Guo, T. Xie, and H. Mei. Time-aware test-case prioritization using integer linear programming. In International Symposium on Software Testing and Analysis (ISSTA), pages 213–224, 2009.

Digital Library

Cited By

Zemín LGodio ACornejo CDegiovanni RGutiérrez Brida SRegis GAguirre NFrias M(2025)An Empirical Study on the Suitability of Test-based Patch Acceptance CriteriaACM Transactions on Software Engineering and Methodology10.1145/370297134:3(1-20)Online publication date: 20-Feb-2025
https://dl.acm.org/doi/10.1145/3702971
Nguyen TZhang XArcaini PIshikawa FVo H(2025)Automated program repair for variability bugs in software product line systemsJournal of Systems and Software10.1016/j.jss.2024.112152221:COnline publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1016/j.jss.2024.112152
Xing YWang SSun SHe XSun KLi QBalzarotti DXu W(2024)What IF is not enough? fixing null pointer dereference with contextual checkProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3698977(1367-1382)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3698977
Show More Cited By

Index Terms

The strength of random search on automated program repair
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools

Recommendations

Does Genetic Programming Work Well on Automated Program Repair?
ICCIS '13: Proceedings of the 2013 International Conference on Computational and Information Sciences

Automated program repair has made some important progress in the recent decade. One well-known repair tool is GneProg, which automates the patch generation process according to the guidance of genetic programming. Although GenProg has successfully fixed ...
Enhanced evolutionary automated program repair by finer‐granularity ingredients and better search algorithms
Summary
Bug repair is time consuming and tedious, which hampers software maintenance. To alleviate the burden, automated program repair (APR) is proposed and has been fruitful in the last decade. Evolutionary repair is the seminal work of this field and ...

MicroRepair is a new automated program repair approach that enlarges the search space by breaking statements into finer‐granularity ingredients. It outperforms state‐of‐the‐art evolutionary APR approaches in terms of precision, correctly repairing 26 ...
Enhanced Evolutionary Automated Program Repair by Finer-Granularity Ingredients and Better Search Algorithms
Internetware '22: Proceedings of the 13th Asia-Pacific Symposium on Internetware

Bug repair is time-consuming and tedious, which hampers software maintenance. To alleviate the burden, automated program repair (APR) is proposed and has been fruitful in the last decade. Evolutionary repair is the seminal work of this field and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE 2014: Proceedings of the 36th International Conference on Software Engineering

May 2014

1139 pages

ISBN:9781450327565

DOI:10.1145/2568225

General Chair:
Pankaj Jalote
IIIT-Delhi, India
,
Program Chairs:
Lionel Briand
University of Luxembourg, Luxembourg
,
André van der Hoek
University of California, Irvine, USA

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '14

Sponsor:

SIGSOFT

ICSE '14: 36th International Conference on Software Engineering

May 31 - June 7, 2014

Hyderabad, India

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

240
Total Citations
View Citations
1,272
Total Downloads

Downloads (Last 12 months)83
Downloads (Last 6 weeks)17

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zemín LGodio ACornejo CDegiovanni RGutiérrez Brida SRegis GAguirre NFrias M(2025)An Empirical Study on the Suitability of Test-based Patch Acceptance CriteriaACM Transactions on Software Engineering and Methodology10.1145/370297134:3(1-20)Online publication date: 20-Feb-2025
https://dl.acm.org/doi/10.1145/3702971
Nguyen TZhang XArcaini PIshikawa FVo H(2025)Automated program repair for variability bugs in software product line systemsJournal of Systems and Software10.1016/j.jss.2024.112152221:COnline publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1016/j.jss.2024.112152
Xing YWang SSun SHe XSun KLi QBalzarotti DXu W(2024)What IF is not enough? fixing null pointer dereference with contextual checkProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3698977(1367-1382)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3698977
Fei ZGe JLi CWang TLi YZhang HHuang LLuo B(2024)Patch Correctness Assessment: A SurveyACM Transactions on Software Engineering and Methodology10.1145/370297234:2(1-50)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3702972
Lin BWang SWen MChen LMao XChristakis MPradel M(2024)One Size Does Not Fit All: Multi-granularity Patch Generation for Better Automated Program RepairProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680381(1554-1566)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680381
Ouyang YYang JZhang LChristakis MPradel M(2024)Benchmarking Automated Program Repair: An Extensive Study on Both Real-World and Artificial BugsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3652140(440-452)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3652140
Zhang JCambronero JGulwani SLe VPiskac RSoares GVerbruggen G(2024)PyDex: Repairing Bugs in Introductory Python Assignments using LLMsProceedings of the ACM on Programming Languages10.1145/36498508:OOPSLA1(1100-1124)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3649850
Yuan YHuyen PTan SMechtaev SKhurshid S(2024)ARJA-e for the First International Competition on Automated Program RepairProceedings of the 5th ACM/IEEE International Workshop on Automated Program Repair10.1145/3643788.3648019(50-52)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643788.3648019
Gao CYang WYe JXue YSun J(2024)sGuard+: Machine Learning Guided Rule-Based Automated Vulnerability Repair on Smart ContractsACM Transactions on Software Engineering and Methodology10.1145/364184633:5(1-55)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3641846
Zirak AHemmati H(2024)Improving Automated Program Repair with Domain AdaptationACM Transactions on Software Engineering and Methodology10.1145/363197233:3(1-43)Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1145/3631972
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten