Evolving Software: Combining Online Learning with Mutation-Based Stochastic Search
Abstract
1 Introduction
2 Repair Model and Research Questions
3 Background
3.1 Online Learning Algorithms
3.2 Evolving Software
3.2.1 Neutral Mutations.
3.2.2 Search Landscapes.
3.3 Automated Program Repair
3.3.1 Test Oracle Problem.
3.3.2 Terminology.
3.4 MWU and Regret

4 Characterizing the Search Space
4.1 Combining Neutral Mutations

4.2 Optimal Combinations of Neutral Mutations
Program | Description | Language | kLoC | Single Mutation Patch Probability | Patch Probability at Peak | Optimal N (No. of Mutations) | Advantage |
---|---|---|---|---|---|---|---|
gzip-2009-08-16 | Compression | C | 480 | 0.270% | 7.70% | 48 | 28.4 |
gzip-2009-09-26 | Compression | C | 480 | 0.028% | 0.90% | 34 | 32.0 |
libtiff-2005-12-14 | Image Processing | C | 77 | 0.810% | 14.00% | 46 | 17.2 |
lighttpd-1806-1807 | Web Server | C | 62 | 0.041% | 1.50% | 11 | 36.2 |
ccrypt | Encryption | C | 7 | 0.980% | 39.50% | 43 | 39.9 |
look | Dictionary | C | 1 | 1.580% | 69.60% | 110 | 43.9 |
units | Conversion | C | 1 | 0.022% | 2.20% | 271 | 99.5 |
Chart 13 | Plotting | Java | 96 | 0.100% | 0.10% | 1 | 1.0 |
Chart 26 | Plotting | Java | 96 | 0.050% | 1.50% | 85 | 30.0 |
Closure 13 | JavaScript Compiler | Java | 90 | 1.050% | 5.15% | 13 | 4.9 |
Closure 19 | JavaScript Compiler | Java | 90 | 0.150% | 2.15% | 91 | 14.3 |
Closure 22 | JavaScript Compiler | Java | 90 | 0.500% | 0.50% | 1 | 1.0 |
Closure 107 | JavaScript Compiler | Java | 90 | 0.140% | 0.23% | 2 | 1.67 |
Lang 10 | Java Language | Java | 22 | 0.100% | 0.10% | 1 | 1.0 |
Math 8 | Math Utilities | Java | 85 | 0.500% | 5.45% | 75 | 10.9 |
Math 28 | Math Utilities | Java | 85 | 0.850% | 0.85% | 1 | 1.0 |
Math 80 | Math Utilities | Java | 85 | 0.090% | 0.32% | 9 | 3.5 |

4.3 Modeling the Observed Distribution
4.4 Do We Observe Similar Dynamics in Other Programs?
5 The MWRepair Algorithm




5.1 Summary
6 Experimental Results
6.1 RQ 4: Does MWRepair Improve the Ability of Java-Based GenProg Tools to Discover Patches and (Unique) Repairs?
Algorithm | No. of Defects | Plausible Patches (Repairs) | Average Success Rate | Variance | p-Value |
---|---|---|---|---|---|
MWRepair | 357 | 54 (16) | 0.151 (0.045) | 0.129 (0.043) | |
jGenProg | 357 | 21 (6) | 0.059 (0.017) | 0.056 (0.017) | 5.87E-7 (7.35E-3) |
GenProg-A | 357 | 30 (2) | 0.084 (0.006) | 0.077 (0.006) | 8.16E-4 (1.62E-4) |
Algorithm | Chart (26) | Lang (65) | Math (106) | Time (27) | Closure (133) | Total (357) |
---|---|---|---|---|---|---|
ACS | 2 (2) | 3 (3) | 16 (11) | 1 (1) | 0 (0) | 22 (17) |
ARJA-p | 15 (8) | 22 (9) | 42 (17) | 5 (1) | 0 (0) | 84 (35) |
AVATAR | 12 (5) | 13 (4) | 17 (3) | 0 (0) | 15 (7) | 57 (19) |
CapGen | 4 (4) | 5 (5) | 13 (13) | 0 (0) | 0 (0) | 22 (22) |
Cardumen | 4 (2) | 0 (0) | 6 (1) | 0 (0) | 2 (0) | 12 (3) |
CoCoNuT\(\dagger\) | 7 (7) | 7 (7) | 16 (16) | 1 (1) | 9 (9) | 40 (40) |
ConFix | 13 (4) | 15 (5) | 37 (6) | 6 (1) | 21 (6) | 92 (22) |
CURE\(\dagger\) | 10 (10) | 9 (9) | 19 (19) | 1 (1) | 14 (14) | 53 (53) |
DEAR | 8 (8) | 8 (8) | 21 (21) | 3 (3) | 7 (7) | 47 (47) |
DLFix | 5 (5) | 5 (5) | 12 (12) | 1 (1) | 6 (6) | 29 (29) |
DynaMoth | 6 (0) | 2 (0) | 13 (1) | 1 (0) | 0 (0) | 22 (1) |
FixMiner | 14 (5) | 2 (0) | 14 (7) | 1 (0) | 2 (0) | 33 (12) |
GenPat | 8 (4) | 11 (4) | 13 (4) | 1 (0) | 7 (6) | 40 (18) |
HDRepair* | 2 (0) | 6 (2) | 7 (4) | 1 (0) | 7 (0) | 23 (6) |
Hercules | 9 (8) | 15 (10) | 29 (20) | 5 (3) | 14 (9) | 72 (50) |
JAID* | 11 (5) | 22 (6) | 28 (9) | 2 (1) | 27 (13) | 90 (34) |
jKali | 4 (0) | 4 (2) | 8 (1) | 0 (0) | 8 (3) | 24 (6) |
jMutRepair | 4 (1) | 2 (0) | 11 (2) | 0 (0) | 5 (2) | 22 (5) |
kPAR | 13 (3) | 18 (1) | 21 (4) | 1 (0) | 10 (2) | 63 (10) |
LSRepair | 3 (3) | 8 (8) | 7 (7) | 0 (0) | 0 (0) | 18 (18) |
Nopol | 6 (0) | 6 (1) | 18 (0) | 1 (0) | 0 (0) | 31 (1) |
NPEFix | 5 (0) | 0 (0) | 3 (0) | 0 (0) | 0 (0) | 8 (0) |
RSRepair-A | 4 (0) | 3 (0) | 12 (0) | 0 (0) | 22 (4) | 41 (4) |
Restore | 4 (4) | 6 (6) | 9 (9) | 1 (1) | 21 (21) | 41 (41) |
RewardRepair | 4 (4) | 3 (3) | 14 (14) | 1 (1) | 7 (7) | 29 (29) |
SelfAPR\(\dagger\) | 9 (6) | 12 (6) | 24 (16) | 3 (0) | 23 (16) | 71 (44) |
SequenceR\(\dagger\) | 3 (3) | 2 (2) | 9 (6) | 0 (0) | 5 (3) | 19 (14) |
SimFix | 8 (3) | 16 (5) | 25 (10) | 0 (0) | 19 (7) | 68 (25) |
SketchFix* | 8 (6) | 4 (3) | 8 (7) | 1 (0) | 5 (3) | 26 (19) |
ssFix | 7 (2) | 12 (5) | 26 (6) | 4 (0) | 11 (1) | 60 (14) |
TBar | 16 (7) | 21 (6) | 23 (8) | 0 (0) | 12 (3) | 72 (24) |
VarFix | 0 (0) | 0 (0) | 24 (11) | 0 (0) | 11 (6) | 35 (17) |
GenProg-A | 5 (0) | 1 (0) | 9 (0) | 0 (0) | 15 (2) | 30 (2) |
jGenProg | 5 (0) | 2 (0) | 12 (4) | 0 (0) | 2 (2) | 21 (6) |
\(\downarrow\) | \(\downarrow\) | \(\downarrow\) | \(\downarrow\) | \(\downarrow\) | \(\downarrow\) | \(\downarrow\) |
MWRepair | 7 (1) | 10 (3) | 20 (5) | 1 (0) | 16 (7) | 54 (16) |
Combined Algs. | 24 (17) | 43 (29) | 75 (48) | 11 (5) | 72 (41) | 225 (140) |

6.2 RQ 5: Do Patches Found by MWRepair Minimize to Single Mutations or Can MWRepair Discover Multi-Edit patches?

6.3 Multi-Edit Repairs

6.4 RQ 6: How Many of the Patches MWRepair Identifies Are Correct Repairs?
Same Location | Different Location | Total | |
---|---|---|---|
Same Modification | 56.25% | 6.25% | 62.5% |
Different Modification | 31.25% | 6.25% | 37.5% |
Total | 87.50% | 12.5% |
6.5 RQ 7: Do Neutral Mutations Remain Neutral or Are They Invalidated over Time by Changes to the Test Suite as a Software Project Matures?

6.6 RQ 8: How Does MWRepair-Enhanced GenProg Compare to the State of the Art?

6.7 Summary of Results
7 Discussion
7.1 Applicability of MWU to Other Domains
7.2 Generalization of MWRepair to Other APR Algorithms
7.3 Search Space Exploration
7.4 Limitations
7.5 Benchmark Quality
8 Future Work
8.1 Efficiency Considerations
8.2 Modeling the Search Space
8.3 Other Search Space Formulations
9 Conclusion
Acknowledgments
Footnote
A MWRepair Evaluation Supplement: ManyBugs
A.1 Can We Find the Optimum with Online Learning?
A.2 ManyBugs
Program | MWRepair | GenProg | TrpAutoRepair | Kali | SPR | Prophet | Angelix | F1X | SOSRepair | Novelty |
---|---|---|---|---|---|---|---|---|---|---|
gzip-2009-08-16 | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | |||
gzip-2009-09-26 | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | |||
libtiff-2005-12-14 | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | |||||
libtiff-2005-12-21 | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | ||
libtiff-2006-02-27 | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | |||||
libtiff-2007-11-02 | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | |||
libtiff-2007-11-23 | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | |||||
libtiff-2009-08-28 | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | |||
libtiff-2010-12-13 | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | ||||
lighttpd-1806-1807 | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | |||
python-69223-69224 | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | |||||
python-70098-70101 | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) | \(\checkmark\) |
Defect | MWRepair Consistency | GenProg Consistency | MWRepair Cost | GenProg Cost |
---|---|---|---|---|
gzip-2009-08-16 | 100% | 30% | 35.64 | 130.7 |
gzip-2009-09-26 | 100% | 0% | 340.45 | \(\infty\) |
python-69223-69224 | 27% | 0% | 101.1 | \(\infty\) |
python-70098-70101 | 3% | 0% | 132.0 | \(\infty\) |
libtiff-2005-12-14 | 100% | 90% | 4.0 | 20.8 |
libtiff-2005-12-21 | 100% | 100% | 22.41 | 20.8 |
libtiff-2006-02-27 | 100% | 100% | 4.33 | 20.8 |
libtiff-2007-11-02 | 100% | 80% | 3.05 | 20.8 |
libtiff-2007-11-23 | 100% | 80% | 9.80 | 20.8 |
libtiff-2009-08-28 | 100% | 100% | 2.89 | 20.8 |
libtiff-2010-12-13 | 100% | 100% | 11.08 | 20.8 |
lighttpd-1806-1807 | 64% | 50% | 251.2 | 44.1 |
ccrypt | 100% | 100% | 53.1 | 32.3 |
look | 100% | 99% | 1.6 | 20.1 |
units | 100% | 7% | 83.13 | 61.7 |
References
Index Terms
- Evolving Software: Combining Online Learning with Mutation-Based Stochastic Search
Recommendations
Combining Online Learning with Mutation-Based Stochastic Search to Repair Buggy Programs
GECCO '24 Companion: Proceedings of the Genetic and Evolutionary Computation Conference CompanionThis article summarizes recent work in the field of Automated Program Repair that was published in Transactions on Evolutionary Learning and Optimization as Evolving Software: Combining Online Learning with Mutation-Based Stochastic Search. Automated ...
Harmony search with differential mutation based pitch adjustment
GECCO '11: Proceedings of the 13th annual conference on Genetic and evolutionary computationHarmony search (HS), as an emerging metaheuristic technique mimicking the improvisation behavior of musicians, has demonstrated strong efficacy in solving various numerical and real-world optimization problems. This work presents a harmony search with ...
Island-based Cuckoo Search with elite opposition-based learning and multiple mutation methods for solving optimization problems
AbstractThe island Cuckoo Search (iCSPM) algorithm is a variation of Cuckoo Search that uses the island model and highly disruptive polynomial mutation to solve optimization problems. This article introduces an improved iCSPM algorithm called iCSPM with ...
Comments
Information & Contributors
Information
Published In

- Editors:
- Juergen Branke,
- Manuel López-Ibáñez
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Funding Sources
- NSF
- DARPA
- AFRL
- Santa Fe Institute
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 488Total Downloads
- Downloads (Last 12 months)329
- Downloads (Last 6 weeks)53
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in