Skip to main content
SpringerLink
Log in
Book cover

International Conference on Parallel Problem Solving from Nature

PPSN 2022: Parallel Problem Solving from Nature – PPSN XVII pp 3–18Cite as

Digging into Semantics: Where Do Search-Based Software Repair Methods Search?

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13399))

Abstract

Search-based methods are a popular approach for automatically repairing software bugs, a field known as automated program repair (APR). There is increasing interest in empirical evaluation and comparison of different APR methods, typically measured as the rate of successful repairs on benchmark sets of buggy programs. Such evaluations, however, fail to explain why some approaches succeed and others fail. Because these methods typically use syntactic representations, i.e., source code, we know little about how the different methods explore their semantic spaces, which is relevant for assessing repair quality and understanding search dynamics. We propose an automated method based on program semantics, which provides quantitative and qualitative information about different APR search-based techniques. Our approach requires no manual annotation and produces both mathematical and human-understandable insights. In an empirical evaluation of 4 APR tools and 34 defects, we investigate the relationship between search-space exploration, semantic diversity and repair success, examining both the overall picture and how the tools’ search unfolds. Our results suggest that population diversity alone is not sufficient for finding repairs, and that searching in the right place is more important than searching broadly, highlighting future directions for the research community.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Ackling, T., Alexander, B., Grunert, I.: Evolving patches for software repair. In: GECCO 2011, Dublin, Ireland, pp. 1427–1434. ACM (2011). https://doi.org/10.1145/2001576.2001768

  2. Arcuri, A.: Evolutionary repair of faulty software. Appl. Soft Comput. 11(4), 3494–3514 (2011)

    Article  Google Scholar 

  3. Beadle, L., Johnson, C.G.: Semantic analysis of program initialisation in genetic programming. Genet. Program. Evolvable Mach. 10(3), 307–337 (2009). https://doi.org/10.1007/s10710-009-9082-5. https://link.springer.com/article/10.1007/s10710-009-9082-5

  4. Cashin, P., Martinez, C., Weimer, W., Forrest, S.: Understanding automatically-generated patches through symbolic invariant differences. In: ASE 2019, San Diego, USA, pp. 411–414. IEEE (November 2019). https://doi.org/10.1109/ASE.2019.00046

  5. Ding, Z.Y.: Patch quality and diversity of invariant-guided search-based program repair. arXiv (March 2020). https://arxiv.org/abs/2003.11667v1

  6. Ding, Z.Y., Lyu, Y., Timperley, C., Le Goues, C.: Leveraging program invariants to promote population diversity in search-based automatic program repair. In: 2019 IEEE/ACM International Workshop on Genetic Improvement (GI), pp. 2–9. IEEE (2019)

    Google Scholar 

  7. Eiben, A.E., Smith, J.E.: Introduction to Evolutionary Computing. Natural Computing Series, vol. 53. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-662-05094-1

  8. Ernst, M.D., Czeisler, A., Griswold, W.G., Notkin, D.: Quickly detecting relevant program invariants. In: Proceedings of the 22nd International Conference on Software Engineering, pp. 449–458 (2000)

    Google Scholar 

  9. Ernst, M.D., et al.: The Daikon system for dynamic detection of likely invariants. Sci. Comput. Program. 69(1–3), 35–45 (2007). https://doi.org/10.1016/j.scico.2007.01.015

  10. Feldt, R.: Generating diverse software versions with genetic programming: an experimental study. IEE Proc. Softw. 145(6), 228–236 (1998)

    Article  Google Scholar 

  11. Fry, Z.P., Landau, B., Weimer, W.: A human study of patch maintainability. In: ISSTA 2012, Minneapolis, USA, p. 177. ACM (2012). https://doi.org/10.1145/2338965.2336775. http://dl.acm.org/citation.cfm?doid=2338965.2336775

  12. Gazzola, L., Micucci, D., Mariani, L.: Automatic software repair: a survey. IEEE Trans. Softw. Eng. 45(1), 34–67 (2017). https://doi.org/10.1109/TSE.2017.2755013

    Article  Google Scholar 

  13. Harrand, N., Allier, S., Rodriguez-Cancio, M., Monperrus, M., Baudry, B.: A journey among Java neutral program variants. Genet. Program Evolvable Mach. 20(4), 531–580 (2019). https://doi.org/10.1007/s10710-019-09355-3

    Article  Google Scholar 

  14. Jiang, J., Xiong, Y., Zhang, H., Gao, Q., Chen, X.: Shaping program repair space with existing patches and similar code. In: ISSTA 2018, Amsterdam, Netherlands, vol. 18, pp. 298–309. ACM (July 2018). https://doi.org/10.1145/3213846.3213871. https://dl.acm.org/doi/10.1145/3213846.3213871

  15. Just, R., Jalali, D., Ernst, M.D.: Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In: ISSTA 2014, San Jose, USA, pp. 437–440. ACM (July 2014). https://doi.org/10.1145/2610384.2628055. http://dl.acm.org/citation.cfm?doid=2610384.2628055

  16. Kou, R., Higo, Y., Kusumoto, S.: A capable crossover technique on automatic program repair. In: IWESEP 2016, Osaka, Japan, pp. 45–50. IEEE (2016). https://doi.org/10.1109/IWESEP.2016.15

  17. Lance, G.N., Williams, W.T.: A general theory of classificatory sorting strategies: 1. Hierarchical systems. Comput. J. 9(4), 373–380 (1967)

    Article  Google Scholar 

  18. Langdon, W.B., Veerapen, N., Ochoa, G.: Visualising the search landscape of the triangle program. In: McDermott, J., Castelli, M., Sekanina, L., Haasdijk, E., García-Sánchez, P. (eds.) EuroGP 2017. LNCS, vol. 10196, pp. 96–113. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-55696-3_7

    Chapter  Google Scholar 

  19. Le, X.B.D., Thung, F., Lo, D., Goues, C.L.: Overfitting in semantics-based automated program repair. Empir. Softw. Eng. 23(5), 3007–3033 (2018)

    Article  Google Scholar 

  20. Le Goues, C., Dewey-Vogt, M., Forrest, S., Weimer, W.: A systematic study of automated program repair: fixing 55 out of 105 bugs for \$8 each. In: ICSE 2012, Zürich, Switzerland, pp. 3–13. IEEE (2012). https://doi.org/10.1109/ICSE.2012.6227211

  21. Le Goues, C., et al.: The ManyBugs and IntroClass benchmarks for automated repair of C programs. IEEE Trans. Softw. Eng. 41(12), 1236–1256 (2015)

    Google Scholar 

  22. Le Goues, C., Nguyen, T., Forrest, S., Weimer, W.: GenProg: a genetic method for automatic software repair. IEEE Trans. Softw. Eng. 38(1), 54–72 (2012). https://doi.org/10.1109/TSE.2011.104

    Article  Google Scholar 

  23. Le Goues, C., Pradel, M., Roychoudhury, A.: Automated program repair (December 2019). https://doi.org/10.1145/3318162. https://dl.acm.org/doi/10.1145/3318162

  24. Le Goues, C., Weimer, W., Forrest, S.: Representations and operators for improving evolutionary software repair. In: Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation, pp. 959–966 (2012)

    Google Scholar 

  25. Liu, K., Koyuncu, A., Kim, D., Bissyandé, T.F.: TBAR: revisiting template-based automated program repair. In: ISSTA 2019, Beijing, China, pp. 43–54. ACM (July 2019). https://doi.org/10.1145/3293882.3330577. https://dl.acm.org/doi/10.1145/3293882.3330577

  26. Liu, K., et al.: A critical review on the evaluation of automated program repair systems. J. Syst. Softw. 171, 110817 (2021)

    Article  Google Scholar 

  27. Long, F., Amidon, P., Rinard, M.: Automatic inference of code transforms for patch generation. In: ESEC/FSE 2017, Paderborn, Germany, vol. Part F1301, pp. 727–739. ACM (August 2017). https://doi.org/10.1145/3106237.3106253. https://dl.acm.org/doi/10.1145/3106237.3106253

  28. Long, F., Rinard, M.: Prophet: automatic patch generation via learning from successful patches. Technical report, MIT-CSAIL (July 2015). www.csail.mit.edu

  29. Long, F., Rinard, M.: Staged program repair with condition synthesis. In: ESEC/FSE 2015, Bergamo, Italy, pp. 166–178. ACM (August 2015). https://doi.org/10.1145/2786805.2786811. https://dl.acm.org/doi/10.1145/2786805.2786811

  30. Long, F., Rinard, M.: An analysis of the search spaces for generate and validate patch generation systems. In: ICSE 2016, Austin, Texas, May, vol. 14–22, pp. 702–713. IEEE Computer Society (May 2016). https://doi.org/10.1145/2884781.2884872

  31. Monperrus, M.: Automatic software repair: a bibliography. ACM Comput. Surv. (CSUR) 51(1), 17 (2018)

    Google Scholar 

  32. Motwani, M., Sankaranarayanan, S., Just, R., Brun, Y.: Do automated program repair techniques repair hard and important bugs? Empir. Softw. Eng. 23(5), 2901–2947 (2018). https://doi.org/10.1007/s10664-017-9550-0. https://link.springer.com/article/10.1007/s10664-017-9550-0

  33. Motwani, M., Soto, M., Brun, Y., Just, R., Le Goues, C.: Quality of automated program repair on real-world defects. IEEE Trans. Softw. Eng. 48, 637–661 (2020)

    Article  Google Scholar 

  34. Nilizadeh, A., Leavens, G.T., Le, X.B.D., Păsăreanu, C.S., Cok, D.R.: Exploring true test overfitting in dynamic automated program repair using formal methods. In: 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), pp. 229–240. IEEE (2021)

    Google Scholar 

  35. Orlov, M., Sipper, M.: Genetic programming in the wild: evolving unrestricted bytecode. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, pp. 1043–1050 (2009)

    Google Scholar 

  36. Petke, J., Brownlee, A.E.I., Alexander, B., Wagner, M., Barr, E.T., White, D.R.: A survey of genetic improvement search spaces. In: GECCO 2019, Prague, Czech Republic, pp. 1715–1721. ACM (July 2019). https://doi.org/10.1145/3319619.3326870. https://dl.acm.org/doi/10.1145/3319619.3326870

  37. Qi, Y., Mao, X., Lei, Y., Dai, Z., Wang, C.: The strength of random search on automated program repair. In: ICSE 2014, Hyderabad, India, pp. 254–265. ACM (2014). https://doi.org/10.1145/2568225.2568254

  38. Qi, Z., Long, F., Achour, S., Rinard, M.: An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In: ISSTA 2015, Baltimore, USA, pp. 24–36. ACM (2015). https://doi.org/10.1145/2771783.2771791

  39. Renzullo, J., Weimer, W., Forrest, S.: Multiplicative weights algorithms for parallel automated software repair. In: 35th IEEE International Parallel and Distributed Processing Symposium (2021)

    Google Scholar 

  40. Renzullo, J., Weimer, W., Moses, M., Forrest, S.: Neutrality and epistasis in program space. In: ICSE 2018, Gothenburg, Sweden, vol. 18, pp. 1–8. IEEE Computer Society (June 2018). https://doi.org/10.1145/3194810.3194812. https://dl.acm.org/doi/10.1145/3194810.3194812

  41. Schulte, E., Forrest, S., Weimer, W.: Automated program repair through the evolution of assembly code. In: ASE 2010, Antwerp, Belgium, pp. 313–316. ACM (2010). https://doi.org/10.1145/1858996.1859059. http://portal.acm.org/citation.cfm?doid=1858996.1859059

  42. Schulte, E., Fry, Z.P., Fast, E., Weimer, W., Forrest, S.: Software mutational robustness. Genet. Program. Evolvable Mach. 15(3), 281–312 (2014). https://doi.org/10.1007/s10710-013-9195-8. https://link.springer.com/article/10.1007/s10710-013-9195-8

  43. Smith, E.K., Barr, E.T., Le Goues, C., Brun, Y.: Is the cure worse than the disease? Overfitting in automated program repair. In: ESEC/FSE 2015, Bergamo, Italy, pp. 532–543. ACM (2015). https://doi.org/10.1145/2786805.2786825

  44. Soto, M.: Improving patch quality by enhancing key components of automatic program repair. In: 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1230–1233. IEEE (2019)

    Google Scholar 

  45. Staats, M., Hong, S., Kim, M., Rothermel, G.: Understanding user understanding: determining correctness of generated program invariants. In: ISSTA 2012, Minneapolis, MN, p. 188. ACM (2012). https://doi.org/10.1145/2338965.2336776. http://dl.acm.org/citation.cfm?doid=2338965.2336776

  46. Tan, S.H., Yoshida, H., Prasad, M.R., Roychoudhury, A.: Anti-patterns in search-based program repair. In: ESEC/FSE 2016, November, vol. 13–18, pp. 727–738. ACM, New York (November 2016). https://doi.org/10.1145/2950290.2950295. https://dl.acm.org/doi/10.1145/2950290.2950295

  47. Urli, S., Yu, Z., Seinturier, L., Monperrus, M., Monperrus, M.: How to design a program repair bot? Insights from the repairnator project. In: ICSE-SEIP 2018, vol. 10 (2018). https://doi.org/10.1145/3183519

  48. Veerapen, N., Daolio, F., Ochoa, G.: Modelling genetic improvement landscapes with local optima networks. In: GECCO 2017, vol. 6, pp. 1543–1548. ACM, New York (July 2017). https://doi.org/10.1145/3067695.3082518. https://dl.acm.org/doi/10.1145/3067695.3082518

  49. Veerapen, N., Ochoa, G.: Visualising the global structure of search landscapes: genetic improvement as a case study. Genet. Program. Evolvable Mach. 19(3), 317–349 (September 2018). https://doi.org/10.1007/s10710-018-9328-1

  50. Vessey, I., Weber, R.: Some factors affecting program repair maintenance: an empirical study. Commun. ACM 26(2), 128–134 (1983)

    Article  Google Scholar 

  51. Villanueva, O.M., Trujillo, L., Hernandez, D.E.: Novelty search for automatic bug repair. In: GECCO 2020, Cancun, Mexico, pp. 1021–1028. ACM (2020). https://doi.org/10.1145/3377930.3389845. https://dl.acm.org/doi/10.1145/3377930.3389845

  52. Wang, S., et al.: Automated patch correctness assessment: how far are we? ASE 2020, 968–980 (2020). https://doi.org/10.1145/3324884.3416590

    Article  Google Scholar 

  53. Weimer, W., Fry, Z.P., Forrest, S.: Leveraging program equivalence for adaptive program repair: models and first results. In: ASE 2013, Silicon Valley, USA, pp. 356–366. IEEE (2013). https://doi.org/10.1109/ASE.2013.6693094

  54. Weimer, W., Nguyen, T., Le Goues, C., Forrest, S.: Automatically finding patches using genetic programming. In: ICSE 2009, Vancouver, Canada, pp. 364–367. IEEE (2009). https://doi.org/10.1109/ICSE.2009.5070536

  55. Wen, M., Chen, J., Wu, R., Hao, D., Cheung, S.C.: Context-aware patch generation for better automated program repair. In: ICSE 2018, Pittsburgh, Pennsylvania, January, vol. 2018, pp. 1–11. IEEE Computer Society (2018). https://doi.org/10.1145/3180155.3180233

  56. Yang, D., Qi, Y., Mao, X.: Evaluating the strategies of statement selection in automated program repair. In: Bu, L., Xiong, Y. (eds.) SATE 2018. LNCS, vol. 11293, pp. 33–48. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04272-1_3

    Chapter  Google Scholar 

  57. Yuan, Y., Banzhaf, W.: Making better use of repair templates in automated program repair: a multi-objective approach. In: Evolution in Action: Past, Present and Future. GEC, pp. 385–407. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-39831-6_26

    Chapter  Google Scholar 

Download references

Acknowledgements

We gratefully acknowledge the partial support of the NSF (CCF 2211749, 2141300, 1763674, 1908633, and CICI 2115075), DARPA (N6600120C4020, FA8750-19C-0003, HR001119S0089-AMP-FP-029), and AFRL (FA8750-19-1-0501).

Author information

Authors and Affiliations

  1. University of Michigan, Ann Arbor, MI, 48109, USA

    Hammad Ahmad & Westley Weimer

  2. Arizona State University, Tempe, AZ, 85281, USA

    Padriac Cashin & Stephanie Forrest

Authors
  1. Hammad Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Padriac Cashin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stephanie Forrest
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Westley Weimer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hammad Ahmad .

Editor information

Editors and Affiliations

  1. TU Dortmund, Dortmund, Germany

    Günter Rudolph

  2. Leiden University, Leiden, The Netherlands

    Anna V. Kononova

  3. Shinshu University, Nagano, Japan

    Hernán Aguirre

  4. Technische Universität Dresden, Dresden, Germany

    Pascal Kerschke

  5. University of Stirling, Stirling, UK

    Gabriela Ochoa

  6. Jožef Stefan Institute, Ljubljana, Slovenia

    Tea Tušar

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ahmad, H., Cashin, P., Forrest, S., Weimer, W. (2022). Digging into Semantics: Where Do Search-Based Software Repair Methods Search?. In: Rudolph, G., Kononova, A.V., Aguirre, H., Kerschke, P., Ochoa, G., Tušar, T. (eds) Parallel Problem Solving from Nature – PPSN XVII. PPSN 2022. Lecture Notes in Computer Science, vol 13399. Springer, Cham. https://doi.org/10.1007/978-3-031-14721-0_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-14721-0_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-14720-3

  • Online ISBN: 978-3-031-14721-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation