Skip to main content
SpringerLink
Log in

Generational Computation Reduction in Informal Counterexample-Driven Genetic Programming

  • Conference paper
  • First Online:
Genetic Programming (EuroGP 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14631))

Included in the following conference series:

  • 116 Accesses

Abstract

Counterexample-driven genetic programming (CDGP) uses specifications provided as formal constraints to generate the training cases used to evaluate evolving programs. It has also been extended to combine formal constraints and user-provided training data to solve symbolic regression problems. Here we show how the ideas underlying CDGP can also be applied using only user-provided training data, without formal specifications. We demonstrate the application of this method, called “informal CDGP,” to software synthesis problems. Our results show that informal CDGP finds solutions faster (i.e. with fewer program executions) than standard GP. Additionally, we propose two new variants to informal CDGP, and find that one produces significantly more successful runs on about half of the tested problems. Finally, we study whether the addition of counterexample training cases to the training set is useful by comparing informal CDGP to using a static subsample of the training set, and find that the addition of counterexamples significantly improves performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Notes

  1. 1.

    This paper expands on a poster paper that we published in GECCO 2020 [18].

  2. 2.

    Every individual in the first generation passes the training set, since it is initially empty.

  3. 3.

    The datasets for these problems are available at https://github.com/thelmuth/program-synthesis-benchmark-datasets.

  4. 4.

    https://github.com/lspector/Clojush.

  5. 5.

    Note that the diversity does not go all the way to 0, since even if one parent created all of the children in the next generation, some of those children are likely to display different behaviors from the parent.

References

  1. Błądek, I., Krawiec, K.: Solving symbolic regression problems with formal constraints. In: GECCO 2019: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 977–984. ACM, Prague (2019). https://doi.org/10.1145/3321707.3321743

  2. Błądek, I., Krawiec, K.: Counterexample-driven genetic programming for symbolic regression with formal constraints. IEEE Trans. Evol. Comput. 27(5), 1327–1339 (2023). https://doi.org/10.1109/TEVC.2022.3205286

    Article  Google Scholar 

  3. Błądek, I., Krawiec, K., Swan, J.: Counterexample-driven genetic programming: heuristic program synthesis from formal specifications. Evol. Comput. 26(3), 441–469 (2018). https://doi.org/10.1162/evco_a_00228

    Article  Google Scholar 

  4. Boldi, R., et al.: The problem solving benefits of down-sampling vary by selection scheme. In: Proceedings of the Companion Conference on Genetic and Evolutionary Computation, pp. 527–530 (2023)

    Google Scholar 

  5. Boldi, R., et al.: Informed down-sampled lexicase selection: identifying productive training cases for efficient problem solving (2023). https://doi.org/10.48550/arXiv.2301.01488, arXiv:2301.01488

  6. Boldi, R., Lalejini, A., Helmuth, T., Spector, L.: A static analysis of informed down-samples. In: Proceedings of the Companion Conference on Genetic and Evolutionary Computation, GECCO 2023 Companion, pp. 531–534. Association for Computing Machinery, New York (2023). https://doi.org/10.1145/3583133.3590751

  7. Dreossi, T., Ghosh, S., Yue, X., Keutzer, K., Sangiovanni-Vincentelli, A., Seshia, S.A.: Counterexample-guided data augmentation (2018)

    Google Scholar 

  8. Ferguson, A.J., Hernandez, J.G., Junghans, D., Lalejini, A., Dolson, E., Ofria, C.: Characterizing the effects of random subsampling on lexicase selection. In: Banzhaf, W., Goodman, E., Sheneman, L., Trujillo, L., Worzel, B. (eds.) Genetic Programming Theory and Practice XVII. GEC, pp. 1–23. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-39958-0_1

    Chapter  Google Scholar 

  9. Helmuth, T., Frazier, J.G., Shi, Y., Abdelrehim, A.F.: Human-driven genetic programming for program synthesis: a prototype. In: Proceedings of the Companion Conference on Genetic and Evolutionary Computation, GECCO 2023 Companion, pp. 1981–1989. Association for Computing Machinery, New York (2023). https://doi.org/10.1145/3583133.3596373

  10. Helmuth, T., McPhee, N.F., Pantridge, E., Spector, L.: Improving generalization of evolved programs through automatic simplification. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2017, pp. 937–944. ACM, Berlin (2017). https://doi.org/10.1145/3071178.3071330, http://doi.acm.org/10.1145/3071178.3071330

  11. Helmuth, T., McPhee, N.F., Spector, L.: Effects of lexicase and tournament selection on diversity recovery and maintenance. In: GECCO 2016 Companion: Proceedings of the Companion Publication of the 2016 Annual Conference on Genetic and Evolutionary Computation, pp. 983–990. ACM, Denver (2016). https://doi.org/10.1145/2908961.2931657, http://doi.acm.org/10.1145/2908961.2931657

  12. Helmuth, T., McPhee, N.F., Spector, L.: The impact of hyperselection on lexicase selection. In: Friedrich, T. (ed.) GECCO 2016: Proceedings of the 2016 Annual Conference on Genetic and Evolutionary Computation, pp. 717–724. ACM, Denver (2016). https://doi.org/10.1145/2908812.2908851

  13. Helmuth, T., McPhee, N.F., Spector, L.: Program synthesis using uniform mutation by addition and deletion. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2018, pp. 1127–1134. ACM, Kyoto (2018). https://doi.org/10.1145/3205455.3205603, http://doi.acm.org/10.1145/3205455.3205603

  14. Helmuth, T., Spector, L.: General program synthesis benchmark suite. In: GECCO 2015: Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, pp. 1039–1046. ACM, Madrid (2015). https://doi.org/10.1145/2739480.2754769, http://doi.acm.org/10.1145/2739480.2754769

  15. Helmuth, T., Spector, L.: Explaining and exploiting the advantages of down-sampled lexicase selection. In: Artificial Life Conference Proceedings, pp. 341–349. MIT Press (2020). https://doi.org/10.1162/isal_a_00334, https://www.mitpressjournals.org/doi/abs/10.1162/isal_a_00334

  16. Helmuth, T., Spector, L.: Problem-solving benefits of down-sampled lexicase selection. Artif. Life 27(3–4), 183–203 (2022). https://doi.org/10.1162/artl_a_00341

    Article  Google Scholar 

  17. Helmuth, T., Spector, L., Matheson, J.: Solving uncompromising problems with lexicase selection. IEEE Trans. Evol. Comput. 19(5), 630–643 (2015). https://doi.org/10.1109/TEVC.2014.2362729, http://ieeexplore.ieee.org/stamp/stamp.jsp?tp= &arnumber=6920034

  18. Helmuth, T., Spector, L., Pantridge, E.: Counterexample-driven genetic programming without formal specifications. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, GECCO 2020, pp. 239–240. ACM (2020). https://doi.org/10.1145/3377929.3389983

  19. Hernandez, J.G., Lalejini, A., Dolson, E., Ofria, C.: Random subsampling improves performance in lexicase selection. In: GECCO 2019: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 2028–2031. ACM, Prague (2019). https://doi.org/10.1145/3319619.3326900

  20. Hernandez, J.G., Lalejini, A., Ofria, C.: An exploration of exploration: measuring the ability of lexicase selection to find obscure pathways to optimality. In: Banzhaf, W., Trujillo, L., Winkler, S., Worzel, B. (eds.) Genetic Programming Theory and Practice XVIII. Genetic and Evolutionary Computation, pp. 83–107. Springer, Singapore (2022). https://doi.org/10.1007/978-981-16-8113-4_5

    Chapter  Google Scholar 

  21. Hopcroft, J.E., Ullman, J.D.: Introduction to Automata Theory, Languages, and Computation. Addison-Wesley (1979)

    Google Scholar 

  22. Jackson, D.: Promoting phenotypic diversity in genetic programming. In: Schaefer, R., Cotta, C., Kołodziej, J., Rudolph, G. (eds.) PPSN 2010. LNCS, vol. 6239, pp. 472–481. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15871-1_48

    Chapter  Google Scholar 

  23. Krawiec, K., Błądek, I., Swan, J.: Counterexample-driven genetic programming. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2017, pp. 953–960. ACM, Berlin (2017). https://doi.org/10.1145/3071178.3071224, http://doi.acm.org/10.1145/3071178.3071224

  24. Krawiec, K., Błądek, I., Swan, J., Drake, J.H.: Counterexample-driven genetic programming: stochastic synthesis of provably correct programs. In: Lang, J. (ed.) Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18), pp. 5304–5308. International Joint Conferences on Artificial Intelligence, Stockholm (2018). https://doi.org/10.24963/ijcai.2018/742, https://www.ijcai.org/proceedings/2018/742

  25. Moore, J.M., Stanton, A.: Lexicase selection outperforms previous strategies for incremental evolution of virtual creature controllers. In: Proceedings of the European Conference on Artificial Life, pp. 290–297 (2017). https://doi.org/10.1162/ecal_a_0050_14, https://www.mitpressjournals.org/doi/abs/10.1162/ecal_a_0050_14

  26. Moore, J.M., Stanton, A.: Tiebreaks and diversity: isolating effects in lexicase selection. In: The 2018 Conference on Artificial Life, pp. 590–597 (2018). https://doi.org/10.1162/isal_a_00109, https://www.mitpressjournals.org/doi/abs/10.1162/isal_a_00109

  27. Rice, H.G.: Classes of recursively enumerable sets and their decision problems. Trans. Am. Math. Soc. 74(2), 358–366 (1953). http://www.jstor.org/stable/1990888

  28. Schweim, D., Sobania, D., Rothlauf, F.: Effects of the training set size: a comparison of standard and down-sampled lexicase selection in program synthesis. In: 2022 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8 (2022). https://doi.org/10.1109/CEC55065.2022.9870337

  29. Sivaraman, A., Farnadi, G., Millstein, T., den Broeck, G.V.: Counterexample-guided learning of monotonic neural networks (2020)

    Google Scholar 

  30. Sobania, D., Briesch, M., Röchner, P., Rothlauf, F.: MTGP: combining metamorphic testing and genetic programming (2023)

    Google Scholar 

  31. Spector, L., Klein, J., Keijzer, M.: The push3 execution stack and the evolution of control. In: GECCO 2005: Proceedings of the 2005 conference on Genetic and evolutionary computation, vol. 2, pp. 1689–1696. ACM Press, Washington DC (2005). https://doi.org/10.1145/1068009.1068292, http://www.cs.bham.ac.uk/wbl/biblio/gecco2005/docs/p1689.pdf

  32. Welsch, T., Kurlin, V.: Synthesis through unification genetic programming. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference, GECCO 2020, pp. 1029–1036. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3377930.3390208, https://doi.org/10.1145/3377930.3390208

Download references

Acknowledgements

We thank the members of the PUSH lab for discussions that improved this work. This material is based upon work supported by the National Science Foundation under Grant No. 2117377. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

  1. Hamilton College, Clinton, NY, 13323, USA

    Thomas Helmuth & James Gunder Frazier

  2. Real Chemistry, Boston, MA, 02111, USA

    Edward Pantridge

  3. Amherst College, Amherst, MA, 01002, USA

    Lee Spector

  4. University of Massachusetts, Amherst, MA, 01003, USA

    Lee Spector

Authors
  1. Thomas Helmuth
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Edward Pantridge
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. James Gunder Frazier
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lee Spector
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas Helmuth .

Editor information

Editors and Affiliations

  1. University of Torino, Grugliasco, Italy

    Mario Giacobini

  2. Victoria University of Wellington, Wellington, New Zealand

    Bing Xue

  3. University of Trieste, Trieste, Italy

    Luca Manzoni

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Helmuth, T., Pantridge, E., Frazier, J.G., Spector, L. (2024). Generational Computation Reduction in Informal Counterexample-Driven Genetic Programming. In: Giacobini, M., Xue, B., Manzoni, L. (eds) Genetic Programming. EuroGP 2024. Lecture Notes in Computer Science, vol 14631. Springer, Cham. https://doi.org/10.1007/978-3-031-56957-9_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56957-9_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56956-2

  • Online ISBN: 978-3-031-56957-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation