Skip to main content

Advertisement

SpringerLink
Log in

Proof searching and prediction in HOL4 with evolutionary/heuristic and deep learning techniques

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Interactive theorem provers (ITPs), also known as proof assistants, allow human users to write and verify formal proofs. The proof development process in ITPs can be a cumbersome and time-consuming activity due to manual user interactions. This makes proof guidance and proof automation the two most desired features for ITPs. In this paper, we first provide two evolutionary and heuristic-based proof searching approaches for the HOL4 proof assistant, where a genetic algorithm (GA) and simulated annealing (SA) is used to search and optimize the proofs in different HOL theories. In both approaches, random proof sequences are first generated from a population of frequently occurring HOL4 proof steps that are discovered with sequential pattern mining. Generated proof sequences are then evolved using GA operators (crossover and mutation) and by applying the annealing process of SA till their fitness match the fitness of the target proof sequences. Experiments were done to compare the performance of SA with that of GA. Results have shown that the two proof searching approaches can be used to efficiently evolve the random sequences to obtain the target sequences. However, these approaches lack the ability to learn the proof process, that is important for the prediction of new proof sequences. For this purpose, we propose to use a deep learning technique known as long short-term memory (LSTM). LSTM is trained on various HOL4 theories for proof learning and prediction. Experimental results suggest that combining evolutionary/heuristic and deep learning techniques with proof assistants can greatly facilitate proof finding/optimization and proof prediction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Hasan O, Tahar S (2015) Formal verification methods. In: Encyclopedia of Information Science & Technology, 3rd edn. IGI Global, pp 7162–7170

  2. Kaliszyk C, Chollet F, Szegedy C (2017) Holstep: A machine learning dataset for higher-order logic theorem proving. CoRR arXiv:1703.00426

  3. Slind K, Norrish M (2008) A brief overview of HOL4. In: Proceedings of International Conference on Theorem Proving in Higher Order Logics (TPHOLs), pp 28–32

  4. Bertot Y, Casteran P (2004) Interactive theorem proving and program development: Coq’Art: The calculus of inductive construction. Springer Publisher

  5. Owre S, Shankar N, Rushby J M, Stringer-Calvert D W J (2001) PVS System guide, PVS prover guide PVS language reference. Technical report, SRI International

  6. Wiedijk F (Accessed on 3 January 2020) Formalizing 100 theorems, available at: http://www.cs.ru.nl/~freek/100

  7. Hales T C, Adams M, Bauer G, Dang D T, Harrison J, Hoang T L, Kaliszyk C, Magron V, McLaughlin S, Nguyen T T, Nguyen T Q, Nipkow T, Obua S, Pleso J, Rute J M, Solovyev A, Ta A H T, Tran T N, Trieu D T, Urban J, Vu K K, Zumkeller R (2017) A formal proof of the Kepler conjecture. Forum Math Pi 5 e2:1–29

    MATH  Google Scholar 

  8. Gonthier G, Asperti A, Avigad J, Bertot Y, Cohen C, Garillot F, Roux SL, Mahboubi A, O’Connor R, Biha SO, Pasca I, Rideau L, Solovyev A, Tassi E, Thery L (2013) A machine-checked proof of the Odd Order theorem. In: Proceedings of International Conference on Interactive Theorem Proving (ITP), pp 163–179

  9. Leroy X (2009) Formal verification of a realistic compiler. Commun ACM 52(7):107–115

    Article  Google Scholar 

  10. Blanchette JC, Haslbeck MPL, Matichuk D, Nipkow T (2015) Mining the archive of formal proofs. In: Proceedings of International Conference on Intelligent Computer Mathematics (CICM), pp 3–17

  11. Harrison J, Urban J, Wiedijk F (2014) History of interactive theorem proving. In: Computational Logic, volume 9 of Handbook of the History of Logic, pp 135–214

  12. Kaliszyk C, Urban J (2015) Learning-assisted theorem proving with millions of lemmas. J Symb Comput 69:109–128

    Article  MathSciNet  Google Scholar 

  13. Färber M, Brown CE (2016) Internal guidance for Satallax. In: Proceedings of International Joint Conference on Automated Reasoning (IJCAR), pp 349–361

  14. Gauthier T, Kaliszyk C, Urban J (2017) TacticToe: Learning to reason with HOL4 tactics. In: Proceedings of International Conference on Logic for Programming, Artificial Intelligence and Reasoning (LPAR), pp 125–143

  15. Irving G, Szegedy C, Alemi AA, Eén N, Chollet F, Urban J (2016) Deepmath - Deep sequence models for premise selection. In: Proceedings of Annual Conference on Neural Information Processing Systems (NIPS), pp 2235–2243

  16. Kaliszyk C, Mamane L, Urban J (2014) Machine learning of Coq proof guidance: First experiments. In: Proceedings of International Symposium on Symbolic Computation in Software Science (SCSS), pp 27–34

  17. Nawaz MS, Sun M, Fournier-Viger P (2019) Proof guidance in PVS with sequential pattern mining. In: Proceedings of International Conference on Fundamentals of Software Engineering (FSEN), pp 45–60

  18. Nawaz MZ, Hasan O, Nawaz MS, Fournier-Viger P, Sun M (2020) Proof searching in HOL4 with genetic algorithm. In: Proceedings of Annual ACM Symposium on Applied Computing (SAC), pp 513–520

  19. Huang SY, Chen YP (2017) Proving theorems by using evolutionary search with human involvement. In: Proceedings of Congress on Evolutionary Computation (CEC), pp 1495–1502

  20. Yang LA, Liu JP, Chen CH, Chen YP (2016) Automatically proving mathematical theorems with evolutionary algorithms and proof assistants. In: Proceedings of Congress on Evolutionary Computation (CEC), pp 4421–4428

  21. Koza J R (1993) Genetic programming - On the programming of computers by means of natural selection. MIT Press Cambridge, Massachusetts

    MATH  Google Scholar 

  22. Duncan H (2007) The use of data-mining for the automatic formation of tactics. PhD Thesis, University of Edinburgh, UK

  23. Alama J, Heskes T, Ku̇hlwein D, Tsivtsivadze E, Urban J (2014) Premise selection for mathematics by corpus analysis and kernel methods. J Auto Reaso 52(2):191–213

    Article  MathSciNet  Google Scholar 

  24. Loos SM, Irving G, Szegedy C, Kaliszyk C (2017) Deep network guided proof search. In: Proceedings of International Conference on Logic for Programming, Artificial Intelligence and Reasoning (LPAR), pp 85–105

  25. Wang M, Tang Y, Wang J, Deng J (2017) Premise selection for theorem proving by deep graph embedding. In: Proceedings of Annual Conference on Neural Information Processing Systems (NIPS), pp 2786–2796

  26. Whalen D (2016) Holophrasm: a neural automated theorem prover for higher-order logic. CoRR arXiv:1608.02644

  27. Gauthier T, Kaliszyk C (2015) Premise selection and external provers for HOL4. In: Proceedings of International Conference on Certified Programs and Proofs (CPP), pp 49–57

  28. Kaliszyk C, Urban J (2015) Hol(y)hammer: Online ATP service for HOL light. Mathe Comp Sci 9(1):5–22

    Article  Google Scholar 

  29. Zhang X, Li Y, Hong W, Sun M (2019) Using recurrent neural network to predict tactics for proving component connector properties in Coq. In: Proceedings of International Symposium on Theoretical Aspects of Software Engineering (TASE), pp 107–112

  30. Holland J H (1975) Adaptation in natural and artificial systems. University of Michigan Press, Ann Arbor

  31. Mitchell M (1996) An introduction to genetic algorithms. MIT Press Cambridge, Massachusetts

  32. Hong T, Wang H, Lin W, Lee W Y (2002) Evolution of appropriate crossover and mutation operators in a genetic process. Appl Intell 16(1):7–17

    Article  Google Scholar 

  33. Bertsimas D, Tsitsiklis J (2013) Simulated annealing. Stati Sci 8(1):10–15

    Article  Google Scholar 

  34. Delahaye D, Chaimatanan S, Mongeau M (2019) Simulated annealing: From basics to applications. In: Handbook of Metaheuristics, pp 1–35

  35. Yao X, McKay R I (2001) Simulated evolution and learning: an introduction. Appl Intell 15 (3):151–152

    Article  Google Scholar 

  36. Graves A (2012) Supervised sequence labelling with recurrent neural networks. Springer Publisher

  37. Medsker L, Jain LC (1999) Recurrent neural networks: Design and applications. CRC Press

  38. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neur Comp 9(8):1735–1780

    Article  Google Scholar 

  39. Fournier-Viger P, Lin J C W, Kiran R U, Koh Y S, Thomas R (2017) A survey of sequential pattern mining. Data Sci Patt Recogn 1(1):54–77

    Google Scholar 

  40. Python codes and HOL4 data, available at: github.com/MuhammadzohaibNawaz/PRS-GA-and-SA-in-Python

  41. Nsakanda A L, Price W L, Diaby M, Gravel M (2007) Ensuring population diversity in genetic algorithms: a technical note with application to the cell formation problem. Euro J Operat Rese 178 (2):634–638

    Article  Google Scholar 

  42. Silver D, Huang A, Maddison C J, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T P, Leach M, Kavukcuoglu K, Graepel T, Hassabis D (2016) Mastering the game of Go with deep neural networks and tree search. Nature 529(7587):484–489

    Article  Google Scholar 

  43. Kaiser L, Sutskever I (2016) Neural GPUs learn algorithms In: International Conference on Learning Representations, (ICLR). Poster

  44. Sukhbaatar S, Szlam A, Weston J, Fergus R (2015) End-to-end memory networks. In: Proceedings of Annual Conference on Neural Information Processing Systems (NIPS). pp 2440–2448

  45. Wu Y, Schuster M, Chen Z, Le Q V, Norouzi M, Macherey M, Krikun M, Cao Y, Gao Q, Macherey Q, Klingner J, Shah A, Johnson M, Liu X, Kaiser L, Gouws S, Kato Y, Kudo T, Kazawa H, Stevens K, Kurian G, Patil N, Wang W, Young C, Smith J, Riesa J, Rudnick A, Vinyals O, Corrado G, Hughes M, Dean J (2016) Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR arXiv:1609.08144

  46. Vinyals Q, Le Q V (2015) A neural conversational model. CoRR arXiv:1506.05869

  47. Socher R, Chen D, Manning CD, Ng AY (2013) Reasoning with neural tensor networks for knowledge base completion. In: Proceedings of Annual Conference on Neural Information Processing Systems (NIPS), pp 926–934

  48. Nawaz M S, Malik M, Li Y, Sun M, Lali M I (2019) A survey on theorem provers in formal methods. CoRR arXiv:1912.03028

  49. Huet G P (1973) The undecidability of unification in third order logic. Informa Cont 22 (3):257–267

    Article  MathSciNet  Google Scholar 

  50. Hochreiter S (1998) The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int J Uncertain Fuzz 6(2):107–116

    Article  MathSciNet  Google Scholar 

  51. Olah C (2015) Understanding LSTM networks, available at: https://colah.github.io/posts/2015-08-Understanding-LSTMs

  52. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, Kudlur M, Levenberg J, Monga R, Moore S, Murray DG, Steiner B, Tucker PA, Vasudevan V, Warden P, Wicke M, Yu Y, Zheng X (2016) Tensorflow: A system for large-scale machine learning. In: Proceedings of USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp 265–283

  53. Jones T (1995) Crossover, macromutationand, and population-based search. In: Proceedings of the 6th International Conference on Genetic Algorithms (ICGA), pp 73–80

  54. Kennedy J, Eberhart R (1995) Particle swarm optimization. In: Proceedings of International Conference on Neural Networks (ICNN), pp 1942–1948

  55. Browne C B, Powley E, Whitehouse D, Lucas S M, Cowling I, Rohlfshagen P, Tavener S, Perez D, Samothrakis S, Colton S (2012) A survey of monte carlo tree search methods. IEEE Trans Comp Intell AI 4(1):1–43

    Google Scholar 

  56. Shuan B, Chen J, Li Z (2011) Study on hybrid PS-ACO algorithm. Appl Intell 34(1):64–73

    Article  Google Scholar 

  57. Santo JE (2015) Curry-howard for sequent calculus at last! In: Proceedings of International Conference on Typed Lambda Calculi and Applications (TLCA), pp 165–179

  58. Cho K, van Merrienboer B, Bahdanau D, Bengio Y (2014) On the properties of neural machine translation: Encoder-decoder approaches. In: Proceedings of Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST@EMNLP), pp 103–111

Download references

Funding

The work was partially supported by the Guangdong Science and Technology Department (under grant no. 2018B010107004) and the National Natural Science Foundation of China under grant no. 61772038 and 61532019.

Author information

Author notes

    Authors and Affiliations

    1. School of Humanities and Social Sciences, Harbin Institute of Technology (Shenzhen), Shenzhen, China

      M. Saqib Nawaz & Philippe Fournier-Viger

    2. School of Electrical Engineering and Computer Science, National University of Sciences and Technology, Islamabad, Pakistan

      M. Zohaib Nawaz & Osman Hasan

    3. Department of Computer Science and IT, University of Sargodha, Sargodha, Pakistan

      M. Zohaib Nawaz

    4. School of Mathematical Sciences, Peking University, Beijing, China

      Meng Sun

    Authors
    1. M. Saqib Nawaz
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. M. Zohaib Nawaz
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Osman Hasan
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Philippe Fournier-Viger
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Meng Sun
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Philippe Fournier-Viger.

    Ethics declarations

    Conflict of interests

    The authors declare that they have no conflict of interest.

    Additional information

    Ethical Approval

    This paper does not contain any studies with human participants or animals performed by any of the authors.

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    M. Saqib Nawaz and M. Zohaib Nawaz equally contributed to the paper.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Nawaz, M.S., Nawaz, M.Z., Hasan, O. et al. Proof searching and prediction in HOL4 with evolutionary/heuristic and deep learning techniques. Appl Intell 51, 1580–1601 (2021). https://doi.org/10.1007/s10489-020-01837-7

    Download citation

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s10489-020-01837-7

    Keywords

    Search

    Navigation