Skip to main content
Log in

The Design of the CADE-13 ATP System Competition

  • Published:
Journal of Automated Reasoning Aims and scope Submit manuscript

Abstract

Running a competition for automated theorem proving (ATP) systems is a difficult and arguable venture. However, the potential benefits of such an event by far outweigh the controversial aspects. The motivations for running the CADE-13 ATP System Competition were to contribute to the evaluation of ATP systems, to stimulate ATP research and system development, and to expose ATP systems to researchers both within and outside the ATP community. This article identifies and discusses the issues that determine the nature of such a competition. Choices and motivated decisions for the CADE-13 competition, with respect to the issues, are given.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Astrachan, O., Khera, V. and Kotz, D.: The Internet programming contest: A report and philosophy, in Proc. of the 24th SIGCSE Technical Conference, SIGCSE Bulletin, ACM Press, 1993, pp. 48–52.

  2. Astrachan, O. L. and Loveland, D. W.: Measuring the performance of automated theorem provers, in G. Sutcliffe and C. B. Suttner (eds), Proc. of the CADE-12 Workshop 2C - Evaluation of Automated Theorem Proving Systems, 1994, pp. 37–41.

  3. Boyer, R. S. and Moore, J. S.: A theorem prover for a computational logic, in Proc. of the 10th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 449, Springer-Verlag, 1990, pp. 1–15.

  4. Buro, M. and Kleine-Büning, H.: Report on a SAT Competition. Technical Report FB 17, Mathematik/Informatik, Universität Paderborn, Paderborn, Germany, 1992.

    Google Scholar 

  5. Chen, E., Istrail, S., Johnson, D., Kececioglu, J., Messing, J., Nadeau, J., Pevzner, P., Rice, P., Vingron, M. and Waterman, M.: The 4th DIMACS Interantional Algorithm Implementation Challenge. URL http://dimacs.rutgers.edu/challenge.html, 1994.

  6. Comer, J., Rinewalt, R., Ryan, P. and Poucher, W. B.: The ACM scholastic programming contest - 1977 to 1990, Proceedings of the 21st SIGCSE Technical Symposium on Computer Science Education, SIGCSE Bulletin 22(1) (1990), 256.

    Google Scholar 

  7. Constable, R., Allen, S., Bromly, H., Cleaveland, W., Cremer, J., Harper, R., Howe, D., Knoblock, T., Mendler, N., Panangaden, P., Sasaki, J. and Smith, S.: Implementing Mathematics with the Nuprl Proof Development System, Prentice-Hall, 1986.

  8. Demoen, B.: The 2nd annual Prolog programming contest, The Newsletter for the Association for Logic Programming 9(2) (1996), 3–4.

    Google Scholar 

  9. Dunker, U.: Search Space and Proof Complexity of Theorem Proving Strategies, in G. Sutcliffe and C. B. Suttner (eds), Proc. of the CADE-12 Workshop 2C - Evaluation of Automated Theorem Proving Systems, 1994.

  10. Farmer, W., Guttman, J. and Thayer, J.: IMPS: An interactive mathematical proof system, in Proc. of the 10th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 449, Springer-Verlag, 1990, pp. 653–654.

  11. Gordon, M. and Melham, T.: Introduction to HOL, a Theorem Proving Environment for Higher Order Logic, Cambridge University Press, 1993.

  12. Hanks, S., Pollack, M. E. and Cohen, P. R.: Benchmarks, test beds, controlled experimentation and the design of agent architectures, AI Magazine 14(4) (1993), 17–42.

    Google Scholar 

  13. Heisel, M., Reif, W. and Stephan, W.: Tactical theorem proving in program verification, in M. Stickel (ed.), Proc. of the 10th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 449, Springer-Verlag, 1990, pp. 117–131.

  14. Huang, X., Kerber, M., Kohlhase, M., Melis, E., Nesmith, D., Richts, J. and Siekmann, J.: Ω-MKRP: A proof development environment, in A. Bundy (ed.), Proc. of the 12th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 814, Springer-Verlag, 1994, pp. 788–792.

  15. Kerber, M.: How to prove higher order theorems in first order logic, in Proc. of the 12th International Joint Conference on Artificial Intelligence, IJCAI, 1991, pp. 137–142.

  16. Kopec, D., Newborn, M. and Valvo, M.: The 22nd annual ACM international computer chess championship, Communications of the ACM 35(11) (1992), 100–110.

    Google Scholar 

  17. Lee, S.-J. and Plaisted, D. A.: Eliminating duplication with the hyper-linking strategy, Journal of Automated Reasoning 9(1) (1992), 25–42.

    Google Scholar 

  18. Letz, R.: First-Order Calculi and Proof Procedures for Automated Deduction, PhD thesis, Technical University of Munich, Munich, Germany, 1993.

    Google Scholar 

  19. McCune, W. W.: Otter 3.0 Reference Manual and Guide. Technical Report ANL-94/6, Argonne National Laboratory, Argonne, Illinois, 1994.

  20. Michie, D., Muggleton, D., Page, D. and Srinivasan, A.: 3 Inductive Learning Competitions. URL ftp://ftp.comlab.ox.ac.uk/pub/Packages/ILP, 1994.

  21. Ohlbach, H. J.: Semantics based translation methods for modal logics, Journal of Logic and Computation 1(5) (1991), 691–746.

    Google Scholar 

  22. Overbeek, R.: The CADE-11 competitions: A personal view, Journal of Automated Reasoning 11(3) (1993), 315–316.

    Google Scholar 

  23. Owre, S., Rushby, J. M. and Shankar, N.: PVS: A prototype verification system, in D. Kapur (ed.), Proc. of the 11th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 607, Springer-Verlag, 1992, pp. 748–752.

  24. Paulson, L. C.: Isabelle: The next 700 theorem provers, Logic and Computer Science (1990), 361–386.

  25. Pelletier, F. J.: The philosophy of automated theorem proving, in J. Mylopolous and R. Reiter (eds), Proc. of the 12th International Joint Conference on Artificial Intelligence, IJCAI, 1991, pp. 1039–1045.

  26. Plaisted, D. A.: The search efficiency of theorem proving strategies, in A. Bundy (ed.), Proc. of the 12th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 814, Springer-Verlag, 1994, pp. 57–71.

  27. Portoraro, F. D.: Symlog: Automated advice in Fitch-style proof construction, in A. Bundy (ed.), Proc. of the 12th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 814, Springer-Verlag, 1994, pp. 802–806.

  28. Prehofer, C. and Slind, K.: Theorem proving in interactive verification systems, in G. Sutcliffe and C. B. Suttner (eds), Proc. of the CADE-12 Workshop 2C - Evaluation of Automated Theorem Proving Systems, 1994.

  29. G. Sutcliffe and C. B. Suttner: ATP System Results for the TPTP Problem Library (up to TPTP v1.1.3), 1995. Technical Report 95/16, Department of Computer Science, James Cook University, Townsville, Australia; Technical Report AR-95-06, Institut für Informatik, Technische Universität München, Munich, Germany.

    Google Scholar 

  30. Sutcliffe, G. and Suttner, C. B.: Special issue: The CADE-13 ATP system competition, Journal of Automated Reasoning 18(2) (1997).

  31. Sutcliffe, G. and Suttner, C. B.: The results of the CADE-13 ATP system competition, Journal of Automated Reasoning 18(2) (1997).

  32. Sutcliffe, G., Suttner, C. B. and Yemenis, T.: The TPTP problem library, in A. Bundy (ed.), Proc. of the 12th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 814, Springer-Verlag, 1994, pp. 252–266.

  33. Suttner, C. B. and Sutcliffe, G.: The TPTP Problem Library (TPTP v1.2.1), 1996. Technical Report AR-96-02, Institut für Informatik, Technische Universität München, Munich, Germany; Technical Report 96/09, Department of Computer Science, James Cook University, Townsville, Australia.

    Google Scholar 

  34. Wadsworth, H. M.: Handbook of Statistical Methods for Engineers and Scientists, McGraw-Hill, 1990.

  35. Wang, T. C. and Goldberg, A.: RVF: An automated formal verification system, in D. Kapur (ed.), Proc. of the 11th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 607, Springer-Verlag, 1992, pp. 735–739.

  36. Wos, L., Winker, S., McCune, W. W., Overbeek, R., Lusk, E. and Stevens, R.: Automated reasoning contributes to mathematics and logic, in M. Stickel (ed.), Proc. of the 10th International Conference on Automated Deduction, Lecture Notes in Artificial Intelligence 449, Springer-Verlag, 1990, pp. 485–499.

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

About this article

Cite this article

Suttner, C., Sutcliffe, G. The Design of the CADE-13 ATP System Competition. Journal of Automated Reasoning 18, 139–162 (1997). https://doi.org/10.1023/A:1005802523220

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1005802523220

Navigation