Abstract
Systems competitions play a fundamental role in the advancement of the state of the art in several automated reasoning fields. The goal of such events is to answer the question: “Which system should I buy?”. In this paper, we consider voting systems as an alternative to other procedures which are well established in automated reasoning contests. Our research is aimed to compare methods that are customary in the context of social choice, with methods that are targeted to artificial settings, including a new hybrid method that we introduce.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Sutcliffe, G., Suttner, C.: The CADE ATP System Competition (April 22, 2007), http://www.cs.miami.edu/~tptp/CASC
Le Berre, D., Simon, L.: The SAT Competition (April 22, 2007), http://www.satcompetition.org
Long, D., Fox, M.: The 3rd International Planning Competition: Results and Analysis. Artificial Intelligence Research 20, 1–59 (2003)
van Dongen, M.R.C.: Introduction to the Solver Competition. In: CPAI 2005 proceedings (2005)
Barrett, C.W., de Moura, L., Stump, A.: SMT-COMP: Satisfiability Modulo Theories Competition. In: Etessami, K., Rajamani, S.K. (eds.) CAV 2005. LNCS, vol. 3576, pp. 20–23. Springer, Heidelberg (2005)
Le Berre, D., Simon, L., Tacchella, A.: Challenges in the QBF arena: the SAT’03 evaluation of QBF solvers. In: Giunchiglia, E., Tacchella, A. (eds.) SAT 2003. LNCS, vol. 2919, Springer, Heidelberg (2004)
Le Berre, D., Narizzano, M., Simon, L., Tacchella, A.: The second QBF solvers evaluation. In: Hoos, H.H., Mitchell, D.G. (eds.) SAT 2004. LNCS, vol. 3542, Springer, Heidelberg (2005)
Narizzano, M., Pulina, L., Tacchella, A.: The third QBF solvers comparative evaluation. Journal on Satisfiability, Boolean Modeling and Computation 2, 145–164 (2006), available on-line at http://jsat.ewi.tudelft.nl/
Hooker, J.N.: Testing Heuristics: We Have It All Wrong. Journal of Heuristics 1, 33–42 (1996)
Saari, D.G.: Chaotic Elections! A Mathematician Looks at Voting. American Mathematical Society (2001)
Smith, W.D.: Range voting (April 22, 2007), available on-line at http://www.math.temple.edu/~wds/homepage/rangevote.pdf
Schulze, M.: A New Monotonic and Clone-Independent Single-Winner Election Method. Voting Matters, 9–19 (2003)
Narizzano, M., Pulina, L., Taccchella, A.: QBF solvers competitive evaluation (QBFEVAL), http://www.qbflib.org/qbfeval
RoboCup, http://www.robocup.org
Arrow, K.J., Sen, A.K., Suzumura, K. (eds.): Handbook of Social Choice and Welfare, vol. 1. Elsevier, Amsterdam (2002)
Conitzer, V., Sandholm, T.: Common Voting Rules as Maximum Likelihood Estimators. In: EC-05. 6th ACM Conference on Electronic Commerce. LNCS, pp. 78–87. Springer, Heidelberg (2005)
Pulina, L.: Empirical Evaluation of Scoring Methods. In: Proc. STAIRS 2006. Frontiers in Artificial Intelligence and Applications, vol. 142, pp. 108–119 (2006)
Narizzano, M., Pulina, L., Tacchella, A.: Competitive Evaluation of QBF Solvers: noisy data and scoring methods. Technical report, STAR-Lab - University of Genoa (May 2006)
Giunchiglia, E., Narizzano, M., Pulina, L., Tacchella, A.: Quantified Boolean Formulas satisfiability library (QBFLIB), www.qbflib.org
Van Gelder, A., Le Berre, D., Biere, A., Kullmann, O., Simon, L.: Purse-Based Scoring for Comparison of Exponential-Time Programs (2006) (unpublished draft)
Narizzano, M., Pulina, L., Tacchella, A.: Competitive Evaluation of Automated Reasoning Tools: Statistical Testing and Empirical Scoring. In: EMAA 2006. First Workshop on Empirical Methods for the Analysis of Algorithms (2006)
Schulze, M.: Extending schulze’s method to obtain an overall ranking. Personal communications
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Narizzano, M., Pulina, L., Tacchella, A. (2007). Ranking and Reputation Systems in the QBF Competition. In: Basili, R., Pazienza, M.T. (eds) AI*IA 2007: Artificial Intelligence and Human-Oriented Computing. AI*IA 2007. Lecture Notes in Computer Science(), vol 4733. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74782-6_10
Download citation
DOI: https://doi.org/10.1007/978-3-540-74782-6_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74781-9
Online ISBN: 978-3-540-74782-6
eBook Packages: Computer ScienceComputer Science (R0)