Abstract
We propose a new model to quantitatively estimate the accuracy of artificial agents over cognitive tasks of approximable complexities. The model is derived by introducing notions from algorithmic information theory into a well-known (psychometric) measurement paradigm called Item Response Theory (IRT). A lower bound on accuracy can be guaranteed with respect to task complexity and the breadth of its solution space using our model. This in turn permits formulating the relationship between agent selection cost, task difficulty and accuracy as optimisation problems. Further results indicate some of the settings over which a group of cooperative agents can be more or less accurate than individual agents or other groups.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
For simplicity and without loss of generality, 1 / m is used in Eq. 1 to replace the probability \(p_{rand}\) of an agent randomly guessing (one of) the correct solutions to the problem.
- 2.
More sophisticated voting rules such as Borda count, harmonic rule, maximin and Copeland require the subject to output a concrete ranking over all possible alternatives of the test/task, which inhibits our ability of making exact predictions. Yet, one can still analytically place \( \min \) and \(\max \) bounds on team accuracy using different sampling techniques.
References
Bien, Z., Bang, W.C., Kim, D.Y., Han, J.S.: Machine intelligence quotient: its measurements and applications. Fuzzy Sets Syst. 127(1), 3–16 (2002)
Birnbaum, A.: Some latent trait models and their use in inferring an examinee’s ability. In: Statistical Theories of Mental Test Scores, pp. 395–479 (1968)
Chmait, N.: Understanding and measuring collective intelligence across different cognitive systems: an information-theoretic approach (extended abstract). In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI-17 Doctoral Consortium, Melbourne, Australia (2017, to appear)
Chmait, N., Dowe, D.L., Li, Y.F., Green, D.G., Insa-Cabrera, J.: Factors of collective intelligence: how smart are agent collectives? In: Proceedings of 22nd European Conference on Artificial Intelligence ECAI, Frontiers in Artificial Intelligence and Applications, vol. 285, pp. 542–550. IOS Press (2016)
De Ayala, R.J.: The Theory and Practice of Item Response Theory. Guilford Publications, New York (2013)
Dowe, D.L., Hajek, A.R.: A computational extension to the turing test. In: Proceedings 4th Conference of the Australasian Cognitive Science Society, University of Newcastle, NSW, Australia (1997)
Dowe, D.L., Hernández-Orallo, J.: IQ tests are not for machines, yet. Intelligence 40(2), 77–81 (2012)
Dowe, D.L., Hernández-Orallo, J., Das, P.K.: Compression and intelligence: social environments and communication. In: Schmidhuber, J., Thórisson, K.R., Looks, M. (eds.) AGI 2011. LNCS (LNAI), vol. 6830, pp. 204–211. Springer, Heidelberg (2011). doi:10.1007/978-3-642-22887-2_21
Gottfredson, L.S.: Mainstream science on intelligence: an editorial with 52 signatories, history, and bibliography. Intelligence 24(1), 13–23 (1997)
Hernández-Orallo, J.: Beyond the Turing test. J. Log. Lang. Inf. 9(4), 447–466 (2000)
Hernández-Orallo, J.: The Measure of All Minds: Evaluating Natural and Artificial Intelligence. Cambridge University Press, New York (2016)
Hernández-Orallo, J., Dowe, D.L.: Measuring universal intelligence: towards an anytime intelligence test. Artif. Intell. 174(18), 1508–1539 (2010)
Hernández-Orallo, J., Insa-Cabrera, J., Dowe, D.L., Hibbard, B.: Turing machines and recursive turing tests. In: AISB/IACAP 2012 Symposium Revisiting Turing and his Test, pp. 28–33 (2012)
Insa-Cabrera, J., Dowe, D.L., España-Cubillo, S., Hernández-Lloreda, M.V., Hernández-Orallo, J.: Comparing humans and AI agents. In: Schmidhuber, J., Thórisson, K.R., Looks, M. (eds.) AGI 2011. LNCS (LNAI), vol. 6830, pp. 122–132. Springer, Heidelberg (2011). doi:10.1007/978-3-642-22887-2_13
Klein, G.A., King, J.A.: A test for the performance of knowledge-based systems: AIQ. In: Proceedings of AAAI Workshop on Validation and Verification of Expert System, Menlo Park, CA (1988)
Kolmogorov, A.N.: Three approaches to the quantitative definition of information. Probl. Inf. Transm. 1(1), 1–7 (1965)
Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, Hoboken (2004)
Legg, S., Hutter, M.: Universal intelligence: a definition of machine intelligence. Mind. Mach. 17(4), 391–444 (2007)
Lempel, A., Ziv, J.: On the complexity of finite sequences. IEEE Trans. Inf. Theory 22(1), 75–81 (1976)
Levin, L.A.: Universal sequential search problems. Probl. Inf. Transm. 9(3), 265–266 (1973)
Li, M., Vitányi, P.: An Introduction to Kolmogorov Complexity and Its Applications, 3rd edn. Springer, New York (2008)
Lord, F.M., Novick, M.R.: Statistical Theories of Mental Test Scores. Addison-Wesley, Menlo Park (1968)
Martínez-Plumed, F., Prudêncio, R.B., Martínez-Usó, A., Hernández-Orallo, J.: Making sense of item response theory in machine learning. In: Proceedings of 22nd European Conference on Artificial Intelligence (ECAI), Frontiers in Artificial Intelligence and Applications, vol. 285, pp. 1140–1148 (2016)
Raven, J.C., Court, J.H.: Raven’s Progressive Matrices and Vocabulary Scales. Oxford Psychologists Press, Oxford (1998)
Roid, G.H.: Stanford-Binet Intelligence Scales. Riverside Publishing, Itasca (2003)
Sanghi, P., Dowe, D.L.: A computer program capable of passing I.Q. tests. In: Slezak, P. (ed.) Proceedings of 4th International Conference on Cognitive Science (ICCS/ASCS-2003), pp. 570–575, Australia, July 2003
Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27(3), 379–423 (1948)
Shapley, L., Grofman, B.: Optimizing group judgmental accuracy in the presence of interdependencies. Public Choice 43(3), 329–343 (1984)
Solomonoff, R.J.: A preliminary report on a general theory of inductive inference. Report ZTB-138. Zator Co 131, Cambridge, MA (1960)
Spearman, C.: General intelligence, objectively determined and measured. Am. J. Psychol. 15(2), 201–292 (1904)
Thurstone, L.L.: Primary Mental Abilities. Chicago Press, Chicago (1938)
Turing, A.M.: Computing machinery and intelligence. Mind 59, 433–460 (1950)
Wechsler, D.: Wechsler Adult Intelligence Scale-Fourth. Pearson, San Antonio (2008)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Chmait, N., Dowe, D.L., Li, YF., Green, D.G. (2017). An Information-Theoretic Predictive Model for the Accuracy of AI Agents Adapted from Psychometrics. In: Everitt, T., Goertzel, B., Potapov, A. (eds) Artificial General Intelligence. AGI 2017. Lecture Notes in Computer Science(), vol 10414. Springer, Cham. https://doi.org/10.1007/978-3-319-63703-7_21
Download citation
DOI: https://doi.org/10.1007/978-3-319-63703-7_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-63702-0
Online ISBN: 978-3-319-63703-7
eBook Packages: Computer ScienceComputer Science (R0)