Abstract
We describe a new approach for understanding the difficulty of designing efficient learning algorithms. We prove that the existence of an efficient learning algorithm for a circuit class C in Angluin’s model of exact learning from membership and equivalence queries or in Valiant’s PAC model yields a lower bound against C. More specifically, we prove that any subexponential time, determinstic exact learning algorithm for C (from membership and equivalence queries) implies the existence of a function f in EXP NP such that \(f \not\in C\). If C is PAC learnable with membership queries under the uniform distribution or Exact learnable in randomized polynomial time, we prove that there exists a function f ∈BPEXP (the exponential time analog of BPP) such that \(f {\not\in} C\).
For C equal to polynomial-size, depth-two threshold circuits (i.e., neural networks with a polynomial number of hidden nodes), our result shows that efficient learning algorithms for this class would solve one of the most challenging open problems in computational complexity theory: proving the existence of a function in EXP NP or BPEXP that cannot be computed by circuits from C. We are not aware of any representation-independent hardness results for learning polynomial-size depth-2 neural networks.
Our approach uses the framework of the breakthrough result due to Kabanets and Impagliazzo showing that derandomizing BPP yields non-trivial circuit lower bounds.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Pitt, L., Valiant, L.: Computational limitations on learning from examples. Journal of the ACM 35, 965–984 (1988)
Gold, E.A.: Complexity of automaton identification from given data. Information and Control 37, 302–320 (1978)
Alekhnovich, Braverman, Feldman, Klivans, Pitassi: Learnability and automatizability. In: FOCS: IEEE Symposium on Foundations of Computer Science (FOCS) (2004)
Kearns, M., Valiant, L.: Cryptographic limitations on learning Boolean formulae and finite automata. Journal of the ACM 41, 67–95 (1994)
Kharitonov, M.: Cryptographic hardness of distribution-specific learning. In: Proceedings of the Twenty-Fifth Annual Symposium on Theory of Computing, pp. 372–381 (1993)
Jackson, J., Klivans, A., Servedio, R.: Learnability beyond AC 0. In: Proceedings of the 34th ACM Symposium on Theory of Computing (2002)
Kabanets, V., Impagliazzo, R.: Derandomizing polynomial identity tests means proving circuit lower bounds. In: Proceedings of the 35th ACM Symposium on the Theory of Computing, pp. 355–364. ACM, New York (2003)
Impagliazzo, R., Wigderson, A.: Randomness vs. time: Derandomization under a uniform assumption. Journal of Computer and System Sciences 63, 672–688 (2001)
Valiant, L.: A theory of the learnable. Communications of the ACM 27, 1134–1142 (1984)
Angluin, D.: Queries and concept learning. Machine Learning 2, 319–342 (1988)
Buhrman, H., Fortnow, L., Thierauf, T.: Nonrelativizing separations. In: Proceedings of the 13th IEEE Conference on Computational Complexity, pp. 8–12. IEEE, New York (1998)
Miltersen, P.B., Vinodchandran, N.V., Watanabe, O.: Super-polynomial versus half-exponential circuit size in the exponential hierarchy. In: Asano, T., et al. (eds.) COCOON 1999. LNCS, vol. 1627, p. 210. Springer, Heidelberg (1999)
Hartmanis, J., Stearns, R.: On the computational complexity of algorithms. Transactions of the American Mathematical Society 117, 285–306 (1965)
Kannan, R.: Circuit-size lower bounds and non-reducibility to sparse sets. Information and Control 55, 40–56 (1982)
Valiant, L.: The complexity of computing the permanent. Theoretical Computer Science 8, 189–201 (1979)
Toda, S.: PP is as hard as the polynomial-time hierarchy. SIAM Journal on Computing 20, 865–877 (1991)
Lipton, R.: New directions in testing. In: Feigenbaum, J., Merritt, M. (eds.) Distributed Computing and Cryptography. DIMACS Series in Discrete Mathematics and Theoretical Computer Science, vol. 2, pp. 191–202. American Mathematical Society, Providence (1991)
Beaver, D., Feigenbaum, J.: Hiding instances in multioracle queries. In: Choffrut, C., Lengauer, T. (eds.) STACS 1990. LNCS, vol. 415, pp. 37–48. Springer, Heidelberg (1990)
Buhrman, H., Homer, S.: Superpolynomial circuits, almost sparse oracles and the exponential hierarchy. In: Shyamasundar, R.K. (ed.) FSTTCS 1992. LNCS, vol. 652, pp. 116–127. Springer, Heidelberg (1992)
Babai, L., Fortnow, L., Nisan, N., Wigderson, A.: BPP has subexponential time simulations unless EXPTIME has publishable proofs. Computational Complexity 3, 307–318 (1993)
Beimel, A., Bergadano, F., Bshouty, N., Kushilevitz, E., Varricchio, S.: On the applications of multiplicity automata in learning. In: Proceedings of the Thirty-Seventh Annual Symposium on Foundations of Computer Science, pp. 349–358 (1996)
Klivans, Shpilka: Learning arithmetic circuits via partial derivatives. In: COLT: Proceedings of the Workshop on Computational Learning Theory. Morgan Kaufmann Publishers, San Francisco (2003)
Bshouty, Hancock, Hellerstein: Learning arithmetic read-once formulas. SICOMP: SIAM Journal on Computing 24 (1995)
Bshouty.: On interpolating arithmetic read-once formulas with exponentiation. JCSS: Journal of Computer and System Sciences 56 (1998)
Linial, N., Mansour, Y., Nisan, N.: Constant depth circuits, fourier transform, and learnability. Journal of the ACM 40, 607–620 (1993)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Fortnow, L., Klivans, A.R. (2006). Efficient Learning Algorithms Yield Circuit Lower Bounds. In: Lugosi, G., Simon, H.U. (eds) Learning Theory. COLT 2006. Lecture Notes in Computer Science(), vol 4005. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11776420_27
Download citation
DOI: https://doi.org/10.1007/11776420_27
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-35294-5
Online ISBN: 978-3-540-35296-9
eBook Packages: Computer ScienceComputer Science (R0)