Skip to main content
SpringerLink
Log in

Sparsity and Complexity of Networks Computing Highly-Varying Functions

  • Conference paper
  • First Online:
Artificial Neural Networks and Machine Learning – ICANN 2018 (ICANN 2018)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11141))

Included in the following conference series:

Abstract

Approximative measures of network sparsity in terms of norms tailored to dictionaries of computational units are investigated. Lower bounds on these norms of real-valued functions on finite domains are derived. The bounds are proven by combining the concentration of measure property of high-dimensional spaces with characterization of dictionaries of computational units in terms of their capacities and coherence measured by their covering numbers. The results are applied to dictionaries used in neurocomputing which have power-type covering numbers. Probabilistic results are illustrated by a concrete construction of a class of functions, computation of which by perceptron networks requires large number of units or it is unstable due to large output weights.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bengio, Y., LeCun, Y.: Scaling learning algorithms towards AI. In: Bottou, L., Chapelle, O., DeCoste, D., Weston, J. (eds.) Large-Scale Kernel Machines. MIT Press, Cambridge (2007)

    Google Scholar 

  2. Ba, L.J., Caruana, R.: Do deep networks really need to be deep? In: Ghahrani, Z., et al. (eds.) Advances in Neural Information Processing Systems, vol. 27, pp. 1–9 (2014)

    Google Scholar 

  3. Bellman, R.: Dynamic Programming. Princeton University Press, Princeton (1957)

    MATH  Google Scholar 

  4. DeVore, R.A., Howard, R., Micchelli, C.: Optimal nonlinear approximation. Manuscripta Mathematica 63, 469–478 (1989)

    Article  MathSciNet  Google Scholar 

  5. Poggio, T., Mhaskar, H., Rosasco, L., Miranda, B., Liao, Q.: Why and when can deep-but not shallow-networks avoid the curse of dimensionality: a review. Int. J. Autom. Comput. https://doi.org/10.1007/s11633-017-1054-2

    Article  Google Scholar 

  6. Kainen, P.C., , V., Vogt, A.: Approximation by neural networks is not continuous. Neurocomputing 29, 47–56 (1999)

    Article  Google Scholar 

  7. Kainen, P.C., , V., Vogt, A.: Geometry and topology of continuous best and near best approximations. J. Approx. Theor. 105, 252–262 (2000)

    Article  MathSciNet  Google Scholar 

  8. Kainen, P.C., , V., Vogt, A.: Continuity of approximation by neural networks in \({L}_p\)-spaces. Ann. Oper. Res. 101, 143–147 (2001)

    Google Scholar 

  9. Bengio, Y., Delalleau, O., Roux, N.L.: The curse of highly variable functions for local kernel machines. In: Advances in Neural Information Processing Systems, vol. 18, pp. 107–114. MIT Press, Cambridge (2006)

    Google Scholar 

  10. , V., Sanguineti, M.: Model complexities of shallow networks representing highly varying functions. Neurocomputing 171, 598–604 (2016)

    Article  Google Scholar 

  11. Barron, A.R.: Neural net approximation. In: Narendra, K.S. (ed.) Proceedings of 7th Yale Workshop on Adaptive and Learning Systems, pp. 69–72. Yale University Press (1992)

    Google Scholar 

  12. , V.: Dimension-independent rates of approximation by neural networks. In: Warwick, K., Kárný, M. (eds.) Computer-Intensive Methods in Control and Signal Processing. The Curse of Dimensionality, pp. 261–270. Birkhäuser, Boston (1997)

    Google Scholar 

  13. , V.: Complexity estimates based on integral transforms induced by computational units. Neural Netw. 33, 160–167 (2012)

    Google Scholar 

  14. , V., Sanguineti, M.: Probabilistic lower bounds for approximation by shallow perceptron networks. Neural Netw. 91, 34–41 (2017)

    Google Scholar 

  15. , V.: Constructive lower bounds on model complexity of shallow perceptron networks. Neural Comput. Appl. 29, 305–315 (2018)

    Google Scholar 

  16. Fine, T.L.: Feedforward Neural Network Methodology. Springer, Heidelberg (1999). https://doi.org/10.1007/b97705

    Book  MATH  Google Scholar 

  17. Plan, Y., Vershynin, R.: One-bit compressed sensing by linear programming. Commun. Pure Appl. Math. 66, 1275–1297 (2013)

    Article  MathSciNet  Google Scholar 

  18. Ito, Y.: Finite mapping by neural networks and truth functions. Math. Sci. 17, 69–77 (1992)

    MathSciNet  MATH  Google Scholar 

  19. Kolmogorov, A.: Asymptotic characteristics of some completely bounded metric spaces. Dokl. Akad. Nauk. SSSR 108, 585–589 (1956)

    Google Scholar 

  20. Matoušek, J.: Lectures on Discrete Geometry. Springer, New York (2002). https://doi.org/10.1007/978-1-4613-0039-7

    Book  MATH  Google Scholar 

  21. Schläfli, L.: Gesamelte Mathematische Abhandlungen, vol. 1. Birkhäuser, Basel (1950)

    Book  Google Scholar 

  22. Makovoz, Y.: Random approximants and neural networks. J. Approx. Theor. 85, 98–109 (1996)

    Article  MathSciNet  Google Scholar 

  23. Haussler, D.: Sphere packing numbers for subsets of the Boolean n-cube with bounded Vapnik-Chervonenkis dimension. J. Comb. Theor. A 69(2), 217–232 (1995)

    Article  MathSciNet  Google Scholar 

  24. Kainen, P.C., , V.: Quasiorthogonal dimension of Euclidean spaces. Appl. Math. Lett. 6(3), 7–10 (1993)

    Article  MathSciNet  Google Scholar 

  25. Kainen, P.C., , V.: Quasiorthogonal dimension. In: Kosheleva, O., Shary, S., Xiang, G., Zapatrin, R. (eds.) Beyond Traditional Probabilistic Data Processing Techniques: Interval, Fuzzy, etc. Methods and Their Applications. Springer (2018, to appear)

    Google Scholar 

  26. Tibshirani, R.: Regression shrinkage and selection via the Lasso. J. Roy. Stat. Soc. B 58, 267–288 (1996)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

V.K. was partially supported by the Czech Grant Foundation grant GA18-23827S and institutional support of the Institute of Computer Science RVO 67985807.

Author information

Authors and Affiliations

  1. Czech Academy of Sciences, Institute of Computer Science, Pod Vodárenskou věží 2, 18207, Prague, Czech Republic

    Věra Kůrková

Authors
  1. Věra Kůrková
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Věra Kůrková .

Editor information

Editors and Affiliations

  1. Czech Academy of Sciences, Prague 8, Czech Republic

    Věra Kůrková

  2. Open University of Cyprus, Latsia, Cyprus

    Yannis Manolopoulos

  3. CITEC Bielefeld University, Bielefeld, Germany

    Barbara Hammer

  4. Democritus University of Thrace, Xanthi, Greece

    Lazaros Iliadis

  5. University of Piraeus, Piraeus, Greece

    Ilias Maglogiannis

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kůrková, V. (2018). Sparsity and Complexity of Networks Computing Highly-Varying Functions. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds) Artificial Neural Networks and Machine Learning – ICANN 2018. ICANN 2018. Lecture Notes in Computer Science(), vol 11141. Springer, Cham. https://doi.org/10.1007/978-3-030-01424-7_52

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-01424-7_52

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-01423-0

  • Online ISBN: 978-3-030-01424-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation