Skip to main content
SpringerLink
Log in

On the Rademacher Complexity of Weighted Automata

  • Conference paper
  • First Online:
Algorithmic Learning Theory (ALT 2015)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 9355))

Included in the following conference series:

Abstract

Weighted automata (WFAs) provide a general framework for the representation of functions mapping strings to real numbers. They include as special instances deterministic finite automata (DFAs), hidden Markov models (HMMs), and predictive states representations (PSRs). In recent years, there has been a renewed interest in weighted automata in machine learning due to the development of efficient and provably correct spectral algorithms for learning weighted automata. Despite the effectiveness reported for spectral techniques in real-world problems, almost all existing statistical guarantees for spectral learning of weighted automata rely on a strong realizability assumption. In this paper, we initiate a systematic study of the learning guarantees for broad classes of weighted automata in an agnostic setting. Our results include bounds on the Rademacher complexity of three general classes of weighted automata, each described in terms of different natural quantities. Interestingly, these bounds underline the key role of different data-dependent parameters in the convergence rates.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Abe, N., Warmuth, M.K.: On the computational complexity of approximating distributions by probabilistic automata. Machine Learning (1992)

    Google Scholar 

  2. Albert, J., Kari, J.: Digital image compression. In: Handbook of weighted automata. Springer (2009)

    Google Scholar 

  3. Baier, C., Größer, M., Ciesinski, F.: Model checking linear-time properties of probabilistic systems. In: Handbook of Weighted automata. Springer (2009)

    Google Scholar 

  4. Bailly, R., Denis, F., Ralaivola, L.: Grammatical inference as a principal component analysis problem. In: ICML (2009)

    Google Scholar 

  5. Bailly, R., Denis, F.: Absolute convergence of rational series is semi-decidable. Inf. Comput. (2011)

    Google Scholar 

  6. Balle, B., Carreras, X., Luque, F., Quattoni, A.: Spectral learning of weighted automata: A forward-backward perspective. Machine Learning (2014)

    Google Scholar 

  7. Balle, B., Hamilton, W., Pineau, J.: Methods of moments for learning stochastic languages: unified presentation and empirical comparison. In: ICML (2014)

    Google Scholar 

  8. Balle, B., Mohri, M.: Spectral learning of general weighted automata via constrained matrix completion. In: NIPS (2012)

    Google Scholar 

  9. Balle, B., Mohri, M.: Learning weighted automata. In: CAI (2015)

    Google Scholar 

  10. Balle, B., Panangaden, P., Precup, D.: A canonical form for weighted automata and applications to approximate minimization. In: Logic in Computer Science (LICS) (2015)

    Google Scholar 

  11. Bartlett, P.L., Mendelson, S.: Rademacher and gaussian complexities: risk bounds and structural results. In: Helmbold, D.P., Williamson, B. (eds.) COLT 2001 and EuroCOLT 2001. LNCS (LNAI), vol. 2111, pp. 224–240. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  12. Berstel, J., Reutenauer, C.: Noncommutative rational series with applications. Cambridge University Press (2011)

    Google Scholar 

  13. Boots, B., Siddiqi, S., Gordon, G.: Closing the learning-planning loop with predictive state representations. In: RSS (2009)

    Google Scholar 

  14. Carlyle, J.W., Paz, A.: Realizations by stochastic finite automata. J. Comput. Syst. Sci. 5(1) (1971)

    Google Scholar 

  15. Cortes, C., Mohri, M., Rastogi, A.: Lp distance and equivalence of probabilistic automata. International Journal of Foundations of Computer Science (2007)

    Google Scholar 

  16. Devroye, L., Lugosi, G.: Combinatorial methods in density estimation. Springer (2001)

    Google Scholar 

  17. Eilenberg, S.: Automata, Languages and Machines, vol. A. Academic Press (1974)

    Google Scholar 

  18. Fliess, M.: Matrices de Hankel. Journal de Mathématiques Pures et Appliquées 53 (1974)

    Google Scholar 

  19. de Gispert, A., Iglesias, G., Blackwood, G., Banga, E., Byrne, W.: Hierarchical phrase-based translation with weighted finite-state transducers and shallow-n grammars. Computational Linguistics (2010)

    Google Scholar 

  20. Hamilton, W.L., Fard, M.M., Pineau, J.: Modelling sparse dynamical systems with compressed predictive state representations. In: ICML (2013)

    Google Scholar 

  21. Hsu, D., Kakade, S.M., Zhang, T.: A spectral algorithm for learning hidden Markov models. In: COLT (2009)

    Google Scholar 

  22. Ishigami, Y., Tani, S.: Vc-dimensions of finite automata and commutative finite automata with k letters and n states. Discrete Applied Mathematics (1997)

    Google Scholar 

  23. Knight, K., May, J.: Applications of weighted automata in natural language processing. In: Handbook of Weighted Automata. Springer (2009)

    Google Scholar 

  24. Koltchinskii, V., Panchenko, D.: Rademacher processes and bounding the risk of function learning. In: High Dimensional Probability II, pp. 443–459. Birkhäuser (2000)

    Google Scholar 

  25. Kuich, W., Salomaa, A.: Semirings, automata, languages. In: EATCS. Monographs on Theoretical Computer Science, vol. 5. Springer-Verlag, Berlin-New York (1986)

    Google Scholar 

  26. Kulesza, A., Jiang, N., Singh, S.: Low-rank spectral learning with weighted loss functions. In: AISTATS (2015)

    Google Scholar 

  27. Kulesza, A., Rao, N.R., Singh, S.: Low-rank spectral learning. In: AISTATS (2014)

    Google Scholar 

  28. Massart, P.: Some applications of concentration inequalities to statistics. In: Annales de la Faculté des Sciences de Toulouse (2000)

    Google Scholar 

  29. Mirsky, L.: A trace inequality of John von Neumann. Monatshefte für Mathematik (1975)

    Google Scholar 

  30. Mohri, M.: Weighted automata algorithms. In: Handbook of Weighted Automata. Monographs in Theoretical Computer Science, pp. 213–254. Springer (2009)

    Google Scholar 

  31. Mohri, M., Pereira, F.C.N., Riley, M.: Speech recognition with weighted finite-state transducers. In: Handbook on Speech Processing and Speech Comm. Springer (2008)

    Google Scholar 

  32. Mohri, M., Rostamizadeh, A., Talwalkar, A.: Foundations of machine learning. MIT press (2012)

    Google Scholar 

  33. Salomaa, A., Soittola, M.: Automata-Theoretic Aspects of Formal Power Series. Springer-Verlag, New York (1978)

    Book  MATH  Google Scholar 

  34. Tropp, J.A.: An Introduction to Matrix Concentration Inequalities (2015). ArXiv abs/1501.01571

  35. Vershynin, R.: Lectures in Geometrical Functional Analysis. Preprint (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, McGill University, Montréal, Canada

    Borja Balle

  2. Courant Institute of Mathematical Sciences, New York, NY, USA

    Mehryar Mohri

  3. Google Research, New York, NY, USA

    Mehryar Mohri

Authors
  1. Borja Balle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mehryar Mohri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Borja Balle .

Editor information

Editors and Affiliations

  1. University of California, La Jolla, California, USA

    Kamalika Chaudhuri

  2. UNIV DEL INSUBRIA, 21100 VARESE, Italy

    CLAUDIO GENTILE

  3. REGINA, Saskatchewan, Canada

    Sandra Zilles

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Balle, B., Mohri, M. (2015). On the Rademacher Complexity of Weighted Automata. In: Chaudhuri, K., GENTILE, C., Zilles, S. (eds) Algorithmic Learning Theory. ALT 2015. Lecture Notes in Computer Science(), vol 9355. Springer, Cham. https://doi.org/10.1007/978-3-319-24486-0_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-24486-0_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-24485-3

  • Online ISBN: 978-3-319-24486-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation