Skip to main content
SpringerLink
Log in

Learning as Data Compression

  • Conference paper
Computation and Logic in the Real World (CiE 2007)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 4497))

Included in the following conference series:

Abstract

In this paper I describe the general principles of learning as data compression. I introduce two-part code optimization and analyze the theoretical background in terms of Kolmogorov complexity. The good news is that the optimal compression theoretically represents the optimal interpretation of the data, the bad news is that such an optimal compression cannot be computed and that an increase in compression not necessarily implies a better theory. I discuss the application of these insights to DFA induction.

This project is supported by a BSIK grant from the Dutch Ministry of Education, Culture and Science (OC&W) and is part of the ICT innovation program of the Ministry of Economic Affairs (EZ).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Adriaans, P., Vitányi, P.: The Power and Perils of MDL, IEEE Trans. Inform. Th. (submitted)

    Google Scholar 

  2. Adriaans, P.W.: The philosophy of learning, Handbook of the philosophy of information. In: Adriaans, P.W., van Benthem, J. (eds.) Handbook of the philosophy of science, Series edited by Gabbay, D. M., Thagard, P., Woods, J. (to appear)

    Google Scholar 

  3. Adriaans, P.W.: Learning Deterministic DEC Grammars Is Learning Rational Numbers. In: Sakakibara, Y., Kobayashi, S., Sato, K., Nishino, T., Tomita, E. (eds.) ICGI 2006. LNCS (LNAI), vol. 4201, pp. 320–326. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  4. Adriaans, P.W.: Using MDL for Grammar Induction, in Grammatical Inference: Algorithms and Applications. In: Sakakibara, Y., Kobayashi, S., Sato, K., Nishino, T., Tomita, E. (eds.) ICGI 2006. LNCS (LNAI), vol. 4201, pp. 293–306. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  5. Cilibrasi, R., Vitányi, P.: Clustering by compression, IEEE Trans. Infomat. Th., Submitted. See http://arxiv.org/abs/cs.CV/0312044

  6. Cilibrasi, R., Vitányi, P.M.B.: Automatic Meaning Discovery Using Google (2004), http://www.citebase.org/abstract?id=oai:arXiv.org:cs/0412098

  7. Domingos, P.: The Role of Occam’s Razor in Knowledge Discovery. Data. Mining and Knowledge Discovery 3(4), 409–425 (1999)

    Article  Google Scholar 

  8. Barron, A., Rissanen, J., Yu, B.: The minimum description length principle in coding and modeling. IEEE Trans. Information Theory 44(6), 2743–2760 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  9. Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (1997)

    MATH  Google Scholar 

  10. Li, M., Vitányi, P.M.B.: An Introduction to Kolmogorov Complexity and Its Applications, 2nd edn. Springer, New York (1997)

    Book  MATH  Google Scholar 

  11. Vereshchagin, N.K., Vitányi, P.M.B.: Kolmogorov’s structure functions and model selection. IEEE Trans. Information Theory 50(12), 3265–3290 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  12. Grünwald, P.D., Langford, J.: Suboptimal behavior of Bayes and MDL in classification under misspecification. Machine Learning (2007)

    Google Scholar 

  13. Gold, E.: Mark, Language Identification in the Limit. Information and Control 10(5), 447–474 (1967)

    Article  MathSciNet  MATH  Google Scholar 

  14. Pitt, L., Warmuth, M.K.: The Minimum Consistent DFA Problem Cannot be Approximated within any Polynomial. Journal of the ACM 40(1), 95–142 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  15. Adriaans, P., Vervoort, M.: The EMILE 4.1 grammar induction toolbox. In: Adriaans, P., Fernau, H., van Zaanen, M. (eds.) ICGI 2002. LNCS (LNAI), vol. 2484, pp. 293–295. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  16. Vervoort, M.: Games, walks and Grammars, Thesis University of Amsterdam (2000)

    Google Scholar 

  17. Lang, K.J., Pearlmutter, B.A., Price, R.A.: Results of the Abbadingo One DFA learning competition and a new evidence-driven state merging algorithm. In: Adriaans, P., Fernau, H., van Zaanen, M. (eds.) ICGI 2002. LNCS (LNAI), vol. 2484, pp. 1–12. Springer, Heidelberg (2002)

    Google Scholar 

  18. van Zaanen, M., Adriaans, P.: Alignment-Based Learning versus EMILE: A Comparison. In: Proceedings of the Belgian-Dutch Conference on Artificial Intelligence (BNAIC), pp. 315–322. Amsterdam, the Netherlands (2001)

    Google Scholar 

  19. Solan, Z., Horn, D., Ruppin, E., Edelman, S.: Unsupervised learning of natural languages. PNAS 102(33), 11629–11634 (2005)

    Article  Google Scholar 

  20. Curnéjols, A., Miclet, L.: Apprentissage artificiel, concepts et algorithmes, Eyrolles (2003)

    Google Scholar 

  21. Gerard Wolff, J.: Unifying Computing And Cognition, The SP Theory and its Applications, CognitionResearch.org.uk (2006)

    Google Scholar 

  22. Wolff, J.G.: Computing As Compression: An Overview of the SP Theory and System. New Generation Comput. 13(2), 187–214 (1995)

    Article  Google Scholar 

  23. Wolff, J.G.: Information Compression by Multiple Alignment, Unification and Search as a Unifying Principle in Computing and Cognition. Journal of Artificial Intelligence Research 19(3), 193–230 (2003)

    Article  Google Scholar 

  24. Dubrovnik, Croatia, de la Higuera, Colin and Adriaans, Pieter and van Zaanen, Menno and Oncina, Jose (eds.): Proceedings of the Workshop and Tutorial on Learning Context-Free Grammars held at the 14th European Conference on Machine Learning (ECML) and the 7th European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD) (2003)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Amsterdam,Kruislaan 419, 1098VA Amsterdam, The Netherlands

    Pieter Adriaans

Authors
  1. Pieter Adriaans
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Dept. of Pure Mathematics, University of Leeds, LS2 9JT, Leeds, UK

    S. Barry Cooper

  2. Institue for Logic, Language and Computation (ILLC), Universiteit van Amsterdam, Plantage Muidergracht 24, 1018, TV Amsterdam, The Netherlands

    Benedikt Löwe

  3. Dipartimento di Scienze Matematiche ed Informatiche “R. Magari”, University of Siena, 53100, Siena, Italy

    Andrea Sorbi

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Adriaans, P. (2007). Learning as Data Compression. In: Cooper, S.B., Löwe, B., Sorbi, A. (eds) Computation and Logic in the Real World. CiE 2007. Lecture Notes in Computer Science, vol 4497. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-73001-9_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-73001-9_2

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-73000-2

  • Online ISBN: 978-3-540-73001-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation