Skip to main content
Springer Nature Link
Log in

Analysis of Czech Web 1T 5-Gram Corpus and Its Comparison with Czech National Corpus Data

  • Conference paper
Text, Speech and Dialogue (TSD 2010)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 6231))

Included in the following conference series:

  • 1479 Accesses

Abstract

In this paper, newly issued Czech Web 1T 5-grams corpus created by Google and LDC is analysed and compared with reference n-gram corpus obtained from Czech National Corpus. Original 5-grams from both corpora were post-processed and statistical trigram language models of various vocabulary sizes and parameters were created. The comparison of various corpus statistics such as unique and total word and n-gram counts before and after post-processing is presented and discussed, especially with the focus on clearing Web 1T data from invalid tokens. The tools from HTK Toolkit were used for the evaluation and accuracy, OOV rates and perplexity were measured using sentence transcriptions from Czech SPEECON database.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Nouza, J., Ždánský, J., David, P., Červa, P., Kolorenč, J., Nejedlová, D.: Fully Automated System for Czéch Spoken Broadcast Transcription with Very Large (300K+) Lexicon. In: Interspeech 2005, Lisboa, Portugal (September 2005)

    Google Scholar 

  2. Pražák, A., Muller, L., Psutka, J.: LIVE TV SUBTITLING – Fast 2-pass LVCSR System for Online Subtitling. In: SIGMAP 2007. INSTICC PRESS, Lisabon (2007)

    Google Scholar 

  3. Brants, T., Franz, A.: Web 1T 5-gram, 10 European Languages, version 1. Linguistic Data Consortium, Philadelphia (2009), Web page http://www.ldc.upenn.edu

  4. Czéch National Corpus: Český národní korpus (Czéch National Corpus) – SYN2006PUB. Institute of the Czéch National Corpus FF UK, Praha (2006), http://www.korpus.cz

  5. Linguistic Data Consortium: Home page (2010), http://www.ldc.upenn.edu

  6. Brants, T., Franz, A.: Web 1T 5-gram, version 1. Linguistic Data Consortium, Philadelphia (2006), Web page http://www.ldc.upenn.edu

  7. Czéch National Corpus: Home page. Institute of the Czéch National Corpus FF UK, Praha (2010), http://www.korpus.cz

  8. Young, S., et al.: The Hidden Markov Model Toolkit (HTK), Version 3.4.1, Cambridge (2009), http://htk.eng.cam.ac.uk

  9. Moreno, A.: LC-StarII. Lexica and Corpora for Speech-to-Speech Translation Components, http://www.lc-star.org

  10. Pollák, P., Černocký, J., Smrž, P.: LC-STAR CSCZ. Czéch lexicon for ASR and TTS (October 2008), http://www.lc-star.org

  11. Pollák, P., Černocký, J.: Czéch SPEECON Adult Database (November 2003), http://www.speechdat.org/speecon

  12. Iskra, D., Grosskopf, B., Marasek, K., van den Heuvel, H., Diehl, F., Kiessling, A.: SPEECON – Speech Databases for Consumer Devices: Database Specification and Validation. In: Proc. of LREC 2002 (May 2002)

    Google Scholar 

  13. Young, S., et al.: The HTK Book, Version 3.4.1, Cambridge (2009), http://htk.eng.cam.ac.uk

  14. Mikolov, T., Oparin, I., Glembek, O., Burget, L., Karafiát, M., Černocký, J.: Použití mluvených korpusů ve vývoji systému pro rozpoznávání přednášek (Use of spoken corpora in the development of system for recognition of Czéch lectures). In: Proc. of Čeština v mluveném korpusu (Czéch in Spoken Corpus), Praha (2007)

    Google Scholar 

  15. Mikolov, T.: Language Models for Automatic Speech Recognition of Czéch Lectures. In: Proc. of STUDENT EEICT 2008, Brno (April 2008)

    Google Scholar 

  16. Byrne, W., Hajič, J., Ircing, P., Krbec, P., Psutka, J.: Morpheme Based Language Models for Speech Recognition in Czéch. In: Proc. of Text, Speech, and Dialog 2000, Brno (2000)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dept. of Circuit Theory, Czech Technical University, Technická 2, 166 27, Prague, Czech Republic

    Václav Procházka & Petr Pollák

Authors
  1. Václav Procházka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Petr Pollák
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Faculty of Informatics, Masaryk University, Brno, Czech Republic

    Petr Sojka

  2. Faculty of Informatics, Masaryk University, Botanická 68a, 602 00, Brno, Czech Republic

    Aleš Horák

  3. Faculty of Informatics, Masaryk University, Botanická 68a, CZ-602 00, Brno, Czech Republic

    Ivan Kopeček

  4. Faculty of Informatics, Department of Computer Graphics and Design, Masaryk University, Botanická 68a, 602 00, Brno, Czech Republic

    Karel Pala

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Procházka, V., Pollák, P. (2010). Analysis of Czech Web 1T 5-Gram Corpus and Its Comparison with Czech National Corpus Data. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds) Text, Speech and Dialogue. TSD 2010. Lecture Notes in Computer Science(), vol 6231. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15760-8_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-15760-8_24

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-15759-2

  • Online ISBN: 978-3-642-15760-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics