Abstract
The output of a speech recognition system is a continuous stream of words that has to be post-processed in various ways, out of which punctuation insertion is an essential step. Punctuated text is far more comprehensible to the reader, can be used for subtitling, and is necessary for further NLP processing, such as machine translation. In this article, we describe how state-of-the-art results in the field of punctuation restoration can be utilized in a production-ready business environment in the Czech language. A recurrent neural network based on long short-term memory is employed, making use of various features: textual based on pre-trained word embeddings, prosodic (mainly temporal), morphological, noise information, and speaker diarization. All the features except morphological tags were found to improve our baseline system. As we work in a real-time setup, it is not possible to employ information from the future of the word stream, yet we achieve significant improvements using LSTM. The usage of RNN also allows the model to learn longer dependencies than any n-gram-based language model can, which we find essential for the insertion of question marks. The deployment of an RNN-based model thus leads to a relative 22.6% decrease in punctuation errors and improvement in all metrics but one.
Supported by the Technology Agency of the Czech Republic (No. FW01010468).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The test set is publicly available on: http://newtontech.net/punctuator/tsd2020_testdata.zip.
References
Boháč, M., Blavka, K.: Using suprasegmental information in recognized speech punctuation completion. In: TSD (2014)
Boháč, M., Rott, M., Kovář, V.: Text punctuation: an inter-annotator agreement study. In: TSD (2017)
Che, X., Wang, C., Yang, H., Meinel, C.: Punctuation prediction for unsegmented transcript based on word vector. In: LREC (2016)
Christensen, H., Gotoh, Y., Renals, S.: Punctuation annotation using statistical prosody models. In: ITRW on Prosody in Speech Recognition and Understanding, pp. 35–40 (2001)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–80 (1997)
Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: Proceedings of the 52nd Annual Meeting of the ACL (Volume 1: Long Papers), pp. 655–665 (2014)
Kim, Y.: Convolutional neural networks for sentence classification. In: EMNLP, pp. 1746–1751 (2014)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2014)
Levy, T., Silber-Varod, V., Moyal, A.: The effect of pitch, intensity and pause duration in punctuation detection. In: 2012 IEEE 27th Convention of Electrical and Electronics Engineers in Israel, pp. 1–4 (2012)
Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., Joulin, A.: Advances in pre-training distributed word representations. In: LREC (2018)
Nouza, J., et al.: Making Czech historical radio archive accessible and searchable for wide public. J. Multimedia 7, 159–169 (2012)
Nouza, J., et al.: Speech-to-text technology to transcribe and disclose 100,000+ hours of bilingual documents from historical Czech and Czechoslovak radio archive. In: Interspeech (2014)
Nouza, J., Šafařík, R., Červa, P.: ASR for South Slavic languages developed in almost automated way. In: Interspeech, pp. 3868–3872 (2016)
Öktem, A., Farrús, M., Wanner, L.: Attentional parallel RNNs for generating punctuation in transcribed speech. In: SLSP, pp. 131–142 (2017)
Stolcke, A., et al.: Automatic detection of sentence boundaries and disfluencies based on recognized words. In: ICSLP (1998)
Straková, J., Straka, M., Hajič, J.: Open-source tools for morphology, lemmatization, POS tagging and named entity recognition. In: Proceedings of 52nd Annual Meeting of the ACL: System Demonstrations, pp. 13–18 (2014)
Swerts, M.: Prosodic features at discourse boundaries of different strength. J. Acoust. Soc. Am. 101, 514–21 (1997)
Tilk, O., Alumäe, T.: LSTM for punctuation restoration in speech transcripts. In: Interspeech (2015)
Tilk, O., Alumäe, T.: Bidirectional recurrent neural network with attention mechanism for punctuation restoration. In: Interspeech (2016)
Weingartovà, L.: Identifikace mluvčího v temporální doméně řeči [Speaker identification in the temporal domain of speech]. Ph.D. thesis, Charles University in Prague (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Hlubík, P., Španěl, M., Boháč, M., Weingartová, L. (2020). Inserting Punctuation to ASR Output in a Real-Time Production Environment. In: Sojka, P., Kopeček, I., Pala, K., Horák, A. (eds) Text, Speech, and Dialogue. TSD 2020. Lecture Notes in Computer Science(), vol 12284. Springer, Cham. https://doi.org/10.1007/978-3-030-58323-1_45
Download citation
DOI: https://doi.org/10.1007/978-3-030-58323-1_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58322-4
Online ISBN: 978-3-030-58323-1
eBook Packages: Computer ScienceComputer Science (R0)