Abstract
Continuous Speech Keyword Spotting (CSKS) is the problem of spotting keywords in recorded conversations, when a small number of instances of keywords are available in training data. Unlike the more common Keyword Spotting, where an algorithm needs to detect lone keywords or short phrases like “Alexa”, “Cortana”, “Hi Alexa!”, “Whatsup Octavia?” etc. in speech, CSKS needs to filter out embedded words from a continuous flow of speech, ie. spot “Anna” and “github” in “I know a developer named Anna who can look into this github issue.” Apart from the issue of limited training data availability, CSKS is an extremely imbalanced classification problem. We address the limitations of simple keyword spotting baselines for both aforementioned challenges by using a novel combination of loss functions (Prototypical networks’ loss and metric loss) and transfer learning. Our method improves F1 score by over 10%.
H. Seth and P. Kumar—Authors contributed equally.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Sainath, T.N., Parada, C.: Convolutional neural networks for small-footprint keyword spotting. In: Interspeech (2015)
Tang, R., Lin, J.: Honk: a PyTorch reimplementation of convolutional neural networks for keyword spotting. arXiv preprint arXiv:1710.06554, 18 October 2017
Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., Prenger, R., Satheesh, S., Sengupta, S., Coates, A., Ng, A.Y.: Deep speech: scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567, 17 December 2014
Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: Advances in Neural Information Processing Systems, pp. 4077–4087 (2017)
Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: International Workshop on Similarity-Based Pattern Recognition, vol. 12, pp. 84–92. Springer, Cham (2015)
Weintraub, M.: Keyword-spotting using SRI’s DECIPHER large-vocabulary speech-recognition system. In: 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing 1993, ICASSP 1993, vol. 2, pp. 463–466. IEEE, 27 April 1993
Wilpon, J.G., Rabiner, L.R., Lee, C.H., Goldman, E.R.: Automatic recognition of keywords in unconstrained speech using hidden Markov models. IEEE Transact. Acoust. Speech Sig. Proc. 38(11), 1870–1878 (1990)
Rose, R.C., Paul, D.B.: A hidden Markov model based keyword recognition system. In: 1990 International Conference on Acoustics, Speech, and Signal Processing 1990, ICASSP 1990, pp. 129–132. IEEE, 3 April 1990
Nouza, J., Silovsky, J.: Fast keyword spotting in telephone speech. Radioengineering 18(4), 665–70 (2009)
Lee, H., Pham, P., Largman, Y., Ng, A.Y.: Unsupervised feature learning for audio classification using convolutional deep belief networks. In: Advances in Neural Information Processing Systems, pp. 1096–1104 (2009)
Mohamed, A.R., Dahl, G.E., Hinton, G.: Acoustic modeling using deep belief networks. IEEE Trans. Audio Speech Lang. Process. 20(1), 14–22 (2012)
Grosse, R., Raina, R., Kwong, H., Ng, A.Y.: Shift-invariance sparse coding for audio classification. arXiv preprint arXiv:1206.5241, 20 June 2012
Hinton, G., Deng, L., Yu, D., Dahl, G.E., Mohamed, A.R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T.N., Kingsbury, B.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012)
Dahl, G.E., Yu, D., Deng, L., Acero, A.: Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transact. Audio Speech Lang. Process. 20(1), 30–42 (2012)
Li, K.P., Naylor, J.A., Rossen, M.L.: A whole word recurrent neural network for keyword spotting. In: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing 1992, ICASSP 1992, vol. 2, pp. 81–84. IEEE, 23 March 1992
Fernández, S., Graves, A., Schmidhuber, J.: An application of recurrent neural networks to discriminative keyword spotting. In: International Conference on Artificial Neural Networks, vol. 9, pp. 220–229. Springer, Heidelberg, September 2007
Chen, G., Parada, C., Heigold, G.: Small-footprint keyword spotting using deep neural networks. In: ICASSP, vol. 14, pp. 4087–4091, 4 May 2014
Pons, J., Serrá, J., Serra, X.: Training neural audio classifiers with few data. arXiv preprint arXiv:1810.10274, 24 October 2018
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D.: Matching networks for one shot learning. In: Advances in Neural Information Processing Systems, pp. 3630–3638 (2016)
Kunze, J., Kirsch, L., Kurenkov, I., Krug, A., Johannsmeier, J., Stober, S.: Transfer learning for speech recognition on a budget. arXiv preprint arXiv:1706.00290, 1 June 2017
Choi, K., Fazekas, G., Sandler, M., Cho, K.: Transfer learning for music classification and regression tasks. arXiv preprint arXiv:1703.09179, 27 March 2017
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Honk: A PyTorch Reimplementation of Convolutional Neural Networks for Keyword Spotting. https://github.com/castorini/honk
Rohlicek, J.R., Russell, W., Roukos, S., Gish, H.: Continuous hidden Markov modeling for speaker-independent word spotting. In: 1989 International Conference on Acoustics, Speech, and Signal Processing 1989, ICASSP 1989, pp. 627–630. IEEE, 23 May 1989
Wilpon, J.G., Miller, L.G., Modi, P.: Improvements and applications for key word recognition using hidden Markov modeling techniques. In: 1991 International Conference on Acoustics, Speech, and Signal Processing 1991, ICASSP 1991, pp. 309–312. IEEE, 14 April 1991
Silaghi, M.C., Bourlard, H.: Iterative posterior-based keyword spotting without filler models. In: Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, pp. 213–216, 12 December 1999
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Seth, H., Kumar, P., Srivastava, M.M. (2020). Prototypical Metric Transfer Learning for Continuous Speech Keyword Spotting with Limited Training Data. In: Martínez Álvarez, F., Troncoso Lora, A., Sáez Muñoz, J., Quintián, H., Corchado, E. (eds) 14th International Conference on Soft Computing Models in Industrial and Environmental Applications (SOCO 2019). SOCO 2019. Advances in Intelligent Systems and Computing, vol 950. Springer, Cham. https://doi.org/10.1007/978-3-030-20055-8_26
Download citation
DOI: https://doi.org/10.1007/978-3-030-20055-8_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20054-1
Online ISBN: 978-3-030-20055-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)