Abstract
Transfer learning has achieved state-of-the-art performance across many different areas, requiring magnitudes less labeled data compared to traditional methods. Pre-trained weights are learned in a self-supervised way on large amounts of unlabeled data, which are fine-tuned for the desired downstream task using labeled data. An example of this in the speech domain is the wav2vec2.0 framework, which was originally designed for automatic speech recognition (ASR) but can also be fine-tuned for general sequence classification tasks.
This paper analyses the effects of including pathological speech during the pre-training of wav2vec2.0, where quantized speech representations are learned, on the performance of a fine-tuned pathology detection task. We show that this architecture can be successfully fine-tuned for cleft lip and palate (CLP) detection, where the best-performing model yields an F1-score of \(82.3\%\) when pre-trained on healthy speech only. Our experiments show, that including pathological speech during pre-training drastically degrades the performance on detection of the same pathology for which it was fine-tuned. The worst-performing model was pre-trained exclusively on CLP speech, resulting in an F1-score of \(33.9\%\). Whilst performed experiments only focus on CLP, the magnitude of the results suggest, that other pathologies will also follow this trend.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Arasteh, S.T., Weise, T., Schuster, M., et al.: The effect of speech pathology on automatic speaker verification-a large-scale study. arXiv preprint arXiv:2204.06450 (2022)
Baevski, A., et al.: wav2vec 2.0: a framework for self-supervised learning of speech representations. In: Advances in Neural Information Processing Systems, vol. 33, pp. 12449–12460 (2020)
Baumann, I., Wagner, D., Braun, F., et al.: The importance of speech stimuli for pathologic speech classification. arXiv preprint arXiv:2210.15941 (2022)
Bayerl, S.P., Wagner, D., Nöth, E., Riedhammer, K.: Detecting dysfluencies in stuttering therapy using wav2vec 2.0. arXiv preprint arXiv:2204.03417 (2022)
Bhattacharyya, N.: The prevalence of voice problems among adults in the United States. Laryngoscope 124(10), 2359–2362 (2014)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Evain, S., et al.: LeBenchmark: a reproducible framework for assessing self-supervised representation learning from speech. arXiv preprint arXiv:2104.11462 (2021)
Fox, A.: PLAKSS-psycholinguistische analyse kindlicher sprechstörungen. Swets & Zeitlinger, Frankfurt am Main (2002)
Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 369–376 (2006)
Hendrycks, D., Gimpel, K.: Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415 (2016)
Jang, E., Gu, S., Poole, B.: Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 (2016)
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
MacDonald, R.L., et al.: Disordered speech data collection: lessons learned at 1 million utterances from project euphonia. In: Proceedings of the Interspeech 2021, pp. 4833–4837 (2021). https://doi.org/10.21437/Interspeech.2021-697
Maier, A., Haderlein, T., et al.: PEAKS - a system for the automatic evaluation of voice and speech disorders. Speech Commun. 51(5), 425–437 (2009)
Morris, M.A., Meier, S.K., Griffin, J.M., Branda, M.E., Phelan, S.M.: Prevalence and etiologies of adult communication disabilities in the United States: results from the 2012 National Health Interview Survey. Disabil. Health J. 9(1), 140–144 (2016)
Park, D.S., Chan, W., et al.: SpecAugment: a simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779 (2019)
Pérez-Toro, P.A., et al.: Alzheimer’s detection from English to Spanish using acoustic and linguistic embeddings. In: Proceedings of the Interspeech 2022, pp. 2483–2487 (2022)
Schneider, S., Baevski, A., Collobert, R., Auli, M.: wav2vec: unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862 (2019)
Schu, G., et al.: On using the UA-Speech and TORGO databases to validate automatic dysarthric speech classification approaches. arXiv preprint arXiv:2211.08833 (2022)
Triantafyllopoulos, A., et al.: Distinguishing between pre-and post-treatment in the speech of patients with chronic obstructive pulmonary disease. arXiv preprint arXiv:2207.12784 (2022)
Venugopalan, S., et al.: Comparing supervised models and learned speech representations for classifying intelligibility of disordered speech on selected phrases. arXiv preprint arXiv:2107.03985 (2021)
Wolf, T., Debut, L., et al.: Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020)
Yeo, E.J., Choi, K., Kim, S., Chung, M.: Automatic severity assessment of dysarthric speech by using self-supervised model with multi-task learning. arXiv preprint arXiv:2210.15387 (2022)
Acknowledgements
We gratefully acknowledge funding for this study by Friedrich-Alexander-University Erlangen-Nuremberg, Medical Valley e.V. and Siemens Healthineers AG within the framework of d.hip campus.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Weise, T. et al. (2023). Impact of Including Pathological Speech in Pre-training on Pathology Detection. In: Ekštein, K., Pártl, F., Konopík, M. (eds) Text, Speech, and Dialogue. TSD 2023. Lecture Notes in Computer Science(), vol 14102. Springer, Cham. https://doi.org/10.1007/978-3-031-40498-6_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-40498-6_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-40497-9
Online ISBN: 978-3-031-40498-6
eBook Packages: Computer ScienceComputer Science (R0)