Skip to main content
Springer Nature Link
Log in

Speech Processing and Prosody

  • Conference paper
  • First Online:
Text, Speech, and Dialogue (TSD 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11697))

Included in the following conference series:

Abstract

The prosody of the speech signal conveys information over the linguistic content of the message: prosody structures the utterance, and also brings information on speaker’s attitude and speaker’s emotion. Duration of sounds, energy and fundamental frequency are the prosodic features. However their automatic computation and usage are not obvious. Sound duration features are usually extracted from speech recognition results or from a force speech-text alignment. Although the resulting segmentation is usually acceptable on clean native speech data, performance degrades on noisy or not non-native speech. Many algorithms have been developed for computing the fundamental frequency, they lead to rather good performance on clean speech, but again, performance degrades in noisy conditions. However, in some applications, as for example in computer assisted language learning, the relevance of the prosodic features is critical; indeed, the quality of the diagnostic on the learner’s pronunciation will heavily depend on the precision and reliability of the estimated prosodic parameters. The paper considers the computation of prosodic features, shows the limitations of automatic approaches, and discusses the problem of computing confidence measures on such features. Then the paper discusses the role of prosodic features and how they can be handled for automatic processing in some tasks such as the detection of discourse particles, the characterization of emotions, the classification of sentence modalities, as well as in computer assisted language learning and in expressive speech synthesis.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    ORFEO project: http://www.projet-orfeo.fr/.

References

  1. Bartkova, K., Dargnat, M., Jouvet, D., Lee, L.: Annotation of discourse particles in French over a large variety of speech corpora. In: ACor4French - Les corpus annotés du français, TALN 2017 - Traitement Automatique des Langues Naturelles. Orléans, France, June 2017. https://hal.inria.fr/hal-01585540

  2. Bartkova, K., Jouvet, D.: Automatic detection of the prosodic structures of speech utterances. In: Železný, M., Habernal, I., Ronzhin, A. (eds.) SPECOM 2013. LNCS (LNAI), vol. 8113, pp. 1–8. Springer, Cham (2013). https://doi.org/10.1007/978-3-319-01931-4_1

    Chapter  Google Scholar 

  3. Bartkova, K., Jouvet, D.: Links between manual punctuation marks and automatically detected prosodic structures. In: Speech Prosody 2014, Dublin, Ireland, May 2014. https://hal.archives-ouvertes.fr/hal-00998031

  4. Bartkova, K., Jouvet, D.: Analysis of prosodic correlates of emotional speech data. In: 9th Tutorial and Research Workshop on Experimental Linguistics, ExLing 2018, Paris, France, August 2018. https://hal.inria.fr/hal-01889932

  5. Bartkova, K., Jouvet, D., Delais-Roussarie, E.: Prosodic parameters and prosodic structures of French emotional data. In: Speech Prosody 2016, Boston, USA, May 2016. https://hal.inria.fr/hal-01293516

  6. Benzeghiba, M., et al.: Automatic speech recognition and speech variability: a review. Speech Commun. 49, 763–786 (2007). https://hal.inria.fr/inria-00616506

    Article  Google Scholar 

  7. Bisani, M., Ney, H.: Joint-sequence models for grapheme-to-phoneme conversion. Speech Commun. 50(5), 434–451 (2008)

    Article  Google Scholar 

  8. Boersma, P.: Accurate short-term analysis of the fundamental frequency and the harmonics-to-noise ratio of a sampled sound. In: Proceedings of the Institute of Phonetic Sciences, Amsterdam, vol. 17, pp. 97–110 (1993)

    Google Scholar 

  9. Boersma, P., Weenink, D.: Praat: doing phonetics by computer [computer program]. Version 6.0.20 (2011)

    Google Scholar 

  10. Bonneau, A., et al.: Gestion d’erreurs pour la fiabilisation des retours automatiques en apprentissage de la prosodie d’une langue seconde. Traitement Automatique des Langues 53(3) (2013). https://hal.inria.fr/hal-00834278

  11. Camacho, A., Harris, J.G.: A sawtooth waveform inspired pitch estimator for speech and music. J. Acoust. Soc. Am. 124(3), 1638–1652 (2008)

    Article  Google Scholar 

  12. de Cheveigné, A., Kawahara, H.: YIN, a fundamental frequency estimator for speech and music. J. Acoust. Soc. Am. 111(4), 1917–1930 (2002)

    Article  Google Scholar 

  13. Dargnat, M., Bartkova, K., Jouvet, D.: Discourse particles in French: prosodic parameters extraction and analysis. In: Dediu, A.-H., Martín-Vide, C., Vicsi, K. (eds.) SLSP 2015. LNCS (LNAI), vol. 9449, pp. 39–49. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-25789-1_5. https://hal.inria.fr/hal-01184197

    Chapter  Google Scholar 

  14. Deng, B., Jouvet, D., Laprie, Y., Steiner, I., Sini, A.: Towards confidence measures on fundamental frequency estimations. In: IEEE International Conference on Acoustics, Speech and Signal Processing, New Orleans, USA, March 2017. https://hal.inria.fr/hal-01493168

  15. Eskenazi, M.: An overview of spoken language technology for education. Speech Commun. 51(10), 832–844 (2009)

    Article  Google Scholar 

  16. European Language Resources Association (ELRA): Speecon manually pitch-marked reference database for Spanish, ISLRN : 866–498-919-979-7, ELRA ID: ELRA-S0218, Catalogue ELRA. (http://catalog.elra.info/)

  17. Fohr, D., Mella, O., Jouvet, D.: De l’importance de l’homogénéisation des conventions de transcription pour l’alignement automatique de corpus oraux de parole spontanée. In: 8es Journées Internationales de Linguistique de Corpus (JLC2015). Orléans, France, September 2015. https://hal.inria.fr/hal-01183352

  18. Ghahremani, P., BabaAli, B., Povey, D., Riedhammer, K., Trmal, J., Khudanpur, S.: A pitch extraction algorithm tuned for automatic speech recognition. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 2494–2498 (2014)

    Google Scholar 

  19. Illina, I., Fohr, D., Jouvet, D.: Multiple pronunciation generation using grapheme-to-phoneme conversion based on conditional random fields. In: XIV International Conference “Speech and Computer” (SPECOM 2011), Kazan, Russia, September 2011. https://hal.inria.fr/inria-00616325

  20. Illina, I., Fohr, D., Jouvet, D.: Génération des prononciations de noms propres à l’aide des champs aéatoires conditionnels. In: JEP-TALN-RECITAL 2012, Grenoble, France, June 2012. https://hal.inria.fr/hal-00753381

  21. Jouvet, D., Bartkova, K.: Acoustical frame rate and pronunciation variant statistics. In: Dediu, A.-H., Martín-Vide, C., Vicsi, K. (eds.) SLSP 2015. LNCS (LNAI), vol. 9449, pp. 123–134. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-25789-1_12. https://hal.inria.fr/hal-01184195

    Chapter  Google Scholar 

  22. Jouvet, D., Bartkova, K., Dargnat, M., Lee, L.: Analysis and automatic classification of some discourse particles on a large set of french spoken corpora. In: 5th International Conference on Statistical Language and Speech Processing, SLSP 2017, Le Mans, France, October 2017. https://hal.inria.fr/hal-01585567

  23. Jouvet, D., Laprie, Y.: Performance analysis of several pitch detection algorithms on simulated and real noisy speech data. In: 25th European Signal Processing Conference, EUSIPCO 2017, Kos, Greece, August 2017. https://hal.inria.fr/hal-01585554

  24. Jouvet, D., Mesbahi, L., Bonneau, A., Fohr, D., Illina, I., Laprie, Y.: Impact of pronunciation variant frequency on automatic non-native speech segmentation. In: 5th Language and Technology Conference - LTC 2011, Poznan, Poland, pp. 145–148, November 2011. https://hal.archives-ouvertes.fr/hal-00639118

  25. Jurafsky, D., et al.: Automatic detection of discourse structure for speech recognition and understanding. In: 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, pp. 88–95. IEEE (1997)

    Google Scholar 

  26. Kawahara, H., de Cheveigné, A., Banno, H., Takahashi, T., Irino, T.: Nearly defect-free F0 trajectory extraction for expressive speech modifications based on STRAIGHT. In: Interspeech, pp. 537–540 (2005)

    Google Scholar 

  27. Kawahara, H., Estill, J., Fujimura, O.: Aperiodicity extraction and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modification and synthesis system straight. In: MAVEBA, pp. 59–64 (2001)

    Google Scholar 

  28. Kawahara, H., Katayose, H., De Cheveigné, A., Patterson, R.D.: Fixed point analysis of frequency to instantaneous frequency mapping for accurate estimation of F0 and periodicity. In: Eurospeech, pp. 2781–2784 (1999)

    Google Scholar 

  29. Kolář, J., Lamel, L.: Development and evaluation of automatic punctuation for French and English speech-to-text. In: Thirteenth Annual Conference of the International Speech Communication Association (2012)

    Google Scholar 

  30. Král, P., Kleckova, J., Cerisara, C.: Sentence modality recognition in French based on prosody. In: International Conference on Enformatika, Systems Sciences and Engineering-ESSE, vol. 8, pp. 185–188. Citeseer (2005)

    Google Scholar 

  31. Kulkarni, A., Vincent, C., Denis, J.: Layer adaptation for transfer of expressivity in speech synthesis. In: 9th Language and Technology Conference Proceedings of LTC 2019 (2019)

    Google Scholar 

  32. Lanjewar, R.B., Chaudhari, D.: Speech emotion recognition: a review. Int. J. Innov. Technol. Explor. Eng. (IJITEE) 2, 68–71 (2013)

    Google Scholar 

  33. Lee, L., Bartkova, K., Dargnat, M., Jouvet, D.: Prosodic and pragmatic values of discourse particles in French. In: 9th Tutorial and Research Workshop on Experimental Linguistics, ExLing 2018, Paris, France, August 2018. https://hal.inria.fr/hal-01889925

  34. Lee, L., Bartkova, K., Jouvet, D., Dargnat, M., Yvon, K.: Can prosody meet pragmatics? Case of discourse particles in French. In: To Appear in Proceedings of ICPhS 2019. International Congress of Phonetic Sciences (2019)

    Google Scholar 

  35. Margolis, A., Ostendorf, M.: Question detection in spoken conversations using textual conversations. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers-Volume 2, pp. 118–124. Association for Computational Linguistics (2011)

    Google Scholar 

  36. Martin, P.: Prosodic and rhythmic structures in French. Linguistics 25(5), 925–950 (1987)

    Article  Google Scholar 

  37. Mesbahi, L., Jouvet, D., Bonneau, A., Fohr, D., Illina, I., Laprie, Y.: Reliability of non-native speech automatic segmentation for prosodic feedback. In: Workshop on Speech and Language Technology in Education - SLaTE 2011. ISCA, Venise, August 2011. https://hal.inria.fr/inria-00614930

  38. Novak, J.R., Minematsu, N., Hirose, K.: Phonetisaurus: exploring grapheme-to-phoneme conversion with joint n-gram models in the WFST framework. Nat. Lang. Eng. 22(6), 907–938 (2016)

    Article  Google Scholar 

  39. Orosanu, L., Jouvet, D.: Combining lexical and prosodic features for automatic detection of sentence modality in French. In: Dediu, A.-H., Martín-Vide, C., Vicsi, K. (eds.) SLSP 2015. LNCS (LNAI), vol. 9449, pp. 207–218. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-25789-1_20. https://hal.inria.fr/hal-01184196

    Chapter  Google Scholar 

  40. Orosanu, L., Jouvet, D., Fohr, D., Illina, I., Bonneau, A.: Combining criteria for the detection of incorrect entries of non-native speech in the context of foreign language learning. In: SLT 2012–4th IEEE Workshop on Spoken Language Technology, Miami, United States, December 2012. https://hal.inria.fr/hal-00753458

  41. Pirker, G., Wohlmayr, M., Petrik, S., Pernkopf, F.: A pitch tracking corpus with evaluation on multipitch tracking scenario. In: Interspeech, pp. 1509–1512 (2011)

    Google Scholar 

  42. Quang, V.M., Castelli, E., Yên, P.N.: A decision tree-based method for speech processing: question sentence detection. In: Wang, L., Jiao, L., Shi, G., Li, X., Liu, J. (eds.) FSKD 2006. LNCS (LNAI), vol. 4223, pp. 1205–1212. Springer, Heidelberg (2006). https://doi.org/10.1007/11881599_150

    Chapter  Google Scholar 

  43. Rao, K., Peng, F., Sak, H., Beaufays, F.: Grapheme-to-phoneme conversion using long short-term memory recurrent neural networks. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4225–4229. IEEE (2015)

    Google Scholar 

  44. Schröder, M.: Expressive speech synthesis: past, present, and possible futures. In: Tao, J., Tan, T. (eds.) Affective Information Processing, pp. 111–126. Springer, London (2009). https://doi.org/10.1007/978-1-84800-306-4_7

    Chapter  Google Scholar 

  45. Schuller, B.W.: Speech emotion recognition: two decades in a nutshell, benchmarks, and ongoing trends. Commun. ACM 61(5), 90–99 (2018)

    Article  Google Scholar 

  46. Segal, N., Bartkova, K.: Prosodic structure representation for boundary detection in spontaneous French. In: Proceedings of ICPhS, pp. 1197–1200 (2007)

    Google Scholar 

  47. Sethu, V., Epps, J., Ambikairajah, E.: Speech based emotion recognition. In: Ogunfunmi, T., Togneri, R., Narasimha, M.S. (eds.) Speech and Audio Processing for Coding, Enhancement and Recognition, pp. 197–228. Springer, New York (2015). https://doi.org/10.1007/978-1-4939-1456-2_7

    Chapter  Google Scholar 

  48. Shadiev, R., Hwang, W.Y., Huang, Y.M.: Review of research on mobile language learning in authentic environments. Comput. Assist. Lang. Learn. 30(3–4), 284–303 (2017)

    Article  Google Scholar 

  49. Sorin, A., et al.: The ETSI extended distributed speech recognition (DSR) standards: client side processing and tonal language recognition evaluation. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 1, pp. 129–132 (2004)

    Google Scholar 

  50. Stede, M., Schmitz, B.: Discourse particles and discourse functions. Mach. Transl. 15(1–2), 125–147 (2000)

    Article  Google Scholar 

  51. Strömbergsson, S.: Today’s most frequently used f 0 estimation methods, and their accuracy in estimating male and female pitch in clean speech. In: Interspeech 2016, pp. 525–529 (2016)

    Google Scholar 

  52. Talkin, D.: A robust algorithm for pitch tracking (RAPT). In: Kleijn, W.B., Paliwal, K.K. (eds.) Speech Coding and Synthesis, pp. 495–518. Elsevier, Amsterdam (1995)

    Google Scholar 

  53. Viberg, O., Grönlund, Å.: Mobile assisted language learning: a literature review. In: 11th World Conference on Mobile and Contextual Learning (2012)

    Google Scholar 

  54. Witt, S.M., Young, S.J.: Phone-level pronunciation scoring and assessment for interactive language learning. Speech Commun. 30(2–3), 95–108 (2000)

    Article  Google Scholar 

  55. Wu, Z., Watts, O., King, S.: Merlin: an open source neural network speech synthesis system. In: SSW, pp. 202–207 (2016)

    Google Scholar 

  56. Yao, K., Zweig, G.: Sequence-to-sequence neural net models for grapheme-to-phoneme conversion. arXiv preprint arXiv:1506.00196 (2015)

  57. Zen, H., et al.: The HMM-based speech synthesis system (HTS) version 2.0. In: SSW, pp. 294–299. Citeseer (2007)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Université de Lorraine, CNRS, Inria, LORIA, 54000, Nancy, France

    Denis Jouvet

Authors
  1. Denis Jouvet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Denis Jouvet .

Editor information

Editors and Affiliations

  1. University of West Bohemia, Pilsen, Czech Republic

    Kamil Ekštein

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jouvet, D. (2019). Speech Processing and Prosody. In: Ekštein, K. (eds) Text, Speech, and Dialogue. TSD 2019. Lecture Notes in Computer Science(), vol 11697. Springer, Cham. https://doi.org/10.1007/978-3-030-27947-9_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-27947-9_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-27946-2

  • Online ISBN: 978-3-030-27947-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics