Skip to main content
Springer Nature Link
Log in

Harnessing Uncertainty - Multi-label Dysfluency Classification with Uncertain Labels

  • Conference paper
  • First Online:
Speech and Computer (SPECOM 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13721))

Included in the following conference series:

  • 1093 Accesses

Abstract

Manually labelled datasets inherently contain errors or uncertain/imprecise labelling as sometimes experts cannot agree or are not sure. This issue is even more prominent in multi-label datasets as some labels may be missing. However, discarding samples with high uncertainty may lead to the loss of valuable data.

In this paper, we study two datasets where the uncertainty is explicit in the expert annotations. We give an overview of the different approaches available to deal with uncertainty and evaluate them on two dysfluency datasets. Our results show that adopting methods that embrace uncertainty leads to better results than using only labels with high certainty and performs better than current state of the art results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Chen, P., Liao, B.B., Chen, G., Zhang, S.: Understanding and utilizing deep neural networks trained with noisy labels. In: International Conference on Machine Learning, pp. 1062–1070. PMLR (2019)

    Google Scholar 

  2. Frénay, B., Verleysen, M.: Classification in the presence of label noise: a survey. IEEE Trans. Neural Netw. Learn. Syst. 25(5), 845–869 (2013)

    Article  Google Scholar 

  3. Han, B., et al.: Co-teaching: robust training of deep neural networks with extremely noisy labels. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

    Google Scholar 

  4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  5. Howell, P., Davis, S., Bartrip, J.: The university college London archive of stuttered speech (UCLASS) (2009)

    Google Scholar 

  6. Huang, J., Qu, L., Jia, R., Zhao, B.: O2U-Net: a simple noisy label detection approach for deep neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3326–3334 (2019)

    Google Scholar 

  7. Jouaiti, M., Dautenhahn, K.: Dysfluency classification in stuttered speech using deep learning for real-time applications. In: ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6482–6486 (2022). https://doi.org/10.1109/ICASSP43922.2022.9746638

  8. Kourkounakis, T., Hajavi, A., Etemad, A.: Detecting multiple speech disfluencies using a deep residual network with bidirectional long short-term memory. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6089–6093. IEEE (2020)

    Google Scholar 

  9. Lea, C., Mitra, V., Joshi, A., Kajarekar, S., Bigham, J.P.: SEP-28k: a dataset for stuttering event detection from podcasts with people who stutter. In: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6798–6802. IEEE (2021)

    Google Scholar 

  10. Malisiewicz, T., Gupta, A., Efros, A.A.: Ensemble of exemplar-SVMs for object detection and beyond. In: 2011 International Conference on Computer Vision, pp. 89–96. IEEE (2011)

    Google Scholar 

  11. Oue, S., Marxer, R., Rudzicz, F.: Automatic dysfluency detection in dysarthric speech using deep belief networks. In: Proceedings of SLPAT 2015: 6th Workshop on Speech and Language Processing for Assistive Technologies, pp. 60–64 (2015)

    Google Scholar 

  12. Ratner, N.B., MacWhinney, B.: Fluency bank: a new resource for fluency research and practice. J. Fluen. Disord. 56, 69–80 (2018)

    Article  Google Scholar 

  13. Shafer, G.: Dempster-Shafer theory. Encycl. Artif. Intell. 1, 330–331 (1992)

    Google Scholar 

  14. Sheikh, S.A., Sahidullah, M., Hirsch, F., Ouni, S.: StutterNet: stuttering detection using time delay neural network. arXiv preprint arXiv:2105.05599 (2021)

  15. Sheikh, S.A., Sahidullah, M., Hirsch, F., Ouni, S.: Machine learning for stuttering identification: review, challenges & future directions. arXiv preprint arXiv:2107.04057 (2021)

  16. Shin, W., Ha, J.W., Li, S., Cho, Y., Song, H., Kwon, S.: Which strategies matter for noisy label classification? Insight into loss and uncertainty. arXiv preprint arXiv:2008.06218 (2020)

  17. Shrivastava, A., Gupta, A., Girshick, R.: Training region-based object detectors with online hard example mining. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 761–769 (2016)

    Google Scholar 

  18. Tachioka, Y.: Knowledge distillation using soft and hard labels and annealing for acoustic model training. In: 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE), pp. 689–690. IEEE (2019)

    Google Scholar 

  19. Thiel, C.: Classification on soft labels is robust against label noise. In: Lovrek, I., Howlett, R.J., Jain, L.C. (eds.) KES 2008. LNCS (LNAI), vol. 5177, pp. 65–73. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-85563-7_14

    Chapter  Google Scholar 

  20. Younes, Z., Abdallah, F., Denœux, T.: An evidence-theoretic k-nearest neighbor rule for multi-label classification. In: Godo, L., Pugliese, A. (eds.) SUM 2009. LNCS (LNAI), vol. 5785, pp. 297–308. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04388-8_23

    Chapter  Google Scholar 

  21. Younes, Z., abdallah, F., Denœux, T.: Evidential multi-label classification approach to learning from data with imprecise labels. In: Hüllermeier, E., Kruse, R., Hoffmann, F. (eds.) IPMU 2010. LNCS (LNAI), vol. 6178, pp. 119–128. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14049-5_13

    Chapter  Google Scholar 

  22. Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O.: Understanding deep learning (still) requires rethinking generalization. Commun. ACM 64(3), 107–115 (2021)

    Article  Google Scholar 

Download references

Acknowledgments

This research was undertaken, in part, thanks to funding from the Canada 150 Research Chairs Program.

Author information

Authors and Affiliations

  1. Electrical and Computer Engineering Department, University of Waterloo, 200 University Ave, Waterloo, ON, N2L3G1, Canada

    Melanie Jouaiti & Kerstin Dautenhahn

Authors
  1. Melanie Jouaiti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kerstin Dautenhahn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Melanie Jouaiti .

Editor information

Editors and Affiliations

  1. Indian Institute of Technology Dharwad, Dharwad, India

    S. R. Mahadeva Prasanna

  2. St. Petersburg Federal Research Center of the Russian Academy of Sciences, St. Petersburg, Russia

    Alexey Karpov

  3. Koneru Lakshmaiah Education Foundation, Vaddeswaram, India

    K. Samudravijaya

  4. KIIT Group of Colleges, Gurugram, India

    Shyam S. Agrawal

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jouaiti, M., Dautenhahn, K. (2022). Harnessing Uncertainty - Multi-label Dysfluency Classification with Uncertain Labels. In: Prasanna, S.R.M., Karpov, A., Samudravijaya, K., Agrawal, S.S. (eds) Speech and Computer. SPECOM 2022. Lecture Notes in Computer Science(), vol 13721. Springer, Cham. https://doi.org/10.1007/978-3-031-20980-2_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20980-2_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20979-6

  • Online ISBN: 978-3-031-20980-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics