Abstract
In order to utilize the large amount of unlabeled sign language resources, unsupervised learning methods are needed. Motivated by the successful results of unsupervised term discovery (UTD) in spoken languages, here we explore how to apply similar methods for sign terms discovery. Our goal is to find the repeating terms from continuous sign videos without any supervision. Using visual features extracted from RGB videos, we show that a k-nearest neighbours based discovery algorithm designed for speech can also discover sign terms. We also run experiments using a baseline UTD algorithm and comment on their differences.
Keywords
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1302–1310 (2017)
McKee, D., McKee, R., Alexander, S.P., Pivac, L.: The online dictionary of New Zealand sign language (2015). http://nzsl.vuw.ac.nz
Dunbar, E., et al.: The zero resource speech challenge 2017. In: Proceeding of IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 323–330 (2017)
Jansen, A., Durme, B.V.: Efficient spoken term discovery using randomized algorithms. In: Proceeding of IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 401–406 (2011)
Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUS. arXiv preprint arXiv:1702.08734 (2017)
Kamper, H.: Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models. In: Proceeding of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6535–3539 (2019)
Kamper, H., Elsner, M., Jansen, A., Goldwater, S.: Unsupervised neural network based feature extraction using weak top-down constraints. In: Proceeding of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5818–5822 (2015)
Kamper, H., Jansen, A., Goldwater, S.: Unsupervised word segmentation and lexicon discovery using acoustic word embeddings. IEEE/ACM Trans. Audio, Speech, Lang. Process. 24(4), 669–679 (2016)
Kelly, D., Mc Donald, J., Markham, C.: Weakly supervised training of a sign language recognition system using multiple instance learning density matrices. IEEE Trans. Syst. Man, Cybern. Part B (Cybernetics) 41(2), 526–541 (2011)
Koller, O., Forster, J., Ney, H.: Continuous sign language recognition: towards large vocabulary statistical recognition systems handling multiple signers. Comput. Vis. Image Understanding 141, 108–125 (2015)
Koller, O., Ney, H., Bowden, R.: Deep hand: how to train a CNN on 1 million hand images when your data is continuous and weakly labelled. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3793–3802 (2016)
Koller, O., Zargaran, S., Ney, H.: Re-Sign: re-aligned end-to-end sequence modelling with deep recurrent CNN-HMMs. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3416–3424 (2017)
Kristoffersen, J.H., et al.: Ordbog over Dansk Tegnsprog (2008). http://www.tegnsprog.dk
Levin, K., Henry, K., Jansen, A., Livescu, K.: Fixed-dimensional acoustic embeddings of variable-length segments in low-resource settings. In: Proceeding of IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 410–415 (2013)
Ludusan, B., et al.: Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems. In: Proceeding of Language Resources and Evaluation Conference (2014)
Nayak, S., Duncan, K., Sarkar, S., Loeding, B.L.: Finding recurrent patterns from continuous sign language sentences for automated extraction of signs. J. Mach. Learn. Res. 13, 2589–2615 (2012)
Ondel, L., Burget, L., Černocký, J.: Variational inference for acoustic unit discovery. Procedia Comput. Sci. 81, 80–86 (2016)
Park, A.S., Glass, J.R.: Unsupervised pattern discovery in speech. IEEE Trans. Audio Speech Lang. Process. 16(1), 186–197 (2008)
Pfister, T., Charles, J., Zisserman, A.: Large-scale learning of sign language by watching TV (using co-occurrences). In: Proceedings of the British Machine Vision Conference, pp. 1–11 (2013)
Polat, K., Saraçlar, M.: Unsupervised term discovery for continuous sign language. In: Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, pp. 189–196. European Language Resources Association (ELRA), Marseille, France (2020)
Sutton, V.: Sign writing. Deaf Action Committee (DAC) for Sign Writing (2000)
Theodorakis, S., Pitsikalis, V., Maragos, P.: Dynamic-static unsupervised sequentiality, statistical subunits and lexicon for sign language recognition. Image Vis. Comput. 32, 533–549 (2014)
Thual, A., Dancette, C., Karadayi, J., Benjumea, J., Dupoux, E.: A k-nearest neighbours approach to unsupervised spoken term discovery. In: Proceeding of 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 491–497 (2018)
Tornay, S., Magimai-Doss, M.: Subunits inference and lexicon development based on pairwise comparison of utterances and signs. Information 10, 298 (2019)
Versteegh, M., et al.: The zero resource speech challenge 2015. In: Proceeding of Interspeech, pp. 3169–3173 (2015)
Yin, P., Starner, T., Hamilton, H., Essa, I., Rehg, J.M.: Learning the basic units in American sign language using discriminative segmental feature selection. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4757–4760 (2009)
Acknowledgments
This work is supported in part by the Scientific and Technological Research Council of Turkey (TUBITAK) under Project 117E059.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Polat, K., Saraçlar, M. (2020). Unsupervised Discovery of Sign Terms by K-Nearest Neighbours Approach. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science(), vol 12536. Springer, Cham. https://doi.org/10.1007/978-3-030-66096-3_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-66096-3_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-66095-6
Online ISBN: 978-3-030-66096-3
eBook Packages: Computer ScienceComputer Science (R0)