Priyanka Ganesan
Rositsa Doneva
ABSTRACT
Sign Language recognition system plays a vital role for the hearing and visually impaired people to make normal communication with the other common people. But deaf and dumb people signs are not understandable to the common person, leading to a communication barrier. Also, there were only 50 certified sign language interpreters in India for a deaf population of around 7 million. To come this communication barrier, an intelligent translator system for isolated and continuous sign language recognition is proposed. In our proposed work, the continuous sign language recognition is dealt as a multi-class classification problem. A Video-based custom Long-Short Term Memory (VidcuLSTM) model was designed and configured with different hyperparameter tuning and optimizers to avoid memorization and overfitting of the translator system. We evaluated the proposed system using Chinese Isolated and Continuous SLR dataset and it is evident from the results that the proposed system outperforms existing state-of-art systems with improved performance in recognition rate by around 5% with reduced WER of around 2.67.
- L. Boppana, R. Ahamed, H. Rane and R. K. Kodali, "Assistive Sign Language Converter for Deaf and Dumb," 2019 International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), 2019, pp. 302-307, doi: 10.1109/iThings/GreenCom/CPSCom/SmartData.2019.00071.Google Scholar
Cross Ref
- Anshul Mittal, Pradeep Kumar, Partha Roy, Raman Balasubramanian, BidyutB. Chaudhari, “A Modified LSTM model for Continuous Sign Language Recognition Using Leap Motion”, IEEE Sensors Journal 19(16):7056-7063, Aug 2019.Google ScholarCross Ref
- Gaolin Fang, Wen Gao, Debin Zhao, “A Real-Time Large Vocabulary Recognition System for Chinese Sign Language”, IEEE Conference on Advances in Multimedia Information Processing, Oct 2016.Google Scholar
- Guilin Yao, Hongxun Yao, Xin Liu, Feng Jiang, “Real Time Large Vocabulary Continuous Sign Language Recognition based on OP/Viterbi Algorithm”, IEEE Conference on Pattern Recognition,2006.Google Scholar
- Tamer Shanableh, Khaled Assaleh, M.Al-Rousan, “Spatio-Temporal Feature-Extraction Techniques for Isolated Gesture Recognition in Arabic Sign Language”, IEEE Transactions On Cybernetics 37(3):641-50, Jul 2007.Google ScholarDigital Library
- Qinkun Xiao, Minying Qin, Peng Gao, Yidan Zhao, “Multimodal Fusion based on LSTM and a Couple Conditional Hidden Markov Model for Chinese Sign Language Recognition, IEEE Access PP(99):1-1,June 2019.Google ScholarCross Ref
- Gaolin Fang, Wen Gao, Debin Zhao, “Large-Vocabulary Continuous Sign Language Recognition based on Transition-Movement Models”, IEEE Transactions on Systems, Man, And Cybernetics, Vol. 37, No. 1, January 2007.Google Scholar
- Gaolin Fang, Wen Gao, Debin Zhao, “Large Vocabulary Sign Language Recognition based on Hierarchical Decision Trees”, IEEE Transactions on Systems Man and Cybernetics, June 2004.Google Scholar
- Cao Dong, Ming C. Leu, Zhaozheng Yin, “American Sign Language Alphabet Recognition Using Microsoft Kinect”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 44-52, 2015.Google Scholar
- Assaleh, K., Al-Rousan, M. Recognition of Arabic Sign Language Alphabet Using Polynomial Classifiers. EURASIP J. Adv. Signal Process. 2005, 507614 (2005). https://doi.org/10.1155/ASP.2005.2136Google ScholarDigital Library
- W. Aly, S. Aly and S. Almotairi, "User-Independent American Sign Language Alphabet Recognition Based on Depth Image and PCANet Features," in IEEE Access, vol. 7, pp. 123138-123150, 2019, doi: 10.1109/ACCESS.2019.2938829.Google ScholarCross Ref
- Prajwal Paudyal, Junghyo Lee, Ayan Banerjee, Sandeep K. S. Gupta, “A Comparison of Techniques for Sign Language Alphabet Recognition Using Armband Wearables”, ACM Transactions on Interactive Intelligent Systems, Volume 9,Issue 2-3, September 2019.Google ScholarDigital Library
- A. M. Rafi, N. Nawal, N. S. N. Bayev, L. Nima, C. Shahnaz and S. A. Fattah, "Image-based Bengali Sign Language Alphabet Recognition for Deaf and Dumb Community," 2019 IEEE Global Humanitarian Technology Conference (GHTC), 2019, pp. 1-7, doi: 10.1109/GHTC46095.2019.9033031.Google ScholarCross Ref
- M. Mohandes, M. Deriche and J. Liu, "Image-Based and Sensor-Based Approaches to Arabic Sign Language Recognition," in IEEE Transactions on Human-Machine Systems, vol. 44, no. 4, pp. 551-557, Aug. 2014, doi: 10.1109/THMS.2014.2318280.Google ScholarCross Ref
- Masood, S., Srivastava, A., Thuwal, H.C., Ahmad, M. (2018). Real-Time Sign Language Gesture (Word) Recognition from Video Sequences Using CNN and RNN. In: Bhateja, V., Coello Coello, C., Satapathy, S., Pattnaik, P. (eds) Intelligent Engineering Informatics. Advances in Intelligent Systems and Computing, vol 695. Springer, Singapore. https://doi.org/10.1007/978-981-10-7566-7_63.Google ScholarCross Ref
- Lean Karlo S. Tolentino, Ronnie O. Serfa Juan, August C. Thio-ac, Maria Abigail B. Pamahoy, Joni Rose R. Forteza, and Xavier Jet O. Garcia, “Static Sign Language Recognition Using Deep Learning”, International Journal of Machine Learning and Computing, Vol. 9, No. 6, December 2019.Google ScholarCross Ref
- Runpeng Cui, Hu Liu, Changshui Zhang, “Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7361-7369, 2017.Google ScholarCross Ref
- Junfu Pu, Wengang Zhou, Houqiang Li, “Iterative Alignment Network for Continuous Sign Language Recognition”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4165-4174, 2019.Google ScholarCross Ref
- Aloysius, N., Geetha, M. Understanding vision-based continuous sign language recognition. Multimed Tools Appl 79, 22177–22209 (2020). https://doi.org/10.1007/s11042-020-08961-z.Google ScholarCross Ref
- O Koller, O Zargaran, H Ney and R Bowden, “Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition”, Proceedings of the British Machine Vision Conference 2016, The British Machine Vision Conference (BMVC) 2016.Google ScholarCross Ref
- Fang, G., Gao, W., Chen, X., Wang, C., Ma, J. (2002). Signer-Independent Continuous Sign Language Recognition Based on SRN/HMM. In: Wachsmuth, I., Sowa, T. (eds) Gesture and Sign Language in Human-Computer Interaction. GW 2001. Lecture Notes in Computer Science, vol 2298. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-47873-6_8.Google ScholarCross Ref
- Wenwen Yang, Jinxu Tao, Zhongfu Ye, “Continuous sign language recognition using level building based on fast hidden Markov model”, Pattern Recognition Letters, Volume 78, Pages 28-35, 2016.Google ScholarDigital Library
- R. Cui, H. Liu and C. Zhang, "A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training," in IEEE Transactions on Multimedia, vol. 21, no. 7, pp. 1880-1891, July 2019, doi: 10.1109/TMM.2018.2889563.Google ScholarCross Ref
- Cheng, K.L., Yang, Z., Chen, Q., Tai, YW. (2020). Fully Convolutional Networks for Continuous Sign Language Recognition. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science, vol 12369. Springer, Cham. https://doi.org/10.1007/978-3-030-58586-0_41.Google ScholarDigital Library
- A. Mittal, P. Kumar, P. P. Roy, R. Balasubramanian and B. B. Chaudhuri, "A Modified LSTM Model for Continuous Sign Language Recognition Using Leap Motion," in IEEE Sensors Journal, vol. 19, no. 16, pp. 7056-7063, 15 Aug.15, 2019, doi: 10.1109/JSEN.2019.2909837.Google ScholarCross Ref
- Camgoz, N.C., Hadfield, S., Koller, O., Bowden, R., “Subunets: end-to-end hand shape and continuous sign language recognition”, Proceedings of IEEE International Conference on Computer Vision, pp. 3075–3084 (2017).Google Scholar
- Guo, D., Zhou, W., Li, H., Wang, M., “Hierarchical LSTM for sign language translation”, Proceedings of AAAI Conference on Artificial Intelligence, pp. 6845–6852 (2018).Google Scholar
- Huang, J., Zhou, W., Zhang, Q., Li, H., Li, W.: Video-based sign language recognition without temporal segmentation. In: Proceedings of AAAI Conference on Artificial Intelligence, pp. 2257–2264 (2018).Google Scholar
- Yang, Z., Shi, Z., Shen, X., Tai, Y.W., “SF-net: structured feature network for continuous sign language recognition”, arXiv preprint arXiv:1908.01341 (2019).Google Scholar
- Pu, J., Zhou, W., Li, H., “Iterative alignment network for continuous sign language recognition”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 4165–4174 (2019).Google Scholar
- T. Prathiba, R. Shantha Selva Kumari, “Content based video retrieval system based on multimodal featuregrouping by KFCM clustering algorithm to promote human–computerinteraction”, Journal of Ambient Intelligence and Humanized Computing, Volume 12, PP. 6215–6229, 2021.Google ScholarCross Ref
- G. Priyanka, M. Prasha Meena, “Survey and Evaluation on Video Summarization Techniques”, Journal of Critical Reviews, Vol 7, Issue 8, 2020.Google Scholar
- S.Arivazhagan, R.NewlinShebiah, V.Sridevi, “Development of video analytic algorithm for anomalydetection in individual and crowd behavior”, International Journal of Applied Engineering Research, ISSN 0973-4562 Vol. 10 No.1, 2015.Google Scholar
- Newlin Shebiah Russel, Arivazhagan Selvaraj,“Fusion of spatial and dynamic CNN streams for action recognition”, Multimedia Systems, Springer, 2021.https://doi.org/10.1145/1177352.1177355Google ScholarDigital Library
- Kasabov, N., NeuCube EvoSpike Architecture for Spatio-Temporal Modelling and Pattern Recognition of Brain Signals, in: Mana, Schwenker and Trentin (Eds) ANNPR, Springer LNAI 7477, 2012, 225-243, https://doi.org/10.1007/978-3-642-33212-8_21.Google ScholarDigital Library
- Tu, E., Kasabov, N., & Yang, J. (2017). Mapping temporal variables into the NeuCube for improved pattern recognition, predictive modelling, and understanding of stream data. IEEE transactions on neural networks and learning systems, 28(6), 1305-1317.Google Scholar
- Kasabov, N., Time-Space, Spiking Neural Networks and Brain-Inspired Artificial Intelligence, Springer Nature (2019) 750p., https://www.springer.com/gp/book/9783662577134Google Scholar
Index Terms
- Spatio-Temporal dependency preserving Cognitive-assisted Continuous Chinese Sign Language Recognition
Index terms have been assigned to the content through auto-classification.
Recommendations
A teaching system of Japanese sign language using sign language recognition and generation
MULTIMEDIA '02: Proceedings of the tenth ACM international conference on MultimediaIn recent years, the number of sign language learners is increasing in Japan. And there are many teaching materials of sign language such as textbooks, videotapes and software for PCs. However, these teaching materials have several problems that ...
Chinese Sign Language Recognition with Batch Sampling ResNet-Bi-LSTM
AbstractSign language has served as a communication medium between the Deaf community and society. Nonetheless, the practice of sign language is not common in Chinese society, along with a lack of professional sign language interpreters. Most existing ...
Mobile sign language translation system for deaf community
W4A '12: Proceedings of the International Cross-Disciplinary Conference on Web AccessibilityNowadays, web technologies are a very efficient way to ensure communication between a large and heterogeneous audience. Furthermore, web information is mainly based on textual and multimedia content and consequently, some people with special needs, such ...
Comments