Ala Addin I. Sidig
Skip Abstract Section
Abstract
Sign language is the major means of communication for the deaf community. It uses body language and gestures such as hand shapes, lib patterns, and facial expressions to convey a message. Sign language is geography-specific, as it differs from one country to another. Arabic Sign language is used in all Arab countries. The availability of a comprehensive benchmarking database for ArSL is one of the challenges of the automatic recognition of Arabic Sign language. This article introduces KArSL database for ArSL, consisting of 502 signs that cover 11 chapters of ArSL dictionary. Signs in KArSL database are performed by three professional signers, and each sign is repeated 50 times by each signer. The database is recorded using state-of-art multi-modal Microsoft Kinect V2. We also propose three approaches for sign language recognition using this database. The proposed systems are Hidden Markov Models, deep learning images’ classification model applied on an image composed of shots of the video of the sign, and attention-based deep learning captioning system. Recognition accuracies of these systems indicate their suitability for such a large number of Arabic signs. The techniques are also tested on a publicly available database. KArSL database will be made freely available for interested researchers.
- Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016).Google Scholar
- Magdy Abull-ela, Mohamed F. Tolba, and Ahmed Samir Elons. 2013. Pulse-coupled neural network feature generation model for Arabic sign language recognition. IET Image Proc. 7, 9 (Dec. 2013), 829--836.Google Scholar
- A. Abdelbaky Ahmed and Saleh Aly. 2014. Appearance-based Arabic sign language recognition using hidden Markov models. In Proceedings of the International Conference on Engineering and Technology (ICET’14). IEEE, 1--6.Google Scholar
- Kinda Al-Fityani and Carol Padden. 2010. Sign language geography in the Arab world. Sign Lang.: Cambridge Surv. (2010), 433--450.Google Scholar
- Omar Al-Jarrah and Alaa Halawani. 2001. Recognition of gestures in Arabic sign language using neuro-fuzzy systems. Artif. Intell. 133, 1--2 (2001), 117--138.Google ScholarDigital Library
- M. AL-Rousan, K. Assaleh, and A. Tala’a. 2009. Video-based signer-independent Arabic sign language recognition using hidden Markov models. Appl. Soft Comput. 9, 3 (June 2009), 990--999.Google Scholar
- Nadia R. Albelwi and Yasser M. Alginahi. 2012. Real-time Arabic sign language (ARSL) recognition. In Proceedings of the International Conference on Communications and Information Technology. 497--501.Google Scholar
- Marco Alfonse, A. Ali, A. Samir Elons, Nagwa L. Badr, and Magdy Aboul-Ela. 2015. Arabic sign language benchmark database for different heterogeneous sensors. In Proceedings of the 5th International Conference on Information & Communication Technology and Accessibility (ICTA’15). IEEE, 1--9.Google ScholarCross Ref
- Miada A. Almasre and Hana Al-Nuaim. 2016. Recognizing Arabic sign language gestures using depth sensors and a KSVM classifier. In Proceedings of the 8th Computer Science and Electronic Engineering Conference (CEEC’16). IEEE, 146--151.Google Scholar
- Abdulaziz Almohimeed, Mike Wald, and R. I. Damper. 2011. Arabic text to Arabic sign language translation system for the deaf and hearing-impaired community. In Proceedings of the 2nd Workshop on Speech and Language Processing for Assistive Technologies. Association for Computational Linguistics, 101--109.Google Scholar
- Saleh Aly and Safaa Mohammed. 2014. Arabic sign language recognition using spatio-temporal local binary patterns and support vector machine. In Advanced Machine Learning Technologies and Applications, Aboul Ella Hassanien, Mohamed F. Tolba, and Ahmad Taher Azar (Eds.). Communications in Computer and Information Science, Vol. 488. Springer International Publishing, 36--45.Google Scholar
- Saleh Aly, Basma Osman, Walaa Aly, and Mahmoud Saber. 2016. Arabic sign language fingerspelling recognition from depth and intensity images. In Proceedings of the 12th International Computer Engineering Conference (ICENCO’16). IEEE, 99--104.Google ScholarCross Ref
- Omar Amin, Hazem Said, Ahmed Samy, and Hoda K. Mohammed. 2015. HMM based automatic Arabic sign language translator using Kinect. In Proceedings of the 10th International Conference on Computer Engineering & Systems (ICCES’15). IEEE, 389--392.Google Scholar
- Arab League Educational Cultural and Scientific Organization. 2006. Second Part of the Unified Arabic Sign Language Dictionary. The League of Arab States & the Supreme Council for Family Affairs, Qatar.Google Scholar
- Khaled Assaleh and M. Al-Rousan. 2005. Recognition of Arabic sign language alphabet using polynomial classifiers. EURASIP J. Adv. Sig. Proc. 2005, 13 (2005), 507614.Google ScholarDigital Library
- Khaled Assaleh, Tamer Shanableh, Mustafa Fanaswala, Harish Bajaj, and Farnaz Amin. 2008. Vision-based system for continuous Arabic Sign Language recognition in user dependent mode. In Proceedings of the 5th International Symposium on Mechatronics and Its Applications. IEEE, 1--5.Google ScholarCross Ref
- Tamás Aujeszky and Mohamad Eid. 2016. A gesture recognition architecture for Arabic sign language communication system. Multimedia Tools Applic. 75, 14 (2016), 8493--8511.Google ScholarDigital Library
- A. L. C. Barczak, N. H. Reyes, M. Abastillas, A. Piccio, and T. Susnjak. 2011. A new 2D static hand gesture colour image dataset for ASL gestures. Res. Lett. Inf. Math. Sci. 15 (2011), 12--20.Google Scholar
- K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. 2014. Return of the devil in the details: Delving deep into convolutional nets. In Proceedings of the British Machine Vision Conference.Google Scholar
- Onno A. Crasborn, Johanna Mesch, Dafydd Waters, Annika Nonhebel, Els Van der Kooij, Bencie Woll, and Brita Bergman. 2007. Sharing sign language data online: Experiences from the ECHO project. Int. J. Corp. Ling. 12, 4 (2007), 535--562.Google ScholarCross Ref
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248--255.Google ScholarCross Ref
- Philippe Dreuw, David Rybach, Thomas Deselaers, Morteza Zahedi, and Hermann Ney. 2007. Speech recognition techniques for a sign language recognition system. Hand 60 (2007), 80.Google Scholar
- Menna El Badawy, A. Samir Elons, Hwaida Sheded, and Mohamed F. Tolba. 2015. A proposed hybrid sensor architecture for Arabic sign language recognition. In Proceedings of the Conference on Intelligent Systems. Springer, 721--730.Google Scholar
- Sergio Escalera, Jordi Gonzàlez, Xavier Baró, Miguel Reyes, Oscar Lopes, Isabelle Guyon, Vassilis Athitsos, and Hugo Escalante. 2013. Multi-modal gesture recognition challenge 2013: Dataset and results. In Proceedings of the 15th ACM International Conference on Multimodal Interaction. ACM, 445--452.Google ScholarDigital Library
- Michael Filhol, Mohamed N. Hadjadj, and Benoît Testu. 2016. A rule triggering system for automatic text-to-sign translation. Univ. Access Inf. Soc. 15, 4 (2016), 487--498.Google ScholarDigital Library
- Nikolaos Gkigkelos and Christos Goumopoulos. 2017. Greek sign language vocabulary recognition using Kinect. In Proceedings of the 21st Pan-Hellenic Conference on Informatics. 1--6.Google ScholarDigital Library
- Fatma Guesmi, Tahani Bouchrika, Olfa Jemai, Mourad Zaied, and Chokri Ben Amar. 2016. Arabic sign language recognition system based on wavelet networks. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 003561--003566.Google ScholarCross Ref
- Ayman Hamed, Nahla A. Belal, and Khaled M. Mahar. 2016. Arabic sign language alphabet recognition based on HOG-PCA using Microsoft Kinect in complex backgrounds. In Proceedings of the IEEE 6th International Conference on Advanced Computing (IACC’16). IEEE, 451--458.Google Scholar
- Mohamed Hassan, Khaled Assaleh, and Tamer Shanableh. 2016. User-dependent sign language recognition using motion detection. In Proceedings of the International Conference on Computational Science and Computational Intelligence (CSCI’16). IEEE, 852--856.Google ScholarCross Ref
- Nada B. Ibrahim, Mazen M. Selim, and Hala H. Zayed. 2017. An automatic Arabic sign language recognition system (ArSLRS). J. King Saud Univ.-comput. Inf. Sci. 30, 4 (2017).Google Scholar
- Mohammed Waleed Kadous. 2002. Temporal Classification: Extending the Classification Paradigm to Multivariate Time Series. Ph.D. Dissertation. University of New South Wales.Google Scholar
- Pradeep Kumar, Himaanshu Gauba, Partha Pratim Roy, and Debi Prosad Dogra. 2017. Coupled HMM-based multi-sensor data fusion for sign language recognition. Pattern Recog. Lett. 86 (2017), 1--8.Google ScholarDigital Library
- Hamzah Luqman and Sabri A. Mahmoud. 2018. Automatic translation of Arabic text-to-Arabic sign language. Univ. Access Inf. Soc. (2018), 1--13.Google Scholar
- Hamzah Luqman and Sabri A. Mahmoud. 2019. A machine translation system from Arabic sign language to Arabic. Univ. Access Inf. Soc. 19 (2019), 1--14.Google Scholar
- Mohamed Mohandes. 2001. Arabic sign language recognition. In Proceedings of the International Conference on Imaging Science, Systems, and Technology. 753--759.Google Scholar
- M. Mohandes, S. A.-Buraiky, T. Halawani, and S. Al-Baiyat. 2004. Automation of the Arabic sign language recognition. In Proceedings of the International Conference on Information and Communication Technologies: From Theory to Applications. IEEE, 479--480.Google Scholar
- Mohamed Mohandes, S . Aliyu, and M. Deriche. 2014a. Arabic sign language recognition using the leap motion controller. In Proceedings of the IEEE 23rd International Symposium on Industrial Electronics (ISIE’14). IEEE, 960--965.Google Scholar
- Mohamed Mohandes and Mohamed Deriche. 2005. Image based Arabic sign language recognition. In Proceedings of the 8th International Symposium on Signal Processing and Its Applications, Vol. 1. IEEE, 86--89.Google ScholarCross Ref
- Mohamed Mohandes and Mohamed Deriche. 2013. Arabic sign language recognition by decisions fusion using Dempster-Shafer theory of evidence. In Proceedings of the Computing, Communications and IT Applications Conference (ComComAp’13). 90--94.Google ScholarCross Ref
- Mohamed Mohandes, Mohamed Deriche, U. Johar, and S. Ilyas. 2012. A signer-independent Arabic sign language recognition system using face detection, geometric features, and a hidden Markov model. Comput. Electric. Eng. 38, 2 (2012), 422--433.Google ScholarDigital Library
- Mohamed Mohandes, Mohamed Deriche, and Junzhao Liu. 2014b. Image-based and sensor-based approaches to Arabic sign language recognition. IEEE Trans. Hum.-mach. Syst. 44, 4 (2014), 551--557.Google ScholarCross Ref
- M. Mohandes, S. I. Quadri, and M. Deriche. 2007. Arabic sign language recognition an image-based approach. In Proceedings of the 21st International Conference on Advanced Information Networking and Applications Workshops (AINAW’07). 272--276.Google Scholar
- Sara Morrissey. 2008. Data-driven Machine Translation for Sign Languages. Ph.D. Dissertation. Dublin City University.Google Scholar
- Davi Hirafuji Neiva and Cleber Zanchettin. 2018. Gesture recognition: A review focusing on sign language in a mobile context. Exp. Syst. Applic. 103, Aug. (2018) 159--183.Google ScholarCross Ref
- Natalia Neverova, Christian Wolf, Graham W. Taylor, and Florian Nebout. 2014. Multi-scale deep learning for gesture detection and localization. In Proceedings of the Workshop at the European Conference on Computer Vision. Springer, 474--490.Google Scholar
- Arab League Educational, Cultural and Scientific Organization. 2000. LAS: First Part of the Unified Arabic Sign Dictionary.Google Scholar
- Nicolas Pugeault and Richard Bowden. 2011. Spelling it out: Real-time ASL fingerspelling recognition. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV Workshops’11). IEEE, 1114--1119.Google ScholarCross Ref
- T. Shanableh and K. Assaleh. 2007. Arabic sign language recognition in user-independent mode. In Proceedings of the International Conference on Intelligent and Advanced Systems. IEEE, 597--600.Google Scholar
- Tamer Shanableh, Khaled Assaleh, and M. Al-Rousan. 2007. Spatio-temporal feature-extraction techniques for isolated gesture recognition in Arabic sign language. IEEE Trans. Syst., Man, Cyber. Part B, Cyber. 37, 3 (June 2007), 641--50.Google ScholarDigital Library
- Samaa M. Shohieb, Hamdy K. Elminir, and A. M. Riad. 2015. SignsWorld atlas: A benchmark Arabic sign language database. J. King Saud Univ.-Comput. Inf. Sci. 27, 1 (2015), 68--76.Google ScholarDigital Library
- Ala addin I. Sidig, Hamzah Luqman, and Sabri A. Mahmoud. 2018. Arabic Sign Language Recognition Using Optical Flow-based Features and HMM. Springer International Publishing, Cham, 297--305.Google Scholar
- Ala addin I. Sidig and Sabri A. Mahmoud. 2018. Trajectory based Arabic sign language recognition. Int. J. Adv. Comput. Sci. Applic. 9, 4 (2018), 283--291.Google Scholar
- Nantinee Soodtoetong and Eakbodin Gedkhaw. 2018. The efficiency of sign language recognition using 3D convolutional neural networks. In Proceedings of the 15th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON’18). IEEE, 70--73.Google ScholarCross Ref
- Chao Sun, Tianzhu Zhang, Bing-Kun Bao, Changsheng Xu, and Tao Mei. 2013. Discriminative exemplar coding for sign language recognition with Kinect. IEEE Trans. Cyber. 43, 5 (Oct. 2013), 1418--1428.Google ScholarCross Ref
- M. F. Tolba, Ahmed Samir, and Magdy Abul-Ela. 2012b. A proposed graph matching technique for Arabic sign language continuous sentences recognition. In Proceedings of the 8th International Conference on Informatics and Systems (INFOS’12). IEEE, MM--14.Google Scholar
- M. F. Tolba, Ahmed Samir, and Magdy Aboul-Ela. 2012a. Arabic sign language continuous sentences recognition using PCNN and graph matching. Neur. Comput. Applic. 23, 3--4 (Aug. 2012), 999--1010.Google Scholar
- Noor Tubaiz, Tamer Shanableh, and Khaled Assaleh. 2015. Glove-based continuous Arabic sign language recognition in user-dependent mode. IEEE Trans. Hum.-mach. Syst. 45, 4 (2015), 526--533.Google ScholarCross Ref
- Ulrich Von Agris and Karl-Friedrich Kraiss. 2007. Towards a video corpus for signer-independent continuous sign language recognition. In Proceedings of Gesture in Human-Computer Interaction and Simulation, International Gesture Workshop, May 23-25, 2007.Google Scholar
- Khairunizam Wan, Nazrul Hamizi Bin Adnan, A. B. Shahriman, Siti Khadijah Za’ba, Mohd Azri Abd Aziz, and Zulkifli Md Yusof. 2012. Gesture recognition b on hand postures and trajectories by using dataglove: A fuzzy probability approach--A review. In Proceedings of International Conference on Man-Machine Systems (ICoMMS'12).Google Scholar
- Frank Weichert, Daniel Bachmann, Bartholomäus Rudak, and Denis Fisseler. 2013. Analysis of the accuracy and robustness of the leap motion controller. Sensors (Switz.) 13, 5 (2013), 6380--6393.Google ScholarCross Ref
- Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning. 2048--2057.Google ScholarDigital Library
- Aliaa A. A. Youssif, Amal Elsayed Aboutabl, and Heba Hamdy Ali. 2011. Arabic sign language (ARSL) recognition system using HMM. Int. J. Adv. Comput. Sci. Applic. 2, 11 (2011).Google Scholar
- Morteza Zahedi, Philippe Dreuw, David Rybach, Thomas Deselaers, and Hermann Ney. 2006. Continuous sign language recognition-approaches from speech recognition and available data resources. In Proceedings of the 2nd Workshop on the Representation and Processing of Sign Languages: Lexicographic Matters and Didactic Scenarios. 21--24.Google Scholar
- Morteza Zahedi, Daniel Keysers, and Hermann Ney. 2005. Pronunciation clustering and modeling of variability for appearance-based sign language recognition. In Proceedings of the International Gesture Workshop. Springer, 68--79.Google Scholar
- Mahmoud M. Zaki and Samir I. Shaheen. 2011. Sign language recognition using a combination of new vision based features. Pattern Recog. Lett. 32, 4 (2011), 572--577.Google ScholarDigital Library
Index Terms
- KArSL: Arabic Sign Language Database
Recommendations
Isolated Arabic Sign Language Recognition Using a Transformer-based Model and Landmark Keypoints
Pose-based approaches for sign language recognition provide light-weight and fast models that can be adopted in real-time applications. This article presents a framework for isolated Arabic sign language recognition using hand and face keypoints. We ...
A machine translation system from Arabic sign language to Arabic
AbstractArabic sign language (ArSL) is one of the sign languages that is used in Arab countries. This language has structure and grammar that differ from spoken Arabic. Available ArSL recognition systems perform direct mapping between the recognized sign ...
Vision-Based Multilingual Sign Language Translation
AbstractLike spoken languages, different countries use different sign languages which include a variety of gestures and movements. This demands the need for a system that can translate one sign language to another. Thus, a computer vision-based sign-to-...
Comments