Abstract
In sign language recognition (SLR) with multimodal data, a sign word can be represented by multiply features, for which there exist an intrinsic property and a mutually complementary relationship among them. To fully explore those relationships, we propose an online early-late fusion method based on the adaptive Hidden Markov Model (HMM). In terms of the intrinsic property, we discover that inherent latent change states of each sign are related not only to the number of key gestures and body poses but also to their translation relationships. We propose an adaptive HMM method to obtain the hidden state number of each sign by affinity propagation clustering. For the complementary relationship, we propose an online early-late fusion scheme. The early fusion (feature fusion) is dedicated to preserving useful information to achieve a better complementary score, while the late fusion (score fusion) uncovers the significance of those features and aggregates them in a weighting manner. Different from classical fusion methods, the fusion is query adaptive. For different queries, after feature selection (including the combined feature), the fusion weight is inversely proportional to the area under the curve of the normalized query score list for each selected feature. The whole fusion process is effective and efficient. Experiments verify the effectiveness on the signer-independent SLR with large vocabulary. Compared either on different dataset sizes or to different SLR models, our method demonstrates consistent and promising performance.
- Jonathan Alon, Vassilis Athitsos, Quan Yuan, and Stan Sclaroff. 2009. A unified framework for gesture recognition and spatiotemporal gesture segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 9 (2009), 1685--1699. Google ScholarDigital Library
- Serge Belongie, Chad Carson, Hayit Greenspan, and Jitendra Malik. 1998. Color- and texture-based image segmentation using em and its application to content-based image retrieval. In IEEE Conference on Computer Vision. 675--682. Google ScholarDigital Library
- Xingyang Cai, Wengang Zhou, Lei Wu, Jiebo Luo, and Houqiang Li. 2016. Effective active skeleton representation for low latency human action recognition. IEEE Transactions on Multimedia 18, 2 (2016), 141--154.Google ScholarDigital Library
- Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, and Richard Bowden. 2016. Using convolutional 3d neural networks for user-independent continuous gesture recognition. In IEEE Conference on Pattern Recognition. 49--54.Google ScholarCross Ref
- Sait Celebi, Ali Selman Aydin, Talha Tarik Temiz, and Tarik Arici. 2013. Gesture recognition using skeleton data with weighted dynamic time warping. In International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. 620--625.Google Scholar
- Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Return of the devil in the details: Delving deep into convolutional nets. Arxiv Preprint Arxiv:1405.3531 (2014).Google Scholar
- Hong Cheng, Lu Yang, and Zicheng Liu. 2016. A survey on 3d hand gesture recognition. IEEE Transactions on Circuits and Systems for Video Technology 29, 9 (2016), 1659--1673. Google ScholarDigital Library
- Cao Dong, Ming Leu, and Zhaozheng Yin. 2015. American sign language alphabet recognition using microsoft kinect. In IEEE Conference on Computer Vision and Pattern Recognition Workshops. 44--52.Google Scholar
- Sergio Escalera, Xavier Baró, Jordi Gonzalez, Miguel A Bautista, Meysam Madadi, Miguel Reyes, Víctor Ponce-López, Hugo J. Escalante, Jamie Shotton, and Isabelle Guyon. 2014. Chalearn looking at people challenge 2014: Dataset and results. In European Conference on Computer Vision Workshop. 459--473.Google Scholar
- Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. 2016. Convolutional two-stream network fusion for video action recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 1933--1941.Google ScholarCross Ref
- Simon Fothergill, Helena Mentis, Pushmeet Kohli, and Sebastian Nowozin. 2012. Instructing people for training gestural interactive systems. In The SIGCHI Conference on Human Factors in Computing Systems. 1737--1746. Google ScholarDigital Library
- Brendan J. Frey and Delbert Dueck. 2007. Clustering by passing messages between data points. Science 315, 5814 (2007), 972--976.Google Scholar
- Dan Guo, Wengang Zhou, Meng Wang, and Houqiang Li. 2016. Sign language recognition based on adaptive hmms with data augmentation. In IEEE Conference on Image Processing. 2876--2880.Google ScholarCross Ref
- Jie Huang, Wengang Zhou, Houqiang Li, and Weiping Li. 2015. Sign language recognition using 3d convolutional neural networks. In IEEE Conference on Multimedia and Expo. 1--6.Google ScholarCross Ref
- Anil Jain, Karthik Nandakumar, and Arun Ross. 2005. Score normalization in multimodal biometric systems. Pattern Recognition 38, 12 (2005), 2270--2285. Google ScholarDigital Library
- Fahad Shahbaz Khan, Rao Muhammad Anwer, Joost Van de Weijer, Andrew D. Bagdanov, Maria Vanrell, and Antonio M. Lopez. 2012a. Color attributes for object detection. In IEEE Conference on Computer Vision and Pattern Recognition. 3306--3313. Google ScholarDigital Library
- Fahad Shahbaz Khan, Joost Van de Weijer, and Maria Vanrell. 2012b. Modulating shape features by color attention for object recognition. International Journal of Computer Vision 98, 1 (2012), 49--64. Google ScholarDigital Library
- Josef Kittler, Mohamad Hatef, Robert P. W. Duin, and Jiri Matas. 1998. On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 3 (1998), 226--239. Google ScholarDigital Library
- Tao Kong, Anbang Yao, Yurong Chen, and Fuchun Sun. 2016. HyperNet: Towards accurate region proposal generation and joint object detection. In IEEE Conference on Computer Vision and Pattern Recognition. 845--853.Google ScholarCross Ref
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In International Conference on Neural Information Processing Systems. 1097--1105. Google ScholarDigital Library
- Alexey Kurakin, Zhengyou Zhang, and Zicheng Liu. 2012. A real time system for dynamic hand gesture recognition with a depth sensor. In European Signal Processing Conference. 1975--1979.Google Scholar
- Shih Yao Lin, Yen Yu Lin, Chu Song Chen, and Yi Ping Hung. 2017. Recognizing human actions with outlier frames by observation filtering and completion. ACM Transactions on Multimedia Computing, Communications, and Applications 13, 3 (2017), 28. Google ScholarDigital Library
- Yushun Lin, Xiujuan Chai, Yu Zhou, and Xilin Chen. 2014. Curve matching from the view of manifold for sign language recognition. In Asian Conference on Computer Vision. 233--246.Google Scholar
- Tao Liu, Wengang Zhou, and Houqiang Li. 2016. Sign language recognition with long short-term memory. In IEEE Conference on Image Processing. 2871--2875.Google ScholarCross Ref
- Wei Liu, Yu Gang Jiang, Jiebo Luo, and Shih Fu Chang. 2011. Noise resistant graph ranking for improved web image search. In IEEE Conference on Computer Vision and Pattern Recognition. 849--856. Google ScholarDigital Library
- Zhen Liu, Houqiang Li, Wengang Zhou, Richang Hong, and Qi Tian. 2015. Uniting keypoints: Local visual information fusion for large-scale image search. IEEE Transactions on Multimedia 17, 4 (2015), 538--548.Google ScholarDigital Library
- Pavlo Molchanov, Xiaodong Yang, Shalini Gupta, Kihwan Kim, Stephen Tyree, and Jan Kautz. 2016. Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network. In IEEE Conference on Computer Vision and Pattern Recognition. 4207--4215.Google ScholarCross Ref
- Karthik Nandakumar, Yi Chen, Sarat C. Dass, and Anil K. Jain. 2008. Likelihood ratio-based biometric score fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (2008), 342--347. Google ScholarDigital Library
- Natalia Neverova, Christian Wolf, Giulio Paci, Giacomo Sommavilla, Graham Taylor, and Florian Nebout. 2013. A multi-scale approach to gesture detection and recognition. In IEEE Conference on Computer Vision Workshops. 484--491. Google ScholarDigital Library
- Natalia Neverova, Christian Wolf, Graham Taylor, and Florian Nebout. 2016. Moddrop: Adaptive multi-modal gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 8 (2016), 1692--1706.Google ScholarDigital Library
- Tomas Pfister, James Charles, and Andrew Zisserman. 2013. Large-scale learning of sign language by watching tv (using co-occurrences). In British Machine Vision Conference.Google Scholar
- Lionel Pigou, Aäron Van Den Oord, Sander Dieleman, Mieke Van Herreweghe, and Joni Dambre. 2016. Beyond temporal pooling: Recurrence and temporal convolutions for gesture recognition in video. International Journal of Computer Vision (2016), 1--10.Google Scholar
- Lingyan Ran, Yanning Zhang, Qilin Zhang, and Tao Yang. 2017. Convolutional neural network-based robot navigation using uncalibrated spherical images. Sensors 17, 6 (2017), 1341.Google ScholarCross Ref
- Zhou Ren, Junsong Yuan, and Zhengyou Zhang. 2011. Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera. In ACM Conference on Multimedia. 1093--1096. Google ScholarDigital Library
- Stan Salvador and Philip Chan. 2007. Toward accurate dynamic time warping in linear time and space. Intelligent Data Analysis 11, 5 (2007), 561--580. Google ScholarDigital Library
- Chao Sun, Tianzhu Zhang, BingKun Bao, Changsheng Xu, and Tao Mei. 2013. Discriminative exemplar coding for sign language recognition with kinect. IEEE Transactions on Cybernetics 43, 5 (2013), 1418--1428.Google ScholarCross Ref
- Chao Sun, Tianzhu Zhang, and Changsheng Xu. 2015. Latent support vector machine modeling for sign language recognition with Kinect. ACM Transactions on Intelligent Systems and Technology 6, 2 (2015), 20. Google ScholarDigital Library
- Shaoyan Sun, Wengang Zhou, Qi Tian, and Houqiang Li. 2016. Scalable object retrieval with compact image representation from generic object regions. ACM Transactions on Multimedia Computing, Communications, and Applications 12, 2 (2016), 29. Google ScholarDigital Library
- Oriol Ramos Terrades, Ernest Valveny, and Salvatore Tabbone. 2009. Optimal classifier fusion in a non-Bayesian probabilistic framework. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 9 (2009), 1630--1644. Google ScholarDigital Library
- Jun Wan, Yibing Zhao, Shuai Zhou, Isabelle Guyon, Sergio Escalera, and Stan Z. Li. 2016. Chalearn looking at people rgb-d isolated and continuous datasets for gesture recognition. In IEEE Conference on Computer Vision and Pattern Recognition Workshops. 56--64.Google Scholar
- Hanjie Wang, Xiujuan Chai, Yu Zhou, and Xilin Chen. 2015. Fast sign language recognition benefited from low rank approximation. In IEEE Conference and Workshops on Automatic Face and Gesture Recognition. 1--6.Google Scholar
- Jiang Wang, Zicheng Liu, Ying Wu, and Junsong Yuan. 2012b. Mining actionlet ensemble for action recognition with depth cameras. In IEEE Conference on Computer Vision and Pattern Recognition. 1290--1297. Google ScholarDigital Library
- Meng Wang, Xian Sheng Hua, Richang Hong, Jinhui Tang, Guo Jun Qi, and Yan Song. 2009. Unified video annotation via multigraph learning. IEEE Transactions on Circuits and Systems for Video Technology 19, 5 (2009), 733--746. Google ScholarDigital Library
- Meng Wang, Hao Li, Dacheng Tao, Ke Lu, and Xindong Wu. 2012a. Multimodal graph-based reranking for web image search. IEEE Transactions on Image Processing 21, 11 (2012), 4649--4661. Google ScholarDigital Library
- Meng Wang, Changzhi Luo, Bingbing Ni, Jun Yuan, Jianfeng Wang, and Shuicheng Yan. 2017. First-person daily activity recognition with manipulated object proposals and non-linear feature fusion. IEEE Transactions on Circuits and Systems for Video Technology 99 (2017), 1--1.Google Scholar
- Di Wu, Lionel Pigou, Pieter-Jan Kindermans, L. E. Nam, Ling Shao, Joni Dambre, and Jean-Marc Odobez. 2016b. Deep dynamic neural networks for multimodal gesture segmentation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 8 (2016), 1583--1597.Google ScholarDigital Library
- Zuxuan Wu, Yu Gang Jiang, Jun Wang, Jian Pu, and Xiangyang Xue. 2014. Exploring inter-feature and inter-class relationships with deep neural networks for video classification. In ACM Conference on Multimedia. 167--176. Google ScholarDigital Library
- Zuxuan Wu, Yu Gang Jiang, Xi Wang, Hao Ye, and Xiangyang Xue. 2016a. Multi-stream multi-class fusion of deep networks for video classification. In ACM Conference on Multimedia. 791--800. Google ScholarDigital Library
- Xiaoshan Yang, Tianzhu Zhang, and Changsheng Xu. 2016. Semantic feature mining for video event understanding. ACM Transactions on Multimedia Computing, Communications, and Applications 12, 4 (2016), 55. Google ScholarDigital Library
- Jun Ye, Hao Hu, Guo Jun Qi, and Kien A. Hua. 2017. A temporal order modeling approach to human action recognition from multimodal sensor data. ACM Transactions on Multimedia Computing, Communications, and Applications 13, 2 (2017), 14. Google ScholarDigital Library
- Jihai Zhang, Wengang Zhou, Chao Xie, Junfu Pu, and Houqiang Li. 2016. Chinese sign language recognition with adaptive HMM. In IEEE Conference on Multimedia and Expo. 1--6.Google ScholarCross Ref
- Qilin Zhang and Gang Hua. 2015. Multi-view visual recognition of imperfect testing data. In ACM Conference on Multimedia. 561--570. Google ScholarDigital Library
- Qilin Zhang, Gang Hua, Wei Liu, Zicheng Liu, and Zhengyou Zhang. 2014. Can visual recognition benefit from auxiliary information in training? In Asian Conference on Computer Vision. 65--80.Google Scholar
- Shaoting Zhang, Ming Yang, Timothee Cour, Kai Yu, and Dimitris N. Metaxas. 2012. Query specific fusion for image retrieval. In European Conference on Computer Vision. 660--673.Google Scholar
- Xin Zhao, Xue Li, Chaoyi Pang, Quan Z. Sheng, Sen Wang, and Mao Ye. 2014. Structured streaming skeleton -- a new feature for online human gesture recognition. ACM Transactions on Multimedia Computing, Communications, and Applications 11, 1s (2014), 22. Google ScholarDigital Library
- Liang Zheng, Shengjin Wang, Lu Tian, Fei He, Ziqiong Liu, and Qi Tian. 2015. Query-adaptive late fusion for image search and person re-identification. In IEEE Conference on Computer Vision and Pattern Recognition. 1741--1750.Google ScholarCross Ref
Index Terms
- Online Early-Late Fusion Based on Adaptive HMM for Sign Language Recognition
Recommendations
Early versus late fusion in semantic video analysis
MULTIMEDIA '05: Proceedings of the 13th annual ACM international conference on MultimediaSemantic analysis of multimodal video aims to index segments of interest at a conceptual level. In reaching this goal, it requires an analysis of several information streams. At some point in the analysis these streams need to be fused. In this paper, ...
Coupled HMM-based multi-sensor data fusion for sign language recognition
A novel multi-sensor framework is proposed for Sign Language Recognition (SLR).The framework recognize dynamic sign words performed by hearing impaired persons.A recognition combination framework using Coupled-HMM (CHMM) is proposed for SLR.Results ...
DCA‐based unimodal feature‐level fusion of orthogonal moments for Indian sign language dataset
Sign language recognition system classifies signs made by hand gestures. An adequate number of features are required to represent the shape variations of sign language. As compared to individual feature set, a combination of features can be effective due ...
Comments