Biaozhang Huang
ABSTRACT
In the field of semi-supervised human action recognition, the effective utilization of both labeled and unlabeled data remains a central and challenging pursuit. To address this issue, we present an innovative framework (DSCF-AEC) that combines a Dual-stream Cross Fusion network (DSCF) with an Ambiguous Exclude Contrastive Learning (AEC) module. Specifically, our Dual-stream Cross Fusion network utilizes the ST-GCN as encoder, independently encoding two augmented versions of the joint and bone streams, which are subsequently cross-fused to achieve enhanced representation. To further bolster the performance, we designed the AEC module. This module constructs a memory bank capable of distinguishing reliable positive and negative samples, while ambiguous samples are excluded. This strategic approach ensures that, through contrastive learning, the model is trained solely on meaningful and trustworthy samples. Extensive experiments on NTU RGB+D and NW-UCLA datasets validate the effectiveness of our approach. The results indicate that, our proposed method significantly outperforms other existing methods.
- Mostafa A Abdelrazik, Abdelhaliem Zekry, and Wael A Mohamed. 2023. Efficient hybrid algorithm for human action recognition. Journal of Image and Graphics 11, 1 (2023), 72–81.Google Scholar
Cross Ref
- Tasweer Ahmad, Junaid Rafique, Hassam Muazzam, and Tahir Rizvi. 2015. Using discrete cosine transform based features for human action recognition. Journal of Image and Graphics 3, 2 (2015), 96–101.Google ScholarCross Ref
- David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. 2019. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785 (2019).Google Scholar
- Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems 33 (2020), 9912–9924.Google Scholar
- Sung Ho Choi, Kyeong-Beom Park, Dong Hyeon Roh, Jae Yeol Lee, Mustafa Mohammed, Yalda Ghasemi, and Heejin Jeong. 2022. An integrated mixed reality system for safety-aware human-robot collaboration using deep learning and digital twin generation. Robotics and Computer-Integrated Manufacturing 73 (2022), 102258.Google ScholarDigital Library
- Yilin Dong, Xinde Li, Jean Dezert, Rigui Zhou, Changming Zhu, Lai Wei, and Shuzhi Sam Ge. 2021. Evidential reasoning with hesitant fuzzy belief structures for human activity recognition. IEEE Transactions on Fuzzy Systems 29, 12 (2021), 3607–3619.Google ScholarDigital Library
- Yves Grandvalet and Yoshua Bengio. 2005. Semi-supervised learning by entropy minimization. CAP 367 (2005), 281–296.Google Scholar
- Liang-Yan Gui, Kevin Zhang, Yu-Xiong Wang, Xiaodan Liang, José MF Moura, and Manuela Veloso. 2018. Teaching robots to predict human motion. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 562–567.Google ScholarDigital Library
- Tianyu Guo, Hong Liu, Zhan Chen, Mengyuan Liu, Tao Wang, and Runwei Ding. 2022. Contrastive learning from extremely augmented skeleton sequences for self-supervised action recognition. 36, 1 (2022), 762–770.Google Scholar
- Muhammad Hassan, Tasweer Ahmad, Nudrat Liaqat, Ali Farooq, Syed Asghar Ali, and Syed Rizwan Hassan. 2014. A review on human actions recognition using vision based techniques. Journal of Image and Graphics 2, 1 (2014), 28–32.Google ScholarCross Ref
- Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9729–9738.Google ScholarCross Ref
- Lipeng Ke, Kuan-Chuan Peng, and Siwei Lyu. 2022. Towards to-at spatio-temporal focus for skeleton-based action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 1131–1139.Google ScholarCross Ref
- Boeun Kim, Hyung Jin Chang, Jungho Kim, and Jin Young Choi. 2022. Global-local motion transformer for unsupervised skeleton-based action learning. In European Conference on Computer Vision. Springer, 209–225.Google ScholarDigital Library
- Yongqiang Kong, Zhengang Wei, and Shanshan Huang. 2018. Automatic analysis of complex athlete techniques in broadcast taekwondo video. Multimedia Tools and Applications 77 (2018), 13643–13660.Google ScholarDigital Library
- Hema S Koppula and Ashutosh Saxena. 2015. Anticipating human activities using object affordances for reactive robotic response. IEEE transactions on pattern analysis and machine intelligence 38, 1 (2015), 14–29.Google Scholar
- Naresh Kumar and Nagarajan Sukavanam. 2018. Motion trajectory for human action recognition using fourier temporal features of skeleton joints. Journal of Image and Graphics 6, 2 (2018), 174–180.Google ScholarCross Ref
- Dong-Hyun Lee 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, Vol. 3. Atlanta, 896.Google Scholar
- Heqing Li, Xinde Li, Zhentong Zhang, Chuanfei Hu, Fir Dunkin, and Shuzhi Sam Ge. 2023. ESUAV-NI: Endogenous Security Framework for UAV Perception System Based on Neural Immunity. IEEE Transactions on Industrial Informatics (2023).Google Scholar
- Jingyuan Li and Eli Shlizerman. 2020. Iterate & cluster: Iterative semi-supervised action recognition. arXiv preprint arXiv:2006.06911 (2020).Google Scholar
- Jingyuan Li and Eli Shlizerman. 2020. Sparse semi-supervised action recognition with active learning. arXiv preprint arXiv:2012.01740 (2020).Google Scholar
- Duohan Liang, Guoliang Fan, Guangfeng Lin, Wanjun Chen, Xiaorong Pan, and Hong Zhu. 2019. Three-stream convolutional neural network with multi-task and ensemble learning for 3d action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 0–0.Google ScholarCross Ref
- Lilang Lin, Sijie Song, Wenhan Yang, and Jiaying Liu. 2020. Ms2l: Multi-task self-supervised learning for skeleton based action recognition. In Proceedings of the 28th ACM International Conference on Multimedia. 2490–2498.Google ScholarDigital Library
- Yunyao Mao, Wengang Zhou, Zhenbo Lu, Jiajun Deng, and Houqiang Li. 2022. Cmd: Self-supervised 3d action representation learning with cross-modal mutual distillation. In European Conference on Computer Vision. Springer, 734–752.Google ScholarDigital Library
- Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence 41, 8 (2018), 1979–1993.Google Scholar
- Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. 2016. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1010–1019.Google ScholarCross Ref
- Weijie Sheng and Xinde Li. 2021. Multi-task learning for gait-based identity recognition and emotion recognition using attention enhanced temporal graph convolutional network. Pattern Recognition 114 (2021), 107868.Google ScholarCross Ref
- Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. 2019. Skeleton-based action recognition with directed graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 7912–7921.Google ScholarCross Ref
- Xiangbo Shu, Binqian Xu, Liyan Zhang, and Jinhui Tang. 2023. Multi-Granularity Anchor-Contrastive Representation Learning for Semi-Supervised Skeleton-Based Action Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 6 (2023), 7559–7576. https://doi.org/10.1109/TPAMI.2022.3222871Google ScholarDigital Library
- Chenyang Si, Xuecheng Nie, Wei Wang, Liang Wang, Tieniu Tan, and Jiashi Feng. 2020. Adversarial self-supervised learning for semi-supervised 3d action recognition. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16. Springer, 35–51.Google Scholar
- Yi-Fan Song, Zhang Zhang, Caifeng Shan, and Liang Wang. 2022. Constructing stronger and faster baselines for skeleton-based action recognition. IEEE transactions on pattern analysis and machine intelligence 45, 2 (2022), 1474–1488.Google ScholarCross Ref
- Fida Mohammad Thoker, Hazel Doughty, and Cees GM Snoek. 2021. Skeleton-contrastive 3D action representation learning. In Proceedings of the 29th ACM international conference on multimedia. 1655–1663.Google ScholarDigital Library
- Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa. 2014. Human action recognition by representing 3d skeletons as points in a lie group. In Proceedings of the IEEE conference on computer vision and pattern recognition. 588–595.Google ScholarDigital Library
- Jiang Wang, Xiaohan Nie, Yin Xia, Ying Wu, and Song-Chun Zhu. 2014. Cross-view action modeling, learning and recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2649–2656.Google ScholarDigital Library
- Minsi Wang, Bingbing Ni, and Xiaokang Yang. 2020. Learning multi-view interactional skeleton graph for action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020).Google Scholar
- Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3733–3742.Google ScholarCross Ref
- Binqian Xu, Xiangbo Shu, and Yan Song. 2022. X-invariant contrastive augmentation and representation learning for semi-supervised skeleton-based action recognition. IEEE Transactions on Image Processing 31 (2022), 3852–3867.Google ScholarCross Ref
- Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.Google ScholarCross Ref
- Siyuan Yang, Jun Liu, Shijian Lu, Meng Hwa Er, and Alex C Kot. 2021. Skeleton cloud colorization for unsupervised 3d action representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 13423–13433.Google ScholarCross Ref
- Seyma Yucer and Yusuf Sinan Akgul. 2018. 3D Human Action Recognition with Siamese-LSTM Based Deep Metric Learning. Journal of Image and Graphics 6, 1 (2018).Google ScholarCross Ref
- Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. 2019. S4l: Self-supervised semi-supervised learning. In Proceedings of the IEEE/CVF international conference on computer vision. 1476–1485.Google ScholarCross Ref
Index Terms
- Integrating Dual-Stream Cross Fusion and Ambiguous Exclude Contrastive Learning for Enhanced Human Action Recognition
Recommendations
Deep semi-supervised learning with contrastive learning and partial label propagation for image data
AbstractDeep semi-supervised learning is becoming an active research topic because it jointly utilizes labeled and unlabeled samples in training deep neural networks. Recent advances are mainly focused on inductive semi-supervised learning ...
Learning correlations for human action recognition in videos
Human action recognition in realistic videos is an important and challenging task. Recent studies demonstrate that multi-feature fusion can significantly improve the classification performance for human action recognition. Therefore, a number of ...
Boosting semi-supervised learning with Contrastive Complementary Labeling
AbstractSemi-supervised learning (SSL) approaches have achieved great success in leveraging a large amount of unlabeled data to learn deep models. Among them, one popular approach is pseudo-labeling which generates pseudo labels only for those unlabeled ...
Comments