Abstract
In the typical video action classification scenario, it is critical to extract the temporal-spatial information in the videos with complex 3D convolution neural networks, which significantly expand both the computation cost and memory costs. In this paper, we propose a novel 1-bit 3D convolution block named 3D BitConv, capable of compressing the 3D convolutional networks while maintaining a high level of accuracy. Due to its high flexibility, the proposed 3D BitConv block can be directly embedded in the latest 3D convolutional backbone. Consequently, we binarize two representative 3D convolutional neural networks (C3D and ResNet3D) and validate the accuracy of action recognition tasks. The results of two widely used datasets demonstrate that the two proposed binary 3D networks achieve impressive performance at a lower cost. Furthermore, we carry out an extensive ablation analysis to test and verify the efficacy of the components in 3D BitConv.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: CVPR, pp. 6299–6308 (2017)
Chen, T., Zhang, Z., Ouyang, X., Liu, Z., Shen, Z., Wang, Z.: BNN-BN=?: training binary neural networks without batch normalization. In: CVPR. pp. 4619–4629 (2021)
Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks: training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 (2016)
Darabi, S., Belbahri, M., Courbariaux, M., Nia, V.P.: BNN+: Improved binary network training (2018)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR, pp. 248–255. IEEE (2009)
Feichtenhofer, C., Fan, H., Malik, J., He, K.: SlowFast networks for video recognition. In: ICCV, pp. 6202–6211 (2019)
Fernando, B., Gavves, E., Oramas, J.M., Ghodrati, A., Tuytelaars, T.: Modeling video evolution for action recognition. In: CVPR, pp. 5378–5387 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Jokic, P., Emery, S., Benini, L.: Binaryeye: A 20 KFPS streaming camera system on FPGA with real-time on-device image recognition using binary neural networks. In: SIES, pp. 1–7 (2018)
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.: Large-scale video classification with convolutional neural networks. In: CVPR, pp. 1725–1732 (2014)
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: HMDB: a large video database for human motion recognition. In: ICCV, pp. 2556–2563 (2011)
Li, R., Wang, Y., Liang, F., Qin, H., Yan, J., Fan, R.: Fully quantized network for object detection. In: CVPR, pp. 2810–2819 (2019)
Li, Z., Ni, B., Zhang, W., Yang, X., Wen, G.: Performance guaranteed network acceleration via high-order residual quantization. In: ICCV, pp. 2603-2611 (2017)
Lin, X., Zhao, C., Pan, W.: Towards accurate binary convolutional neural network. NIPS (2017)
Liu, Z., Wu, B., Luo, W., Yang, X., Liu, W., Cheng, K.T.: Bi-real net: Enhancing the performance of 1-bit CNNs with improved representational capability and advanced training algorithm. In: ECCV, pp. 722–737 (2018)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. NIPS (2019)
Peng, H., Chen, S.: BDNN: Binary convolution neural networks for fast object detection. PRL, pp. 91–97 (2019)
Phan, H., He, Y., Savvides, M., Shen, Z., et al.: MoBiNet: a mobile binary network for image classification. In: WACV, pp. 3453–3462 (2020)
Qin, H., Cai, Z., Zhang, M., Ding, Y., Zhao, H., Yi, S., Liu, X., Su, H.: Bipointnet: Binary neural network for point clouds. arXiv preprint arXiv:2010.05501 (2020)
Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: XNOR-Net: ImageNet classification using binary convolutional neural networks. In: ECCV (2016)
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Adv. Neural Inf. Proc. Syst. (NeurIPS), pp. 568–576 (2014)
Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: ICCV, pp. 4489–4497 (2015)
Wei, Y., Pan, X., Qin, H., Ouyang, W., Yan, J.: Quantization mimic: towards very tiny CNN for object detection. In: ECCV, pp. 267–283 (2018)
Xu, S., Li, Y., Zhao, J., Zhang, B., Guo, G.: POEM: 1-bit point-wise operations based on expectation-maximization for efficient point cloud processing. arXiv preprint arXiv:2111.13386 (2021)
Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: CVPR, pp. 6848–6856 (2018)
Zhao, J., et al.: Data-adaptive binary neural networks for efficient object detection and recognition. PRL, pp. 239–245 (2022)
Acknowledgements
This work was supported in part by National Natural Science Foundation of China (61976220 and 61832017), Beijing Outstanding Young Scientist Program (BJJWZYJH012019100020098), and the Research Seed Funds of School of Interdisciplinary Studies, Renmin University of China.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Han, H., Lu, Z., Wen, JR. (2023). Binary Neural Network for Video Action Recognition. In: Dang-Nguyen, DT., et al. MultiMedia Modeling. MMM 2023. Lecture Notes in Computer Science, vol 13833. Springer, Cham. https://doi.org/10.1007/978-3-031-27077-2_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-27077-2_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-27076-5
Online ISBN: 978-3-031-27077-2
eBook Packages: Computer ScienceComputer Science (R0)