Abstract
As one of the key tasks of video processing, video object segmentation technology is the foundation of high-level computer vision application. The spatio-temporal context information in the video is of great significance for video object segmentation. Existing algorithms usually introduce spatio-temporal context information with pre-trained models such as optical flow for segmentation, which will result in sub-optimal solution and huge computational resource consumption. To address the above problem, this paper proposes an end-to-end lightweight video object segmentation model based on ConvGRU. A convolutional neural network is used to extract the visual features of each frame, and recursive neural network is used to extract the spatio-temporal context information of the whole video. The ConvGRU is used to achieve the deep fusion of visual features and spatial-temporal context information. The MobileNet-based lightweight algorithm can meet the demand for practical application and solve the problem of high consumption for computing resources. Experiments on DAVIS2016 dataset show that our method is competitive with similar state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wang, Z., Liu, S., Zhang, J., Chen, S., Guan, Q.: A spatio-temporal CRF for human interaction understanding. IEEE Trans. Circuits Syst. Video Technol. 27, 1647–1660 (2017)
Wang, Z., et al.: Understanding human activities in videos: a joint action and interaction learning approach. Neurocomputing 321, 216–226 (2018)
Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 724–732 (2016)
Caelles, S., Maninis, K.K., Pont-Tuset, J., Leal-Taixe, L., Cremers, D., Van Gool, L.: One-shot video object segmentation. In: 30th IEEE Conference on Computer Vision and Pattern Recognition, pp. 5320–5329 (2017)
Caelles, S., Chen, Y., Pont-Tuset, J., Van Gool, L.: Semantically-guided video object segmentation. arXiv preprint arXiv:1704.01926 (2017)
Yang, L., Wang, Y., Xiong, X., Yang, J., Katsaggelos, A.K.: Efficient video object segmentation via network modulation. In: 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6499–6507 (2018)
Perazzi, F., Khoreva, A., Benenson, R., Schiele, B., Sorkine-Hornung, A.: Learning video object segmentation from static images. In: 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3491–3500 (2016)
Li, X., et al.: Video object segmentation with re-identification. arXiv preprint arXiv:1708.00197 (2017)
Wang, W., Shen, J., Porikli, F., Yang, R.: Semi-supervised video object segmentation with super-trajectories. IEEE Trans. Pattern Anal. Mach. Intell. 41, 985–998 (2019)
Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilitieS. Proc. Natl. Acad. Sci. U.S.A. Biol. Sci. 79, 2554–2558 (1982)
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W., Woo, W.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Advances in Neural Information Processing Systems 28 (Nips 2015) 28 (2015)
Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Xu, N., et al.: YouTube-VOS: sequence-to-sequence video object segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11209, pp. 603–619. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01228-1_36
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.-C.: MobileNetV2: inverted residuals and linear bottlenecks. In: 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510–4520 (2018)
Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5Â MB model size. arXiv preprint arXiv:1602.07360 (2016)
Huang, G., Liu, S., van der Maaten, L., Weinberger, K.Q.: CondenseNet: an efficient DenseNet using learned group convolutions. In: 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2752–2761 (2018)
Ma, N., Zhang, X., Zheng, H.T., Sun, J.: ShuffleNet V2: practical guidelines for efficient CNN architecture design. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11218, pp. 122–138. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_8
Zhang, X., Zhou, X., Lin, M., Sun, J.: ShuffleNet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856 (2018)
Xie, G., Wang, J., Zhang, T., Lai, J., Hong, R., Qi, G.-J.: Interleaved structured sparse convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8847–8856 (2018)
Valipour, S., Siam, M., Jagersand, M., Ray, N.: Recurrent fully convolutional networks for video segmentation. In: 17th IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 29–36 (2016)
Tokmakov, P., Inria, K.A., Schmid, C.: Learning video object segmentation with visual memory. In: 16th IEEE International Conference on Computer Vision (ICCV), pp. 4491–4500. (2017)
Zhang, Y., Yao, R., Jiang, Q., Zhang, C., Wang, S.: Video object segmentation with weakly temporal information. KSII Trans. Internet Inf. Syst. 13, 1434–1449 (2019)
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 61772530, 61572505, U1610124, and 61806206, in part by the State’s Key Project of Research and Development Plan of China under Grant 2016YFC0600900, in part by the Six Talent Peaks Project in Jiangsu Province under Grant 2018-XYDXX-044 and 2015-DZXX-010, in part by the Natural Science Foundation of Jiangsu Province under Grant BK20171192, BK20180639.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Yao, R., Zhang, Y., Gao, C., Zhou, Y., Zhao, J., Liang, L. (2019). Lightweight Video Object Segmentation Based on ConvGRU. In: Lin, Z., et al. Pattern Recognition and Computer Vision. PRCV 2019. Lecture Notes in Computer Science(), vol 11858. Springer, Cham. https://doi.org/10.1007/978-3-030-31723-2_37
Download citation
DOI: https://doi.org/10.1007/978-3-030-31723-2_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-31722-5
Online ISBN: 978-3-030-31723-2
eBook Packages: Computer ScienceComputer Science (R0)