Abstract
Due to the complex and diverse structure of the portrait boundary, it is a great challenge to segment the portrait from the natural background efficiently and accurately. We propose a new lightweight real-time semantic segmentation network-DDCNet, for portrait segmentation in complex background. Firstly, we propose a deformable depthwise separable convolution block, which combines deformable convolution with depthwise separable convolution, so that the network can fully obtain global and local information while reducing time consumption, and effectively reduce the complexity of the model. Secondly, we propose a detail selection block, which improves the segmentation accuracy of the network by selecting the information supplemented by skip connection. Finally, we propose a novel loss to improve the robustness of portrait segmentation in natural background. Our model has few parameters (0.122M) and FLOPs (0.092G). Experimental results show that our method could efficiently obtain the accurate segmentation image in real-time and achieve state-of-the-art comprehensive performance on the public datasets EG1800 and Conference Video Segmentation Dataset. User study shows that our method is favored by the most testers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Miao, J., Sun, K., Liao, X., Leng, L., Chu, J.: Human segmentation based on compressed deep convolutional neural network. IEEE Access 8, 167,585–167,595 (2020). http://doi.org/10.1109/ACCESS.2020.3023746
Chuang, Y.Y., Curless, B., Salesin, D.H., Szeliski, R.: A Bayesian approach to digital matting. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, vol. 2, pp. II-II (2001). http://doi.org/10.1109/CVPR.2001.990970
Pare, S., Kumar, A., Bajaj, V., Singh, G.K.: A multilevel color image segmentation technique based on cuckoo search algorithm and energy curve. Appl. Soft Comput. 47, 76–102 (2016). https://doi.org/10.1016/j.asoc.2016.05.040
Xu, N., Price, B., Cohen, S., Huang, T.: Deep image matting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2970–2979 (2017). http://doi.org/10.1109/CVPR.2017.41
Poudel, R.P., Liwicki, S., Cipolla, R.: Fast-SCNN: fast semantic segmentation network. arXiv preprint arXiv:1902.04502 (2019)
Park, H., Sjosund, L., Yoo, Y., Monet, N., Bang, J., Kwak, N.: SiNet: extreme lightweight portrait segmentation networks with spatial squeeze module and information blocking decoder. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2066–2074 (2020). http://doi.org/10.1109/WACV45572.2020.9093588
Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: BiSeNet: bilateral segmentation network for real-time semantic segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11217, pp. 334–349. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01261-8_20
Yu, C., Gao, C., Wang, J., Yu, G., Shen, C., Sang, N.: Bisenet v2: bilateral network with guided aggregation for real-time semantic segmentation. Int. J. Comput. Vis. 129(11), 3051–3068 (2021). http://doi.org/10.1007/s11263-021-01515-2
Dai, J., et al.: Deformable convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 764–773 (2017). http://doi.org/10.1109/ICCV.2017.89
Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017). http://doi.org/10.1109/CVPR.2017.195
Paszke, A., Chaurasia, A., Kim, S., Culurciello, E.: Enet: a deep neural network architecture for real-time semantic segmentation. arXiv preprint arXiv:1606.02147 (2016)
Fan, M., et al.: Rethinking BiseNet for real-time semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9716–9725 (2021). http://doi.org/10.1109/CVPR46437.2021.00959
Wang, Y., et al.: LedNet: a lightweight encoder-decoder network for real-time semantic segmentation. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 1860–1864 (2019). http://doi.org/10.1109/ICIP.2019.8803154
Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv 2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018). http://doi.org/10.1109/CVPR.2018.00474
Howard, A., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324 (2019). http://doi.org/10.1109/ICCV.2019.00140
Shen, X., et al.: Automatic portrait segmentation for image stylization. Comput. Graphics Forum 35, 93–102 (2016). http://doi.org/10.1111/cgf.12814
Kuang, Z., Tie, X.: Flow-based video segmentation for human head and shoulders. arXiv preprint arXiv:2104.09752 (2021)
Zhang, S.H., Dong, X., Li, H., Li, R., Yang, Y.L.: PortraitNet: real-time portrait segmentation network for mobile device. Comput. Graph. 80, 104–113 (2019). http://doi.org/10.1016/j.cag.2019.03.007
Park, H., Sjösund, L.L., Yoo, Y., Bang, J., Kwak, N.: Extremec3Net: extreme lightweight portrait segmentation networks using advanced c3-modules. arXiv preprint (2019). https://doi.org/10.48550/arXiv.1908.03093
Mehta, S., Rastegari, M., Caspi, A., Shapiro, L., Hajishirzi, H.: ESPNet: efficient spatial pyramid of dilated convolutions for semantic segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 561–580. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01249-6_34
Acknowledgements
This work was supported by the Shanghai Natural Science Foundation of China under Grant No.19ZR1419100 and the Shanghai talent development funding of China under Grant No. 2021016.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Huang, D., Wu, D., Liu, J., Lv, Y. (2022). DDCNet: A Lightweight Network with Variable Receptive Field for Real-Time Portrait Segmentation in Complex Environment. In: Magnenat-Thalmann, N., et al. Advances in Computer Graphics. CGI 2022. Lecture Notes in Computer Science, vol 13443. Springer, Cham. https://doi.org/10.1007/978-3-031-23473-6_36
Download citation
DOI: https://doi.org/10.1007/978-3-031-23473-6_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23472-9
Online ISBN: 978-3-031-23473-6
eBook Packages: Computer ScienceComputer Science (R0)