Abstract
Image matting aims to extract the accurate foreground opacity mask for a given image. State-of-the-art approaches are usually based on encoder-decoder neural networks and require a large dataset with ground-truth alpha matte to facilitate the training process. However, the alpha matte annotation process is extremely time-consuming and labor-intensive. To lift such a burden, we propose a novel deep learning-based weakly supervised image matting method. It can simultaneously utilize data with and without ground-truth alpha mattes to boost the matting performance. The key idea is to exploit the patch-wise similarity of the alpha mattes without explicitly relying on ground-truth alpha mattes. To this end, we design a novel patch clustering module to cluster patches with similar alpha mattes and subsequently propose a new loss function to supervise the matting network by utilizing the clustering prior. Experimental results show that our proposed method can effectively cluster image patches by their corresponding alpha patches’ similarity and improve the matting performance. To our knowledge, our method is the first to tackle the weakly supervised image matting problem with only trimaps as the annotation.
Y. Zhang and C. Wang—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aksoy, Y., Oh, T.H., Paris, S., Pollefeys, M., Matusik, W.: Semantic soft segmentation. ACM Trans. Graph. 37(4), 72 (2018)
Cai, S., et al.: Disentangled image matting. In: International Conference on Computer Vision, October 2019
Chen, Q., Li, D., Tang, C.K.: KNN matting. IEEE Trans. Pattern Anal. Mach. Intell. 35(9), 2175–2188 (2013)
Chuang, Y.Y., Curless, B., Salesin, D., Szeliski, R.: A bayesian approach to digital matting. In: CVPR, 2001. In: Proceedings of the 2001 IEEE Computer Society Conference on, CVPR 2001, vol. 2, pp. II-II. IEEE (2001)
Dai, Y., Lu, H., Shen, C.: Learning affinity-aware upsampling for deep image matting. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6841–6850. Computer Vision Foundation/IEEE (2021)
Duchon, J.: Splines minimizing rotation-invariant semi-norms in sobolev spaces. In: Schempp, W., Zeller, K. (eds.) Constructive Theory of Functions of Several Variables: Proceedings of a Conference Held at Oberwolfach, Germany, April 25–May 1, 1976. LNM, vol. 571, pp. 85–100. Springer, Cham (1976). https://doi.org/10.1007/BFb0086566
Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. IJCV 88(2), 303–338 (2010)
Gastal, E.S., Oliveira, M.M.: Shared sampling for real-time alpha matting. In: Computer Graphics Forum, pp. 575–584. Wiley Online Library (2010)
Grady, L., Schiwietz, T., Aharon, S., Westermann, R.: Random walks for interactive alpha-matting. In: Proceedings of VIIP, vol. 2005, pp. 423–429 (2005)
He, K., Rhemann, C., Rother, C., Tang, X., Sun, J.: A global sampling method for alpha matting. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2049–2056. IEEE Computer Society (2011)
Hou, Q., Liu, F.: Context-aware image matting for simultaneous foreground and alpha estimation. In: International Conference on Computer Vision, October 2019
Kulharia, V., Chandra, S., Agrawal, A., Torr, P., Tyagi, A.: Box2Seg: attention weighted loss and discriminative feature learning for weakly supervised segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12372, pp. 290–308. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58583-9_18
Levin, A., Lischinski, D., Weiss, Y.: A closed form solution to natural image matting. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 61–68. IEEE (2006)
Levin, A., Rav-Acha, A., Lischinski, D.: Spectral matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(10), 1699–1712 (2008)
Li, J., Zhang, J., Maybank, S.J., Tao, D.: Bridging composite and real: towards end-to-end deep image matting. Int. J. Comput. Vis. 1–21 (2021). https://doi.org/10.1007/s11263-021-01541-0
Li, J., Zhang, J., Tao, D.: Deep automatic natural image matting. In: Zhou, Z. (ed.) Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event/Montreal, Canada, 19–27 August 2021, pp. 800–806. ijcai.org (2021)
Li, Y., Lu, H.: Natural image matting via guided contextual attention. In: AAAI, vol. 34, pp. 11450–11457 (2020)
Lin, S., Ryabtsev, A., Sengupta, S., Curless, B.L., Seitz, S.M., Kemelmacher-Shlizerman, I.: Real-time high-resolution background matting. In: IEEE Conference on Computer Vision and Pattern recognition, pp. 8762–8771, June 2021
Lin, T.-Y., et al.: Microsoft coco: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Liu, J., Yao, Y., Hou, W., Cui, M., Xie, X., Zhang, C., Hua, X.: Boosting semantic human matting with coarse annotations. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 8560–8569. Computer Vision Foundation/IEEE (2020)
Liu, W., Zhang, C., Lin, G., Hung, T.Y., Miao, C.: Weakly supervised segmentation with maximum bipartite graph matching. In: ACMMM (2020)
Liu, Y., Xie, J., Shi, X., Qiao, Y., Huang, Y., Tang, Y., Yang, X.: Tripartite information mining and integration for image matting. In: ICCV, pp. 7555–7564 (2021)
Liu, Z., et al.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012–10022, October 2021
Lu, H., Dai, Y., Shen, C., Xu, S.: Indices matter: learning to index for deep image matting. In: International Conference on Computer Vision, October 2019
Lutz, S., Amplianitis, K., Smolic, A.: AlphaGAN: generative adversarial networks for natural image matting. In: British Machine Vision Conference, p. 259. BMVA Press (2018)
Qiao, Y., Liu, Y., Yang, X., Zhou, D., Xu, M., Zhang, Q., Wei, X.: Attention-guided hierarchical structure aggregation for image matting. In: IEEE Conference on Computer Vision and Pattern Recognition, June 2020
Ren, Z., et al.: Instance-aware, context-focused, and memory-efficient weakly supervised object detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 10598–10607 (2020)
Rhemann, C., Rother, C., Wang, J., Gelautz, M., Kohli, P., Rott, P.: A perceptually motivated online benchmark for image matting. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1826–1833. IEEE (2009)
Ruzon, M.A., Tomasi, C.: Alpha estimation in natural images. In: IEEE Conference on Computer Vision and Pattern Recognition, p. 1018. IEEE (2000)
Santambrogio, F.: Optimal Transport for Applied Mathematicians. PNDETA, vol. 87. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-20828-2
Sengupta, S., Jayaram, V., Curless, B., Seitz, S.M., Kemelmacher-Shlizerman, I.: Background matting: the world is your green screen. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2291–2300 (2020)
Simard, P.Y., Steinkraus, D., Platt, J.C.: Best practices for convolutional neural networks applied to visual document analysis. In: ICDAR 2003, pp. 958–962. IEEE Computer Society (2003)
Sun, J., Jia, J., Tang, C.K., Shum, H.Y.: Poisson matting. ACM Trans. Graph. 23(3), 315–321 (2004)
Sun, Y., Tang, C., Tai, Y.: Semantic image matting. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 11120–11129. Computer Vision Foundation/IEEE (2021)
Sun, Y., et al.: Circle loss: a unified perspective of pair similarity optimization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6397–6406. Computer Vision Foundation/IEEE (2020)
Wang, J., Cohen, M.F., et al.: Image and video matting: a survey. Found. Trends® Comput. Graph. Vis. 3(2), 97–175 (2008)
Xu, N., Price, B.L., Cohen, S., Huang, T.S.: Deep image matting. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, p. 4 (2017)
Yu, Q., et al.: Mask guided matting via progressive refinement network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1154–1163. Computer Vision Foundation/IEEE (2021)
Zhang, D., Han, J., Cheng, G., Yang, M.H.: Weakly supervised object localization and detection: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 44(9), 5866–5885 (2021)
Zhang, Y., et al.: A late fusion CNN for digital matting. In: IEEE Conference on Computer Vision and Pattern Recognition, June 2019
Zhou, F., Tian, Y., Qi, Z.: Attention transfer network for nature image matting. IEEE Trans. Circ. Syst. Video Technol. 31(6), 2192–2205 (2020)
Zhou, Z.H.: A brief introduction to weakly supervised learning. Natl. Sci. Rev. 5(1), 44–53 (2018)
Zou, Z., Li, W., Shi, T., Shi, Z., Ye, J.: Generative adversarial training for weakly supervised cloud matting. In: ICCV, pp. 201–210 (2019)
Acknowledgements
We thank the reviewers for their constructive comments. Weiwei Xu is partially supported by “Pioneer” and “Leading Goose” R &D Program of Zhejiang (No. 2023C01181). This paper is supported by Information Technology Center and State Key Lab of CAD &CG, Zhejiang University.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, Y., Wang, C., Zhang, Y., Bao, H., Xu, W. (2023). Weakly Supervised Image Matting via Patch Clustering. In: Lu, H., et al. Image and Graphics. ICIG 2023. Lecture Notes in Computer Science, vol 14355. Springer, Cham. https://doi.org/10.1007/978-3-031-46305-1_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-46305-1_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46304-4
Online ISBN: 978-3-031-46305-1
eBook Packages: Computer ScienceComputer Science (R0)