Abstract
We propose a new segmentation algorithm based on deep learning. To segment ice hockey players, a fully convolutional network (FCN) is adopted and fine-tuned with our augmented training data. The original FCN has difficulty segmenting small-size objects. To solve this problem, our method divides an input image into four overlapping sub-images and each image is fed into the deep learning network. After obtaining segmentation results from all sub-images, we combine them into a single result. The segmentation results should be consistent over time in video. Thus, our method tracks segments over time and removes false positives that appear for brief periods. Mathematically, we show that our overlapping subdivision process can be interpreted as overlapping domain decomposition methods, which enable the FCN to regularize over consecutive sub-images in training time. Experimental results demonstrate that our method accurately segments ice hockey players when they appear small and when there exists severe background clutter. Our method shows real-time performance.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bhole, C., Pal, C.: Fully automatic person segmentation in unconstrained video using spatio-temporal conditional random fields. Image Vis. Comput. 51(7), 58–68 (2016)
Chen, H.C., Chien, W.J., Wang, S.J.: Contrast-based color image segmentation. IEEE Signal Process. Lett. 11(7), 641–644 (2004)
Chen, L., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. In: CoRR (2017). arXiv:abs/1706.05587
Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Semantic image segmentation with deep convolutional nets and fully connected crf. In: ICLR (2015)
Cheng, G., Han, J., Zhou, P., Xu, D.: Learning rotation-invariant and fisher discriminative convolutional neural networks for object detection. IEEE Trans. Image Process. 28(1), 265–278 (2019)
Duan, Y., Chang, H., Tai, X.C.: Convergent non-overlapping domain decomposition methods for variational image segmentation. J. Sci. Comput. 69(2), 532–555 (2016)
Firsov, D., Lui, S.H.: Domain decomposition methods in image denoising using Gaussian curvature. J. Comput. Appl. Math. 193(2), 460–473 (2006)
Fani, M., Neher, H., Clausi, D.A., Wong, A., Zelek, J.: Hockey action recognition via integrated stacked hourglass network. In: CVPRW (2017)
Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. Int. J. Comput. Vis. 59(2), 167–181 (2004)
Fragkiadaki, K., Arbelaez, P., Felsen, P., Malik, J.: Learning to segment moving objects in videos. In: CVPR (2015)
Hamaguchi, R., Fujita, A., Nemoto, K., Imaizumi, T., Hikosaka, S.: Effective use of dilated convolutions for segmenting small object instances in remote sensing imagery. In: CoRR arXiv:abs/1709.00179 (2017)
Han, J., Chen, H., Liu, N., Yan, C., Li, X.: CNNs-based RGB-D saliency detection via cross-view transfer and multiview fusion. IEEE Trans. Cybernet. 48(11), 3171–3183 (2018)
Kampffmeyer, M., Salberg, A.B., Jenssen, R.: Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks. In: CVPRW (2016)
Kim, W., Kim, M.: On-line detection and segmentation of sports motions using a wearable sensor. Sensors 18(3), 913 (2018)
Li, J., Liang, X., Wei, Y., Xu, T., Feng, J., Yan, S.: Perceptual generative adversarial networks for small object detection. In: CVPR (2017)
Li, Y., Tax, D.M., Loog, M.: Scale selection for supervised image segmentation. Image Vis. Comput. 30(12), 991–1003 (2012)
Liu, W., Yan, C.C., Liu, J., Ma, H.: Deep learning based basketball video analysis for intelligent arena application. Multimedia Tools Appl. 76(23), 24983–25001 (2017)
Liu, Z., Zhang, C., Tian, Y.: 3D-based deep convolutional neural network for action recognition with depth sequences. Image Vis. Comput. 55(2), 93–100 (2016)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)
Lu, X., Li, X.: Multiresolution imaging. IEEE Trans. Cybernet. 44(1), 149–160 (2014)
Lu, X., Li, X., Mou, L.: Semi-supervised multitask learning for scene recognition. IEEE Trans. Cybernet. 45(9), 1967–1976 (2015)
Lu, X., Wang, B., Zheng, X., Li, X.: Exploring models and data for remote sensing image caption generation. IEEE Trans. Geosci. Remote Sens. 56(4), 2183–2195 (2018)
Lu, X., Wu, H., Yuan, Y.: Double constrained NMF for hyperspectral unmixing. IEEE Trans. Geosci. Remote Sens. 52(5), 2746–2758 (2014)
Lu, X., Zheng, X., Li, X.: Latent semantic minimal hashing for image retrieval. IEEE Trans. Image Process. 26(1), 355–368 (2017)
Luo, Z., Mishra, A., Achkar, A., Eichel, J., Li, S., Jodoin, P.M.: Non-local deep features for salient object detection. In: CVPR (2017)
Mumford, D.B., Shah, J.: Optimal approximations by piecewise smooth functions and associated variational problems. Commun. Pure Appl. Math. 42(5), 577–685 (1989)
Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. In: ICCV (2015)
Oh, S., Russell, S., Sastry, S.: Markov chain monte carlo data association for general multiple-target tracking problems. In: ICDC (2004)
Papazoglou, A., Ferrari, V.: Fast object segmentation in unconstrained video. In: ICCV (2013)
Papoutsakis, K.E., Argyros, A.A.: Integrating tracking with fine object segmentation. Image Vis. Comput. 31(10), 771–785 (2013)
Reinbacher, C., Rther, M., Bischof, H.: Fast variational multi-view segmentation through backprojection of spatial constraints. Image Vis. Comput. 30(11), 797–807 (2012)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)
Tai, X.C., Xu, J.: Global and uniform convergence of subspace correction methods for some convex optimization problems. Math. Comput. 71(237), 105–124 (2002)
Tora, M.R., Chen, J., Little, J.J.: Classification of puck possession events in ice hockey. In: CVPR (2017)
Xie, L., Xu, P., Chang, S.F., Divakaran, A., Sun, H.: Structure analysis of soccer video with domain knowledge and hidden markov models. Pattern Recognit. Lett. 25(7), 767775 (2004)
Zagrouba, E., Gamra, S.B., Najjar, A.: Model-based graph-cut method for automatic flower segmentation with spatial constraints. Image Vis. Comput. 32(12), 1007–1020 (2014)
Zhang, D., Han, J., Zhao, L., Meng, D.: Leveraging prior-knowledge for weakly supervised object detection under a collaborative self-paced curriculum learning framework. Int. J. Comput. Vis. 127, 1–18 (2018)
Zhang, W., Lu, X., Li, X.: A coarse-to-fine semi-supervised change detection for multispectral images. IEEE Trans. Geosci. Remote Sens. 56(6), 3587–3599 (2018)
Zhao, X., Qu, Y., Zhang, H.: Sports video segmentation using spectral clustering. J. Multimedia 9(7), 873–878 (2014)
Zhu, J., Liao, S., Lei, Z., Li, S.Z.: Multi-label convolutional neural network based pedestrian attribute classification. Image Vis. Comput. 58(2), 224–229 (2017)
Zhu, J., Liao, S., Lei, Z., Li, S.Z.: Multi-label convolutional neural network based pedestrian attribute classification. Image Vis. Comput. 59(3), 31–43 (2017)
Zitnick, L., Dollar, P.: Edge boxes: Locating object proposals from edges. In: ECCV (2014)
Acknowledgements
This work was supported by ICT R&D program of MSIT / IITP [2017-0-00543] and partly supported by the Chung-Ang University Research Scholarship Grants in 2019.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Park, J., Kwon, D., Choi, B.W. et al. Small object segmentation with fully convolutional network based on overlapping domain decomposition. Machine Vision and Applications 30, 707–716 (2019). https://doi.org/10.1007/s00138-019-01023-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00138-019-01023-x