Generalized playfield segmentation of sport videos using color features
Research highlights
► We propose a method to segment playfields for various sport videos. ► Two schemes which employ pdf of color features are first proposed for clustering. ► A novel scheme is then developed to merge those clusters into four color classes. ► The segmentation based on the four classes effectively segments the playfield.
Introduction
Automated sport video analysis has attracted a great deal of attention because of the appeal of sports to large audiences. Effective annotation of sport videos will allow users to retrieve highlights or important events efficiently and precisely at a later date (Ekin et al., 2003, Hung and Hsieh, 2008, Xie et al., 2004, Zhang and Chang, 2004). An automated means of extracting high-level information from sport videos that is consistent with human semantics is crucial to develop a variety of application systems such as highlighting, summarizing, and indexing/retrieval, as well as for athlete training. Playfields are the main backgrounds of sport scenes, and almost all important actions occur on them. As a result, playfield features provide information that is helpful, for example, for scene analysis (Duan et al., 2005, Shih and Huang, 2005), player tracking (Han et al., 2008, Pallavi et al., 2008, Yu et al., 2006), and the detection or classification of high-level sporting events (Hung and Hsieh, 2008). Therefore, playfield segmentation is an important step in sport video analysis.
Many previous studies of playfield segmentation have focused on one specific sport, such as Barnard and Odobez, 2004, Liu et al., 2005, Yang et al., 2006, Kuo et al., 2008. Most existing methods use color thresholds for specific colors, such as green grass or red sand. As the playfield always has very distinctive attributes, such as clear boundary lines and dense color distribution, a fixed-threshold scheme is most commonly used. However, the fixed-threshold scheme often results in large errors in segmentation because colors vary with changes in illumination.
Some sophisticated methods for background modeling, such as the Gaussian Mixture Model (GMM), have been proposed to describe the distribution of the playfield in feature space. Liu et al. (2004) proposed an adaptive GMM algorithm to model the dominant colors of soccer videos. Barnard and Odobez (2004) proposed a GMM and maximum a posteriori (MAP) adaptation for soccer playfield segmentation. However, this method requires very high computational complexity, and it has only been verified on grass-type playfields. Kuo et al. (2008) presented a playfield segmentation method for baseball videos based on a GMM. The adaptive GMM model is constructed by a novel training pixel-selection scheme, which automatically selects the appropriate pixel(s) from input video for parameter estimation in the expectation–maximization (EM) process. The methods described above are adaptive schemes aimed at improving robustness against lighting variations, but they were still designed for a specific sport or a single type of playfield.
Compared with the methods for a specific domain, few studies have examined general segmentation methods that can be applied to different playfields in videos of various sports. Wang et al. (2004) proposed a GMM-based method to detect multiple colors of some semantic objects, such as the playfield and players’ uniforms. Jiang et al. (2004) proposed a playfield segmentation method based on GMMs and region growing. Assuming field pixels are the dominant components in videos frames, the authors build the GMMs of the field pixels and use these models to detect playfield pixels. Then, a region growing is performed to connect the discrete playfield pixels into regions, eliminate noises and smooth the boundaries of objects.
Playfields are usually composed of various natural and artificial substances. For example, the outdoor fields used for baseball, soccer, and tennis are composed of grass, soil, and sand. Indoor basketball and volleyball courts are mainly constructed of wood and plastic materials. The use of these natural and artificial substances often means that the playfield is characterized by homogenous and distinctive colors. Therefore, color features are usually adopted for segmentation of playfields in sport videos.
In this paper, we propose a generalized playfield segmentation method that is suitable for various types of sport videos. This method first estimates the probability density function (pdf) of the color components (cb, cr) of the input image, and then applies steepest ascent hill-climbing search to the pdf to generate several clusters. Two hill-climbing schemes are designed at this stage. Next, a simple and effective rule is developed to merge the clusters into four color classes: red, green, blue, and gray. Finally, a simple scheme is used to fuse small regions into their adjacent large regions to obtain the segmentation result. The proposed method essentially performs mode seeking in the cb–cr histogram, which thus identifies the dominant color bins. We assume that these modes are mapped to red, green, or blue, which are the most common color components across different types of playfields, or to gray, which corresponds to the average color of non-playfield pixels. The mapping of the four colors is very close to real scenes of general sport videos, and therefore mode seeking will be more effective and efficient than with existing methods as we make a good initial estimate of the modes. The experimental results indicated that our method surpasses the GMM-based method in both segmentation accuracy and computational efficiency. The method proposed here is significantly less computationally intensive than the mean-shift-based method, but it has comparable segmentation accuracy.
Section snippets
Proposed method
Fig. 1 shows a flow chart of the proposed segmentation method. First, a two-dimensional pdf is obtained from the color histogram of an image frame by nonparametric estimation. Second, local maximum clustering with hill climbing is applied to the pdf to requantize the image. Two hill-climbing schemes, including two-dimensional fast hill climbing and its one-dimensional approximation scheme, have been developed. Next, a four-color cluster-merging method is proposed to segment the image into
Experimental results and discussion
The proposed segmentation method includes four stages, as shown in Fig. 1. For the stage of local maximum clustering, we developed two hill-climbing schemes (2D and 1D) for local maximum clustering. For convenience of comparison, we denote the proposed segmentation methods containing 2D and 1D hill climbing as “proposed 2D” and “proposed 1D,” respectively. In our experiments, we applied the proposed methods to several field scenes in various sports, including basketball, baseball, football,
Conclusions
This paper presented a generalized method for playfield segmentation for various types of sport videos. The method mainly includes four processes: estimation of color probability density function (pdf), unsupervised clustering, cluster merging, and region fusion. The first process estimates color pdf with nonparametric estimation. Two unsupervised clustering algorithms based on 2D hill climbing and 1D hill climbing schemes were then designed to group image samples into several clusters. Next, a
Acknowledgements
The authors would like to express their sincere thanks to the anonymous reviewers for their invaluable comments, which helped in the effective presentation of this work. This study was supported in part by the National Science Council, Taiwan, Republic of China, under Grants NSC 97-2221-E-130-017-MY2, NSC 98-2221-E-151-036-MY3, and NSC 98-2811-E-214-151-001.
References (26)
- et al.
Finding salient regions in images: nonparametric clustering for image segmentation and grouping
Comput. Vis. Image Underst.
(1999) - et al.
Structure analysis of soccer video with domain knowledge and hidden markov models
Pattern Recognit. Lett.
(2004) A survey on evaluation methods for image segmentation
Pattern Recognit.
(1996)- et al.
Robust playfield segmentation using MAP adaptation
Internat. Conf. Pattern Recognit.
(2004) - et al.
Mean shift: a robust approach toward feature space analysis
IEEE Trans. Pattern Anal. Machine Intell.
(2002) - et al.
A unified framework for semantic shot classification in sport video
IEEE Trans. Multimedia
(2005) - et al.
Automatic soccer video analysis and summarization
IEEE Trans. Image Process.
(2003) - Feldman, B., 2006. Mean-Shift Clustering....
- et al.
Digital Image Processing
(2002) - et al.
Broadcast court-net sports video analysis using fast 3-D camera modeling
IEEE Trans. Circuits Systems Video Technol.
(2008)
On the use of windows for harmonic analysis with the discrete Fourier transform
Proc. IEEE
An experimental comparison of range image segmentation algorithms
IEEE Trans. Pattern Anal. Mach. Intell.
Event detection of broadcast baseball videos
IEEE Trans. Circuits Systems Video Technol.
Cited by (13)
A survey on player tracking in soccer videos
2017, Computer Vision and Image UnderstandingCitation Excerpt :In addition, playfield detection using GMM outperformed that of histogram and hidden Markov models (HMMs) in Jiang et al. (2004). Alternatively, thresholding (Yoon et al., 2002) or clustering (Hung et al., 2011) based on histogram peaks was applied for detecting playfield pixels. An improved generalized Lloyd algorithm (GLA) (Yu et al., 2007) and multilayer perceptron neural network (MLP) classifier (Kangarloo and Kabir, 2004) were also applied for playfield detection.
Fuzzy rule-based reasoning approach for event detection and annotation of broadcast soccer video
2013, Applied Soft Computing JournalCitation Excerpt :Due to differences in the style of various sports, most published papers in this area are genre-specific. So that, they concentrate on individual sports such as soccer [8–11], basketball [6], cricket [12], tennis [13], volleyball [14], baseball [15] on the other hand, a few works are observed to be genre-independent studies [16,17]. Genre-specific approaches usually produce more promising results rather than genre-independent ones.
Application of Virtual Reality Technology and Unsupervised Video Object Segmentation Algorithm in 3D Model Modeling
2022, Advances in Mathematical PhysicsLow Cost Player Tracking in Field Hockey
2022, Communications in Computer and Information ScienceWushu Competition Field Segmentation Based on Expectation-Maximization Algorithm
2021, 2021 IEEE 4th International Conference on Electronics Technology, ICET 2021Analysis of dynamic movement of elevator doors based on semantic segmentation
2020, Advances in Intelligent Systems and Computing