Semi-automatic object-based video segmentation with labeling of color segments

Abstract

In this paper we propose a semi-automatic method for general object-based segmentation of image sequences. A label field is initialized by the user in the first frame of the sequence and then it is automatically tracked for the rest of the frames based on the color and motion properties of the various objects in the scene. We propose a novel statistical modeling which is based on the local objects’ properties. The locality introduced in our modeling allows the tracking of complex objects that, globally, might be inhomogeneous in their color and motion characteristics. The labeling criterion is the maximization of the joint probability of the labels and the observed color and motion properties. The proposed method utilizes an initial color-based segmentation which is obtained for each frame of the sequence. Both the modeling and the optimization are expressed in terms of the color-segments’ statistical properties. Experimental results are presented on real-image sequences that include complex human motion in cluttered backgrounds.

Introduction

Boosted by the technological advances in the area of communications and computer engineering, there has been an explosion in the amount of the distributed visual information in the last decade. This fact and the applications that are continuously emerging advocate the need for the development of a wide range of methods for distributing and processing the available visual information. Object-based segmentation of image sequences is one of the issues that often arise in the world of video processing and communications. By partitioning each frame into segments that correspond to meaningful objects, we obtain a representation of the scene that allows coding, delivering and viewing the visual information in terms of its actual contents. The application that can benefit are numerous. Content-based functionalities such as object-dependent coding quality or object manipulation for video editing can now be developed. Furthermore, in the context of MPEG-7 [7], a semantically rich representation allows the attachment of meaningful descriptors to the data, thus boost applications that revolve around retrieval of multimedia material.

Activity in the area of object-based segmentation in the last decade soon revealed that fully automatic, general object-based segmentation of image sequences is a chimera. On the one hand, the most popular assumptions that lead to robust schemes—such as the assumption that the camera is static [15] or the assumption that the objects are rigidly moving [18]—are too strict for general scenes. On the other hand, depending on the application and the user, the goal of the segmentation may vary; the problem of general object-based segmentation is therefore inherently ill-posed.

In order to overcome these problems, semi-automatic methods started to emerge. Typically, in these schemes a label field is initialized by the user in the first frame of the sequence and then automatically tracked in the subsequent frames. The challenges in such schemes lie in tracking robustly multiple complex objects in complex environments. Obviously, the global homogeneity assumptions on motion characteristics that lie behind the robustness of many automatic schemes do not hold true for complex objects. Even the assumption that it is possible to decompose the scene based on the kinematic behavior of the depicted objects might not hold true. Other sources of information such as color and/or texture should be also used. For such sources of information the adoption of global homogeneity models is in general more problematic than in the case of motion: objects that are semantically meaningful are unlikely to exhibit global homogeneity in such properties and/or discontinuities might be difficult to be detected. Such inhomogeneity in the objects’ properties makes probable that the detected objects’ borders are attracted by details and structure in other neighboring objects. Temporal evidence can support a correct decision, but it should be used with care; even if the label field of the previous frame is perfectly estimated, the motion field that establishes correspondences with the current frame in notoriously known to contain errors.

The semi-automatic method for object-based labeling (i.e. object-based segmentation) that we propose here is based on all of the above considerations. We follow the dominant paradigm in which the label field is initialized by the user and then automatically tracked for the rest of the sequence. An outline of the proposed method is illustrated in Fig. 1 and a preliminary version of the paper was presented in [17]. The method operates at three levels. At Level 1 (pixel level) a feature vector is estimated for each pixel in the current frame. At Level 2 (segment level) a color-based segmentation method decomposes the current frame in a number of color segments. Subsequently, we estimate the statistical properties of the color segments. Finally, at Level 3 (object level) a labeling based on a probabilistic classification of the color segments takes place. The labeling imposes a “common fate” to the pixels belonging to the same color segment. The classification is based on a novel local statistical modeling of the color and motion properties of the objects in the scene. The classification criterion is the maximization of the joint probability of the label field and the observations (i.e. feature vectors), with respect to the label field. For the maximization a deterministic iterative local search algorithm is proposed.

The relation of our method with other methods in the literature will be addressed in the next section. Let us note here that the novel local modeling that we propose allows the modeling of multiple objects that do not exhibit global homogeneity in their characteristics. Furthermore, we propose explicit modeling of the color and motion characteristics as well as the use of color segments as primary elements in the modeling and in the classification. In this way, we aim at robustness against spurious edges in the background and against errors in the estimation of the motion field that establishes correspondences with the label field in the previous frame. Finally, our scheme requires the manual setting of only one parameter: the degree of the locality of the proposed modeling.

The remainder of the paper is organized as follows. In Section 3 we briefly describe the color-based segmentation method, the user interaction and the motion estimation algorithm. In Section 4 we describe the local modeling of the objects’ properties and in Section 5 we formulate and solve the labeling as an optimization problem. In Section 6 the projection procedure is described. In Section 7 we present experimental results and in Section 8 we draw conclusions.