A fuzzy theoretic approach for video segmentation using syntactic features

Abstract

This paper is concerned with the development of a fuzzy-logic-based framework for segmentation of video sequences. We have proposed a scheme for fuzzification of the frame-to-frame property difference values using the Rayleigh distribution. The difference values have been characterized by fuzzy terms like small, significant, large, etc. These terms have been used to design fuzzy rules for detecting abrupt changes and gradual changes. Fuzzy rules have provided a mechanism for integrating evidences based on different properties. The decompositional inference strategy has been used for fuzzy reasoning over the set of fuzzy rules. Gradual changes have been further classified as fade-in, fade-out and others (including dissolves, wipes, etc). Experimental results have shown that the proposed scheme can detect changes reliably.

Introduction

Partitioning of the video sequence by detecting scene changes is essential for indexing, parsing, characterization and categorization of video. Basically, there are two types of scene changes: abrupt change and gradual change. Camera breaks or abrupt changes are apparently easy to detect because the difference in image properties between consecutive frames is expected to be large. However, the amount of change can vary from one sequence to another depending upon the content of video and context of the scene change. Detection of gradual transition is more difficult because the change takes place over a period of frames. In this paper, we propose a fuzzy theoretic framework for video segmentation. The fuzzy theoretic scheme provides a mechanism for characterizing scene transitions through soft decisions. Subjectivity involved in the video partitioning process can be adequately captured by this framework.

The problem of video segmentation has received considerable attention in the recent times. Zhang et al. (1993) propose segmentation by using a threshold setting on differences of color histograms. They also deal with special effects, such as dissolving sequences, making use of a double threshold technique. Jain et al. (1995) present a model-based method, most suited to well-established video production processes. In particular, video segmentation is formulated as a production-model-based classification problem, and segmentation error measures are also defined. Aigrain et al. (1997) propose a rule-based system for video segmentation, which makes use of temporally located information present in video frames. These rules allow for the identification of more macroscopical changes consistent with the editing practices. In the work by Ardizzonei and Cascia (1996), a technique employing neural network is used. Essentially to detect a scene cut, two successive frames are preprocessed and a pixel-by-pixel difference is computed. This difference is then processed by a multi-layer perceptron that determines if the frames belong to the same shot. Color information present in input sequences is discarded and only luminance is used for the cut detection. Zabih et al. (1995) suggest a feature-based segmentation technique for classifying scene breaks. This approach can be used for detecting dissolves as a sequence of fade-out and fade-in effects. Xiong and Lee (1998) use a step-variable-based algorithm for scene change detection. The MPEG encoded video data can be segmented directly using the technique suggested by Arman et al. (1993). Here the DCT coefficients are analyzed to find frames where camera breaks take place.

Most of these video segmentation techniques characterize scene changes using static thresholds. Due to the time varying and subjective nature of the video data, these thresholds are not expected to work reliably under all possible conditions. Also, these approaches have used different image-based attributes for measuring the change. However, a particular property may not be always adequate for identifying the same type of transitions for different video sequences. In order to overcome these difficulties, we have proposed a fuzzy theoretic framework.

The benefits of a fuzzy framework in characterizing video data are twofold. Firstly, ambiguities and uncertainties involved in the selection of thresholds are taken care of by fuzzy rules. Secondly, because all inputs are represented as fuzzy sets, the complicated process of combination of various feature differences is simplified to straightforward application of standard fuzzy implications and compositions over these fuzzy relations.

In this paper we have presented a general scheme for fuzzifying frame-to-frame property differences. We have also indicated the patterns of fuzzy rules required for classifying scene transitions using multiple features. The actual scheme implemented in this paper is hierarchical in nature. A fusion of various syntactic features has been done to detect shot boundaries in a reliable fashion. An abrupt change is detected using histogram intersection, a gradual change is detected using a combination of pixel difference and histogram intersection while a fade is detected using a combination of pixel difference, histogram intersection and edge-pixel-count. The other syntactic features like motion (Cherfaoui and Bertin, 1995), moments difference (Arman et al., 1994) have also been tested. The motion feature has not been found to be very effective for shot detection as reported by Boreczky and Rowe (1996). There are a large number of false positives due to camera motion. However, we have found that optic-flow-based features are useful for detecting gradual transitions. Experimental results show that the proposed set of features produces reliable results.

The rest of the paper is organized as follows. We briefly discuss various syntactic features used for segmentation in Section 2. In Section 3, we have presented the fuzzy theoretic framework for characterization of the scene transitions. Experimental results are reported in Section 4. Finally, Section 5 presents the conclusions.