Fuzzy constraint satisfaction problem for model-based image interpretation

Abstract

Scene interpretation guided by a generic model benefits from structural representations of objects and their spatial relationships. In this paper, we consider nested conceptual graphs for encoding objects and groups of objects, spatial relationships between objects or between groups of objects, along with the imprecision and uncertainty attached to the formal representations of such relationships. Scene interpretation is then formalized as a graph homomorphism problem for the identification of possibly multiple instances of the model in an image. We propose an extension of fuzzy constraint satisfaction problems (FCSP) to deal with complex objects. In particular, we extend FCSP arc-consistency checking to deal with groups of objects which can be related among them or have a spatial property such as being aligned. The instantiations of the model in the image are obtained by solving a FCSP. This framework is illustrated on the example of interpretation of Earth observation images. A method is proposed to find the instantiations of a nested conceptual graph, representing a generic model of the scene (such as harbor or airport) in an unlabeled image. Experimental results on high resolution satellite images show that the proposed approach successfully recognizes a given spatial configuration and is robust to image segmentation errors. The results demonstrate the interest of using complex spatial relations for the interpretation of images.

Introduction

Image interpretation consists in recognizing different objects which compose a scene, understanding their spatial organization, and providing a description of this scene and a semantic labeling of the image. To interpret an image it is necessary to use information within the image, but also contextual information relevant to the interpretation task. The contextual information should allow us to answer questions such as: which are the objects of interest? How to identify these objects or their parts? And how are these objects or parts related? This information depends on the domain, the objective of the description and the application. Thus when performing an automatic image interpretation a knowledge representation system should be developed. In this system we should be capable of encoding information about objects, spatial relations between objects or object parts, extraction algorithms, etc.

In this paper we focus on the interpretation of very high resolution Earth observation images, as an illustration. These images contain a large amount of information. They are the outcome of the combination of many different intensities that can represent natural concepts such as vegetation, geomorphological and hydrological concepts, man-made objects such as buildings and roads, and artifacts caused by variations in illumination of the terrain, such as shadows [43]. This large quantity of information allows the description of the images at different conceptual levels [23]:

• individual objects, for instance a house, a tree, a road segment,

• land cover type, for instance water, bare land, vegetation,

• complex or composite objects, which are new semantic objects formed by several spatially related individual objects, for instance airports, harbors, train stations, nuclear power plants, toll gates, stadiums, etc.

The conceptual level used to describe the image depends on the objective of the description as well as the resolution of the image. Moreover, Earth observation images contain objects of different sizes, which makes it unfeasible to analyze all the concepts of the image at the same scale. For instance, a building can be identified at a very high resolution using its shape, a city is better identified at a lower resolution as a texture. Therefore, according to the level of concept we are interested in, we should choose an appropriate scale of observation. In this work we concentrate on the interpretation of complex objects. One of the difficulties of this task lies in determining the important details to extract the objects which compose complex objects. Objects belonging to the same complex object can be observable at different scales. Furthermore, some objects in a complex scene cannot always be recognized with traditional methods using spectral and textural features. They often require the recognition of other objects having a spatial relation with them, and then use the spatial relation to identify them. Hence the spatial relations and the spatial arrangements of objects and of complex objects and their parts in the scene are of prime importance for the recognition of complex objects. Another difficulty encountered in the interpretation of complex objects is that the spatial arrangements can be expressed as complex relationships, i.e. spatial relations that are not necessarily expressed between two objects but between a group of objects and an object, or groups of objects having a spatial property, such as alignment. Hence, the interpretation method should be able to encode these complex relationships and deal with them.

In this work we propose a method to identify complex objects in an Earth observation image. We make use of knowledge on the spatial structure of the complex objects to guide their identification in the image. We describe a complex object through a nested conceptual graph. This model allows the representation of spatial information of the complex object, including complex spatial relationships. We assume that the image is already segmented and we only concentrate on the problem of labeling the regions according to the knowledge and information supplied by the model.

We formulate the problem of labeling the image regions as a homomorphism from the model to the image regions. Modeling the problem as a homomorphism permits to map the model to several regions of the image (several instantiations), and thus account for uncertainty within the model. By uncertainty in the model we mean that even if all the objects in the model appear in the image, we are not certain about the number of instantiations. For instance in a harbor, a dock can have none, one or several boats adjacent to it. This type of uncertainty is present in Earth observation images. Moreover, when a model is used to represent the spatial arrangement of the objects that should appear in the scene, other types of information imperfections can be present:

• Imprecision on spatial relations. Many spatial relations can be imprecise by nature. Their satisfaction depends on the context or even on the size of the objects. For instance the relation near has a different interpretation according to the size of the reference object: the distance used to determine whether a building is near an airport is different from the one used to determine whether a building is near a tree.

• Uncertainty with labeling objects in the image. When labeling the objects in the image after a segmentation, labels may be uncertain.

• Imprecision on the objects in the image. This can be due to different causes, such as the discretization of space (passing from a continuous scene to a digital image), or the processing methods (e.g. segmentation).

We express the problem of finding a graph homomorphism as a Fuzzy Constraint Satisfaction Problem (FCSP). Using a FCSP allows us to consider the different sources of imperfections. The imprecision of spatial relations is dealt by modeling the spatial relations as fuzzy relations which are handled by FCSP. The other two types of information imperfections are modeled by using fuzzy membership functions (that consider information from the model) to perform an initial labeling of the objects in the image.

The main two contributions of this paper are (i) an interpretation method that starts from an unlabeled segmented image (possibly imprecisely and inaccurately segmented) and exploits knowledge about the scene to be recognized, and (ii) translate the problem as a FCSP and adapt the FCSP formalism to deal with complex spatial relations. The proposed framework was developed in [44].

We formulate the problem of image interpretation within the framework of Knowledge Based Systems (KBS). In Section 2, we introduce KBS, their use in image interpretation, and discuss the choice of the proposed method. In Section 3, we discuss nested conceptual graphs and their representation capabilities. In Section 4 we explain how the proposed problem can be represented as a Fuzzy CSP. The FCSP formalism has to be extended in order to allow the representation of embedded information, which is done in Section 5. Finally in Section 6, we illustrate the proposed method on two examples.