Elsevier

Image and Vision Computing

Volume 24, Issue 9, 1 September 2006, Pages 1001-1009
Image and Vision Computing

Characterizing the performance of automatic road detection using error propagation

https://doi.org/10.1016/j.imavis.2006.02.018Get rights and content

Abstract

A methodology is introduced to predict the performance of automatic road detection using image examples of typical road types. In contrast to previous work on road detection, the focus is on characterizing the detection performance to achieve reliable performance measures of the detection. It is studied how noise, like road markings, shadows, trees and buildings, influences the detection of road. This noise is modeled using second-order statistics and its effects are calculated using error propagation on the detection equations. The method predicts the performance in terms of detection rate and gives the optimal parameter set that is needed for this detection. Experiments have been conducted on a set of images of typical roads in very high-resolution satellite images.

Introduction

Although digital maps are available for many areas, producers of geospatial data continuously need to keep this data accurate, up-to-date and complete. Not only do they need to control the quality of their data, the high demands of current applications put pressure to deliver products of increasingly higher spatial resolution. Upgrading the spatial accuracy of existing data, however, is a labor intensive and expensive process. In this paper, we focus on the problem of quality assessment of digital maps and more specifically digital road maps. We investigate automatic road detection in very high-resolution (VHR) satellite images, which forms the basis for automated quality assessment and quality control in road GIS databases.

The proposed system for quality assessment is based on object-based spatial registration, where detected objects in the image are registered and compared to corresponding objects in the GIS data [1]. The system consists of two stages: (1) a low-level feature detection process, which extracts roads and junctions using an improved ridge detector, and (2) a high-level matching process, which uses graph matching to find correspondences between the detected image information and the road data (cf. Fig. 1). The graph matching process is driven by the spatial relations between the objects and takes into account different errors that can occur (e.g. spatial inaccuracy, data inconsistencies between image and GIS data). The matched objects can be used to calculate a rubber sheeting transformation between image and GIS data that is able to compensate the local distortions that can occur between the datasets. Additionally the object-to-object mapping is useful to define measures of change between datasets.

Much of the performance of the system depends on the quality of first stage, i.e. the extraction of road information from the images. Although much effort has been spent on designing algorithms for road detection [2], this complex problem is still not fully solved. In contrast to previous efforts, which are aimed at improving the detection, we focus on characterizing the performance of detection. There are several reasons for this:

  • (1)

    The industry needs to be able to assess the benefits that can be gained with image-based road detection for its specific needs. An analysis tool should be ready to assess the expected quality of road detection for a given dataset to be able to decide if the use of this technology can decrease operational costs. Often the results published in the literature are based on a small number of images and are as such insufficient to predict the performance on larger datasets or data of different type (e.g. a different resolution or region). Based on the example images that describe typical roads in the region, a tool is needed that predicts the expected detection rate.

  • (2)

    An important difficulty in using a detection algorithm is that a number of parameters need to be determined by the operator. The choice of the parameter set has a big impact on the detection performance and is dependent on the image content. Since the operator is not necessarily an image-processing expert, a tool should be available to estimate the ‘optimal’ parameter set given example images.

  • (3)

    An object-based quality assessment system naturally depends on the quality of detection in the image. Decisions need to be taken based on the information gathered by the detection, so one needs to be able to estimate the reliability of the information. The high-level matching process, which compares image information with the GIS data, needs information about the reliability of object detection in terms of true and false positive detection. The reason is that the matching process needs to differentiate between inconsistencies due to change and outliers (i.e. false positive detection). Knowledge about the expected number of outliers is used to make this distinction.

The image-processing community has traditionally focused on the development of novel and sophisticated algorithms. However, the importance of performance evaluation and characterization has gained greater acceptance in recent years [3], [4].

In order to compare the performance of different algorithms on the same basis, quantitative performance measures are needed. In [5], a number of methods for assessing the performance of edge detectors are reviewed. An approach is presented which uses manually labeled edges from real images as ground truth to produce a ROC analysis. In [6], different measures combined with statistical ANOVA tests are introduced to objectively evaluate different edge detector schemes. In [7], quality measures are proposed for the evaluation of automatic road extraction. The quality measures comprise completeness, correctness, quality, redundancy, planimetric RMS differences and gap statistics. They aim at evaluating exhaustivity as well as assessing geometrical accuracy. In [8], a methodology is discussed to generate artificial ground truth data for the assessment of corner detectors.

In many applications, it is interesting to evaluate beforehand the ability of feature detectors to perform a given task. Predicting the performance, however, is difficult [9]. Nevertheless, performance prediction as part of performance characterization has received greater attention in recent work. In [10], the performance of edge detection system is evaluated by modeling an ideal ramp edge disturbed with gaussian noise. For this detector, analytic expressions are obtained for the output distributions. From these distributions, performance measures for edge detection are optimized to determine the optimal detector parameters. In [11], the bootstrap formalism is introduced to go from statistical samples to parameter estimation and applied to performance assessment. In [12], the performance accuracy for edge detection and motion estimation is predicted based on the statistical properties of the local context. Context in this case is described by three variables: Gabor components, entropy and signal/noise ratio. A prediction function is determined from training using a logistic regression function. In [13], first-order error propagation is applied to estimate how directional noise in 3D point clouds affects the estimated parameters of rigid body motion.

In this work, we focus on predicting the performance of differential ridge detectors, which are commonly used for tasks like road detection [2]. We study how noise, like activity on the road, shadows, trees and buildings next to the road, influences the detection of road. This noise is modeled using second-order statistics and its effects are calculated using error propagation on the detection equations [20]. Error propagation allows us to predict the detection performance of a road in a given image, based on simple image statistics. This becomes important in the context of automated quality control, where detected image information is used to make decisions about GIS data. Information about the detection quality is paramount in order to use this information. In addition, the optimal parameter set can be determined that gives the best detection, taking into account the type of road, image noise and structural nose. Prediction of the performance can be done in an interactive manner, where the user shows typical road examples and gets in return the estimated performance on his dataset. On the other hand, it can be used during the detection, where in addition to the detected road pixels information is given on the reliability.

Section snippets

Ridge detection

Lines in an image can be seen as narrow valleys or ridges in the intensity surface if one views the image as a terrain model. In [14], [16], different approaches to ridge detection are reviewed. In this work, line detection is performed based on polynomial interpolation to determine pixels belonging to road structures in the image, the ‘facet model’ [19]. This is a standard method for ridge detection. Although our focus is on the facet model, the methodology of error propagation can be applied

Error analysis

We wish to give a more quantitative analysis of the performance of ridge detection. More specifically, we wish to be able to predict the performance of the detector for a given dataset and the respective parameter set that gives optimal results. For this, we analyze the influence that perturbations on the intensity values have on the estimation of the parameters by using error propagation [20]. Additive random perturbations are assumed on the input x. In the case of road detection, the road

Experimental results

In this section, the derived expressions are applied to predict the performance of the detection. For these experiments, an IKONOS panchromatic satellite image has been used, acquired over the city of Ghent, Belgium. The image is a standard GEO product with a 1-m pixel resolution. A number of small image subsets have been selected containing straight examples of typical roads. These subsets have been manually rotated until each road is positioned parallel to the vertical axis. Fig. 4 shows the

Conclusion

In this paper, an analysis of the performance of road detection is introduced. A methodology has been established where based on the image statistics of a road and its immediate surroundings, the performance of the detection can be predicted as well as the optimal parameter set, which is needed for the detection of this type of road. The performance is characterized as a function of the detection rate. Experiments have been conducted on a set of images of typical roads in an IKONOS satellite

Acknowledgements

This research has been conducted with the support of the Belgian Science Policy as part of the STEREO research programme. The authors also gratefully acknowledge the remarks made by the anonymous reviewers.

References (22)

  • J. Mena

    State of the art on automatic road extraction for GIS update: a novel classification

    Pattern Recogn. Lett.

    (2003)
  • P. Danielsson et al.

    Efficient detection of second-degree variations in 2D and 3D images

    J. Visual Commun. Image Represent.

    (2001)
  • S. Gautama et al.

    Image-based change detection of geographic information using spatial constraints

  • P. Courtney et al.

    Performance characterisation in computer vision: the role of statistics in testing and design

  • M. Heath et al.

    Robust Visual method for assessing the relative performance of edge-detection algorithms

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1997)
  • S. Borra et al.

    A framework for performance characterization of intermediate-level grouping modules

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1997)
  • C. Wiedemann et al.

    Empirical evaluation of automatically extracted road axes, CVPR Workshop on Empirical Evaluation Methods in Computer Vision

  • P. Rockett

    Performance assessment of feature detection algorithms: a methodology and case study on corner detectors

    IEEE Trans. Image Process.

    (2003)
  • W. Foerstner

    Ten pros and cons against performance characterisation of vision algorithms

    Mach. Vis. Appl.

    (1997)
  • V. Ramesh, R. Haralick, Performance characterization of edge operators, DARPA Image Understanding Workshop, 1993, pp....
  • Cited by (0)

    View full text