Fusion of 3D-LIDAR and camera data for scene parsing

Highlights

• One geometry segmentation algorithm is proposed to parse scanner pointclouds.

• One efficient multilayer perception classifier is trained to parse camera images.

• We propose one fuzzy logic based fusion method to integrate results of two sensors.

• We propose one Markov random field based temporal fusion method.

• The fused results are more reliable than those of individual sensors.

Abstract

Fusion of information gathered from multiple sources is essential to build a comprehensive situation picture for autonomous ground vehicles. In this paper, an approach which performs scene parsing and data fusion for a 3D-LIDAR scanner (Velodyne HDL-64E) and a video camera is described. First of all, a geometry segmentation algorithm is proposed for detection of obstacles and ground areas from data collected by the Velodyne scanner. Then, corresponding image collected by the video camera is classified patch by patch into more detailed categories. After that, parsing result of each frame is obtained by fusing result of Velodyne data and that of image using the fuzzy logic inference framework. Finally, parsing results of consecutive frames are smoothed by the Markov random field based temporal fusion method. The proposed approach has been evaluated with datasets collected by our autonomous ground vehicle testbed in both rural and urban areas. The fused results are more reliable than that acquired via analysis of only images or Velodyne data.

Introduction

Autonomous situation awareness is an important research aspect for robots and unmanned vehicles. Besides whether the terrain is traversable, they also require more specific object category information to carry out their tasks: e.g., approaching a tree, or the water area. For decades, computer vision approaches have been studied to classify scenes from images. Studies of the human visual system show us that scene perception is a highly complex process of information fusion which involves not just the human eyes, but also other human senses including hearing, tasting, etc. Even within a human vision system, there is clearly fusion of information from color, motion, depth and a whole variety of ways to infer shape, movement and physical characteristics of the things within the view [1]. In other words, efficient perceptual performance often requires integration of multiple sources of information, both within the senses and between them. As a matter of fact, other sensors like infrared laser projector in Kinect [2] and LIDAR scanners [3] have been applied to complement video cameras in recent years.

In this work, in order to help unmanned vehicles to understand their environment, two sensors are used: Velodyne HDL-64E 3D-LIDAR scanner [3] and monocular video camera. A Velodyne scanner provides 3-dimensional but sparse pointcloud of the surrounding environment. The pointcloud is trustworthy for obstacle detection but lacks color and texture information, which is valuable for more detailed categorization of objects. Besides, although Velodyne HDL-64E is a powerful LIDAR scanner in the market, its effective coverage limits within 70 m from the center of the sensor. Considering some time will be taken for information processing and task scheduling, the 70 m distance may not be sufficient for an unmanned moving vehicle to respond. Furthermore, for some tasks, we hope the vehicle can “see” as far as 200 m for advanced planning. On the contrary, images captured by video cameras can easily cover a much broader and further area and provide more discriminative information to classify objects into categories. However, due to the lack of depth information, image-based detection of obstacles of various shapes, sizes and orientations remains challenging. Due to the above-mentioned complementary features between cameras and LIDAR sensors, it is possible to acquire more reliable scene parsing by fusing information derived from these two sensors.

In addition, the sequential scene parsing also requires fusing results of consecutive frames. In fact, even after fusing results of two sensors, the obtained parsing results of consecutive frames may have abrupt changes due to stochastic errors. These abrupt changes of parsing results may confuse the vehicle navigation system. Intuitively, it is possible to obtain more cohesive sequential parsing results by including the temporal fusion.

In this research, we first propose a new way to fuse the results of two sensors by employing fuzzy logic inference [4]. Then we propose a Markov random field based approach to fuse the results of consecutive frames. Fig. 1 illustrates the fusion process. Fuzzy logic is preferable for our application due to its advantages. First, fuzzy logic is built on top of the knowledge and experience of experts. Therefore, it can employ not only results from LIDAR and video camera data but also a priori knowledge. Second, fuzzy logic can model nonlinear functions of arbitrary complexity. This is important as scene parsing is not a trivial problem. Third, fuzzy logic can tolerate imprecise results of two sensors. Moreover, fuzzy logic is a flexible fusion framework so that results of more sensors can be easily integrated to the system in future.

To fuse results of consecutive frames, we propose a Markov random field (MRF) based temporal fusion method [5], [6]. Correspondences between consecutive frames are first estimated by using the dense optical flow method [7]. Then, a MRF model is built to integrate results of multiple consecutive frames. The result of each frame is refined by the Belief Propagation (BP) algorithm [8]. The following contributions have been made in this paper:

• To the best of our knowledge, the proposed approach is the first systematic fuzzy logic inference based fusion work for scene understanding by fusing results of Velodyne 3D-LIDAR scanner and monocular video camera.

• The MRF based temporal fusion method is introduced to obtain cohesive video parsing results. It can smooth whole frame simultaneously by integrating results of multiple consecutive frames.

• We test the proposed approach on datasets collected by our autonomous ground vehicle testbed. The datasets are captured from urban and rural areas either in day or night time. The results validate the robustness and effectiveness of our fusion approach for scene parsing.

A preliminary version of this paper was described in [9]. The current version described here differs from the former in several ways, including: the introduction of MRF based temporal fusion method; comprehensive evaluation of the method with three more datasets; further analysis and discussion of the whole approach, as well as the introduction of more related works about sensor fusion and scene parsing. While the preliminary version in [9] focuses on fuzzy logic based fusion strategies, the current version will provide more details on image parsing techniques, too.

This paper is organized as follows: In Section 2, we briefly survey the sensor fusion and scene parsing literature. After giving the parsing methods for individual sensors, we describe the fuzzy logic based method to fuse the results of two sensors in Section 3.1. The MRF based temporal fusion is presented in Section 3.1. Thorough experiments are conducted in Section 3.1 for evaluation, and in-depth discussion is provided in Section 3.1. We conclude our paper in Section 3.1.