Comparative evaluation of range sensor accuracy for indoor mobile robotics and automated logistics applications

https://doi.org/10.1016/j.robot.2012.08.011Get rights and content

Abstract

3D range sensing is an important topic in robotics, as it is a component in vital autonomous subsystems such as for collision avoidance, mapping and perception. The development of affordable, high frame rate and precise 3D range sensors is thus of considerable interest. Recent advances in sensing technology have produced several novel sensors that attempt to meet these requirements. This work is concerned with the development of a holistic method for accuracy evaluation of the measurements produced by such devices. A method for comparison of range sensor output to a set of reference distance measurements, without using a precise ground truth environment model, is proposed. This article presents an extensive evaluation of three novel depth sensors — the Swiss Ranger SR-4000, Fotonic B70 and Microsoft Kinect. Tests are concentrated on the automated logistics scenario of container unloading. Six different setups of box-, cylinder-, and sack-shaped goods inside a mock-up container are used to collect range measurements. Comparisons are performed against hand-crafted ground truth data, as well as against a reference actuated Laser Range Finder (aLRF) system. Additional test cases in an uncontrolled indoor environment are performed in order to evaluate the sensors’ performance in a challenging, realistic application scenario.

Highlights

► Three novel range sensing devices are compared to an actuated laser range finder. ► A methodology, previously used for spatial reperesntation evaluation is adapted. ► The Kinect structured light camera is found to have the best accuracy at short range.

Introduction

In recent years, a multitude of range sensing devices have become available at more affordable costs. Notably, 2D laser range sensors have demonstrated reliable performance and therefore have been widely used for both research and industrial applications. As the complexity of application scenarios considered in mobile robotics increases, so does the use of 3D range sensors. Although precise commercial 3D laser sensors are available (see for example products from Velodyne1 and Riegl2), their prohibitively high cost has limited their use. Actuated laser range finders (aLRF) have thus been the most widely used 3D range sensors in the mobile robotics community. Usually aLRF sensors utilize commercially available 2D laser sensors of high accuracy and known precision, resulting in a reliable measurement system. Nevertheless, aLRF sensors have several disadvantages—the lack of commercial availability of an integrated system, slow refresh rates on the order of 0.1 Hz, high weight and a large number of moving parts.

Several competing sensor technologies that attempt to solve the problems of aLRF systems have been proposed. Recently, time-of-flight (ToF) and structured light cameras have become more available and more widely used. A recently developed class of ToF sensors operate by emitting modulated infrared light and measuring the phase shift of the reflected signal. Typically, ToF cameras can deliver dense range measurements at high frame rates of up to 50 Hz. Structured light cameras can produce similar measurements, using a projected pattern that is observed by a CCD camera with a known baseline distance. Although the ToF and structured light sensor technologies have a lot of potential for use in mobile robotics, both are affected by several error sources. It is thus very important to evaluate the accuracy of these emerging sensors, compared to that of the actuated LRF.

Over the course of development of ToF sensors, several works have evaluated their accuracy in the context of sensor calibration. Kahlmann  [1] proposes several calibration routines for the SwissRanger SR-2 and SR-3000 cameras. In order to evaluate their effect on the range measurement quality, he proposes to scan a flat wall and compare offsets from the expected distance. Several precisely engineered optical calibration setups, as well as a calibration track line are also used. Linder et al.  [2] also use standard optical calibration setups (checkerboard images), in order to calibrate PMD ToF cameras and increase the accuracy of the depth measurements (the deviation of points on a flat wall target is used as a performance metric). Fuchs and Hirzinger  [3] evaluate distance accuracy using a modified checkerboard pattern and a pair of ToF cameras mounted on an industrial manipulator. Chiabrando et al.  [4] perform a comprehensive evaluation of the SR-4000 ToF camera, also using a pre-fixed optical calibration pattern and a tilted calibration board. While these setups constitute an important test case for ToF cameras, they do not capture the complexity of typical uncontrolled environments encountered by mobile robots. May et al.  [5] consider several real-world environments and measure the distance to known objects in these setups, resulting in a more complete accuracy evaluation. Several works have also implicitly evaluated the accuracy of ToF ranging systems by considering their utility in the context of mapping May et al.  [6], [7], obstacle avoidance  [8] and object modeling Cui et al.  [9].

Prior work dealing with the accuracy of ToF systems has uncovered complex error sources. Features of the environment, such as dark textures, sharp edges, foreground objects and distant bright objects, all introduce measurement errors in the obtained ranges. Although investigations into structured light camera accuracy are lacking, a similar complexity of the error sources can be expected. Thus, the evaluation of both ToF and structured light cameras in strictly controlled environments and engineered test scenarios may not properly reflect on their performance in a real world setting. It is therefore important to develop a procedure for a holistic evaluation of novel range sensing devices for the purposes of mobile robotics applications.

This article closely follows the evaluation methodology, presented in our prior work Stoyanov et al.  [10], but performs a more extensive data collection in varying environments and a more detailed subsequent analysis. Prior to the publication of Stoyanov et al.  [10], no similar comparisons had been reported in literature. A recent work by Wong et al.  [11], however, presents a thorough evaluation of several range sensors in an underground mine environment. The authors perform an analysis, based on the properties of surface patches, estimated in dense regions of the point cloud samples they evaluate. Although the comparison methodology Wong et al. use differs significantly from the one discussed in this work—it is based on the average variance of point samples from the extracted surface patches, there is a similarity in the overall strategy of evaluation without ground truth data.

In this article, we extend the methodology for spatial representation accuracy evaluation from our previous work Stoyanov et al.  [12] to perform a comparison of range sensor measurements. Three integrated 3D range sensors—the SR-4000 and Fotonic B70 ToF cameras and the Microsoft Kinect structured light camera are compared to a standard aLRF sensor. The proposed comparison can be used in cases when the environment is known well, but unlike other comparison methodologies can also operate without making any use of ground truth knowledge of the environment. A major role in the experiments performed is given to the scenario of scanning goods, stacked in a mock-up of a standardized logistics container. This application scenario is motivated by the currently on-going EU project “RobLog—Cognitive Robot for Automation of Logistic Processes”. In the context of the RobLog project, different stackable goods have to be autonomously perceived, recognized and subsequently unloaded from a standard container, in a reliable and scalable manner. As the success of operation of the RobLog system hinges in accurate geometric models, the significance of accurate and reliable dense range measurements is evident. Additionally, the evaluation presented in this work also compares the accuracy of the three sensors in an uncontrolled indoor environment. The experiments in a semi-structured indoor environment are similar to those already presented in our prior work Stoyanov et al.  [10] but contain a larger data set and fix a hardware problem with the panning actuator of the aLRF sensor.

This work proceeds with a description of the accuracy evaluation approach. Section  3 describes the test setup and evaluation environments considered. Section  4 presents an analysis of the obtained results, followed by a summary of the major contributions.

Section snippets

Accuracy evaluation

In order to objectively compare the performance of different 3D range sensors, a well-defined framework is needed. It is also important to define well-formed comparison criteria and obtain meaningful, easy-to-interpret statistical results. The main objective of this article is to propose a method to compare the accuracy of range measurements in an uncontrolled environment, closely related to the application scenario in which the sensors are to be used.

The information returned by 3D range

Evaluation methodology

Using the evaluation procedure proposed in Section  2.2, any number of sensors can be compared against a ground truth reference scan. In this work, we consider four different 3D range sensors, mounted on a mobile platform. The sensor setup and sample output point sets, used in our previous work Stoyanov et al.  [10], are shown in Fig. 2(a). For the purposes of this analysis, the setup was modified and placed on a mobile robot, as shown in Fig. 2(b).

In order to use the evaluation methodology

Results

As mentioned previously, two distinct sets of experiments were performed in the context of this work. In the first set of experiments (Section  4.1), tests were performed in environments with precisely known geometrical properties. In the second part (Section  4.2), evaluations were performed relative to the actuated LRF scanner—the sensor with the best accuracy record. Naturally, the range data collected for the purposes of the first evaluation is used also in the second part of this section.

Discussion

This article presented a comparative evaluation of three integrated 3D range cameras. A novel evaluation methodology, based on the results in spatial representation evaluation presented in  [12], was used to compare the outputs of the three cameras. The proposed approach offers easy to interpret statistics for the accuracy of different sensors, compared to a set of reference range observations. In simple environments, hand-crafted precise geometric models can be used to evaluate absolute sensor

Acknowledgments

This research received partial funding from the European Union Framework Program 7 project “Cognitive Robot for Automation of Logistic Processes (RobLog)” (grant number FP7-270350).

Todor Stoyanov received a Masters degree in Smart Systems from Jacobs University Bremen in 2006 and a Ph.D. in Computer Science from Örebro University in 2012. He is currently a postdoctoral researcher with the Center of Applied Autonomous Sensor Systems (AASS) at Örebro University. His main research interests are in 3D perception and safe autonomous navigation for mobile robots.

References (19)

  • T. Kahlmann, Range imaging metrology: investigation, calibration and development, Ph.D. Thesis, Technische Hochschule...
  • M. Lindner et al.

    Time-of-flight sensor calibration for accurate range sensing

    Computer Vision and Image Understanding

    (2010)
  • S. Fuchs, G. Hirzinger, Extrinsic and depth calibration of ToF-cameras, in: IEEE Computer Society Conference on...
  • F. Chiabrando et al.

    Sensors for 3D imaging: metric evaluation and calibration of a CCD/CMOS time-of-flight camera

    Sensors

    (2009)
  • S. May, B. Werner, H. Surmann, K. Pervolz, 3D time-of-flight cameras for mobile robotics, in: Proc. of IEEE/RSJ Int....
  • S. May, D. Dröschel, S. Fuchs, D. Holz, A. Nuchter, Robust 3D-mapping with time-of-flight cameras, in: Proc. of...
  • K. Pathak, A. Birk, J. Poppinga, Sub-pixel depth accuracy with a time of flight sensor using multimodal Gaussian...
  • D. Dröschel, D. Holz, J. Stückler, S. Behnke, Using time-of-flight cameras with active gaze control for 3D collision...
  • Y. Cui, S. Schuon, D. Chan, S. Thrun, C. Theobalt, 3D shape scanning with a time-of-flight camera, in: IEEE Computer...
There are more references available in the full text version of this article.

Cited by (34)

  • Towards transient experimental water surfaces: A new benchmark dataset for 2D shallow water solvers

    2018, Advances in Water Resources
    Citation Excerpt :

    Moreover, because of their extended use and their low-cost manufacturing, these devices suffer from technical issues such as low thermal stability (G. Guidi, 2016; Mankoff and Russo, 2013) although, arguably, the errors introduced may be acceptable for many applications. In contrast, it has been argued that ToF devices may suffer from motion artifacts (if there is indeed movement in the images), relatively low resolution, they are more expensive and require somewhat more power, setup and equipment to operate (Kahn et al., 2013; Langmann et al., 2012; Stoyanov et al., 2013). The obvious advantage of RGB-D sensors, compared to conventional stereo vision cameras and time-of-flight sensors, is clearly their very affordable cost, which makes them an attractive tool for many researchers despite the aforementioned limitations.

View all citing articles on Scopus

Todor Stoyanov received a Masters degree in Smart Systems from Jacobs University Bremen in 2006 and a Ph.D. in Computer Science from Örebro University in 2012. He is currently a postdoctoral researcher with the Center of Applied Autonomous Sensor Systems (AASS) at Örebro University. His main research interests are in 3D perception and safe autonomous navigation for mobile robots.

Rasoul Mojtahedzadeh graduated with a M.Sc. in Systems, Control and Robotics at the Royal Institute of Technology (KTH) in 2011. He is currently a Ph.D. student at the Örebro University, working on 3D perception, scene analysis and task planning for the European project Cognitive Robot for Automation of Logistic Processes (RobLog).

Henrik Andreasson received the Masters degree in mechatronics from the Royal Institute of Technology, Stockholm, in 2001 and his Ph.D. degree in computer science from Örebro University, Örebro in 2008. He is currently holds a lector position with the Mobile Robot and Olfaction lab at the Applied Autonomous Sensor System (AASS), Örebro, Sweden. His current research interests include mobile robotics, computer vision, and machine learning.

Achim J. Lilienthal is associate professor at AASS, Örebro University, Sweden, where he is leading the Mobile Robotics and Olfaction Lab. He is also research associate at the ESB Logistikfabrik, Reutlingen University, Germany. His main research interests are mobile robot olfaction, rich 3D perception, robot vision, and safe navigation for autonomous transport robots. Achim Lilienthal obtained his Ph.D. in computer science from Tübingen University, Germany and his M.Sc. and B.Sc. in Physics from the University of Konstanz, Germany. The Ph.D. Thesis addresses gas distribution mapping and gas source localisation with a mobile robot. The M.Sc. thesis is concerned with an investigation of the structure of (C60)n+ clusters using gas phase ion chromatography.

View full text