1 Introduction

The RoboCup@Work league was established in 2012 [1] to foster the development and benchmarking of robots in the industrial environment. The main focus of the league is to improve small but versatile robots capable of doing different tasks. Those robots are attractive not only to huge firms being able to afford many robots, but also to small companies. After introducing the league and tests performed in 2016, we present our major improvements in comparison to our status as of 2015 [2], which focused on task managing and failure handling.

2 The LUHbots

The LUHbots team was founded in 2012 at the Institute of Mechatronic Systems at Leibniz Universität Hannover it consists of Bachelor’s and Master’s students from mechanical engineering, computer science, and navigation and field robotics (see Fig. 1a). Most of the founding team members have participated in the research inspired practical lecture RobotChallenge [3]. Today, the team is part of the Hannover Centre for Mechatronics. In 2012, the LUHbots first competed in the RoboCup@Work challenge and were able to win the competition [4]. In 2013, the second place was achieved [5]. In 2015, the LUHbots won both the German Open and the RoboCup in Hefei [2]. This year, in 2016, the LUHbots won the RoboCup in Leipzig.

Fig. 1.
figure 1

RoboCup@Work in 2016

Full size image

3 The RoboCup@Work

In this section, we introduce the tests and major changes of the 2016 RoboCup @Work world championship. Nine teams participated at the World Cup in Leipzig (see Fig. 1b).

Changed Rules in Comparison to 2015: The arena size has been increased from 25 m\(^2\) to 60 m\(^2\). The arena has an entrance and exit area, marked with red-white barrier tape. It is entirely shut either by a wall or by yellow-black barrier tape (see Fig. 2). Different area heights of 0 cm, 5 cm, and 15 cm as well as shelves are available to grasp and place objects. The 0 cm areas are marked by the blue-white barrier tape.

Fig. 2.
figure 2

The RoboCup@Work arena in 2016 (Color figure online)

Full size image

The teams can no longer choose complexity levels. The new referee box randomly generates all task specifications. The number of manipulation objects has been raised to a total of 14 objects. In some tests, objects need to be placed in a red or blue container (see Fig. 3 left). A new obstacle type, the yellow-black barrier tape, is used to mark areas where the robot is not allowed to drive through. Additionally, the Conveyer Belt Test (CBT) was added.

Fig. 3.
figure 3

The picture on the left shows our robot searching for the blue container. The one on the right displays our map used for navigation. (Color figure online)

Full size image

Basic Navigation Test: The purpose of the Basic Navigation Test (BNT) is testing navigation in a static environment. The arena is initially known and can be mapped during a set-up phase (see Fig. 3 right). The task consists of reaching and completely covering a series of markers in a specified orientation without any collisions. Two static obstacles and one barrier tape obstacle are positioned in the arena.

Basic Manipulation Test: The Basic Manipulation Test (BMT) focuses on manipulation tasks. The objective is to successfully grasp five objects and place them on a nearby service area. In comparison to 2015, the teams can no longer choose the order, position, or rotation of the objects.

Basic Transportation Test: The Basic Transportation Test (BTT) combines manipulation and navigation. The referee box sends the task description to the robot. It includes information of start and end positions of the objects to be transported. The task order and the particular transport tasks have to be determined autonomously by the robot. After placing all objects, the robot has to leave the arena through the exit. The order, position, and rotation of the objects as well as the pick-up and place down areas are randomly determined.

Three different BTTs are tested. They differ in the following aspects: the area heights, whether obstacles are put in the arena or not, whether five or seven objects need to be transported, whether objects need to be placed into a blue or red container, and whether decoy objects lie on the areas or not.

Precision Placement Test: The Precision Placement Test (PPT) consists of transporting objects and placing them inside small cavities, which are only a few millimetres larger in “diameter” than the object itself. Initially unknown positions of the cavities increase the complexity.

Conveyer Belt Test: The Conveyer Belt Test (CBT) focuses on dynamic manipulation. The task is to successfully grasp three objects. Two variations of the CBT exist: objects either lie on a conveyer belt or on a rotating table. The objects move with a velocity in the range of 2.5 to 5 \(\mathrm {\frac{cm}{s}}\).

Final: The final is a combination of all the aforementioned tests. Ten objects need to be manipulated and transported. The task time for the final is ten minutes, while all other tests are only five minutes long.

4 Hardware

Our robot is based on the mobile robot KUKA youBot [6], see Fig. 4. Basically, the robot consists of a mobile platform with four Mecanum Wheels [7] and a five degrees-of-freedom (DoF) manipulator which has been remounted to increase the manipulation area [4].

Many improvements have been made this year to rectify the error susceptibility of the robot hardware and to enhance the performance of the robot. In the following chapter, the main changes are described.

Fig. 4.
figure 4

Hardware overview

Full size image

4.1 Computer Upgrade

Unfortunately, the stock built-in Intel Atom computer of the youBot does not provide enough computing power needed to run multiple processes like camera driver, object recognition, and navigation simultaneously. To meet the requirements, the internal PC has been replaced by a Mini-ITX mainboard with an Intel i7-4790T CPU. Moreover, the RAM is increased to 16 GB. For storage, a 256 GB SSD is used. The necessary power of the new PC can not be supplied by the 12 V/5 A output of the youBot power board. The new mainboard’s built-in power supply unit allows for an input between 9 V and 24 V. This way, the PC can be supplied from the 24 V output of the youBot power board. Additionally, the upgrade to USB 3.0 is needed to support the new camera (see 4.3).

4.2 Gripper

The gripper of the youBot has been redesigned. Now it is based on a 3D printed component. The usage of a belt drive allows for the reduction of weight and dimension of the gripper. The micro controller and the power regulator are located inside the 3D printed case of the gripper. Furthermore, a camera mount is included. Compared to the previous version [2], the gripper is powered by the youBot manipulator. Like the original youBot gripper it communicates indirectly over the internal EtherCAT bus of the manipulator. This way, there is no need for externally guided cables. The length and weight of the gripper are comparable to the stock gripper, but the gripping width has been improved significantly from 20 mm to 65 mm. Furthermore, the current feedback of the new servo motor allows for a grip check. Fin ray effect fingers are used for safety reasons. They are flexible and do not break in case of a collision.

4.3 Camera

We upgraded the vision system to the “Intel RealSense Camera SR300” (see Fig. 4b). It is a depth camera, just like the previously used “Creative Senz3D”, having an improved 3D and 2D image quality and a 1080p RGB resolution. Furthermore, the camera uses USB 3.0 instead of USB 2.0 for a higher data transfer. The quality of the USB extension cable is crucial to ensure that the camera keeps running. Therefore, the camera is connected to the internal PC via a high quality USB-cable to enable the high data transfer rate.

4.4 Transportation Plate on the Robot’s Backside

The transportation plate is designed to fit each of the manipulation objects of the RoboCup@Work league. The occurring vibrations while driving are used to slide the objects into their fittings. Even if the objects are not gripped perfectly, grasping errors of one to two centimetres can be resolved. Before taking an object from cargo, it is positioned in the middle of the gripper. Hence, it can be placed with high accuracy. This is crucial, for example during the precision placement test.

5 Approach

Generally satisfied with the results of our approach in 2015 [2], we updated our software to be able to work appropriately with the new changes. Here, we present the changes we have done to stay on top.

5.1 Overview

Our software architecture is based on ROS [8] (see Fig. 5). The yellow nodes (drivers) give access to the sensors. The youBot driver in red can be accessed via the youBot driver API node. The camera data is first processed by the vision node and then filtered and clustered by the observer node, which is triggered by the state machine. The laser scanners and barrier tape detection publish to the navigation stack. The watchdog filters the navigation commands. The task planner and the referee box connection communicate with the state machine. The laser scanner nodes are used unmodified. The standard ROS navigation stack is used, but the global planner has been replaced. As local planner the Dynamic Window Approach (DWA) [9] is used. The youBot driver API is heavily, and the camera driver is slightly modified. All other nodes have been entirely developed by the team.

Fig. 5.
figure 5

Overview of the software architecture

Full size image

5.2 Navigation

Last year, navigation was our primary concern. The ROS DWA local planner of December 2015 was superior to our local planner. Therefore, we changed to the ROS DWA.

Barrier Tape Detection: Three different barrier tapes (yellow-black, red-white and blue-white) have to be tracked. Navigating over a yellow-black barrier tape is counted as a collision. To prevent those collisions, we developed a barrier tape detection algorithm. This node detects the black-yellow tape and displays it on the cost map of the robot (see Fig. 6). To detect the tape, the 2D image of the camera is filtered in the HSV colourspace. The filter is light sensitive. Thus, the colour table needs to be retrained for each new location. Black is difficult to detect on the dark ground. Therefore, it is not tracked. A zero-height point cloud is calculated from the 2D image and the kinematics of the robot.

Fig. 6.
figure 6

Barrier detection (Color figure online)

Full size image

During the competition, we faced the problem of a reflective ground. This results in difficulties in the detection of the tape and the ground. We had to reduce the navigation speed to ensure that the robot can react appropriately.

5.3 Manipulation

The software development kit (SDK) we developed in 2014 still fit the necessary requirements for all tests in 2016. Furthermore, we released the code of the package in April this year. The code and documentation are uploaded to our GitHub:

https://github.com/LUHbots/luh_youbot_os.

https://github.com/LUHbots/luh_youbot_os/wiki.

https://github.com/LUHbots/luh_youbot_os/blob/master/luh_youbot_os_description.pdf.

5.4 Vision

To use the SR300 with ROS, we used the Intel librealsense package from Intel and a custom driver to publish the images in ROS.

Fig. 7.
figure 7

Current vision system

Full size image

Our vision system consists of three major parts (see Fig. 7). The first part is the vision system from last year, the second is the proposed new approach. All information is collected from a third component - the observer. The first approach mainly relies on the IR image of the camera and estimates a contour around an object in this image. Based on the contour, the node calculates the Hu moments and feeds them in a random forest to classify the objects. The debug output of this approach is shown in Fig. 8c.

Fig. 8.
figure 8

Subfigure (a) shows the points from the transformed point cloud belonging to a service area. The black areas are potential objects. In (b) the transformed contours are shown and in blue and green the bounding boxes of the objects. Subfigure (c) shows the results of our vision from last year. (Color figure online)

Full size image

The second component is our new approach, running parallel to the old one. Unlike the old approach, we do not classify, we only calculate features from found objects. In the first step, we transform the point cloud into a coordinate system, whose z axis is perpendicular to the floor. With this transformed point cloud, we run a plain segmentation with random sample consensus (RANSAC) and get a consensus set of points belonging to the area. The result of an exemplary calculation is shown in Fig. 8a. The points, which support our estimated service area, are shown in white. Based on the image we are looking for potential objects on the service area. Our potential objects are gaps in the point cloud of the service area. We estimate the contour of these holes and calculate the centre and the occupied area of this contour.

In the next step, the rotation of the object and the mean colour are calculated based on the RGB image. All calculated features and the name and probability of the classified object are sent to the observer.

The vision runs with 10 Hz. To enhance the object detection we removed the frequency limit of the node.

Observer: Since our previous approach suffered from a lot of misclassifications, we implemented an observer, which clusters the objects, their probabilities, and types. The clustering is done with a density-based spatial clustering of applications with noise (DBSCAN) [10] algorithm clustering based on the x-y-z coordinates of the objects and calculates a mean centre, name, and probability.

We extended the observer in order to be able to perform additional plausibility tests. Therefore, it receives additional features from the vision like the colour and the volume of the object. Thanks to this information it can differentiate hard objects, like the F20_20_G and F20_20_B objects, which can be easily mistaken using our approach we took last year. Furthermore, the observer uses a confusion matrix shown and explained in Fig. 9.

Fig. 9.
figure 9

This figure shows the confusion matrix of our old vision approach (b) compared to our new vision approach (a). It can be seen, that the old vision often mistakes the object AXIS with S40_40_B, but with our new approach the confusion matrix demonstrates a drastic decreasing of the number of misclassifications. (Color figure online)

Full size image

5.5 Testing

We implemented a vision testing framework, which loads our vision and an annotated rosbag. We run the vision and evaluate misclassified objects. With this information, we built a confusion matrix (see Fig. 9), which is fed back to the observer. Hence, it can take this information into account when receiving a classified object.

The state machine is based on SMACH [11], a Python library. It offers introspection and fast development. Python uses just in time compiling. Therefore, a line of code is interpreted at runtime and can crash for syntax errors. Because of various recovery behaviours, a probabilistic task planning and various combinations of tasks, we decided to design automated tests. These tests can be performed on any development PC or on a central server. This server uses Jenkins to schedule the different tests. Since the primary focus is to test the python code and the task planning, the manipulation and navigation is replaced by mocked action servers. The action servers would randomly fail and therefore allow the behaviour to deal with these possible failures. The new version of the referee box allows for random task generation. The Jenkins server schedules several tests per day. After a random task is generated, the behaviour uses the mocked servers to simulate performing the task. All actions are logged and analysed. At the end of the simulation, the points which would have been awarded are calculated. In preparation for the RoboCup event, more than ten thousand simulations were run.

To further increase the testing capabilities, Gazebo is used to simulate navigation. The tests are extended and time keeping is introduced. Using preconfigured values for the manipulation time, it is possible to estimate if the speed is sufficient. The Gazebo youBot model is extended to allow for the detection of collisions and include them into scoring. Since the simulation in Gazebo is computationally expensive, only a few thousand simulations were run.

Simulation: The Gazebo robot simulation is used for an efficient parameter improvement and error localisation of the existing software. The combination of Gazebo and Jenkins is beneficial: the start, end, and data acquisition are automated. Therefore, a big amount of data is collected, that can be used for optimisation. Changes in the current state of the software are refreshed periodically and self-directed. Hence, a problem in new code is found in a short amount of time and without the use of various resources. The optimisation of the navigation is possible if the movement model of the path planning is implemented. A laser scanner extension is available. Standard laser scanner extensions in Gazebo can be adjusted individually. For an automatic evaluation, a collision detector in Gazebo is used.

6 Results

The results of the performed tests are summarised in Table 1.

Table 1. Results of the RoboCup@Work competition in 2016
Full size table

7 Conclusion

This year, rule changes troubled all teams. Besides a referee box and barrier obstacles, new objects resulted in many changes in perception and the manipulation state machine. Although we did not grasp misclassified objects, we had a number of failed grasps. These grasps were mainly due to shadows and reflections but were all handled by grip checks.

Last year, our focus layed on robustness. This year we further increased the robustness. Because of the new tests, which included more actions (e.g. objects to be transported) in less time, speed became more important. During the competition we improved our vision to reduce the number of failed grasps, which resulted in higher CPU load. Even though the frequencies of the nodes were still met and no warnings occurred, the performance was very slow. In two tests, hardware failures lead to a restart. Because of the speed we were unable compensate the penalty.

Our approach aimed to reach all possible points. The majority of the test runs were successful. However, the high number of different possibilities for task specifications did not allow for testing all combinations. In the final run we achieved zero points due to a collision of the manipulator with the wall.

8 Future Work

The vision parameter has to be set individually for every test because of the impact of changing light conditions. We want to improve the stability of the vision. Furthermore, till next year we want to merge the state machine with the manipulation state machine into one C++ state machine.