Abstract
In this paper, we describe the hardware and software components of a fully autonomous prototype delivery vehicle. Equipped with a robotic arm, the demonstrator is capable of delivering packages and picking up new ones by interacting with custom-made delivery boxes. As highly accurate positioning w. r. t. a box is required for successful handover of packages, we track the pose (position and orientation) of the box using a high-resolution on-board camera. The resulting estimate is relayed to our planning and control modules, which ensure that the vehicle reaches its required pose with centimeter-level accuracy.
In order to deliver packages, the car needs to autonomously navigate our test facility, avoiding static and dynamic obstacles while obeying simple traffic rules. As one focus is on the practical challenges encountered when building a prototype, we cover issues ranging from sensor calibration and system identification to perception, planning, control, and the implementation of high-level behaviors. While some of the proposed solutions to these problems are not necessarily novel, they allowed us to demonstrate the vehicle’s capabilities after a development phase of less than 12 months.
Zusammenfassung
In diesem Beitrag beschreiben wir die Hard- und Softwarekomponenten eines vollständig autonomen prototypischen Lieferfahrzeugs. Ausgestattet mit einem Roboterarm, ist dieser Demonstrator in der Lage, Pakete an eigens hierfür gefertigte Postkästen auszuliefern sowie von dort aufzunehmen. Da die erfolgreiche Paketübergabe eine hochpräzise Positionierung relativ zu einem solchen Kasten erfordert, verfolgen wir dessen sog. Pose (Position und Orientierung) mit Hilfe einer hochauflösenden On-Board-Kamera. Das Schätzergebnis wird an unsere Planungs- und Regelungsmodule weitergeleitet, welche dafür sorgen, dass das Fahrzeug seine gewünschte Position zentimetergenau erreicht.
Um Pakete ausliefern zu können, muss das Fahrzeug autonom durch unser Versuchsgelände navigieren, statischen und dynamischen Hindernissen ausweichen und dabei einfache Verkehrsregeln beachten. Da unser Hauptaugenmerk unter anderem auf den praktischen Herausforderungen beim Aufbau eines Prototypen liegt, erstrecken sich die hier angesprochenen Themen von der Sensorkalibrierung und Systemidentifikation über die Wahrnehmung, Planung und Regelung bis hin zur Verhaltensgenerierung. Wenngleich einige der aufgezeigten Problemlösungen nicht unbedingt neuartig sind, so erlaubten sie es uns dennoch, die Leistungsfähigkeit des Systems nach einer Entwicklungszeit von weniger als zwölf Monaten erfolgreich zu demonstrieren.
About the authors
Benjamin C. Heinrich studied engineering cybernetics (biocybernetics) at the University of Stuttgart and the KTH Stockholm. At Ericsson AB, Kista, he worked on dead-time compensation and optimization for mobile networks. Since 2013 he has been researching autonomous driving at the University of the Bundeswehr Munich. His research interests include model predictive control and dead-time compensation.
Thorsten Luettel studied electrical engineering (mechatronics) at the Leibniz Universität Hannover. Since 2006 he has been researching autonomous driving at the University of the Bundeswehr Munich. Recently, he has also led ‘Team MuCAR’ during successful international competitions. His research interests include sensor data fusion and system integration.
Dennis Fassbender studied computer science at the University of Koblenz-Landau. Since 2012 he has been researching autonomous driving at the University of the Bundeswehr Munich. His research interests include trajectory planing, navigation and behavior.
Patrick Burger studied electrical engineering and information technology (robotics & navigation) at the Technical University of Munich. Since 2015 he has been researching autonomous driving at the University of the Bundeswehr Munich. His research interests include SLAM, LiDAR point-cloud segmentation and feature extraction.
Felix Ebert studied electrical engineering and information technology (automation technology) at the Technical University of Munich. Since 2014 he has been researching autonomous driving at the University of the Bundeswehr Munich. His research interests include object relative navigation.
Michael Himmelsbach studied computer science at Humboldt University Berlin. From 2007 to 2015 he has been researching autonomous driving at the University of the Bundeswehr Munich. His research interests include LIDAR environment and object perception with focus on machine learning and pattern recognition techniques. Since 2016 he has been working for a large German automobile company in the field of highly automated driving.
Hanno Jaspers studied applied computer science (robotics) at the TU Dortmund. From 2012 to 2017 he has been researching autonomous driving at the University of the Bundeswehr Munich. His research interests include visual environment perception for autonomous vehicles. Since 2018 he has been working for a large German automobile company in the field of highly automated driving.
Michael Kusenbach studied computer science at University of Koblenz-Landau. Since 2013 he has been researching autonomous driving at the University of the Bundeswehr Munich. His research interests include LiDAR data processing with a focus on object detection and classification.
Georg R. Müller studied mechanical engineering (microtechnology & control technology) at the Technical University of Munich. Since 2012 he has been researching autonomous driving at the University of the Bundeswehr Munich. His research interests include online camera calibration.
Benjamin Naujoks studied technomathematics (scientific computing & optimization) at the Technical University of Dresden. Since 2015 he has been researching autonomous driving at the University of the Bundeswehr Munich. His research interests include probabilistic filter algorithmic, object-detection, especially in LiDAR point-clouds, parallelization with CUDA and machine-learning algorithms.
Felix Orth studied mechanical and process engineering (mechatronics) at the Otto von Guericke University of Magdeburg. During his studies he worked for the Formula Student team UMD-Racing of the University of Magdeburg. After his Diploma, he was employed as a developer and race engineer at Motopark in Oschersleben. Since 2015 Felix is part of the department Autonomous Logistic at StreetScooter in Aachen.
Fabian Schmitt studied mechanical engineering at the RWTH Aachen University. After his Diploma, he worked as research assistant at the Institute for Automotive Engineering of the RWTH. In 2010 he was employed at Streetscooter GmbH as chief vehicle engineer. Since 2015 he is CTO there.
Hans-Joachim Wuensche got his PhD from University of the Bundeswehr Munich in 1987 with Ernst D. Dickmanns, where he co-developed the 4D-approach to computer vision. After many years in management, he returned to the same University to lead the Institute for Autonomous Systems Technology in 2004. His research interests include autonomous robots, especially on- and off-road vehicles exploring and navigating unknown terrain.
References
1. Q. Sun, H. Liu, Q. Yang, and W. Yan, “On the Design for AGVs: Modeling, Path Planning and Localization,” in Proceedings of IEEE International Conference on Mechatronics and Automation (ICMA), Beijing, China, Aug. 2011, pp. 1515–1520.10.1109/ICMA.2011.5985974Search in Google Scholar
2. H. Andreasson, J. Saarinen, M. Cirillo, T. Stoyanov, and A. J. Lilienthal, “Fast, Continuous State Path Smoothing to Improve Navigation Accuracy,” in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA, May 2015, pp. 662–669.10.1109/ICRA.2015.7139250Search in Google Scholar
3. R. Simmons, R. Goodwin, K. Z. Haigh, S. Koenig, and J. O’Sullivan, “A Layered Architecture for Office Delivery Robots,” in Proceedings of First International Conference on Autonomous Agents. ACM, 1997, pp. 245–252.10.1145/267658.267723Search in Google Scholar
4. J. M. Evans, “HelpMate: An Autonomous Mobile Robot Courier for Hospitals,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. 3, Sep. 1994, pp. 1695–1700.Search in Google Scholar
5. J. Bezos, “First-ever #AmazonPrimeAir customer delivery is in the books. …,” 2016, https://twitter.com/JeffBezos/status/809034847121350657.Search in Google Scholar
6. J. O’Malley, “Pirates of the skies,” Engineering & Technology, vol. 12, no. 3, pp. 32–35, 2017.10.1049/et.2017.0302Search in Google Scholar
7. B. Arbanas, A. Ivanovic, M. Car, T. Haus, M. Orsag, T. Petrovic, and S. Bogdan, “Aerial-ground Robotic System for Autonomous Delivery Tasks,” in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, May 2016, pp. 5463–5468.10.1109/ICRA.2016.7487759Search in Google Scholar
8. J. Ziegler, P. Bender, M. Schreiber, H. Lategahn, T. Strauss, C. Stiller, T. Dang, U. Franke, N. Appenrodt, C. G. Keller et al., “Making Bertha Drive – An Autonomous Journey on a Historic Route,” IEEE Intelligent Transportation Systems Magazine, vol. 6, no. 2, pp. 8–20, 2014.10.1109/MITS.2014.2306552Search in Google Scholar
9. P. Petrov, C. Boussard, S. Ammoun, and F. Nashashibi, “A Hybrid Control for Automatic Docking of Electric Vehicles for Recharging,” in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), St. Paul, MN, USA, May 2012, pp. 2966–2971.10.1109/ICRA.2012.6225087Search in Google Scholar
10. E. D. Dickmanns, B. Mysliwetz, and T. Christians, “An Integrated Spatio-Temporal Approach to Automatic Visual Guidance of Autonomous Vehicles,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 20, no. 6, pp. 1273–1284, Dec. 1990.10.1109/21.61200Search in Google Scholar
11. T. Luettel, F. Ebert, D. Fassbender, C. Fries, B. C. Heinrich, H. Jaspers, J. Kallwies, B. Naujoks, and H.-J. Wuensche, “Autonomous driving functions reducing the driver’s cognitive load in off-road scenarios (Assistenz- und Autonomiefunktionen zur Entlastung des Fahrers in Offroad-Szenarien),” at - Automatisierungstechnik, vol. 65, no. 7, 2017.10.1515/auto-2017-0013Search in Google Scholar
12. D. Fassbender, A. Mueller, and H.-J. Wuensche, “Trajectory Planning for Car-Like Robots in Unknown, Unstructured Environments,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Chicago, IL, USA, Sep. 2014.10.1109/IROS.2014.6943071Search in Google Scholar
13. F. Ebert, D. Fassbender, B. Naujoks, and H.-J. Wuensche, “Robust Long-Range Teach-and-Repeat in Non-Urban Environments,” in Proceedings of IEEE Intelligent Transportation Systems Conference (ITSC), Yokohama, Japan, Oct. 2017.10.1109/ITSC.2017.8317767Search in Google Scholar
14. C. Fries, P. Burger, J. Kallwies, B. Naujoks, T. Luettel, and H.-J. Wuensche, “How MuCAR Won the Convoy Scenario at ELROB 2016,” in Proceedings of IEEE Intelligent Transportation Systems Conference (ITSC), Yokohama, Japan, Oct. 2017.10.1109/ITSC.2017.8317711Search in Google Scholar
15. ELROB.ORG, “M-ELROB 2016 Official Results,” 2016, http://www.elrob.org/elrob-2016-results.Search in Google Scholar
16. E. D. Dickmanns, Dynamic Vision for Perception and Control of Motion. Springer Verlag, 2007.Search in Google Scholar
17. F. Orth, F. Schmitt, M. Sommer, and U. Kessels, “Delivery Vehicle and Method for Delivery of Consignments to Different Sites Along a Delivery Route,” Patent US020 170 291 766 A1, 2017. [Online]. Available: https://depatisnet.dpma.de/DepatisNet/depatisnet?action=bibdat&docid=US020170291766A1.Search in Google Scholar
18. B. C. Heinrich, T. Luettel, and H.-J. Wuensche, “A New Control Architecture for MuCAR,” in Proceedings of IEEE Intelligent Vehicles Symposium (IV), Redondo Beach, CA, USA, Jun. 2017.10.1109/IVS.2017.7995976Search in Google Scholar
19. M. Goebl and G. Färber, “A Real-Time-capable Hard-and Software Architecture for Joint Image and Knowledge Processing in Cognitive Automobiles,” in Proceedings of IEEE Intelligent Vehicles Symposium (IV), Istanbul, Turkey, Jun. 2007.10.1109/IVS.2007.4290204Search in Google Scholar
20. D. Schramm, M. Hiller, and R. Bardini, Vehicle Dynamics, 1st ed. Berlin Heidelberg: Springer, 2014.10.1007/978-3-540-36045-2_1Search in Google Scholar
21. Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330–1334, 2000.10.1109/34.888718Search in Google Scholar
22. S. Remy, M. Dhome, J. M. Lavest, and N. Daucher, “Hand-Eye Calibration,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. 2, Grenoble, France, Sep. 1997, pp. 1057–1065.Search in Google Scholar
23. E. Olson, “AprilTag: A robust and flexible visual fiducial system,” in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), May 2011, pp. 3400–3407.10.1109/ICRA.2011.5979561Search in Google Scholar
24. Y. Chen and G. Medioni, “Object Modelling by Registration of Multiple Range Images,” Image and Vision Computing, vol. 10, no. 3, pp. 145–155, 1992.10.1016/0262-8856(92)90066-CSearch in Google Scholar
25. Z. Zhang, “Iterative Point Matching for Registration of Free-Form Curves and Surfaces,” International Journal of Computer Vision, vol. 13, pp. 119–152, Oct. 1994.10.1007/BF01427149Search in Google Scholar
26. A. Segal, D. Haehnel, and S. Thrun, “Generalized-ICP,” in Proceedings of Robotics: Science and Systems, Seattle, WA, USA, Jun. 2009.10.15607/RSS.2009.V.021Search in Google Scholar
27. OpenStreetMap contributors, 2016, https://wiki.openstreetmap.org.Search in Google Scholar
28. P. Bender, J. Ziegler, and C. Stiller, “Lanelets: Efficient Map Representation for Autonomous Driving,” Proceedings of IEEE Intelligent Vehicles Symposium (IV), pp. 420–425, Jun. 2014.10.1109/IVS.2014.6856487Search in Google Scholar
29. H. Jaspers, M. Himmelsbach, and H.-J. Wuensche, “Multi-modal Local Terrain Maps from Vision and LiDAR,” in Proceedings of IEEE Intelligent Vehicles Symposium (IV), Redondo Beach, CA, USA, Jun. 2017.10.1109/IVS.2017.7995863Search in Google Scholar
30. A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard, “OctoMap: An Efficient Probabilistic 3D Mapping Framework Based on Octrees,” Autonomous Robots, vol. 34, pp. 189–206, 2013, software available at http://octomap.github.com.10.1007/s10514-012-9321-0Search in Google Scholar
31. M. Himmelsbach, F. von Hundelshausen, and H.-J. Wuensche, “Fast Segmentation of 3D Point Clouds for Ground Vehicles,” in Proceedings of IEEE Intelligent Vehicles Symposium (IV), San Diego, CA, USA, Jun. 2010.10.1109/IVS.2010.5548059Search in Google Scholar
32. C. Urmson, J. Anhalt, D. Bagnell, C. Baker, R. Bittner, M. N. Clark, J. Dolan, D. Duggins, T. Galatali, C. Geyer, M. Gittleman, S. Harbaugh, M. Hebert, T. M. Howard, S. Kolski, A. Kelly, M. Likhachev, M. McNaughton, N. Miller, K. Peterson, B. Pilnick, R. Rajkumar, P. Rybski, B. Salesky, Y.-W. Seo, S. Singh, J. Snider, A. Stentz, W. Whittaker, Z. Wolkowicki, J. Ziglar, H. Bae, T. Brown, D. Demitrish, B. Litkouhi, J. Nickolaou, V. Sadekar, W. Zhang, J. Struble, M. Taylor, M. Darms, and D. Ferguson, “Autonomous Driving in Urban Environments: Boss and the Urban Challenge,” Journal of Field Robotics, vol. 25, no. 8, pp. 425–466, 2008.10.1002/rob.20255Search in Google Scholar
33. Y. Bar-Shalom, T. Kirubarajan, and X.-R. Li, Estimation with Applications to Tracking and Navigation. New York, NY, USA: John Wiley & Sons, Inc., 2002.10.1002/0471221279Search in Google Scholar
34. T. Gindele, D. Jagszent, B. Pitzer, and R. Dillmann, “Design of the planner of Team AnnieWAY’s autonomous vehicle used in the DARPA Urban Challenge 2007,” in Proceedings of IEEE Intelligent Vehicles Symposium (IV), Eindhoven, The Netherlands, Jun. 2008, pp. 1131–1136.10.1109/IVS.2008.4621268Search in Google Scholar
35. D. Ferguson, T. M. Howard, and M. Likhachev, “Motion Planning in Urban Environments,” Journal of Field Robotics, vol. 25, no. 11–12, pp. 939–960, 2008.10.1002/rob.20265Search in Google Scholar
36. M. McNaughton, C. Urmson, J. M. Dolan, and J.-W. Lee, “Motion Planning for Autonomous Driving with a Conformal Spatiotemporal Lattice,” in Proceedings of IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, 2011, pp. 4889–4895.10.1109/ICRA.2011.5980223Search in Google Scholar
37. U. Schwesinger, M. Rufli, P. Furgale, and R. Siegwart, “A Sampling-Based Partial Motion Planning Framework for System-Compliant Navigation along a Reference Path,” in Proceedings of IEEE Intelligent Vehicles Symposium (IV), Gold Coast, QLD, Australia, 2013, pp. 391–396.10.1109/IVS.2013.6629500Search in Google Scholar
38. D. Dolgov, S. Thrun, M. Montemerlo, and J. Diebel, “Path Planning for Autonomous Vehicles in Unknown Semi-structured Environments,” The International Journal of Robotics Research, vol. 29, no. 5, pp. 485–501, 2010.10.1177/0278364909359210Search in Google Scholar
39. D. Fassbender, B. C. Heinrich, and H.-J. Wuensche, “Motion Planning for Autonomous Vehicles in Highly Constrained Urban Environments,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, South Korea, Oct. 2016.10.1109/IROS.2016.7759692Search in Google Scholar
40. J. M. Snider, “Automatic Steering Methods for Autonomous Automobile Path Tracking,” Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA, Tech. Rep. CMU-RI-TR-09-08, Feb. 2009.Search in Google Scholar
41. B. C. Heinrich, D. Fassbender, and H.-J. Wuensche, “Precise Object-Relative Positioning for Car-Like Robots,” in Proceedings of IEEE Intelligent Transportation Systems Conference (ITSC), Rio de Janeiro, Brazil, Nov. 2016, pp. 1720–1726.10.1109/ITSC.2016.7795790Search in Google Scholar
42. M. Zeitz, “Differential Flatness: A Useful Method also for Linear SISO Systems (Differenzielle Flachheit: Eine nützliche Methodik auch für lineare SISO-Systeme),” at - Automatisierungstechnik, vol. 58, no. 1, pp. 5–14, 2010.10.1524/auto.2010.0815Search in Google Scholar
43. G. E. Alefeld, F. A. Potra, and Y. Shi, “Algorithm 748: Enclosing Zeros of Continuous Functions,” ACM Transactions on Mathematical Software, vol. 21, no. 3, pp. 327–344, Sep. 1995.10.1145/210089.210111Search in Google Scholar
44. R. Goldman, “Curvature Formulas for Implicit Curves and Surfaces,” Computer Aided Geometric Design, vol. 22, no. 7, pp. 632–658, Oct. 2005.10.1016/j.cagd.2005.06.005Search in Google Scholar
45. H. K. Khalil, Nonlinear Systems, 3rd ed. Upper Saddle River, NJ, USA: Prentice Hall, 1996.Search in Google Scholar
46. D. Fassbender, B. C. Heinrich, T. Luettel, and H.-J. Wuensche, “An Optimization Approach to Trajectory Generation for Autonomous Vehicle Following,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada, Sep. 2017.10.1109/IROS.2017.8206213Search in Google Scholar
47. M. A. Fischler and R. C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,” Communications of the ACM, vol. 24, no. 6, pp. 381–395, 1981.10.1145/358669.358692Search in Google Scholar
48. H. Jaspers, G. R. Mueller, and H.-J. Wuensche, “High Accuracy Model-Based Object Pose Estimation for Autonomous Recharging Applications,” in Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Lake Placid, NY, USA, Mar. 2016.10.1109/WACV.2016.7477635Search in Google Scholar
49. C. Fries and H.-J. Wuensche, “Real-time Unsupervised Feature Model Generation for a Vehicle Following System,” in Proceedings of IEEE Intelligent Transportation Systems Conference (ITSC), Rio de Janeiro, Brazil, Nov. 2016.10.1109/ITSC.2016.7795950Search in Google Scholar
50. M. Kusenbach, M. Himmelsbach, and H.-J. Wuensche, “A New Geometric 3D LiDAR Feature for Model Creation and Classification of Moving Objects,” in Proceedings of IEEE Intelligent Vehicles Symposium (IV), Gothenburg, Sweden, Jun. 2016, pp. 272–278.10.1109/IVS.2016.7535397Search in Google Scholar
51. A. Elfes, “Using Occupancy Grids for Mobile Robot Perception and Navigation,” Computer, vol. 22, no. 6, pp. 46–57, Jun. 1989.10.1109/2.30720Search in Google Scholar
Article note
On our website (http://www.mucar3.de/at2018-delivery) we present videos of the system described in this paper.
© 2018 Walter de Gruyter GmbH, Berlin/Boston
