Abstract
This article investigates methods for achieving more general manipulation capabilities for mobile manipulation platforms, which produce legible behavior in human living environments. To achieve generality and legibility, we combine two control mechanisms. First of all, experience- and observation-based learning of skills is applied to routine tasks, so that the repetitive and stereotypical character of everyday activity is exploited. Second, we use planning, reasoning, and search for novel tasks which have no stereotypical solution. We apply these ideas to the learning and use of action-related places, to the model-based visual recognition and localization of objects, and the learning and application of reaching strategies and motions from humans. We demonstrate the integration of these mechanisms into a single low-level control system for autonomous manipulation platforms.
Similar content being viewed by others
References
Albu-Schaffer, A., Haddadin, S., Ott, Ch., Stemmer, A., Wimbock, T., & Hirzinger, G. (2007). The dlr lightweight robot: design and control concepts for robots in human environments. Industrial Robot: An International Journal, 34(5), 376–385.
Asfour, T., Regenstein, K., Azad, P., Schroder, J., Bierbaum, A., Vahrenkamp, N., & Dillmann, R. (2006). Armar-iii: An integrated humanoid platform for sensory-motor control. In Proceedings of the IEEERAS/RSJ international conference on humanoid robots (Humanoids06) (pp. 169–175).
Beetz, M., Stulp, F., Radig, B., Bandouch, J., Blodow, N., Dolha, M., Fedrizzi, A., Jain, D., Klank, U., Kresse, I., Maldonado, A., Marton, Z., Mösenlechner, L., Ruiz, F., Rusu, R. B., & Tenorth, M. (2008). The assistive kitchen—a demonstration scenario for cognitive technical systems. In IEEE 17th international symposium on robot and human interactive communication (RO-MAN), Muenchen, Germany. Invited paper.
Belker, T. (2004). Plan projection, execution, and learning for mobile robot control. PhD thesis, Department of Applied Computer Science, University of Bonn.
Berenson, D., Diankov, R., Nishiwaki, K., Kagami, S., & Kuffner, J. (2007). Grasp planning in complex scenes. In IEEE-RAS international conference on humanoid robots.
Berenson, D., Srinivasa, S., Ferguson, D., Romea, A. C., & Kuffner, J. (2009). Manipulation planning with workspace goal regions. In Proceedings of the IEEE international conference on robotics and automation (ICRA).
Billard, A., Calinon, S., Dillmann, R., & Schaal, S. (2008). Robot programming by demonstration. In Springer handbook of robotics. Berlin: Springer. Chap. 59.
Borst, C., Fischer, M., Haidacher, S., Liu, H., & Hirzinger, G. (2003). Dlr hand ii: Experiments and experiences with an anthropomorphic hand. In 2003 IEEE international conference on robotics and automation (ICRA), Taipei, Taiwan, 14–19 September 2003.
Calinon, S., Guenter, F., & Billard, A. (2007). On learning, representing and generalizing a task in a humanoid robot. IEEE Transactions on Systems, Man and Cybernetics, 37(2), 286–298. Special issue on robot learning by observation, demonstration and imitation.
Cheng, G., Hyon, S., Morimoto, J., Ude, A., Hale, J. G., Colvin, G., Scroggin, W., & Jacobsen, S. C. (2007). Cb: A humanoid research platform for exploring neuroscience. Journal of Advanced Robotics, 21(10), 1097–1114.
Dearden, A., & Demiris, Y. (2005). Learning forward models for robotics. In Proceedings of the nineteenth international joint conference on artificial Intelligence (IJCAI) (pp. 1440–1445).
Fedrizzi, A., Moesenlechner, L., Stulp, F., & Beetz, M. (2009). Transformational planning for mobile manipulation based on action-related places. In Proceedings of the international conference on advanced robotics (ICAR).
Flash, T., & Hochner, B. (2005). Motor primitives in vertebrates and invertebrates. Current Opinion in Neurobiology, 15, 660–666.
Fox, M., Gough, J., & Long, D. (2006). Using learned action models in execution monitoring. In Proceedings of UK planning and scheduling SIG.
Gerkey, B., Vaughan, R. T., & Howard, A. (2003). The Player/Stage Project: Tools for multi-robot and distributed sensor systems. In Proceedings of the 11th international conference on advanced robotics (ICAR) (pp. 317–323).
Haigh, K. Z. (1998). Situation-dependent learning for interleaved planning and robot execution. Ph.D. thesis, School of Computer Science, Carnegie Mellon University.
Harris, C. M., & Wolpert, D. M. (1998). Signal-dependent noise determines motor planning. Nature, 394(20), 780–784.
Hoffmann, H., Pastor, P., Park, D.-H., & Schaal, S. (2009). Biologically-inspired dynamical systems for movement generation: Automatic real-time goal adaptation and obstacle avoidance. In IEEE international conference on robotics and automation.
Hofhauser, A., Steger, C., & Navab, N. (2008). Harmonic deformation model for edge based template matching. In Third international conference on computer vision theory and applications (Vol. 2, pp. 75–82).
Horswill, I. (1996). Integrated systems and naturalistic tasks. In Strategic directions in computing research AI working group.
Ijspeert, A. J., Nakanishi, J., & Schaal, S. (2002). Movement imitation with nonlinear dynamical systems in humanoid robots. In International conference on robotics and automation (ICRA2002).
Jenkins, O., Bodenheimer, R., & Peters, R. (2006). Manipulation manifolds: Explorations into uncovering manifolds in sensory-motor spaces. In International conference on development and learning (ICDL).
Julier, S., & Uhlmann, J. K. (1996). A general method for approximating nonlinear transformations of probability distributions (Technical report). Dept. of Engineering Science, University of Oxford.
Katz, D., Horrell, E., Yang, Y., Burns, B., Buckley, T., Grishkan, A., Zhylkovskyy, V., Brock, O., & Miller, E. (2006). The UMass mobile manipulator UMan: an experimental platform for autonomous mobile manipulation. In Workshop on manipulation in human environments at robotics: science and systems.
Kemp, C., Edsinger, A., & Torres-Jara, E. (2007). Challenges for robot manipulation in human environments. IEEE Robotics and Automation Magazine, 14(1), 20–29.
Klank, U., Zia, M. Z., & Beetz, M. (2009). 3D model selection from an Internet database for robotic vision. In International conference on robotics and automation (ICRA).
Kragic, D., Björkman, M., Christensen, H. I., & Eklundh, J. O. (2005). Vision for robotic object manipulation in domestic settings. Robotics and Autonomous Systems, 52(1), 85–100.
LaValle, S. M. (2006). Planning algorithms. Cambridge: Cambridge University Press.
Lepetit, V., & Fua, P. (2006). Keypoint recognition using randomized trees. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(9), 1465–1479.
Maciejewski, A. A., & Klein, C. A. (1989). The singular value decomposition: Computation and applications to robotics. International Journal of Robotics Research, 8(6), 63–79.
McDermott, D. (1991). A reactive plan language (Research Report YALEU/DCS/RR-864). Yale University.
Metta, G., Fitzpatrick, P., & Natale, L. (2006). YARP: Yet Another Robot Platform. International Journal of Advanced Robotics Systems, 3(1). Special issue on Software Development and Integration in Robotics.
Okada, K., Kojima, M., Sagawa, Y., Ichino, T., Sato, K., & Inaba, M. (2006). Vision based behavior verification system of humanoid robot for daily environment tasks. In Proceedings of the 6th IEEE-RAS international conference on humanoid robots (humanoids) (pp. 7–12).
Osada, R., Funkhouser, T., Chazelle, B., & Dobkin, D. (2002). Shape distributions. ACM Transactions on Graphics (TOG), 21(4), 807–832.
Oztop, E., Franklin, D. W., Chaminade, T., & Cheng, G. (2005). Human-humanoid interaction: Is a humanoid robot perceived as a human? International Journal of Humanoid Robotics, 2(4), 537–559.
Park, D.-H., Hoffmann, H., & Schaal, S. (2008). Movement reproduction and obstacle avoidance with dynamic movement primitives and potential fields. In International conference on humanoid robots.
Petrovskaya, A., & Ng, A. Y. (2007). Probabilistic mobile manipulation in dynamic environments, with application to opening doors. In International joint conference on artificial intelligence (IJCAI-07).
Roduit, P., Martinoli, A., & Jacot, J. (2007). A quantitative method for comparing trajectories of mobile robots using point distribution models. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 2441–2448).
Rosenbaum, D. A., Cohen, R. G., Meulenbroek, R. G. J., & Vaughan, J. (2006). Plans for grasping objects. In M. L. Latash & F. Lestienne (Eds.), Motor control and learning (pp. 9–25). New York: Springer.
Roweis, S., & Saul, L. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500), 2323–2326.
Rusu, R. B., Marton, Z. C., Blodow, N., Dolha, M., & Beetz, M. (2008). Towards 3D point cloud based object maps for household environments. Robotics and Autonomous Systems Journal. (Special issue on Semantic Knowledge).
Saxena, A., Driemeyer, J., & Ng, A. Y. (2008). Robotic grasping of novel objects using vision. The International Journal of Robotics Research, 27(2), 157.
Schaal, S. (1999). Is imitation learning the route to humanoid robots? Trends in Cognitive Sciences, 3(6), 233–242.
Smits, R., De Laet, T., Claes, K., Soetens, P., De Schutter, J., & Bruyninckx, H. (2008). Orocos: A software framework for complex sensor-driven robot tasks. IEEE Robotics and Automation Magazine.
Srinivasa, S., Ferguson, D., Weghe, M. V., Diankov, R., Berenson, D., Helfrich, C., & Strasdat, H. (2008). The robotic busboy: Steps towards developing a mobile robotic home assistant. In International conference on intelligent autonomous systems.
Stulp, F., & Beetz, M. (2008). Refining the execution of abstract actions with learned action models. Journal of Artificial Intelligence Research (JAIR), 32, June 2008.
Stulp, F., Koska, W., Maldonado, A., & Beetz, M. (2007). Seamless execution of action sequences. In Proceedings of the IEEE international conference on robotics and automation (ICRA) (pp. 3687–3692).
Stulp, F., Kresse, I., Maldonado, A., Ruiz, F., Fedrizzi, A., & Beetz, M. (2009, to appear). Compact models of human reaching motions for robotic control in everyday manipulation tasks. In Proceedings of the 8th international conference on development and learning (ICDL).
Tenorth, M., Nyga, D., & Beetz, M. (2009). Understanding and executing instructions for everyday manipulation tasks from the world wide web (Technical report). IAS Group, Technische Universität München, Fakultät für Informatik.
Todorov, E. (2004). Optimality principles in sensorimotor control. Nature Neuroscience, 7(9), 907–915.
Wiedemann, C., Ulrich, M., & Steger, C. (2008). Recognition and tracking of 3d objects. In G. Rigoll (Ed.), Lecture notes in computer science : Vol. 5096. Pattern recognition (pp. 132–141). Berlin: Springer.
Wilhelm, A., Huissoon, J., Melek, W., Clark, C., Fuchs, M., & Hirzinger, G. (2007). Design of a wheeled mobile robotic platform with variable footprint. In IEEE/RSJ conference on intelligent robotics and systems.
PR2 Robot. (2008). www.willowgarage.com/pages/robots/pr2-overview.
Wimmer, M., Stulp, F., Pietzsch, S., & Radig, B. (2008). Learning local objective functions for robust face model fitting. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 30(8), 1357–1370.
Wolpert, D., & Ghahramani, Z. (2000). Computational principles of movement neuroscience. Nature Neuroscience Supplement, 3, 1212–1217.
Zacharias, F., Borst, Ch., & Hirzinger, G. (2007). Capturing robot workspace structure: representing robot capabilities. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 3229–3236).
Zacharias, F., Borst, Ch., & Hirzinger, G. (2008). Positioning mobile manipulators to perform constrained linear trajectories. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 2578–2584).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Beetz, M., Stulp, F., Esden-Tempski, P. et al. Generality and legibility in mobile manipulation. Auton Robot 28, 21–44 (2010). https://doi.org/10.1007/s10514-009-9152-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10514-009-9152-9