Skip to main content
SpringerLink
Log in

Model-based autonomous system for performing dexterous, human-level manipulation tasks

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

This article presents a model based approach to autonomous dexterous manipulation, developed as part of the DARPA Autonomous Robotic Manipulation Software (ARM-S) program. Performing human-level manipulation tasks is achieved through a novel combination of perception in uncertain environments, precise tool use, forceful dual-arm planning and control, persistent environmental tracking, and task level verification. Deliberate interaction with the environment is incorporated into planning and control strategies, which, when coupled with world estimation, allows for refinement of models and precise manipulation. The system takes advantage of sensory feedback immediately with little open-loop execution, attempting true autonomous reasoning and multi-step sequencing that adapts in the face of changing and uncertain environments. A tire change scenario utilizing human tools, discussed throughout the article, is used to described the system approach. A second scenario of cutting a wire is also presented, and is used to illustrate system component reuse and generality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. Personal communication at PI meetings.

References

  • Backes, P. G. (1994). Dual-arm supervisory and shared control task description and execution. Robotics and Autonomous Systems, 12, 29–54.

    Article  Google Scholar 

  • Bagnell, J. A. D., Cavalcanti, F., Cui, L., Galluzzo, T., Hebert, M., Kazemi, M., Klingensmith, M., Libby, J., Liu, T.Y., Pollard, N., Pivtoraiko, M., Valois, J. S., & Zhu, R. (2012). An integrated system for autonomous robotics manipulation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 2955–2962).

  • Bajracharya, M., Ma, J., Howard, A., & Matthies, L. (2013). Real-time 3d stereo mapping in complex dynamic environments. In: IEEE International Conference on Robotics and Automation Workshop: Semantic Perception Mapping and Exploration.

  • Beetz, M., Klank, U., Kresse, I., Maldonado, A., Mosenlechner, L., Pangercic, D., Ruhr, T., & Tenorth, M. (2011). Robotic roommates making pancakes. In: IEEE-RAS International Conference on Humanoid Robots (pp. 529–536).

  • Bohren, J., Rusu, R., Jones, E., Marder-Eppstein, E., Pantofaru, C., Wise, M., Mosenlechner, L., Meeussen, W., & Holzer, S. (2011). Towards autonomous robotic butlers: Lessons learned with the pr2. In: IEEE International Conference on Robotics and Automation (pp. 5568–5575).

  • Chitta, S., Jones, E., Ciocarlie, M., & Hsiao, K. (2012). Mobile manipulation in unstructured environments: Perception, planning, and execution. IEEE Robotics Automation Magazine, 19(2), 58–71.

    Article  Google Scholar 

  • Coumans, E., et al. (2013). Bullet physics library. http://bulletphysics.org.

  • Diankov, R., & Kuffner, J. (2008). OpenRAVE: a planning architecture for autonomous robotics (Technical Report CMU-RI-TR-08-34). Pittsburgh, PA: Robotics Institute. Accessed 13 Nov 2013.

  • Dynamics and Real-Time Simulation (DARTS) Lab (2013). http://dartslab.jpl.nasa.gov. Accessed 13 Nov 2013.

  • Gallagher, G., Srinivasa, S. S., Bagnell, J. A., & Ferguson, D. (2009). GATMO: A generalized approach to tracking movable objects. In: IEEE International Conference on Robotics and Automation.

  • Hackett, D., Pippine, J., Watson, A., Sullivan, C., & Pratt, G. (2013). An overview of the DARPA autonomous robotic manipulation (ARM) program. Journal of the Robotics Society of Japan (to appear).

  • Hebert, P. (2013). Estimation and inference for grasping and manipulation tasks using vision and kinesthetic sensors. Ph.D. thesis, California Institute of Technology.

  • Hebert, P., Hudson, N., Ma, J., & Burdick, J. (2011). Fusion of stereo vision, force-torque, and joint sensors for estimation of in-hand object location. In: IEEE International Conference on Robotics and Automation.

  • Hebert, P., Hudson, N., Ma, J., & Burdick, J. (2013). Dual arm estimation for coordinated bi-manual manipulation. In: IEEE International Conference on Robotics and Automation.

  • Hebert, P., Hudson, N., Ma, J., Howard, T., & Burdick, J. (2013). Action inference: The next best touch for model-based object localization, parametrization, and identification. In: IEEE International Conference on Robotics and Automation.

  • Hebert, P., Hudson, N., Ma, J., Howard, T., Fuchs, T., & Burdick, J. (2012). Combined shape, appearance and silhouette for object manipulation. In: IEEE International Conference on Robotics and Automation.

  • Howard, T., Green, C., Kelly, A., & Ferguson, D. (2008). State space sampling of feasible motions for high-performance mobile robot navigation in complex environments. Journal of Field Robotics, 25(6–7), 325–345.

    Article  Google Scholar 

  • Hudson, N., Howard, T., Ma, J., Jain, A., Bajracharya, M., Kuo, C., et al. (2012). End-to-end dexterous manipulation with deliberate interactive estimation. In: IEEE International Conference on Robotics and Automation.

  • Jain, A. (2010). Robot and multibody dynamics: Analysis and algorithms. New York: Springer.

    Google Scholar 

  • Jain, A. (2013). Structure based modeling and computational architecture for robotic systems. In: IEEE International Conference on Robotics and Automation.

  • Jain, A., Crean, C., Kuo, C., von Bremen, H., & Myint, S. (2012). Minimal coordinate formulation of contact dynamics in operational space. In: Robotics Science and Systems, Sydney, Australia.

  • Jain, A., & Kemp, C. C. (2010). EL-E: An assistive mobile manipulator that autonomously fetches objects from flat surfaces. Autonomous Robots, 28, 45–64.

    Article  Google Scholar 

  • Kalakrishnan, M., Pastor, P., Righetti, L., & Schaal, S. (2013). Learning objective functions for manipulation. In: IEEE International Conference on Robotics and Automation.

  • Kazemi, M., Valois, J. S., Bagnell, J. A., & Pollard, N. (2012). Robust object grasping using force compliant motion primitives. In: Proceedings of Robotics: Science and Systems, Sydney, Australia.

  • Knepper, R., Srinivasa, S., & Mason, M. (2010). Hierarchical planning architectures for mobile manipulation tasks in indoor environments. In: IEEE International Conference on Robotics and Automation (pp. 1985–1990).

  • Kuffner, J. J., & LaValle, S. (2000). Rrt-connect: An efficient approach to single-query path planning. In: IEEE International Conference on Robotics and Automation (Vol. 2, pp. 995–1001).

  • Ma, J., Susca, S., Bajracharya, M., Matthies, L., Malchano, M., & Wooden, D. (2012). Robust multi-sensor, day/night 6-dof pose estimation for a dynamic legged vehicle in gps-denied environments. In: IEEE International Conference on Robotics and Automation.

  • Maitin-Shepard, J., Cusumano-Towner, M., Lei, J., & Abbeel, P. (2010). Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. In: IEEE International Conference on Robotics and Automation (pp. 2308–2315).

  • Pastor, P., Kalakrishnan, M., Chitta, S., Theodorou, E., & Schaal, S. (2011). skill learning and task outcome prediction for manipulation. In: IEEE International Conference on Robotics and Automation.

  • Quigley, M., Berger, E., & Ng, A. (2007). Stair: Hardware and software architecture. In: AAAI Robotics Workshop.

  • Srinivasa, S., Ferguson, D., Helfrich, C., Berenson, D., Romea, A. C., Diankov, R., et al. (2010). HERB: A home exploring robotic butler. Autonomous Robots, 28, 520.

    Article  Google Scholar 

  • Xue, Z., Ruehl, S. W., Hermann, A., Kerscher, T., & Dillmann, R. (2012). Autonomous grasp and manipulation planning using a tof camera. Robotics and Autonomous Systems, 60(3), 387–395.

    Article  Google Scholar 

Download references

Acknowledgments

The research described in this publication was carried out at the Jet Propulsion Laboratory, California Institute of Technology, with funding from the DARPA Autonomous Robotic Manipulation Software Track (ARM-S) program through an agreement with NASA.

Author information

Author notes

  1. Calvin Kuo

    Present address: Stanford University, Stanford, CA, USA

  2. Thomas Howard

    Present address: Massachusetts Institute of Technology, Cambridge, MA, USA

Authors and Affiliations

  1. Jet Propulsion Laboratory, 4800 Oak Grove Drive, Pasadena, 91109-8099, CA, USA

    Nicolas Hudson, Jeremy Ma, Paul Hebert, Abhinandan Jain, Max Bajracharya, Calvin Kuo, Thomas Howard, Larry Matthies & Paul Backes

  2. California Institute of Technology, 1200 E California Blvd, Pasadena, 91125, CA, USA

    Thomas Allen, Rangoli Sharan, Matanya Horowitz & Joel Burdick

Authors
  1. Nicolas Hudson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeremy Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul Hebert
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Abhinandan Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Max Bajracharya
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Thomas Allen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Rangoli Sharan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Matanya Horowitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Calvin Kuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Thomas Howard
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Larry Matthies
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Paul Backes
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Joel Burdick
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicolas Hudson.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (m4v 38880 KB)

Supplementary material 2 (wmv 19423 KB)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hudson, N., Ma, J., Hebert, P. et al. Model-based autonomous system for performing dexterous, human-level manipulation tasks. Auton Robot 36, 31–49 (2014). https://doi.org/10.1007/s10514-013-9371-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-013-9371-y

Keywords

Search

Navigation