Skip to main content
SpringerLink
Log in

The DARPA PerceptOR evaluation experiments

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

This article describes the conduct of six evaluation experiments for the Perception for Off-Road Robotics program. Key distinctions of the testing methodology include conduct of the experiments by a group independent from the developers, unrehearsed experiments that provide little advance knowledge of the test courses, and blind experiments that do not allow the system operators to see the test courses until testing has completed.

The article presents quantified, objective performance metrics for the systems evaluated. The basis for evaluation is 296 runs traveling 130 km in 110 hr. The results show significant progress over the course of the program, reducing the Emergency-Stops per kilometer by a factor of 22, reducing the uplink data volume per unit distance by a factor of 46 and the downlink data volume per unit distance by a factor of 3.

At the end of Phase III, typical performance in desert terrain by the most reliable system achieved travel speed of 66 cm/s covering 90% of the distance in autonomous mode.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  • Ansar, A., Huertas, A., Matthies, L., and Goldberg, S. 2004. Enhancement of stereo at range discontinuities. In Proc. of the SPIE UGV Technology IV. Orlando, FL, Vol. 5422, pp. 24–35.

  • DARPA, Grand Challenge. 2004. http://www.darpa.mil/grand challenge/ParticipantsConference.html

  • Huertas, A., Matthies, L., and Rankin, A. 2005. Stereo-vision based tree traversability analysis for autonomous off-road navigation. In Proc. IEEE Work-shop on Applications of Computer Vision, Breckenridge.

  • Kelly, A., Stentz, A., Amidi, O., Bode, M., Bradley, D., Diaz-Calderon, A., Happold, M., Herman, H., Mandelaum, R., Pilarski, T., Rander, P., Thayer, S., Vallidis, N., and Warner, R. 2004. Toward reliable off-road autonomous vehicle operating in challenging environments. In Proc. International Symposium on Experimental Robotics, Singapore.

  • Kluge, K. and Morgenthaler, M. 2004. Multi-horizon reactive and deliberative planning for autonomous cross-country navigation. In Proc. SPIE Conference 5422: Unmanned Ground Vehicle Technology VI, pp. 461–472.

  • Krotkov, E. and Blitch, J. 1999. The DARPA tactical mobile robotics program. Intl. Journal of Robotics Research, 18(7):769–776.

    Article  Google Scholar 

  • Matthies, L., Bellutta, P., and McHenry, M. 2003a. Detecting water hazards for autonomous off-road navigation. In Proc. SPIE Conference 5083: Unmanned Ground Vehicle Technology V. Orlando, FL, pp. 231–242.

  • Matthies, L., Bergh, C., Castano, A., Macedo, J., and Manduchi, R. 2003b. Obstacle detection in foliage with ladar and radar. In Proc. 11th International Symposium of Robotics Research. Siena, Italy.

  • Matthies, L. and Rankin, A. 2003. Negative obstacle detection by thermal signature. In Proc. IEEE Conf. on Intelligent Robots and Systems (IROS). Las Vegas.

  • Ollis, M. and Jochem, T. 2003. Structural method for obstacle detection and terrain classification. In Proc. SPIE AeroSense Conf. Unmanned Ground Vehicle Technology V. Orlando, Vol. 5083.

  • Rankin, A., Bergh, C., Goldberg, S., Bellutta, P., Huertas, A., and Matthies, L. 2005. Passive perception system for day/night autonomous off-road navigation. In Proc. SPIE Defense and Security Symposium: Unmanned Ground Vehicle Technology VI Conference. Orlando, pp. 343–358.

  • Spofford, J., Herron, J., Anhalt, D., Morgenthaler, M., and DeHerrera, C. 2003. Off-road perception testbed vehicle design and evaluation, in unmanned ground vehicle technology V, In Proc. SPIE AeroSense Conference. Orlando, Vol. 5083, pp. 347–357.

  • Stentz, A., Kelly, A., Herman, H., Rander, P., Amidi, O., and Mandelbaum, R. 2002. Integrated air/ground vehicle system for semi-autonomous off-road navigation. In Proc. Association for Unmanned Vehicle Systems International. Orlando.

  • Stentz, A., Kelly, A., Rander, P., Herman, H., Amidi, O., Mandelbaum, R. Salgian, G., and Pedersen, J. 2003. Real-time, multi-perspective perception for unmanned ground vehicles. In Proc. Association for Unmanned Vehicle Systems International, Baltimore.

  • Technology Development for Army Unmanned Ground Vehicles. 2002. Committee on Army Unmanned Ground Vehicle Technology, Board on Army Science and Technology. Division on Engineering and Physical Sciences, National Research Council, The National Academies Press, Washington, D.C.

  • Vandapel, N., Donamukkala, R., and Hebert, M. 2003. Quality assessment of traversability maps from aerial LIDAR data for an unmanned ground vehicle. In Proc. Intl. Conf. on Intelligent Robots and Systems (IROS), Las Vegas.

  • Vandapel, N., Donamukkala, R., and Hebert, M. 2006. Unmanned ground vehicle navigation using aerial ladar data. Intl. J. of Robotics Research, 25(1).

Download references

Acknowledgments

The authors would like to acknowledge and thank the members of the PerceptOR Government Team for their outstanding and sustained contributions: Derek Baker, Mac Dorsey, David Neuman, Bernie Roberts (System Planning Corporation); Jim Cerulli, Dave Fish, Daniel Jordan, Aaron McCracken, Tom Ortiz, Chad Sullivan (Maritime Applied Physics Corporation); Lily Bengfort, Shawn Doyle, Steve Durbano, Neil Fenwick, Joe Goldstein, Tim Krout, Chip Stewart (CENGEN); Dan Krueger, Greg Peri, Frank Russell (Southwest Research Institute); Matt Anderson, Scott Bauer, and Mark McKay (Idaho National Environmental and Engineering Laboratory).

Author information

Authors and Affiliations

  1. Griffin Technologies, P.O. Box 276, Wynnewood, PA, 19096, USA

    Eric Krotkov

  2. SAIC, 8303 N. Mopac, Suite B-450, Austin, TX, 78759, USA

    Scott Fish

  3. DARPA/IPTO, 3701 North Fairfax Drive, Arlington, VA, 22203-1714, USA

    Larry Jackel

  4. SwRI, 6220 Culbra RD, San Antonio, TX, 78228, USA

    Bill McBride

  5. MAPC, 1850 Frankfurst Avenue, Baltimore, MD, 21226, USA

    Mike Perschbacher

  6. SPC, 1000 Wilson Boulevard, Arlington, VA, 22209-2211, USA

    Jim Pippine

Authors
  1. Eric Krotkov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Scott Fish
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Larry Jackel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bill McBride
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mike Perschbacher
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jim Pippine
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eric Krotkov.

Additional information

Eric Krotkov is the President of Griffin Technologies, founded in 2001 as a consulting and research firm specializing in robotics and machine perception. He has served as an executive in industry, a program manager at DARPA, and a research professor at CMU. Dr. Krotkov earned his Ph.D. degree in Computer and Information Science from the University of Pennsylvania, for pioneering work in active computer vision.

Scott Fish is an Assistant Vice President for Technology at Science Applications International Corporation (SAIC) in Austin. He has been actively involved in robotics research and development as a DARPA Program Manager and is currently serving as the Chief Engineer for Unmanned Ground Vehicles for the Army Future Combat Systems Program. Dr Fish received his Doctorate in Mechanical Engineering from the University of Maryland.

Larry Jackel as a Program Manager at DARPA, where he has focused on applying machine learning and perception techniques to ground robots. For most of his scientific career Jackel was a manager and researcher in Bell Labs and then AT&T Labs. He has created and managed research groups in Microscience and Microfabrication, in Machine Learning and Pattern Recognition, and in Carrier-Scale Telecom Services. Jackel holds a PhD in Experimental Physics from Cornell University with a thesis in superconducting electronics. He is a Fellow of the American Physical Society and the IEEE.

Bill McBride USMC (ret.), is a Staff Engineer in the Department of Electronics Systems and Robotics at Southwest Research Institute. Under a contract from the ITIC, he established the Small Robotic Vehicle Test Bed at the Institute in 1988, which is now recognized as one of the pre-eminent test facilities in the nation. He holds a bachelor degree in systems engineering and a master degree in electronics engineering from the U.S. Naval Academy and U.S. Naval Postgraduate School, respectively.

Mike Perschbacher serves as Vice President and Principal Engineer of the Maritime Applied Physics Corporation. Mr. Perschbacher has particular expertise in the design, construction, and testing of unmanned systems (in particular ground) and advanced marine/air vehicles (including hybrids). He earned dual B.S. degrees in Aerospace Engineering and Ocean Engineering from Virigina Tech.

Jim Pippine is the lead for System Planning Corporation Ground Robotics Team which has supported DARPA for over a decade. Mr. Pippine spent over five years providing test and evaluation support to several Department of Defense unmanned vehicle programs and four years as an Air Force officer conducting research and development in directed energy systems. Mr. Pippine received his B.S. in Chemical Engineering from Clarkson University. Michael Perschbacher serves as Vice President and Principal Engineer of the Maritime Applied Physics Corporation. Mr. Perschbacher has particular expertise in the design, construction, and testing of unmanned systems (in particular ground) and advanced marine/air vehicles (including hybrids). He earned dual B.S. degrees in Aerospace Engineering and Ocean Engineering from Virigina Tech. Jim Pippine is the lead for System Planning Corporation Ground Robotics Team which has supported DARPA for over a decade. Mr. Pippine spent over five years providing test and evaluation support to several Department of Defense unmanned vehicle programs and four years as an Air Force officer conducting research and development in directed energy systems. Mr. Pippine received his B.S. in Chemical Engineering from Clarkson University.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Krotkov, E., Fish, S., Jackel, L. et al. The DARPA PerceptOR evaluation experiments. Auton Robot 22, 19–35 (2007). https://doi.org/10.1007/s10514-006-9000-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-006-9000-0

Keywords

Search

Navigation