Skip to main content
SpringerLink
Log in

Computer Vision on Mars

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Increasing the level of spacecraft autonomy is essential for broadening the reach of solar system exploration. Computer vision has and will continue to play an important role in increasing autonomy of both spacecraft and Earth-based robotic vehicles. This article addresses progress on computer vision for planetary rovers and landers and has four main parts. First, we review major milestones in the development of computer vision for robotic vehicles over the last four decades. Since research on applications for Earth and space has often been closely intertwined, the review includes elements of both. Second, we summarize the design and performance of computer vision algorithms used on Mars in the NASA/JPL Mars Exploration Rover (MER) mission, which was a major step forward in the use of computer vision in space. These algorithms did stereo vision and visual odometry for rover navigation and feature tracking for horizontal velocity estimation for the landers. Third, we summarize ongoing research to improve vision systems for planetary rovers, which includes various aspects of noise reduction, FPGA implementation, and vision-based slip perception. Finally, we briefly survey other opportunities for computer vision to impact rovers, landers, and orbiters in future solar system exploration missions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ali, K. et al. 2005. Attitude and Position Estimation on the Mars Exploration Rovers. In IEEE Conference on Systems, Man, and Cybernetics.

  • Amidi, O., Kanade, T., and Miller, J.R. 1998. Vision-Based Autonomous Helicopter Research at Carnegie Mellon Robotics Institute 1991–1997. In American Helicopter Society International Conference, Heli, Japan.

  • Amidi, O., Kanade, T., and Fujita, K. 1999. A Visual Odometer for Autonomous Helicopter Flight. Journal of Robotics and Autonomous Systems, 28:185–193.

    Article  Google Scholar 

  • Andrade, C., Ben Amar, F., Bidaud, P., and Chatila, R. 1998. Modeling robot-soil interaction for planetary rover motion control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 576–581.

  • Angelova, A., Matthies, L., and Perona, P. 2006. Slip prediction using visual information. Robotics Science and Systems Conference.

  • Angelova, A., Matthies, L., Sibley, G., and Perona, P. 2006. Learning to predict slip for ground robots. In IEEE International Conference on Robotics and Automation.

  • Ansar, A., Castano, A., and Matthies, L. 2004. Enhanced real-time Stereo using Bilateral Filtering. In 2nd International Symposium on 3D Data Processing, Visualization, and Transmission, Thessaloniki, Greece.

  • Bares, J., Hebert, M., Kanade, T., Krotkov, E., Mitchell, T., Simmons, R., and Whittaker, W.L. 1989. Ambler: An Autonomous Rover for Planetary Exploration. IEEE Computer, 22(6):18–26.

    Google Scholar 

  • Bares, J. and Wettergreen, D. 1999. Dante II: Technical Description, Results and Lessons Learned. International Journal of Robotics Research, 18(7):621–649.

  • Bekker, M.G. 1964. Mechanics of locomotion and lunar surface vehicle concepts. Transactions of the Society of Automotive Engineers, 72:549–569.

    Google Scholar 

  • Biesiadecki, J. et al. 2005. Mars Exploration Rover surface operations: Driving Opportunity at Meridiani Planum. In IEEE Conference on Systems, Man, and Cybernetics.

  • Biesiasdecki, J. and Maimone, M. 2006. The Mars Exploration Rover Surface Mobility Flight Software: Driving ambition. In IEEE Aerospace Conference.

  • Bodt, B.A. and Camden, R.S. 2004. Technology Readiness Level Six and Autonomous Mobility. In Proc. SPIE Vol. 5422: Unmanned Ground Vehicle Technology VI, pp. 302–313.

  • Bornstein, J.A. and Shoemaker, C.M. 2003. Army Ground Robotics Research Program. In Proc. SPIE Vol. 5083: Unmanned Ground Vehicle Technolgy V, pp. 303–310.

  • Burt, P., Wixson, L., and Salgian, G. 1995. Electronically directed ‘focal’ stereo. In International Conference on Computer Vision.

  • Cheng, Y., Johnson, A., Olson, C., and Matthies, L. 2003. Optical Landmark Detection for Spacecraft Navigation. In 13th Annual AAS/AIAA Space Flight Mechanics Meeting.

  • Cheng, Y. and Miller, J. 2003. Autonomous Landmark Based Spacecraft Navigation System. In 13th Annual AAS/AIAA Space Flight Mechanics Meeting.

  • Cheng, Y., Goguen, J., Johnson, A., Leger, C., Matthies, L., San Martin, M., and Willson, R. 2004. The Mars Exploration Rovers Descent Image Motion Estimation System. IEEE Intelligent Systems Magazine, May/June, pp. 13–21.

  • Cheng, Y., Johnson, A., and Matthies, L. 2005. MER-DIMES: A Planetary Landing Application of Computer Vision. In IEEE Conf. on Computer Vision and Pattern Recognition, San Diego, pp. 806–813.

  • Cheng, Y., Maimone, M., and Matthies, L. 2006. Visual Odometry on the Mars Exploration Rovers. IEEE Robotics and Automation Magazine, Vol. 13, No. 2, pp. 54–62.

  • Di, K., Xu, F., Wang, J., Niu, C., Serafy, F., Zhou, R., Li, R., and Matthies, L. 2005. Surface imagery Based Mapping and Rover Localization for the 2003 Mars Exploration Rover Mission. In Proc. ASPRS Annual Conference, MD, Baltimore.

  • Eustice, R., Singh, H., Leonard, J., Walter, M., and Ballard, R. 2005. Visually navigation the RMS Titanic with SLAM information filters. In Proc. Robotics Science and Systems Conference.

  • Gennery, D.B. 1980. Modelling the Environment of an Exploring Vehicle by Means of Stereo Vision, PhD thesis, Computer Science Department, Stanford University.

  • Goldberg, S., Maimone, M., and Matthies, L. 2002 Stereo vision and Rover Navigation Software for Planetary Exploration. IEEE Aerospace Conference, Big Sky, MO, Vol. 5, pp. 2025–2036.

  • Guerra, C. and Kanade, T. 1985. A systolic algorithm for stereo matching. In VLSI: Algorithms and Architectures, P. Bertolazzi and F. Luccio (Eds.), Elsevier Science Publishers B.V. (North-Holland), pp. 103–112.

  • Hapke, B. 1986. Bidirectional reflectance spectroscopy. IV. The extinction coefficient and the opposition effect. Icarus, 67:264–280.

    Article  Google Scholar 

  • Hebert, M., Kanade, T., and Kweon, I. 1988. 3-D Vision Techniques for Autonomous Vehicles. Technical report CMU-RI-TR-88-12, Robotics Institute, Carnegie Mellon University.

  • Hebert, M., Krotkov, E., and Kanade, T. 1989. A Perception System for a Planetary Rover. In Proc. 28th IEEE Conference on Decision and Control, pp. 1151–1156.

  • Hebert, M., Bolles, B., Gothard, B., Matthies, L., and Rosenbloom, M. 1996. Mobility for the Unmanned Ground Vehicle. In Reconnaissance, Surveillance, and Target Recognition (RSTA) for the Unmanned Ground Vehicle, O. Firschein (Ed.), PRC Inc.

  • Helmick, D., Cheng, Y., Clouse, D., and Matthies, L. 2004. Path following using visual odometry for a Mars Rover in high-slip environments. In IEEE Aerospace Conference, Big Sky, MO.

  • Hirschmuller, H., Innocent, P.R., and Garibaldi, J. 2002. Real-time correlation-based stereo vision with reduced border errors. International Journal of Computer Vision, 47:229–246.

    Article  Google Scholar 

  • Iagnemma, K., Kang, S., Shibly, H., and Dubowsky, S. 2004. On-line terrain parameter estimation for wheeled mobile robots with application to planetary rovers. IEEE Transactions on Robotics, 20(5):921–927.

    Article  Google Scholar 

  • Johnson, A., Montgomery, J., and Matthies, L. 2005. Vision Guided Landing of an Autonomous Helicopter in Hazardous Terrain. In IEEE International Conference on Robotics and Automation, Barcelona, Spain.

  • Johnson, A., Willson, R., Goguen, J., Alexander, J., and Meller, D. 2005. Field testing of the Mars Exploration Rovers Descent Image Motion Estimation System. In IEEE International Conference on Robotics and Automation, Barcelona, Spain, pp. 4463–4469.

  • Johnson, A., Willson, R., Cheng, Y., Goguen, J., Leger, C., SanMartin, M., and Matthies, L. Design Through Operation of an Image-Based Velocity Estimation System for Mars Landing. Joint issue of the International Journal of Computer Vision and the International Journal of Robotics Research, Special Issue on Vision and Robotics, to appear.

  • Kanade, T. and Ikeuchi, K. 1991. Introduction to the Special Issue on Physical Modeling in Computer Vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(7):609–610.

    Google Scholar 

  • Kanade, T., Yoshida, A., Oda, K., Kano, H., and Tanaka, M. 1996. A Stereo Machine for Video-rate Dense Depth Mapping and Its New Applications. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp. 196–202.

  • Kanade, T., Amidi, O., and Ke, Q. 2004. Real-time and 3D vision for Autonomous Small and Micro Air Vehicles. In IEEE Conference on Decision and Control.

  • Klinker, G., Shafer, S., and Kanade, T. 1990. A Physical Approach to Color Image Understanding. International Journal of Computer Vision, 4:7–38.

    Article  Google Scholar 

  • Konolige, K. 1997. Small Vision Systems: Hardware and implementation. In Proc. 8th International Symposium on Robotics Research, Hayama, Japan.

  • Krotkov, E. and Simmons, R. 1996. Perception, planning, and Control for Autonomous Walking with the Ambler Planetary Rover. International Journal of Robotics Research, 15(2):155–180.

    Article  Google Scholar 

  • Krotkov, E. and Blitch, J. 1999. The DARPA Tactical Mobile Robotics Program. International Journal of Robotics Research, 18(7):769–776.

    Google Scholar 

  • Krotkov, E., Fish, S., Jackel, L., McBride, B., Pershbacher, M., and Pippine, J. 2007. The DARPA PerceptOR evaluation experiments. Autonomous Robots, to appear.

  • Leger, C. et al. 2005. Mars Exploration Rover Surface Operations: Driving Spirit at Gusev Crater. In IEEE Conference on Systems, Man, and Cybernetics.

  • Levine, M.D., O’Handley, D.A., and Yagi, G.M. 1973. Computer determination of depth maps. Computer Graphics and Image Processing, 2:131–150.

    Google Scholar 

  • Lewis, R.A. and Johnston, A.R. 1977. A scanning laser rangefinder for a robotic vehicle. In Proc. 5th IJCAI, Cambridge, MA, pp. 762–768.

  • Li, R., Archinal, B., Arvidson, R., Bell, J., Christensen, P., Crumpler, L., Des Marais, D., Di, K., Duxbury, T., Golombek, M., Grant, J., Greeley, R., Gunn, J., Johnson, A., Kirk, R., Maimone, M., Matthies, L, Malin, M., Parker, T., Sims, M., Thompson, S., Squyres, S., and Soderblom, L. 2006. Rover localization and topographic mapping at the landing site of Gusev Crater, Mars. Journal of Geophysical Research–-Planets, 111(E1).

  • Lindemann, R. and Voorhees, C. 2005. Mars Exploration Rover Mobility Assembly Design, Test, and Performance. In IEEE Conference on Systems, Man, and Cybernetics.

  • Maimone, M., Biesadecki, J., Tunstel, E., Cheng, Y., and Leger, C. 2006. Surface Navigation and Mobility Intelligence on the Mars Exploration Rovers. In Intelligence for Space Robotics, TSI Press, Albuquerque, NM.

  • Maki, J. et al. 2003. Mars Exploration Rover Engineering Cameras. Journal of Geophysical Research, 108(E12).

  • Mallet, A., Lacroix, S., and Gallo, L. 2000. Position estimation in outdoor environments using pixel tracking and stereovision. In IEEE International Conference on Robotics and Automation, pp. 3519–2524.

  • Masrani, D.K. and MacLean, W.J. 2006. A real-time large disparity range stereo-system using FPGAs. In Asian Conference on Computer Vision, Hyderabad, India.

  • Matthies, L. and Shafer, S, 1987. Error Modelling in Stereo Navigation. IEEE Journal of Robotics and Automation, RA-3(3).

  • Matthies, L. 1992. Stereo Vision for Planetary Rovers: Stochastic Modeling to Near Real-time Implementation. International Journal of Computer Vision, 8(1).

  • Matthies, L., Gat, E., Harrison, R., Wilcox, B., Volpe, R., and Litwin, T. 1995. Mars Microrover Navigation: Performance Evaluation and Enhancement. Autonomous Robots, 2(4).

  • Matthies, L., Kelly, A., Litwin, T., and Tharp, G. 1996. Obstacle Detection for Unmanned Ground Vehicles: A Progress Report. In Robotics Research: The Seventh International Symposium, G. Giralt and G. Hirzinger (Eds.), Springer, pp. 475–486.

  • Matthies, L., Xiong, Y., Hogg, R., Zhu, D., Rankin, A., Kennedy, B., Hebert, M., Maclachlan, R., Won, C., and Frost, T. 2002. A Portable, Autonomous, Urban Reconnaissance Robot. Robotics and Autonomous Systems, 40(2).

  • Matthies, L. and Rankin, A. 2003. Negative obstacle detection by thermal signature. In IEEE Conf. on Intelligent Robots and Systems, Las Vegas, NV, pp. 906–913.

  • Matthies, L., Bellutta, P., and McHenry, M. 2003. Detecting Water Hazards for Autonomous Off-Road Navigation. In SPIE Conf. on Unmanned Ground Vehicles, Orlando, FL.

  • Mettala, E.G. 1992. The OSD Tactical Unmanned Ground Vehicle Program. In Proc. DARPA Image Understanding Workshop, MorganKaufmann Publishers, pp. 159–172.

  • Mettler, B., Tischler, M.B., and Kanade, T. 2001. System Identification Modeling of a Small-Scale Unmanned Helicopter. Journal of the American Helicopter Society.

  • Miller, J.R. and Amidi, O. 1998. 3-D Site mapping with the CMU Autonomous Helicopter. In Proc. 5th International Conf. on Intelligent Autonomous Systems (IAS-5).

  • Montgomery, J., Johnson, A., Roumeliotis, S.I., and Matthies, L. The JPL Autonomous Helicopter Testbed: A Platform for Planetary Exploration Technology Research and Development. Journal of Field Robotics, to appear.

  • Moravec, H.P. 1980. Obstacle Avoidance and Navigation in the Real World by a Seeing Robot Rover, PhD thesis, Computer Science Department, Stanford University.

  • Moravec, H.P. 1983. The Stanford Cart and the CMU Rover. Proceedings of the IEEE, 71(7):872–884.

    Article  Google Scholar 

  • Nayar, S., Ikeuchi, K., and Kanade, T. 1991. Surface reflection: Physical and Geomerical Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(7):611–634.

    Article  Google Scholar 

  • Nister, D., Naroditsky, O., and Bergen, J. 2006. Visual odometry for ground vehicle applications. Journal of Field Robotics, 23(1):3–20.

    Article  Google Scholar 

  • A Scientific Rationale for Mobility in Planetary Environments. 1999. Committee on Planetary and Lunar Exploration, National Research Council. The National Academies Press.

  • Technology Development for Army Unmanned Ground Vehicles, 2002. Committee on Army Unmanned Ground Vehicle Technology, Board on Army Science and Technology, Division of Engineering and Physical Sciences, National Research Council, The National Academies Press, Washington, D.C.

  • O’Handley, D.A. 1973. Scene Analysis in Support of a Mars Rover. Computer Graphics and Image Processing, 2:281–297.

  • Okutomi, M. and Kanade, T. 1993. A Multiple-Baseline Stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(4):353–363.

    Article  Google Scholar 

  • Pomerleau, D. and Jochem, T. 1996. Rapidly Adapting Machine Vision for Automated Vehicle Steering. IEEE Expert: Special Issue on Intelligent Systems and their Applications, 11(2):19–27.

    Google Scholar 

  • Schaal, S. and Atkeson, C. 1998. Constructive incremental learning from only local information. Neural Computation, 10(8):2047–2084.

    Article  Google Scholar 

  • Se, S., Lowe, D., and Little, J. 2002. Mobile robot localization and mapping with uncertainty using scale-invarianet visual landmarks. International Journal of Robotics Research, 21(8).

  • Shimizu, M. and Okutomi, M. 2001. Precise Sub-Pixel Estimation on Area-Based Matching. In International Conference on Computer Vision.

  • Shoemaker, C.M. and Bornstein, J.A. 2000. Overview and Update of the Demo III Experimental Unmanned Vehicle program. In Proc. SPIE Vol. 4024: Unmanned Ground Vehicle Technology II, pp. 212–220.

  • Squyres, S. and Knoll, A.H. 2005. Sedimentary Rocks at Meridiani Planum: Origin, Diagenesis, and Implications for Life on Mars. Earth and Planetary Science Letters, Special Issue on Sedimentary Geology at Meridiani Planum, Mars, Vol. 240, No. 1, pp. 1–10.

  • Stein, A., Huertas, A., and Matthies, L. 2006. Attenuating Stereo Pixel-Locking via Affine Window Adaptation. In IEEE International Conference on Robotics and Automation.

  • Szeliski, R., and Scharstein, D. 2002. Symmetric Sub-Pixel Stereo Matching. In European Conference on Computer Vision, pp. 525–540.

  • Thompson, A.M. 1977. The navigation system of the JPL robot. In Proc. 5th IJCAI, Cambridge, MA, pp. 749–757.

  • Thorpe, C., Hebert, M., Kanade, T., and Shafer, S. 1991a. Toward Autonomous Driving: The CMU Navlab. Part I: Perception. IEEE Expert, 6(4):31–42.

    Article  Google Scholar 

  • Thorpe, C., Hebert, M., Kanade, T., and Shafer, S. 1991b. Toward Autonomous Driving: The CMU Navlab. Part II: System and architecture. IEEE Expert, 6(4):44–52.

    Article  Google Scholar 

  • Thrun, S. 2001. A Probabilistic Online Mapping Algorithm for Teams of Mobile Robots. International Journal of Robotics Research, 20(5):335–363.

    Article  Google Scholar 

  • Tomasi, C. and Kanade, T. 1992. Shape and Motion from Image Streams under Orthography: A Factorization Method. International Journal of Computer Vision, 9(2):137–154.

    Article  Google Scholar 

  • Varma, M. and Zisserman, A. 2005. A statistical approach to texture classification from single images. International Journal of Computer Vision, 62.

  • Stereo-on-a-Chip [STOC] Stereo Head User Manual 1.1, 2006. Videre Design.

  • Vijayakumar, S., D’Souza, A., and Schaal, S. 2005. Incremental online learning in high dimensions. Neural Computation, 17(12):2602–2634.

    Article  MathSciNet  Google Scholar 

  • Villalpando, C. 2006. Acceleration of Stereo Correlation in Verilog. In 9th Military and Aerospace Programmable Logic Devices (MAPLD) International Conference, Washington, D.C.

  • Wettergreen, D., Thorpe, C., and Whittaker, W.L. 1993. Exploring Mount Erebus by Walking Robot. Robotics and Autonomous Systems.

  • Wilcox, B.H. and Gennery, D.B. 1987. A Mars rover for the 1990s. Journal of the British Interplanetary Society, pp. 483–488.

  • Wilcox, B. and Nguyen, T. 1998. Sojourner on Mars and Lessons Learned for Future Planetary Rovers. In Proc. 28th International Conf. on Environmental Systems (ICES’98), Danvers, MA, Society of Automotive Engineers.

  • Williamson, T. 1998. A High-Performance Stereo Vision System for Obstacle Detection, technical report CMU-RI-TR-98-24, Robotics Institute, Carnegie Mellon University.

  • Willson, R., Johnson, A., and Goguen, D. 2005. MOC2DIMES: A Camera Simulator for the Mars Exploration Rover Descent Image Motion Estimation System. In 8th International Symposium on Artificial Intelligence, Robotics and Automation in Space (iSAIRAS), Munich, Germany.

  • Willson, R., Maimone, M., Johnson, A., and Scherr, L. 2005. An Optical Model for Image Artifacts Produced by Dust Particles on Lenses. In 8th International Symposium on Artificial Intelligence, Robotics, and Automation in Space (i-SAIRAS).

  • Wold, H. 1966. Estimation of principal components and related models by iterative least squares. Multivariate Analysis, P.R. Krishnaiah (Ed.), Academic Press, New York, pp. 391–420.

  • Woodfill, J., Gordon, G., and Buck, R. 2004. Tyzx DeepSea high speed stereo vision system. In IEEE Workshop on Real Time 3-D Sensors and their Use, Washington, D.C.

  • Xiong, Y. and Matthies, L. 1997. Error Analysis of a Real-Time Stereo System. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1087–1093.

Download references

Author information

Authors and Affiliations

  1. Jet Propulsion Laboratory, California Institute of Technology, 4800 Oak Grove Drive, Pasadena, CA, 91109, USA

    Larry Matthies, Mark Maimone, Andrew Johnson, Yang Cheng, Reg Willson, Carlos Villalpando, Steve Goldberg & Andres Huertas

  2. Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA, 15213, USA

    Andrew Stein

  3. California Institute of Technology, 1200 East California Boulevard, Pasadena, CA, 91125, USA

    Anelia Angelova

Authors
  1. Larry Matthies
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark Maimone
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrew Johnson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yang Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Reg Willson
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Carlos Villalpando
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Steve Goldberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Andres Huertas
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Andrew Stein
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Anelia Angelova
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Larry Matthies.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Matthies, L., Maimone, M., Johnson, A. et al. Computer Vision on Mars. Int J Comput Vis 75, 67–92 (2007). https://doi.org/10.1007/s11263-007-0046-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-007-0046-z

Keywords

Search

Navigation