Skip to main content
SpringerLink
Log in

Efficient Fourier-Based Approach for Detecting Orientations and Occlusions in Epipolar Plane Images for 3D Scene Modeling

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

This paper presents a Fourier-based approach for automatically constructing a 3D panoramic model of a natural scene from a video sequence. The video sequences could be captured by an unstabilized camera mounted on a moving platform on a common road surface. As the input of the algorithms, “seamles” panoramic view images (PVIs) and epipolar plane images (EPIs) are generated after image stabilization if the camera is unstabilized. A novel panoramic EPI analysis method is proposed that combines the advantages of both PVIs and EPIs efficiently in three important steps: locus orientation detection in the Fourier frequency domain, motion boundary localization in the spatio-temporal domain, and occlusion/resolution recovery only at motion boundaries. The Fourier energy-based approaches in literature were usually for low-level local motion analysis and are therefore not accurate for 3D reconstruction and are also computationally expensive. Our panoramic EPI analysis approach is both accurate and efficient for 3D reconstruction. Examples of layered panoramic representations for large-scale 3D scenes from real world video sequences are given.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Adelson, E.H. and Bergen, J.R. 1985. Spatiotemporal energy model for the perception of motion. J. Opt. Soc. Am., A2:284–299.

    Google Scholar 

  • Allmen, M. and Dyer, C.R. 1991. Long range spatiotemporal motion understanding using spatiotemporal flow curves. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 303–309.

  • Baillard, C. and Zisserman, A. 1999. Automatic reconstruction of piecewise planar models from multiple views. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 559–565.

  • Baker, H.H. and Bolles, R.C. 1989. Generalizing epipolar-plane image analysis on the spatiotemporal surface. Int. J. Computer Vision, 3:33–49.

    Google Scholar 

  • Baker, S., Szeliski, R., and Anandan, P. 1998. A layered approach to stereo reconstruction. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 434–441.

  • Black, M.J. and Jepson, A.D. 1996. Estimating optical flow in segmented images using variable-order parametric models with local deformations. IEEE Trans Pattern Analysis and Machine Intelligence, 18(10):972–986.

    Google Scholar 

  • Bolles, R.C., Baker, H.H., and Marimont, D.H. 1987. Epipolar-plane image analysis: An approach to determining structure from motion. Int. J. Computer Vision, 1(1):7–55.

    Google Scholar 

  • Chang, N.L. and Zakhor, A. 1997. View generation for three-dimensional scene from video sequence. IEEE Trans on Image Processing, 6(4):584–598.

    Google Scholar 

  • Chang, N.L. and Zakhor, A. 2001. Constructing a multivalued representation for view synthesis. Int. J. of Computer Vision, 45(2):157–190.

    Google Scholar 

  • Chen, S.E. 1995. QuickTime VR—An image based approach to virtual environment navigation. In ACM Conf. Proc. SIGGRAPH 95, pp. 29–38.

  • Collins, R. 1996. A space-sweep approach to true multi-image matching. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 358–363.

  • Collins, R., Jaynes, C., Cheng, Y., Wang, X., Stolle, F., Schultz, H., Hanson, A., and Riseman, E. 1998. The Ascender System: Automated Site Modeling from Multiple Aerial Images,” Computer Vision and Image Understanding, 72(2):143–162.

    Google Scholar 

  • Coorg, S. and Teller, S. 1998. Automatic extraction of textured vertical facades from pose imagery. MIT LCS TR-729.

  • Dalmia, A.K. and Trivedi, M. 1996. High speed extraction of 3D structure of selectable quality using a translating camera. Computer Vision and Image Understanding, 64(1):97–110.

    Google Scholar 

  • Debevec, P., Taylor, C., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In ACM Conf. Proc. SIGGRPAH 96, pp. 11–20.

  • Faugeras, O., Robert, L., Laveau, S., Csurka, G., Zeller, C., Gauclin, C., and Zoghlami, I. 1998. 3-D reconstruction of urban scenes from image sequences. Computer Vision and Image Understanding, 69(3):292–309.

    Google Scholar 

  • Fleet, D.J., Black, M.J., and Jepson, A.D. 1998. Motion feature detection using steerable flow fields. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 274–281.

  • Freeman, W.T. and Adelson, E.H. 1991. The design and use of steerable filters. IEEE Trans Pattern Analysis and Machine Intelligence, 13(9):891–906.

    Google Scholar 

  • Hansen, M., Anandan, P., Dana, K., van de Wal, G., and Burt, P. 1994. Real-time scene stabilization and mosaic construction. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 54–62.

  • Heeger, D.J. 1987. Optical flow from spatio-temporal filters. In Proc. IEEE Int. Conf. Computer Vision, pp. 181–190.

  • Ishiguro, H., Yamamoto, M., and Tsuji, S. 1990. Omni-directional stereo for making global map. In Proc. IEEE Int. Conf. Computer Vision, pp. 540–547.

  • Jahne, B. 1991. Digital Image Processing, Concept, Algorithms and Scientific Applications. Springer-Verlag.

  • Li, Y., Shum, H.-Y., Tang, C.-K., and Szeliski, R. 2004. Stereo reconstruction from multiperspective panoramas. IEEE Trans. Pattern Analysis and Machine Intelligence, 26(1):44–62.

    Google Scholar 

  • McMillan, L. and Bishop, G. 1995. Plenoptic modeling: An image-based rendering system. In ACM Conf. Proc. SIGGRAPH 95, pp. 39–46.

  • Morimoto, C. and Chellappa, R. 1997. Fast 3-D stabilization and mosaic construction. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 660–665.

  • Murray, D.W. 1995. Recovering range using virtual multicamera stereo. Computer Vision and Image Understanding, 61(2):285–291.

    Google Scholar 

  • Nayar, S. and Karmarkar, 2000. 360 x 360 mosaics. In IEEE Conf. Computer Vision and Pattern Recognition, II:388–395.

    Google Scholar 

  • Niyogi, S.A. 1995. Detecting kinetic occlusion. In Proc. IEEE Int. Conf. Computer Vision, pp. 1044–1049.

  • Peleg, S. and Ben-Ezra, M. 1999. Stereo panorama with a single camera. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 395–401.

  • Peleg, S., Ben-Ezra, M., and Pritch, Y., 2001. OmniStereo: Panoramic stereo imaging. IEEE Trans. on PAMI, 23(3):279–290.

    Google Scholar 

  • Peleg, S. and Herman, J. 1997. Panoramic mosaics by manifold projection. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 338–343.

  • Peleg, S., Rousso, B., Rav-Akha, A., and Zomet, A. 2000. Mosaicing on adaptive manifolds. IEEE Trans. Pattern Recognition and Machine Analysis, 22(10):1144–1154.

    Google Scholar 

  • Rademacher, P. and Bishop, G. 1998. Multiple-center-of-projection images. In Proc. SIGGRAPH'98, pp. 199–206.

  • Sawhney, H.S. and Ayer, S. 1996. Compact representation of videos through dominant and multiple motion estimation. IEEE Trans Pattern Analysis and Machine Intelligence, 18(8):814–830.

    Google Scholar 

  • Sawhney, H.S., Kumar, R., Gendel, G., Bergen, J., Dixon, D., and Paragano, V. 1998. VideoBrush TM: Experiences with consumer video mosaicing. In IEEE Workshop on Application of Computer Vision, pp. 56–62.

  • Shade, J., Gortler, S., He. L., and Szeliski, R. 1998. Layered depth image. In Proc. SIGGRAPH'98, pp. 231–242.

  • Shum, H.-Y. and Szeliski, R. 2000. Construction of panoramic image mosaics with global and local alignment. Int. J. of Computer Vision, 36(2):101–130.

    Google Scholar 

  • Shum, H.-Y., Han, M., and Szeliski, R. 1998. Interactive construction of 3D models from panoramic mosaics. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 427–433.

  • Shum, H.-Y. and Szeliski, R., 1999. Stereo reconstruction from multi-perspective panoramas. In Proc. IEEE Int. Conf. Computer Vision, pp. 14–21.

  • Shum, H.-Y., Kalai, A., and Seitz, S.M. 1999. Omnivergent stereo. In Proc. IEEE Int. Conf. Computer Vision, pp. 22–29.

  • Szeliski, R. 1999. A multi-view approach to motion and stereo. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 157–163.

  • Wang, J. and Adelson, E.H. 1994. Representation moving images with layers. IEEE Trans. on Image Processing, 3(5):625–638.

    Google Scholar 

  • Xiong, Y. and Turkowski, K. 1997. Creating image-based VR using a self-calibrating fisheye lens. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 237–243.

  • Zheng, J.Y. and Tsuji, S. 1992. Panoramic representation for route recognition by a mobile robot. Int. J. Computer Vision, 9(1):55–76.

    Google Scholar 

  • Zheng, J.Y. and Tsuji, S. 1998. Generating dynamic projection images for scene representation and understanding. Computer Vision and Image Understanding, 72(3):237–256.

    Google Scholar 

  • Zhu, Z., Xu, G., and Lin, X. 1998. Constructing 3Dnatural scene from video sequences with vibrating motions. In Proc. IEEE Virtual Reality Annual International Symposium (VRAIS-98), pp. 105–112.

  • Zhu, Z., Xu, G., and Lin, X., 1999. Panoramic EPI generation and analysis of video from a moving platform with vibration. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 531–537.

  • Zhu, Z. 2001. Full View Spatio-Temporal Visual Navigation—Imaging, Modeling and Representation of Real Scenes, China Higher Education Press, December 2001 (based on his Ph.D. Thesis, Department of Computer Science and Technology, Tsinghua University, 1997. English Version may be found at http://www-cs. engr.ccny.cuny.edu/~zhu/PhD-Thesis/).

  • Zhu, Z. and Hanson, A.R. 2001. 3D LAMP: A new layered panoramic representation. In Proc. IEEE Int. Conf. Computer Vision, vol. II, pp. 723–730.

    Google Scholar 

  • Zhu, Z., Riseman, E.M., and Hanson, A.R. 2001. Parallel-perspective stereo mosaics. In Proc. IEEE Int. Conf. Computer Vision, vol. I, pp. 345–352.

    Google Scholar 

  • Zhu, Z., Riseman, E.M. and Hanson, A.R., 2004. Generalized parallel-perspective stereo mosaics from airborne videos. IEEE Trans. Pattern Analysis and Machine Intelligence, 26(2):226–237.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, The City College, The City University of New York, New York, NY, 10031, USA

    Zhigang Zhu

  2. Department of Computer Science and Technology, Tsinghua University, Beijing, 100084, People's Republic of China

    Guangyou Xu & Xueyin Lin

Authors
  1. Zhigang Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guangyou Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xueyin Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhu, Z., Xu, G. & Lin, X. Efficient Fourier-Based Approach for Detecting Orientations and Occlusions in Epipolar Plane Images for 3D Scene Modeling. International Journal of Computer Vision 61, 233–258 (2005). https://doi.org/10.1023/B:VISI.0000045325.10439.3f

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/B:VISI.0000045325.10439.3f

Search

Navigation