Abstract
The problem of image registration subsumes a number of problems and techniques in multiframe image analysis, including the computation of optic flow (general pixel-based motion), stereo correspondence, structure from motion, and feature tracking. We present a new registration algorithm based on spline representations of the displacement field which can be specialized to solve all of the above mentioned problems. In particular, we show how to compute local flow, global (parametric) flow, rigid flow resulting from camera egomotion, and multiframe versions of the above problems. Using a spline-based description of the flow removes the need for overlapping correlation windows, and produces an explicit measure of the correlation between adjacent flow estimates. We demonstrate our algorithm on multiframe image registration and the recovery of 3D projective scene geometry. We also provide results on a number of standard motion sequences.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Adelson, E. H. and Bergen, J. R. 1985. Spatiotemporal energy models for the perception of motion. Journal of the Optical Society of America, A2(2):284–299.
Amit, Y. 1993. Anon-linear variational problem for image matching. unpublished manuscript (from Newton Institute).
Anandan, P. 1989. A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 2(3):283–310.
Bajcsy, R. and Broit, C. 1982. Matching of deformed images. In Sixth International Conference on Pattern Recognition (ICPRs'82), IEEE Computer Society Press: Munich, Germany, pp. 351–353.
Bajcsy, R. and Kovacic, S. 1989. Multiresolution elastic matching. Computer Vision, Graphics, and Image Processing, 46(1):1–21.
Barnard, S. T. and Fischler, M. A. 1982. Computational stereo. Computing Surveys, 14(4):553–572.
Barron, J. L., Fleet, D. J., and Beauchemin, S. S. 1994. Performance of optical flow techniques. International Journal of Computer Vision, 12(1):43–77.
Beier, T. and Neely, S. 1992. Feature-based image metamorphosis. Computer Graphics (SIGGRAPHs'92), 26(2):35–42.
Bergen, J. R., Anandan, P., Hanna, K. J., and Hingorani, R. 1992. Hierarchical model-based motion estimation. In Second European Conference on Computer Vision (ECCVs'92), Santa Margherita Liguere, Springer-Verlag: Italy, pp. 237–252.
Beymer, D., Shashua, A., and Poggio, T. 1993. Example based image analysis and synthesis. A. I. Memo 1431, Massachusetts Institute of Technology.
Blake, A., Curwen, R., and Zisserman, A. 1993. A framework for spatio-temporal control in the tracking of visual contour. International Journal of Computer Vision, 11(2):127–145.
Bolles, R. C., Baker, H. H., and Marimont, D. H. 1987. Epipolar-plane image analysis: An approach to determining structure from motion. International Journal of Computer Vision, 1:7–55.
Brown, L. G. 1992. A survey of image registration techniques. Computing Surveys, 24(4):325–376.
Burr, D. J. 1981. A dynamic model for image registration. Computer Graphics and Image Processing, 15(2):102–112.
Burt, P. J. and Adelson, E. H. 1983. The Laplacian pyramid as a compact image code. IEEE Transactions on Communications, COM-31(4):532–540.
Carlbom, I., Terzopoulos, D., and Harris, K. M. 1991. Reconstructing and visualizing models of neuronal dendrites. In Scientific Visualization of Physical Phenomena, N. M. Patrikalakis (Ed.), Springer-Verlag: New York, pp. 623–638.
Dhond, U. R. and Aggarwal, J. K. 1989. Structure from stereo—A review. IEEE Transactions on Systems, Man, and Cybernetics, 19(6):1489–1510.
Dreschler, L. and Nagel, H.-H. 1982. Volumetric model and 3D trajectory of a moving car derived from monocular TV frame sequences of a stree scene. Computer Graphics and Image Processing, 20:199–228.
Enkelmann, W. 1988. Investigations of multigrid algorithms for estimation of optical flow fields in image sequences. Computer Vision, Graphics, and Image Processing, pp. 150–177.
Farin, G. E. 1992. Curves and Surfaces for Computer Aided Geometric Design. Academic Press: Boston, Massachusetts, 3rd edition.
Faugeras, O. D. 1992. What can be seen in three dimensions with an uncalibrated stereo rig? In Second European Conference on Computer Vision (ECCVs'92), Santa Margherita Liguere, Springer-Verlag: Italy, pp. 563–578.
Fleet, D. and Jepson, A. 1990. Computation of component image velocity from local phase information. International Journal of Computer Vision, 5:77–104.
Fuh, C.-S. and Maragos, P. 1991. Motion displacement estimation using an affine model for image matching. Optical Engineering, 30(7):881–887.
Geiger, D., Ladendorf, B., and Yuille, A. 1992. Occlusions and binocular stereo. In Second European Conference on Computer Vision (ECCVs'92), Santa Margherita Liguere, Springer-Verlag, Italy, pp. 425–433.
Gennert, M. A. 1988. Brightness-based stereo matching. In Second International Conference on Computer Vision (ICCVs'88), IEEE Computer Society Press: Tampa, Florida, pp. 139–143.
Goshtasby, A. 1986. Piecewise linear mapping functions for image registration. Pattern Recognition, 19(6):459–466.
Goshtasby, A. 1988. Image registration by local approximation methods. Image and Vision Computing, 6(4):255–261.
Hanna, K. J. 1991. Direct multi-resolution estimation of ego-motion and structure from motion. In IEEE Workshop on Visual Motion, IEEE Computer Society Press: Princeton, New Jersey, pp. 156–162.
Hartley, R. and Gupta, R. 1993. Computing matched-epipolar projections. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPRs'93), IEEE Computer Society Press: New York, pp. 549–555.
Hartley, R., Gupta, R., and Chang, T. 1992. Stereo from uncalibrated cameras. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPRs'92), IEEE Computer Society Press: Champaign, Illinois, pp. 761–764,.
Heeger, D. J. 1987. Optical flow from spatiotemporal filters. In First International Conference on Computer Vision (ICCVs'87), IEEE Computer Society Press: London, England, pp. 181–190.
Hildreth, E. C. 1986. Computing the velocity field along contours. In Motion: Representation and Perception, N. I. Badler and J. K. Tsotsos (Eds.), North-Holland, New York, pp. 121–127.
Horn, B. K. P. and Schunck, B. G. 1981. Determining optical flow. Artificial Intelligence, 17:185–203.
Horn, B. K. P. and Weldon, E. J., Jr. 1988. Direct methods for recovering motion. International Journal of Computer Vision, 2(1):51–76.
Kass, M., Witkin, A., and Terzopoulos, D. 1988. Snakes: Active contour models. International Journal of Computer Vision, 1(4):321–331.
Koenderink, J. J. and van Doorn, A. J. 1991. Affine structure from motion. Journal of the Optical Society of America A, 8:377–385,538.
Le Gall, D. 1991. MPEG: A video compression standard for multimedia applications. Communications of the ACM, 34(4):44–58.
Lucas, B. D. 1984. Generalized Image Matching by the Method of Differences. Ph. D. Thesis, Carnegie Mellon University.
Lucas, B. D. and Kanade, T. 1981. An iterative image registration technique with an application in stereo vision. In Seventh International Joint Conference on Artificial Intelligence (IJCAI-81), Vancouver, pp. 674–679.
Manmatha, R. and Oliensis, J. 1992. Measuring the affine transform —I: Scale and rotation. Technical Report 92-74, University of Massachussets, Amherst, Massachussets.
Matthies, L. H., Szeliski, R., and Kanade, T. 1989. Kalman filter-based algorithms for estimating depth from image sequences. International Journal of Computer Vision, 3:209–236.
Menet, S., Saint-Marc, P., and Medioni, G. 1990. B-snakes: implementation and applications to stereo. In Image Understanding Workshop, Morgan Kaufmann Publishers: Pittsburgh, Pennsylvania, pp. 720–726.
Mohr, R., Veillon, L., and Quan, L. 1993. Relative 3D reconstruction using multiple uncalibrated images. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPRs'93), New York, pp. 543–548.
Nagel, H.-H. 1987. On the estimation of optical flow: Relations between different approaches and some new results. Artificial Intelligence, 33:299–324.
Okutomi, M. and Kanade, T. 1992. A locally adaptive window for signal matching. International Journal of Computer Vision, 7(2):143–162.
Okutomi, M. and Kanade, T. 1993. A multiple baseline stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(4):353–363.
Poggio, T., Torre, V., and Koch, C. 1985. Computational vision and regularization theory. Nature, 317(6035):314–319.
Press, W. H., Flannery, B. P., Teukolsky, S. A., and Vetterling, W. T. 1992. Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press: Cambridge, England, 2nd edition.
Quam, L. H. 1984. Hierarchical warp stereo. In Image Understanding Workshop, Science Applications International Corporation: New Orleans, Louisiana, pp. 149–155.
Rehg, J. and Witkin, A. 1991. Visual tracking with deformation models. In IEEE International Conference on Robotics and Automation, IEEE Computer Society Press: Sacramento, California, pp. 844–850.
Sethi, I. K. and Jain, R. 1987. Finding trajectories of feature points in a monocular image sequence. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-9(1):56–73.
Shi, J. and Tomasi, C. 1994. Good features to track. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPRs'94), IEEE Computer Society: Seattle, Washington, pp. 593–600.
Simoncelli, E. P., Adelson, E. H., and Heeger, D. J. 1991. Probability distributions of optic flow. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPRs'91), IEEE Computer Society Press: Maui, Hawaii, pp. 310–315.
Singh, A. 1990. An estimation-theoretic framework for image-flow computation. In Third International Conference on Computer Vision (ICCVs'90), IEEE Computer Society Press: Osaka, Japan, pp. 168–177.
Szeliski, R. 1989. Bayesian Modeling of Uncertainty in Low-Level Vision. Kluwer Academic Publishers: Boston, Massachusetts.
Szeliski, R. 1990. Fast surface interpolation using hierarchical basis functions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(6):513–528.
Szeliski, R. 1996. Video mosaics for virtual environments. IEEE Computer Graphics and Applications, 16(2):22–30.
Szeliski, R. and Kang, S. B. 1994. Recovering 3D shape and motion from image streams using nonlinear least squares. Journal of Visual Communication and Image Representation, 5(1):10–28.
Szeliski, R. and Kang, S. B. 1995. Direct methods for visual scene reconstruction. In IEEE Workshop on Representations of Visual Scenes, Cambridge, Massachusetts, pp. 26–33.
Szeliski, R. and Shum, H.-Y. 1996. Motion estimation with quadtree splines. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(12):1199–1210.
Szeliski, R., Kang, S. B., and Shum, H.-Y. 1995. A parallel feature tracker for extended image sequences. In IEEE International Symposium on Computer Vision, Coral Gables, Florida, pp. 241–246.
Terzopoulos, D. 1986. Regularization of inverse visual problems involving discontinuities. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-8(4):413–424.
Tomasi, C. and Kanade, T. 1992. Shape and motion from image streams under orthography: A factorization method. International Journal of Computer Vision, 9(2):137–154.
Witkin, A., Terzopoulos, D., and Kass, M. 1987. Signal matching through scale space. International Journal of Computer Vision, 1:133–144.
Wolberg, G. 1990. Digital Image Warping. IEEE Computer Society Press: Los Alamitos, California.
Xu, G., Tsuji, S., and Asada, M. 1987. Amotion stereo method based on coarse-to-fine control strategy. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-9(2):332–336.
Zheng, Q. and Chellappa, R. 1992. Automatic feature point extraction and tracking in image sequences for arbitrary camera motion. Technical Report CAR-TR-628, Computer Vision Laboratory, Center for Automation Research, University of Maryland.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Szeliski, R., Coughlan, J. Spline-Based Image Registration. International Journal of Computer Vision 22, 199–218 (1997). https://doi.org/10.1023/A:1007996332012
Issue Date:
DOI: https://doi.org/10.1023/A:1007996332012