Skip to main content
Springer Nature Link
Log in

Adaptive Non-rigid Registration and Structure from Motion from Image Trajectories

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

This paper addresses the problem of registering a known 3D model to a set of 2D deforming image trajectories. The proposed approach can adapt to a scenario where the 3D model to register is not an exact description of the measured image data. This results in finding a 2D–3D registration, given the complexity of having both 2D deforming data and a coarse description of the image observations. The method acts in two distinct phases. First, an affine step computes a factorization for both the 2D image data and the 3D model using a joint subspace decomposition. This initial solution is then upgraded by finding the best projection to the image plane complying with the metric constraints given by a scaled orthographic camera. Both steps are computed efficiently in closed-form with the additional feature of being robust to degenerate motions which may possibly affect the 2D image data (i.e. lack of relevant rigid motion). A further extension of the approach allows to compute the full 3D deformations of the shape given the first initial (rigid) registration. This step results in solving a Non-rigid Structure from Motion (NRSfM) problem using the 3D known shape as a prior. Experimental results show the robustness of the method in registration tasks such as pose estimation and 3D reconstruction when degenerate image motion is present.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Algorithm 2
Fig. 2
Algorithm 3
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 15
Fig. 13
Fig. 14
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Marques, M., & Costeira, J. (2009). Estimating 3d shape from degenerate sequences with missing data. Computer Vision and Image Understanding, 113(2), 261–272.

    Article  Google Scholar 

  • Akhter, I., Sheikh, Y., & Khan, S. (2009). In defense of orthonormality constraints for nonrigid structure from motion. In Proc. IEEE conference on computer vision and pattern recognition, Miami, Florida.

    Google Scholar 

  • Bartoli, A., Pizarro, D., & Loog, M. (2010). Stratified generalized procrustes analysis. In Proceedings of the British machine vision conference (pp. 70.1–70.10).

    Google Scholar 

  • Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S., & Sayd, P. (2008). Coarse-to-fine low-rank structure-from-motion. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska (pp. 1–8).

    Google Scholar 

  • Basri, R., Jacobs, D., & Kemelmacher, I. (2007). Photometric stereo with general, unknown lighting. International Journal of Computer Vision, 72(3), 239–257.

    Article  Google Scholar 

  • Björck, Å. (1996). Numerical methods for least squares problems. SIAM: Philadelphia.

    Book  MATH  Google Scholar 

  • Brand, M. (2005). A direct method for 3d factorization of nonrigid motion observed in 2d. In Proc. IEEE conference on computer vision and pattern recognition (pp. 122–128), San Diego, California.

    Google Scholar 

  • Bregler, C., Hertzmann, A., & Biermann, H. (2000). Recovering non-rigid 3d shape from image streams. In Proc. IEEE conference on computer vision and pattern recognition, Hilton Head, South Carolina (pp. 690–696).

    Google Scholar 

  • Del Bue, A. (2008). A factorization approach to structure from motion with shape priors. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska (pp. 1–8).

    Google Scholar 

  • Del Bue, A. (2010). Adaptive metric registration of 3d models to non-rigid image trajectories. In K. Daniilidis, P. Maragos, & N. Paragios (Eds.), Lecture Notes in Computer Science: Vol. 6313. 11th European conference on computer vision (ECCV 2010), Crete, Greece (pp. 87–100). Berlin: Springer.

    Chapter  Google Scholar 

  • Del Bue, A., Lladó, X., & Agapito, L. (2006). Non-rigid metric shape and motion recovery from uncalibrated images using priors. In Proc. IEEE conference on computer vision and pattern recognition, New York, NY (pp. 1191–1198).

    Google Scholar 

  • Del Bue, A., Smeraldi, F., & Agapito, L. (2007). Non-rigid structure from motion using ranklet–based tracking and non-linear optimization. Image and Vision Computing, 25(3), 297–310.

    Article  Google Scholar 

  • Del Bue, A., Xavier, J., Agapito, L., & Paladini, M. (2012). Bilinear modelling via augmented Lagrange multipliers (BALM). IEEE Trans. Pattern Anal. Machine Intell. 34(8) 1496–1508.

    Article  Google Scholar 

  • Forsyth, D., Ioffe, S., & Haddon, J. (1999). Bayesian structure from motion. In Proc. 7th international conference on computer vision 1, Kerkyra, Greece (p. 660).

    Chapter  Google Scholar 

  • Hansen, P. (1998). Rank-deficient and discrete Ill-posed problems: numerical aspects of linear inversion. Society for Industrial Mathematics.

    Book  Google Scholar 

  • Olsen, S., & Bartoli, A. (2008). Implicit non-rigid structure-from-motion with priors. Journal of Mathematical Imaging and Vision, 31(2), 233–244.

    Article  MathSciNet  Google Scholar 

  • Solem, J., & Kahl, F. (2005). Surface reconstruction using learned shape models. Advances in neural information processing systems 17

  • Stegmann, M. B., Ersbøll, B. K., & Larsen, R. (2003). FAME—a flexible appearance modelling environment. IEEE Transactions on Medical Imaging, 22(10), 1319–1331.

    Article  Google Scholar 

  • Sturm, J. (1999). Using SeDuMi 1.02, a Matlab toolbox for optimization over symmetric cones. Optimization Methods & Software, 11(1), 625–653.

    Article  MathSciNet  Google Scholar 

  • Tomasi, C., & Kanade, T. (1992). Shape and motion from image streams under orthography: a factorization approach. International Journal of Computer Vision, 9(2), 137–154.

    Article  Google Scholar 

  • Torresani, L., Hertzmann, A., & Bregler, C. (2008). Non-rigid structure-from-motion: estimating shape and motion with hierarchical priors. In IEEE transactions on pattern analysis and machine intelligence (pp. 878–892).

    Google Scholar 

  • Triggs, B., McLauchlan, P., Hartley, R. I., & Fitzgibbon, A. (2000). Bundle adjustment—a modern synthesis. In W. Triggs, A. Zisserman, & R. Szeliski (Eds.), Vision algorithms: theory and practice, LNCS (pp. 298–375). Berlin: Springer citeseer.nj.nec.com/triggs00bundle.html.

    Chapter  Google Scholar 

  • Xiao, J., Baker, S., Matthews, I., & Kanade, T. (2004). Real-time combined 2d+3d active appearance models. In Proceedings of the IEEE conference on computer vision and pattern recognition, vol. 2 (pp. 535–542).

    Google Scholar 

  • Xiao, J., Chai, J., & Kanade, T. (2006). A closed-form solution to non-rigid shape and motion recovery. International Journal of Computer Vision, 67(2), 233–246.

    Article  Google Scholar 

  • Yezzi, A. J., & Soatto, S. (2003). Deformation: deforming motion, shape average and the joint registration and approximation of structures in images. International Journal of Computer Vision, 53(2), 153–167.

    Article  Google Scholar 

Download references

Acknowledgements

This work was partially supported by Fundação para a Ciência e a Tecnologia (ISR/IST pluriannual funding) through the POS_Conhecimento Program (include FEDER funds) and grant PTDC/EEA-ACR/72201/2006, “MODI—3D Models from 2D Images”. E. Muñoz, J. Xiao and J. Peyras kindly made available sequences used in the experimental section.

Author information

Authors and Affiliations

  1. Pattern Analysis and Computer Vision (PAVIS), Istituto Italiano di Tecnologia (IIT), Via Morego 30, 16163, Genova, Italy

    Alessio Del Bue

Authors
  1. Alessio Del Bue
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alessio Del Bue.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

(WMV 4.1 MB)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Del Bue, A. Adaptive Non-rigid Registration and Structure from Motion from Image Trajectories. Int J Comput Vis 103, 226–239 (2013). https://doi.org/10.1007/s11263-012-0577-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-012-0577-9

Keywords

Search

Navigation