Abstract
This paper addresses the problem of registering a known 3D model to a set of 2D deforming image trajectories. The proposed approach can adapt to a scenario where the 3D model to register is not an exact description of the measured image data. This results in finding a 2D–3D registration, given the complexity of having both 2D deforming data and a coarse description of the image observations. The method acts in two distinct phases. First, an affine step computes a factorization for both the 2D image data and the 3D model using a joint subspace decomposition. This initial solution is then upgraded by finding the best projection to the image plane complying with the metric constraints given by a scaled orthographic camera. Both steps are computed efficiently in closed-form with the additional feature of being robust to degenerate motions which may possibly affect the 2D image data (i.e. lack of relevant rigid motion). A further extension of the approach allows to compute the full 3D deformations of the shape given the first initial (rigid) registration. This step results in solving a Non-rigid Structure from Motion (NRSfM) problem using the 3D known shape as a prior. Experimental results show the robustness of the method in registration tasks such as pose estimation and 3D reconstruction when degenerate image motion is present.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Marques, M., & Costeira, J. (2009). Estimating 3d shape from degenerate sequences with missing data. Computer Vision and Image Understanding, 113(2), 261–272.
Akhter, I., Sheikh, Y., & Khan, S. (2009). In defense of orthonormality constraints for nonrigid structure from motion. In Proc. IEEE conference on computer vision and pattern recognition, Miami, Florida.
Bartoli, A., Pizarro, D., & Loog, M. (2010). Stratified generalized procrustes analysis. In Proceedings of the British machine vision conference (pp. 70.1–70.10).
Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S., & Sayd, P. (2008). Coarse-to-fine low-rank structure-from-motion. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska (pp. 1–8).
Basri, R., Jacobs, D., & Kemelmacher, I. (2007). Photometric stereo with general, unknown lighting. International Journal of Computer Vision, 72(3), 239–257.
Björck, Å. (1996). Numerical methods for least squares problems. SIAM: Philadelphia.
Brand, M. (2005). A direct method for 3d factorization of nonrigid motion observed in 2d. In Proc. IEEE conference on computer vision and pattern recognition (pp. 122–128), San Diego, California.
Bregler, C., Hertzmann, A., & Biermann, H. (2000). Recovering non-rigid 3d shape from image streams. In Proc. IEEE conference on computer vision and pattern recognition, Hilton Head, South Carolina (pp. 690–696).
Del Bue, A. (2008). A factorization approach to structure from motion with shape priors. In Proc. IEEE conference on computer vision and pattern recognition, Anchorage, Alaska (pp. 1–8).
Del Bue, A. (2010). Adaptive metric registration of 3d models to non-rigid image trajectories. In K. Daniilidis, P. Maragos, & N. Paragios (Eds.), Lecture Notes in Computer Science: Vol. 6313. 11th European conference on computer vision (ECCV 2010), Crete, Greece (pp. 87–100). Berlin: Springer.
Del Bue, A., Lladó, X., & Agapito, L. (2006). Non-rigid metric shape and motion recovery from uncalibrated images using priors. In Proc. IEEE conference on computer vision and pattern recognition, New York, NY (pp. 1191–1198).
Del Bue, A., Smeraldi, F., & Agapito, L. (2007). Non-rigid structure from motion using ranklet–based tracking and non-linear optimization. Image and Vision Computing, 25(3), 297–310.
Del Bue, A., Xavier, J., Agapito, L., & Paladini, M. (2012). Bilinear modelling via augmented Lagrange multipliers (BALM). IEEE Trans. Pattern Anal. Machine Intell. 34(8) 1496–1508.
Forsyth, D., Ioffe, S., & Haddon, J. (1999). Bayesian structure from motion. In Proc. 7th international conference on computer vision 1, Kerkyra, Greece (p. 660).
Hansen, P. (1998). Rank-deficient and discrete Ill-posed problems: numerical aspects of linear inversion. Society for Industrial Mathematics.
Olsen, S., & Bartoli, A. (2008). Implicit non-rigid structure-from-motion with priors. Journal of Mathematical Imaging and Vision, 31(2), 233–244.
Solem, J., & Kahl, F. (2005). Surface reconstruction using learned shape models. Advances in neural information processing systems 17
Stegmann, M. B., Ersbøll, B. K., & Larsen, R. (2003). FAME—a flexible appearance modelling environment. IEEE Transactions on Medical Imaging, 22(10), 1319–1331.
Sturm, J. (1999). Using SeDuMi 1.02, a Matlab toolbox for optimization over symmetric cones. Optimization Methods & Software, 11(1), 625–653.
Tomasi, C., & Kanade, T. (1992). Shape and motion from image streams under orthography: a factorization approach. International Journal of Computer Vision, 9(2), 137–154.
Torresani, L., Hertzmann, A., & Bregler, C. (2008). Non-rigid structure-from-motion: estimating shape and motion with hierarchical priors. In IEEE transactions on pattern analysis and machine intelligence (pp. 878–892).
Triggs, B., McLauchlan, P., Hartley, R. I., & Fitzgibbon, A. (2000). Bundle adjustment—a modern synthesis. In W. Triggs, A. Zisserman, & R. Szeliski (Eds.), Vision algorithms: theory and practice, LNCS (pp. 298–375). Berlin: Springer citeseer.nj.nec.com/triggs00bundle.html.
Xiao, J., Baker, S., Matthews, I., & Kanade, T. (2004). Real-time combined 2d+3d active appearance models. In Proceedings of the IEEE conference on computer vision and pattern recognition, vol. 2 (pp. 535–542).
Xiao, J., Chai, J., & Kanade, T. (2006). A closed-form solution to non-rigid shape and motion recovery. International Journal of Computer Vision, 67(2), 233–246.
Yezzi, A. J., & Soatto, S. (2003). Deformation: deforming motion, shape average and the joint registration and approximation of structures in images. International Journal of Computer Vision, 53(2), 153–167.
Acknowledgements
This work was partially supported by Fundação para a Ciência e a Tecnologia (ISR/IST pluriannual funding) through the POS_Conhecimento Program (include FEDER funds) and grant PTDC/EEA-ACR/72201/2006, “MODI—3D Models from 2D Images”. E. Muñoz, J. Xiao and J. Peyras kindly made available sequences used in the experimental section.
Author information
Authors and Affiliations
Corresponding author
Electronic Supplementary Material
Below is the link to the electronic supplementary material.
(WMV 4.1 MB)
Rights and permissions
About this article
Cite this article
Del Bue, A. Adaptive Non-rigid Registration and Structure from Motion from Image Trajectories. Int J Comput Vis 103, 226–239 (2013). https://doi.org/10.1007/s11263-012-0577-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-012-0577-9