Abstract
In this paper we address the problem of recovering 3D non-rigid structure from a sequence of images taken with a stereo pair. We have extended existing non-rigid factorization algorithms to the stereo camera case and presented an algorithm to decompose the measurement matrix into the motion of the left and right cameras and the 3D shape, represented as a linear combination of basis-shapes. The added constraints in the stereo camera case are that both cameras are viewing the same structure and that the relative orientation between both cameras is fixed. Our focus in this paper is on the recovery of flexible 3D shape rather than on the correspondence problem. We propose a method to compute reliable 3D models of deformable structure from stereo images. Our experiments with real data show that improved reconstructions can be achieved using this method. The algorithm includes a non-linear optimization step that minimizes image reprojection error and imposes the correct structure to the motion matrix by choosing an appropriate parameterization. We show that 3D shape and motion estimates can be successfully disambiguated after bundle adjustment and demonstrate this on synthetic and real image sequences. While this optimization step is proposed for the stereo camera case, it can be readily applied to the case of non-rigid structure recovery using a monocular video sequence.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Aanaes, H. and Kahl, F. 2002, Estimation of deformable structure and motion. In Workshop on Vision and Modelling of Dynamic Scenes, ECCV'02, Copenhagen, Denmark.
Bar-Itzhack, I.Y. 2000. New method for extracting the quaternion from a rotation matrix. Journal of Guidance, Control and Dynamics, 23(3):1085–1087.
Brand, M. 2001. Morphable models from video. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, Kauai, Hawaii.
Brand, M. and Bhotika, R. 2001. Flexible flow for 3D nonrigid tracking and shape recovery. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, Kauai, Hawaii, pp. 315–322.
Bregler, C., Hertzmann, A., and Biermann, H. 2000. Recovering non-rigid 3D shape from image streams. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, Hilton Head, South Carolina, pp. 690–696.
Del Bue, A. and Agapito, L. 2004. Non-rigid 3D shape recovery using stereo factorization. In Asian Conference on Computer Vision (ACCV2004), vol. 1. Jeju, South Korea.
Del Bue, A., Smeraldi, F., and Agapito, L. 2004, Non-rigid structure from motion using nonparametric tracking and non-linear optimization. In Workshop in Articulated and Nonrigid Motion ANM04, held in Conjunction with CVPR2004. Washington.
Essa, I. and Basu, S. 1996. Modeling, tracking and interactive animation of facial expressions and head movements using input from video. In Proceedings of Computer Animation Conference. Geneva, Switzerland.
Horn, B. 1987, Closed form solutions of absolute orientation using unit quaternions. J. Optical Soc. of America A. 4(4): 629–642.
Irani, M. 1999, Multi-frame optical flow estimation using subspace constraints. In Proc. 7th International Conference on Computer Vision, Kerkyra, Greece.
Parke, F.I. and Waters, K. 1996. Computer Facial Animation. A.K. Peters, Ltd.
Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D.H. 1998. Synthesising realistic facial expressions from photographs. In Proceedings of the ACM SIGGRAPH Conference on Computer Graphics.
Sugaya, Y. and Kanatani, K. 2004. Extending interrupted feature point tracking for 3-D affine reconstruction. IEICE Transactions on Information and System, E87-D(4):1031–1039.
Tan, J. and Ishikawa, S. 2001. Deformable shape recovery by factorization based on a spatiotemporal measurement matrix. Computer Vision and Image Understanding, 82:101–109.
Tomasi, C. and Kanade, T. 1991. Shape and motion from image streams: A factorization method. International Journal in Computer Vision, 9(2):137–154.
Torresani, L., Yang, D., Alexander, E., and Bregler, C. 2001. Tracking and modeling non-rigid objects with rank constraints. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, Kauai, Hawaii.
Tresadern, P. and Reid, I. 2003. Synchronizing image sequences of non-rigid objects. In Proc. British Machine Vision Conference, Norwich.
Triggs, B., McLauchlan, P., Hartley, R., and Fitzgibbon., A. 2000. Bundle adjustment---A modern synthesis. In W. Triggs, A. Zisserman, and R. Szeliski (Eds.), Vision Algorithms: Theory and Practice, LNCS. Springer Verlag, pp. 298–375.
Vetter, T. and Blanz, V. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of the ACM SIGGRAPH Conference on Computer Graphics, pp. 187–194.
Xiao, J., Chai, J., and Kanade, T. 2004. A closed-form solution to non-rigid shape and motion recovery. In The 8th European Conference on Computer Vision (ECCV 2004).
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Del Bue, A., Agapito, L. Non-Rigid Stereo Factorization. Int J Comput Vision 66, 193–207 (2006). https://doi.org/10.1007/s11263-005-3958-5
Received:
Revised:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/s11263-005-3958-5