Abstract
We present a novel automatic method for high resolution, non-rigid dense 3D point tracking. High quality dense point clouds of non-rigid geometry moving at video speeds are acquired using a phase-shifting structured light ranging technique. To use such data for the temporal study of subtle motions such as those seen in facial expressions, an efficient non-rigid 3D motion tracking algorithm is needed to establish inter-frame correspondences. The novelty of this paper is the development of an algorithmic framework for 3D tracking that unifies tracking of intensity and geometric features, using harmonic maps with added feature correspondence constraints. While the previous uses of harmonic maps provided only global alignment, the proposed introduction of interior feature constraints allows to track non-rigid deformations accurately as well. The harmonic map between two topological disks is a diffeomorphism with minimal stretching energy and bounded angle distortion. The map is stable, insensitive to resolution changes and is robust to noise. Due to the strong implicit and explicit smoothness constraints imposed by the algorithm and the high-resolution data, the resulting registration/deformation field is smooth, continuous and gives dense one-to-one inter-frame correspondences. Our method is validated through a series of experiments demonstrating its accuracy and efficiency.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akgul, Y., & Kambhamettu, C. (1999). Recovery and tracking of continuous 3d surfaces from stereo data using a deformable dual-mesh. In IEEE international conference on computer vision (pp. 765–772).
Allen, B., Curless, B., & Popović, Z. (2003). The space of human body shapes: reconstruction and parameterization from range scans. ACM Transactions on Graphics, 22(3), 587–594.
Basu, S., Oliver, N., & Pentland, A. (1998). 3d lip shapes from video: a combined physical-statistical model. Speech Communication, 26(1-2), 131–148.
Beauchemin, S. S., & Barron, J. L. (1995). The computation of optical flow. ACM Computing Surveys, 27(3), 433–466.
Besl, P., & McKay, N. (1992). A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2).
Black, M. J., & Yacoob, Y. (1995). Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In IEEE international conference on computer vision (pp. 374–381).
Blanz, V., & Vetter, T. (2003). Face recognition based on fitting a 3d morphable model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(9), 1063–1074.
Brand, M., & Bhotika, R. (2001). Flexible flow for 3d nonrigid tracking and shape recovery. In IEEE computer vision and pattern recognition (Vol. I, pp. 315–322).
Bronstein, A. M., Bronstein, M. M., & Kimmel, R. (2005). Three-dimensional face recognition. International Journal of Computer Vision, 64(1), 5–30.
Bronstein, A. M., Bronstein, M. M., & Kimmel, R. (2006a). Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching. Proceedings of the National Academy of Sciences, 103(5), 1168–1172.
Bronstein, A. M., Bronstein, M. M., & Kimmel, R. (2006b). Efficient computation of isometry-invariant distances between surfaces. SIAM Journal of Scientific Computing, 28(5), 1812–1836.
Chai, J., Xiao, J., & Hodgins, J. (2003). Vision-based control of 3d facial animation. In ACM SIGGRAPH/Eurographics symposium on computer animation (pp. 193–206).
Chen, Y., & Medioni, G. G. (1991). Object modeling by registration of multiple range images. In IEEE conference on robotics and automation (pp. 2724–2729).
Davis, J., Ramamoorthi, R., & Rusinkiewicz, S. (2003). Spacetime stereo: a unifying framework for depth from triangulation. In IEEE computer vision and pattern recognition (pp. 359–366).
DeCarlo, D., & Metaxas, D. (2002). Adjusting shape parameters using model-based optical flow residuals. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(6), 814–823.
Dimitrijevic, M., Ilic, S., & Fua, P. (2004). Accurate face models from uncalibrated and ill-lit video sequences. In IEEE computer vision and pattern recognition (Vol. II, pp. 1034–1041).
Eck, M., DeRose, T., Duchamp, T., Hoppe, H., Lounsbery, M., & Stuetzle, W. (1995). Multiresolution analysis of arbitrary meshes. In ACM SIGGraph, computer graphics (pp. 173–182).
Eells, J., & Sampson, J. H. (1964). Harmonic mappings of Riemannian manifolds. American Journal of Mathematics, 86, 109–160.
Essa, I. A., & Pentland, A. P. (1997). Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 757–763.
Evans, L. C. (1998). Partial differential equations. Providence: American Mathematical Society.
Gokturk, S. B., Bouguet, J. Y., & Grzeszczuk, R. (2001). A data-driven model for monocular face tracking. In IEEE international conference on computer vision (pp. 701–708).
Goldenstein, S. K., Vogler, C., & Metaxas, D. (2003). Statistical cue integration in dag deformable models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(7), 801–813.
Gu, X., & Yau, S. (2003). Surface classification using conformal structures. In IEEE international conference on computer vision (pp. 701–708).
Guenter, B., Grimm, C., Wood, D., Malvar, H., & Pighin, F. (1998). Making faces. In ACM SIGGraph, computer graphics (pp. 55–66).
Huang, P. S., & Zhang, S. (2004). High resolution, real time 3-d shape acquisition. In IEEE workshop on real-time 3D sensors and their use (joint with CVPR’04).
Huang, X., Paragios, N., & Metaxas, D. (2003). Establishing local correspondences towards compact representations of anatomical structures. In International conference on medical image computing and computer assisted intervention (pp. 926–934).
Huang, X., Zhang, S., Wang, Y., Metaxas, D., & Samaras, D. (2004). A hierarchical framework for high resolution facial expression tracking. In IEEE workshop on articulated and nonrigid motion.
Hughes, T. (1987). The finite element method. New York: Prentice-Hall.
Kalberer, G. A., & Van Gool, L. (2001). Face animation based on observed 3d speech dynamics. In IEEE conference on computer animation.
Lien, J. J., Kanade, T. K., Zlochower, A. Z., Cohn, J. F., & Li, C. C. (1998). Subtly different facial expression recognition and expression intensity estimation. In IEEE computer vision and pattern recognition (pp. 853–859).
Litke, N., Droske, M., Rumpf, M., & Schröder, P. (2005). An image processing approach to surface matching. In Eurographics symposium on geometry processing (pp. 207–241).
Noh, J.-Y., & Neumann, U. (2001). Expression cloning. In ACM SIGGraph, computer graphics (pp. 277–288).
O’Neill, B. (2001). Elementary differential geometry. New York, Academic Press.
Pighin, F., Szeliski, R., & Salesin, D. (1999). Resynthesizing facial animation through 3d model-based tracking. In IEEE international conference on computer vision (pp. 143–150).
Ramanan, D., & Forsyth, D. A. (2003). Finding and tracking people from the bottom up. In IEEE computer vision and pattern recognition (Vol. II, pp. 467–474).
Rittscher, J., Blake, A., & Roberts, S. J. (2002). Towards the automatic analysis of complex human body motions. Image and Vision Computing, 20(12), 905–916.
Rusinkiewicz, S., & Hall-Holt, O. (2002). Levoy Marc. Real-time 3d model acquisition. In ACM SIGGraph, computer graphics (Vol. 1281, pp. 438–446).
Schoen, R., & Yau, S. T. (1997). Lectures on harmonic maps. Cambridge: International Press, Harvard University.
Sharon, E., & Mumford, D. (2004). 2d-shape analysis using conformal mapping. In IEEE computer vision and pattern recognition (Vol. II, pp. 350–357).
Tao, H., & Huang, T. S. (1999). Explanation-based facial motion tracking using a piecewise Bezier volume deformation model. In IEEE computer vision and pattern recognition (Vol. I, pp. 611–617).
Tomasi, C., Petrov, S., & Sastry, A. (2003). 3d tracking = classification + interpolation. In IEEE international conference on computer vision (pp. 1441–1448).
Torresani, L., Yang, D. B., Alexander, E. J., & Bregler, C. (2001). Tracking and modeling non-rigid objects with rank constraints. In IEEE computer vision and pattern recognition (Vol. I, pp. 493–500)
Wang, Y., Huang, X., Lee, C.-S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., & Huang, P. (2004). High resolution acquisition, learning and transfer of dynamic 3-d facial expressions. Computer Graphics Forum, 23(3), 677–686.
Wen, Z., & Huang, T. S. (2003). Capturing subtle facial motions in 3d face tracking. In IEEE international conference on computer vision (pp. 1343–1350).
Witkin, A. P., Terzopoulos, D., & Kass, M. (1987). Signal matching through scale space. International Journal of Computer Vision, 1(2), 133–144.
Xiao, J., Baker, S., Matthews, I., & Kanade, T. (2004). Real-time combined 2d+3d active appearance models. In IEEE computer vision and pattern recognition (Vol. II, pp. 535–542)
Yezzi, A. J., & Soatto, S. (2003). Deformotion: Deforming motion, shape average and the joint registration and approximation of structures in images. International Journal of Computer Vision, 53(2), 153–167.
Zhang, Z. (1994). Iterative point matching for registration of free-form curves and surfaces. International Journal of Computer Vision, 13(2), 119–152.
Zhang, D., & Hebert, M. (1999). Harmonic maps and their applications in surface matching. In IEEE computer vision and pattern recognition (Vol. II, pp. 524–530).
Zhang, L., Snavely, N., Curless, B., & Seitz, S. M. (2004). Spacetime faces: high resolution capture for modeling and animation. ACM SIGGraph, Computer Graphics, 23(3), 548–558.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wang, Y., Gupta, M., Zhang, S. et al. High Resolution Tracking of Non-Rigid Motion of Densely Sampled 3D Data Using Harmonic Maps. Int J Comput Vis 76, 283–300 (2008). https://doi.org/10.1007/s11263-007-0063-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-007-0063-y