Chong He
ABSTRACT
Non-rigid human surface tracking systems have many important applications in virtual reality and mixed reality. However, current systems can hardly be applied to interactive scenarios for their performance or accuracy limitations. One of the biggest challenges is efficiently estimating accurate correspondences between the template human model and tracking data. To bridge the gap, we propose a hierarchical feature matching framework for computing accurate dense correspondences between human shapes in real-time. For input human mesh model and depth scan, we train a fully convolutional network to produce dense feature descriptors with local similarity on human surface. Base on that, correspondences are found by performing fast hierarchical matching on segmented human body. Our approach is robust to large motion and deformation, its efficiency is validated in related real-time scenarios, while its accuracy has been proven comparable to state-of-the-art offline methods.
- Shafaei, A. and Little, J.J., 2016, June. Real-Time Human Motion Capture with Multiple Depth Cameras. In Computer and Robot Vision (CRV), 2016 13th Conference on (pp. 24--31). IEEE.Google Scholar
- Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M. and Moore, R., 2013. Real-time human pose recognition in parts from single depth images. Communications of the ACM, 56(1), pp. 116--124. Google ScholarDigital Library
- Ye, M. and Yang, R., 2014. Real-time simultaneous pose and shape estimation for articulated objects using a single depth camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2345--2352). Google ScholarDigital Library
- Barmpoutis, A., 2013. Tensor body: Real-time reconstruction of the human body and avatar synthesis from RGB-D. IEEE transactions on cybernetics, 43(5), pp. 1347--1356.Google Scholar
- Zhang, L., Sturm, J., Cremers, D. and Lee, D., 2012, October. Real-time human motion tracking using multiple depth cameras. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on (pp. 2389--2395). IEEE.Google Scholar
- Chen, Q. and Koltun, V., 2015. Robust nonrigid registration by convex optimization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2039--2047). Google ScholarDigital Library
- Newcombe, R.A., Fox, D. and Seitz, S.M., 2015. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 343--352).Google Scholar
- Taylor, J., Shotton, J., Sharp, T. and Fitzgibbon, A., 2012, June. The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on (pp. 103--110). IEEE. Google ScholarDigital Library
- Van Kaick, O., Zhang, H., Hamarneh, G. and Cohen-Or, D., 2011, September. A survey on shape correspondence. In Computer Graphics Forum (Vol. 30, No. 6, pp. 1681--1707). Blackwell Publishing Ltd.Google Scholar
- Vlasic, D., Baran, I., Matusik, W. and Popović, J., 2008, August. Articulated mesh animation from multi-view silhouettes. In ACM Transactions on Graphics (TOG) (Vol. 27, No. 3, p. 97). ACM. Google ScholarDigital Library
- Wei, L., Huang, Q., Ceylan, D., Vouga, E. and Li, H., 2016. Dense human body correspondences using convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1544--1553).Google Scholar
- Han, X., Leung, T., Jia, Y., Sukthankar, R. and Berg, A.C., 2015. Matchnet: Unifying feature and metric learning for patch-based matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3279--3286).Google Scholar
- Long, J.L., Zhang, N. and Darrell, T., 2014. Do convnets learn correspondence?. In Advances in Neural Information Processing Systems (pp. 1601--1609). Google ScholarDigital Library
- Masci, J., Boscaini, D., Bronstein, M. and Vandergheynst, P., 2015. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the IEEE international conference on computer vision workshops (pp. 37--45). Google ScholarDigital Library
- Ciresan, D., Giusti, A., Gambardella, L.M. and Schmidhuber, J., 2012. Deep neural networks segment neuronal membranes in electron microscopy images. In Advances in neural information processing systems (pp. 2843--2851). Google ScholarDigital Library
- Farabet, C., Couprie, C., Najman, L. and LeCun, Y., 2013. Learning hierarchical features for scene labeling. IEEE transactions on pattern analysis and machine intelligence, 35(8), pp. 1915--1929. Google ScholarDigital Library
- Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R. and LeCun, Y., 2013. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229.Google Scholar
- Long, J., Shelhamer, E. and Darrell, T., 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3431--3440).Google Scholar
- Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J. and Davis, J., 2005, July. SCAPE: shape completion and animation of people. In ACM Transactions on Graphics (TOG) (Vol. 24, No. 3, pp. 408--416). ACM. Google ScholarDigital Library
- Bogo, F., Romero, J., Loper, M. and Black, M.J., 2014. FAUST: Dataset and evaluation for 3D mesh registration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3794--3801). Google ScholarDigital Library
- Rodolà, E., Rota Bulo, S., Windheuser, T., Vestner, M. and Cremers, D., 2014. Dense non-rigid shape correspondence using random forests. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4177--4184). Google ScholarDigital Library
- Kim, V.G., Lipman, Y. and Funkhouser, T., 2011, August. Blended intrinsic maps. In ACM Transactions on Graphics (TOG) (Vol. 30, No. 4, p. 79). ACM. Google ScholarDigital Library
Index Terms
- Real-time Human Body Correspondences for Motion Tracking
Recommendations
Human Body Parts Tracking Using Torso Tracking: Applications to Activity Recognition
ITNG '12: Proceedings of the 2012 Ninth International Conference on Information Technology - New GenerationsThis paper proposed an approach for human body part tracking which is based on torso tracking. The main goal in this paper is tracking main human body parts such as torso, head and hands. In the proposed method for human body part tracking, we are using ...
Self-occlusion handling for human body motion tracking from 3D ToF image sequence
3DVP '10: Proceedings of the 1st international workshop on 3D video processingA 3D Time-of-flight (ToF) image is very useful to accurately track the human body motion due to its precision. However, the ToF image can not provide occluded 3D data because it also has a limitation of camera viewpoint. This paper proposes a self-...
Real-Time Tracking and Reproduction of 3D Human Body Motion
ICIAP '01: Proceedings of the 11th International Conference on Image Analysis and ProcessingAbstract: Non intrusive human body tracking is a key issue in advanced human-computer interaction, with applications ranging from virtual reality to videoconference and telepresence. This paper describes a system for vision-based tracking of body ...
Comments