Xavier P. Burgos-Artizzu
ABSTRACT
One of the main issues of current Head-Mounted Displays (HMD) is that they hide completely the wearer's face. This can be an issue in social experiences where two or more users want to share the 3D immersive experience. We propose a novel method to recover the face of the user in real-time. First, we learn the user appearance offline by building a 3D textured model of his head from a series of pictures. Then, by calibrating the camera and tracking the HMD's position in real-time we reproject the model on top of the video frames mimicking exactly the user's head pose. Finally, we remove the HMD and replace the occluded part of the face in a seamingless manner by performing image in-painting with the background. We further propose an extension to detect facial expressions on the visible part of the face and use it to change the upper face model accordingly. We show the promise of our method via some qualitative results on a variety of users.
- Autodesk Inc. 123d catch app. http://www.123dapp.com/catch.Google Scholar
- Baillard, C., and Zisserman, A. 2000. A plane-sweep strategy for the 3d reconstruction of buildings from multiple images. International Archives of Photogrammetry and Remote Sensing 33, B2; PART 2, 56--62.Google Scholar
- Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. 2000. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., 417--424. Google ScholarDigital Library
- Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., and Nayar, S. K. 2008. Face swapping: Automatically replacing faces in photographs. ACM Trans. Graph. 27, 3, 39:1--39:8. Google ScholarDigital Library
- Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In SIGGRAPH. Google ScholarDigital Library
- Burgos-Artizzu, X., Perona, P., and Dollar, P. 2013. Robust face landmark estimation under occlusion. In International Conference in Computer Vision (ICCV), IEEE. Google ScholarDigital Library
- Chen, Y., and Medioni, G. 1992. Object modelling by registration of multiple range images. Image and vision computing 10, 3, 145--155. Google ScholarDigital Library
- Cheng, Y.-T., Tzeng, V., Liang, Y., Wang, C.-C., Chen, B.-Y., Chuang, Y.-Y., and Ouhyoung, M. 2009. 3d-model-based face replacement in video. In SIGGRAPH '09: Posters. Google ScholarDigital Library
- Cootes, T., Edwards, G., and Taylor, C. 2001. Active appearance models. PAMI 23, 6, 681--685. Google ScholarDigital Library
- Dale, K., Sunkavalli, K., Johnson, M. K., Vlasic, D., Matusik, W., and Pfister, H. 2011. Video face replacement. ACM Trans. Graph. 30, 6, 130:1--130:10. Google ScholarDigital Library
- D. Lin, and Tang, X. 2007. Quality-driven face occlusion detection and recovery. In CVPR.Google Scholar
- Faugeras, O., and Luong, Q. 2004. The geometry of multiple images: the laws that govern the formation of multiple images of a scene and some of their applications. MIT Press. Google ScholarDigital Library
- Garrido-Jurado, S., Noz Salinas, R. M., Madrid-Cuevas, F., and Marín-Jiménez, M. 2014. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition 47, 6, 2280--2292. Google ScholarDigital Library
- Gross, R., Matthews, I., Cohn, J., Kanade, T., and Baker, S. 2008. Multi-pie. In FG.Google Scholar
- Hosoi, T., Nagashima, S., Kobayashi, K., Ito, K., and Aoki, T. 2012. Restoring occluded regions using fw-pca for face recognition. In Computer Vision and Pattern Recognition Workshops (CVPRW).Google Scholar
- Huang, D., and De La Torre, F. 2012. Facial action transfer with personalized bilinear regression. In Computer Vision--ECCV 2012. Springer, 144--158. Google ScholarDigital Library
- Hwang, B.-W., and Lee, S.-W. 2003. Reconstruction of partially damaged face images based on a morphable face model. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 3, 365--372. Google ScholarDigital Library
- Lepetit, V., F. Moreno-Noguer, and P. Fua. 2009. Epnp: An accurate o(n) solution to the pnp problem. International Journal Computer Vision 81, 2. Google ScholarDigital Library
- Li, H., Trutoiu, L., Olszewski, K., Wei, L., Trutna, T., Hsieh, P.-L., Nicholls, A., and Ma, C. 2015. Facial performance sensing head-mounted display. In SIGGRAPH. Google ScholarDigital Library
- Mo, Z., Lewis, J., and Neumann, U. 2004. Face inpainting with local linear representations. In BMVC.Google Scholar
- OCULUS VR. Oculus rift dk2. https://www.oculus.com/dk2/.Google Scholar
- Pearson, K. 1901. On lines and planes of closest fit to systems of points in space. Philosophical Magazine.Google Scholar
- Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. In ACM Transactions on Graphics (TOG), vol. 22(3), ACM, 313--318. Google ScholarDigital Library
- Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. In ACM Transactions on Graphics (TOG), vol. 24(3), ACM, 426--433. Google ScholarDigital Library
- Yu, D., and T., S. 2008. Using targeted statistics for face regeneration. In IEEE International Conference on Automatic Face and Gesture Recognition.Google ScholarCross Ref
Index Terms
- Real-time expression-sensitive HMD face reconstruction
Recommendations
Multi-pose face reconstruction and Gabor-based dictionary learning for face recognition
AbstractMany methods based on data and model analysis have been proposed to overcome the adverse effect of poses, occlusion, and illumination on face recognition. Frontal face generation from multi-pose faces is still a widely studied and challenging ...
Occlusion recovery face recognition based on information reconstruction
AbstractThe face data in the wild have a face occlusion, complex background, and inestimable posture, leading to more sample classification errors. To alleviate the problem of face occlusion, we apply facial information reconstruction to occluded face ...
Reconstruction based face occlusion elimination for recognition
SRC is a recent breakthrough on occluded face recognition, but raw SRC is slow and inaccurate because both occlusion elimination and face recognition are confused and finished once in @?"1-optimization. In this paper, a reconstruction based occlusion ...
Comments