Dong-Hyun Hwang
Hideki Koike
ABSTRACT
In this work, we propose a simple yet effective method for synthesizing a pseudo-2.5D scene from a monocular video for mixed reality (MR) content. We also propose the ParaPara system, which applies this method. Most previously proposed systems convert real-world objects into 3D graphic models using expensive equipment; this is a barrier for individuals or small groups to create MR content. ParaPara uses four points in an image and their manually estimated distances to synthesize MR content by applying deep neural networks and simple image processing techniques to monocular videos. The synthesized content can be observed through an MR head-mounted display, and spatial mapping and spatial sound are applied to support the interaction between the real-world and MR content. The proposed system is expected to reduce the entry barriers to create MR content because it can create such content from a large number of previously captured videos.
Supplemental Material
- T. Kanade, P. Rander, and P. J. Narayanan. 1997. Virtualized reality: constructing virtual worlds from real scenes. IEEE MultiMedia 4, 1 (Jan. 1997), 34-- 47. Google Scholar
Digital Library
- Canon Global. Canon announces development of the Free Viewpoint Video System virtual camera system that creates an immersive viewing experience. (Sep. 2017). Retrieved Jan 10, 2018 from http://global.canon/en/news/2017/20170921.htmlGoogle Scholar
- Youichi Horry, Ken-Ichi Anjyo, and Kiyoshi Arai. 1997. Tour into the Picture: Using a Spidery Mesh Interface to Make Animation from a Single Image. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 225--232. Google ScholarDigital Library
- Itaru Kitahara, Yuichi Ohta, Hideo Saito, Shinji Akimichi, Tooru Ono, and Takeo Kanade. 2002. Recording of multiple videos in a large-scale space for large-scale virtualized reality. Journal of the Institute of Image Information and Television Engineers 56, 8 (Aug. 2002), 1328--1333.Google ScholarCross Ref
- Y. Kameda, T. Koyama, Y. Mukaigawa, F. Yoshikawa, and Y. Ohta. 2004. Free viewpoint browsing of live soccer games. In 2004 IEEE International Conference on Multimedia and Expo (ICME '04). 747--750.Google Scholar
- P. Goorts, S. Maesen, M. Dumont, S. Rogmans, and P. Bekaert. 2014. Free Viewpoint Video for Soccer using Histogram-Based Validity Maps in Plane Sweeping. In Proceedings of the Ninth International Conference on Computer Vision Theory and Applications (VISAPP '14). 378--386.Google Scholar
- N. Inamoto and H. Saito. 2005. Free viewpoint video synthesis and presentation from multiple sporting videos. In 2005 IEEE International Conference on Multimedia and Expo (ICME '05). 40--49.Google Scholar
- Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. 2015. Highquality Streamable Free-viewpoint Video. ACM Trans. Graph. 34, 4, Article 69 (July. 2015), 13 pages. Google ScholarDigital Library
- Sergio Orts-Escolano et al. 2016. Holoportation: Virtual 3D Teleportation in Real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 741--754. Google ScholarDigital Library
- N. Dalal and B. Triggs. 2005. Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '05). 886-- 893. Google ScholarDigital Library
- Z. Cao, T. Simon, S. E. Wei, and Y. Sheikh. 2017. Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR '17). 1302--1310.Google Scholar
- Zoran Zivkovic and Ferdinand van der Heijden. 2006. Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recognition Letters 27, 7 (May. 2006), 773--780. Google ScholarDigital Library
- Tomas Akenine-Möller, Eric Haines, and Naty Hoffman. 2008. Real-Time Rendering (3rd. ed.). A. K. Peters, Ltd., Natick, MA, USA.Google Scholar
- Michael S. Landy and J. Anthony Movshon (Eds.). 1991. Computational Models of Visual Processing. Massachusetts Institute of Technology Press, Cambridge, MA, USA. Google ScholarDigital Library
Index Terms
- ParaPara: Synthesizing Pseudo-2.5D Content from Monocular Videos for Mixed Reality
Recommendations
Garden: A Mixed Reality Experience Combining Virtual Reality and 3D Reconstruction
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing SystemsGarden is a Mixed Reality (MR) experience that combines both Virtual Reality (VR) and Augmented Reality (AR), and lets players transform their environment into a virtual garden they can play in. This is done by doing both stereoscopic rendering and 3D ...
Through Tinted Eyeglasses
EIC Roy Want introduces the special issue on cross-reality environments and discusses alternate realities including virtual reality, augmented reality, embodied virtuality, cross-reality, and mixed reality.
Mixed Reality MIDI Keyboard Demonstration
AM '17: Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music ExperiencesThe Mixed Reality MIDI Keyboard is a prototype designed to augment virtual reality experiences through the inclusion of a physical interface which aligns the user's senses with the virtual environment. It also serves as a platform on which the uses of ...
Comments