Skip to main content
Springer Nature Link
Log in

Interacting with Real Objects in Virtual Worlds

  • Chapter
  • First Online:
Real VR – Immersive Digital Reality

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11900))

Abstract

In a virtual reality (VR) experience, the manner in which a user interacts with the virtual environment and computer generated objects in the scene greatly effects the feeling of immersion. Traditionally, VR systems use controllers as a means of facilitating interaction, with a sequence of button presses corresponding to a particular action. However, controllers do not accurately model the intuitive way to interact with a real-world object and offer limited tactile feedback. Alternatively, a physical object can be tracked and used to regulate the behaviour of a virtual object. In this chapter, we review a range of approaches which use the tracked behaviour of a physical object to control elements of the virtual environment. These virtual props have the potential to be used as a more immersive alternative to the traditional controllers. We discuss how motion capture systems and external sensors can be used to track rigid and non-rigid objects, in order to drive the motion of computer generated 3D models. We then consider two neural network based tracking solutions and explain how these can be used for transporting real objects into virtual environments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Andrychowicz, M., et al.: Learning dexterous in-hand manipulation. CoRR arXiv:1808.00177 (2018)

  2. Artec: 3D Object Scanner Artec Eva. https://www.artec3d.com/portable-3d-scanners/artec-eva

  3. Chen, Y., Medioni, G.: Object modeling by registration of multiple range images. Image Vis. Comput. 10, 145–155 (1992)

    Article  Google Scholar 

  4. Choy, C.B., Xu, D., Gwak, J.Y., Chen, K., Savarese, S.: 3D-R2N2: a unified approach for single and multi-view 3D object reconstruction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 628–644. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_38

    Chapter  Google Scholar 

  5. Cook, R.D., Malkus, D.S., Plesha, M.E., Witt, R.J.: Concepts and applications of finite element analysis, 3rd edn. Wiley, New York (1989)

    MATH  Google Scholar 

  6. De Maesschalck, R., Jouan-Rimbaud, D., Massart, D.L.: The mahalanobis distance. Chemometr. Intell. Lab. Syst. 50(1), 1–18 (2000)

    Article  Google Scholar 

  7. Dou, P., Shah, S.K., Kakadiaris, I.A.: End-to-end 3D face reconstruction with deep neural networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    Google Scholar 

  8. Dreamscape: Dreamscape immersive. https://dreamscapeimmersive.com/

  9. Elbrechter, C., Haschke, R., Ritter, H.: Bi-manual robotic paper manipulation based on real-time marker tracking and physical modelling. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1427–1432, September 2011. https://doi.org/10.1109/IROS.2011.6094742

  10. HTC: Discover virtual reality beyond imagination. https://www.vive.com/uk/

  11. HTC: Vive tracker. https://www.vive.com/uk/vive-tracker/

  12. Immersive, D.: Alien zoo. https://dreamscapeimmersive.com/adventures/details/alienzoo01

  13. Insider, V.F.: AMC and nickelodeon partner with dreamscape immersive, large scale VR experiences follow. https://www.vrfitnessinsider.com/amc-nickelodeon-partner-with-dreamscape-immersive-large-scale-vr-experiences-follow/

  14. Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7122–7131 (2018)

    Google Scholar 

  15. Kanazawa, A., Kovalsky, S., Basri, R., Jacobs, D.: Learning 3D deformation of animals from 2D images. In: Computer Graphics Forum, vol. 35, pp. 365–374. Wiley Online Library (2016)

    Google Scholar 

  16. Kanazawa, A., Zhang, J.Y., Felsen, P., Malik, J.: Learning 3D human dynamics from video. In: Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  17. Kausch, L., Hilsmann, A., Eisert, P.: Template-based 3D non-rigid shape estimation from monocular image sequences. In: Proceedings of the Conference on Vision, Modeling and Visualization, pp. 37–44. Eurographics Association (2017)

    Google Scholar 

  18. Leizea, I., Álvarez, H., Aguinaga, I., Borro, D.: Real-time deformation, registration and tracking of solids based on physical simulation. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 165–170. IEEE (2014)

    Google Scholar 

  19. Li, H., Yu, J., Ye, Y., Bregler, C.: Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. (TOG) 32(4), 42 (2013). https://doi.org/10.1145/2461912.2462019

    Article  MATH  Google Scholar 

  20. Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM Trans. Graph. 34(6), 248:1–248:16 (2015). (Proc. SIGGRAPH Asia)

    Article  Google Scholar 

  21. Microsoft: Microsoft Hololens|mixed reality technology for business. https://www.microsoft.com/en-us/hololens

  22. Mondjar-Guerra, V., Garrido-Jurado, S., Muoz-Salinas, R., Marn-Jimnez, M.J., Medina-Carnicer, R.: Robust identification of fiducial markers in challenging conditions. Expert Syst. Appl. 93(C), 336–345 (2018). https://doi.org/10.1016/j.eswa.2017.10.032

    Article  Google Scholar 

  23. Newcombe, R.A., Fox, D., Seitz, S.M.: Dynamicfusion: reconstruction and tracking of non-rigid scenes in real-time. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 343–352 (2015)

    Google Scholar 

  24. Oculus: Oculus rift. https://www.oculus.com/rift/

  25. Park, Y., Lepetit, V., Woo, W.: Multiple 3D object tracking for augmented reality. In: Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 117–120. IEEE Computer Society (2008)

    Google Scholar 

  26. Paulus, C.J., Haouchine, N., Cazier, D., Cotin, S.: Augmented reality during cutting and tearing of deformable objects. In: 2015 IEEE International Symposium on Mixed and Augmented Reality, pp. 54–59. IEEE (2015)

    Google Scholar 

  27. Petit, A., Lippiello, V., Fontanelli, G.A., Siciliano, B.: Tracking elastic deformable objects with an RGB-D sensor for a pizza chef robot. Robot. Auton. Syst. 88, 187–201 (2017)

    Article  Google Scholar 

  28. Pumarola, A., Agudo, A., Porzi, L., Sanfeliu, A., Lepetit, V., Moreno-Noguer, F.: Geometry-aware network for non-rigid shape prediction from a single view, June 2018

    Google Scholar 

  29. Rambach, J., Pagani, A., Stricker, D.: [poster] augmented things: enhancing AR applications leveraging the Internet of Things and universal 3D object tracking. In: 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), pp. 103–108. IEEE (2017)

    Google Scholar 

  30. Remondino, F.: From point cloud to surface: the modeling and visualization problem. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XXXIV–5/W10, 1–11 (2003). https://doi.org/10.3929/ethz-a-004655782. ISSN 1682-1750

    Article  Google Scholar 

  31. Romero, J., Tzionas, D., Black, M.J.: Embodied hands: modeling and capturing hands and bodies together. ACM Trans. Graph. 36(6), 245:1–245:17 (2017). https://doi.org/10.1145/3130800.3130883. (Proc. SIGGRAPH Asia)

    Article  Google Scholar 

  32. Salzmann, M., Pilet, J., Ilic, S., Fua, P.: Surface deformation models for nonrigid 3D shape recovery. IEEE Trans. Pattern Anal. Mach. Intell. 29(8), 1481–1487 (2007). https://doi.org/10.1109/TPAMI.2007.1080

    Article  Google Scholar 

  33. Schulman, J., Lee, A., Ho, J., Abbeel, P.: Tracking deformable objects with point clouds. In: 2013 IEEE International Conference on Robotics and Automation, pp. 1130–1137. IEEE (2013)

    Google Scholar 

  34. Sridhar, S., Mueller, F., Zollhöfer, M., Casas, D., Oulasvirta, A., Theobalt, C.: Real-time joint tracking of a hand manipulating an object from RGB-D input. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 294–310. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_19

    Chapter  Google Scholar 

  35. Taylor, C., McNicholas, R., Cosker, D.: VRProp-net: real-time interaction with virtual props. In: ACM SIGGRAPH 2019 Posters, SIGGRAPH 2019, pp. 31:1–31:2. ACM, New York (2019). https://doi.org/10.1145/3306214.3338548

  36. Taylor, C., Mullanay, C., McNicholas, R., Cosker, D.: VR props: an end-to-end pipeline for transporting real objects into virtual and augmented environments. In: 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE (2019)

    Google Scholar 

  37. Tipping, M.E., Bishop, C.M.: Probabilistic principal component analysis. J. R. Stat. Soc.: Ser. B (Stat. Methodol.) 61(3), 611–622 (1999)

    Article  MathSciNet  Google Scholar 

  38. Tjaden, H., Schwanecke, U., Schomer, E.: Real-time monocular pose estimation of 3D objects using temporally consistent local color histograms. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 124–132 (2017)

    Google Scholar 

  39. Tsoli, A., Argyros, A.A.: Joint 3D tracking of a deformable object in interaction with a hand. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 484–500 (2018)

    Google Scholar 

  40. Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., Gall, J.: Capturing hands in action using discriminative salient points and physics simulation. CoRR abs/1506.02178 (2015). http://arxiv.org/abs/1506.02178

  41. Vicon: Motion capture systems. https://www.vicon.com/

  42. Vicon: Origin by vicon. https://www.vicon.com/press/2018-08-13/origin-by-vicon

  43. to VR, R.: Optitrack shows hundreds of simultaneously tracked objects in a single VR experience. https://www.roadtovr.com/optitrack-hundreds-of-tracked-objects-jenga-gdc-2019/amp/

  44. Welch, G., Bishop, G., et al.: An Introduction to the Kalman Filter (1995)

    Google Scholar 

  45. Xiang, Y., Schmidt, T., Narayanan, V., Fox, D.: PoseCNN: a convolutional neural network for 6D object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199 (2017)

  46. Zhang, H., Bo, Z.H., Yong, J.H., Xu, F.: Interactionfusion: real-time reconstruction of hand poses and deformable objects in hand-object interactions. ACM Trans. Graph. 38(4), 48:1–48:11 (2019). https://doi.org/10.1145/3306346.3322998

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Bath, Bath, UK

    Catherine Taylor & Darren Cosker

Authors
  1. Catherine Taylor
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Darren Cosker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Darren Cosker .

Editor information

Editors and Affiliations

  1. TU Braunschweig, Brunswick, Germany

    Marcus Magnor

  2. Facebook Zurich, Zürich, Switzerland

    Alexander Sorkine-Hornung

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Taylor, C., Cosker, D. (2020). Interacting with Real Objects in Virtual Worlds. In: Magnor, M., Sorkine-Hornung, A. (eds) Real VR – Immersive Digital Reality. Lecture Notes in Computer Science(), vol 11900. Springer, Cham. https://doi.org/10.1007/978-3-030-41816-8_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-41816-8_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-41815-1

  • Online ISBN: 978-3-030-41816-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics