Skip to main content
SpringerLink
Log in

Augmented immersion: video cutout and gesture-guided embedding for gaming applications

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper introduces a novel framework, Gesture and Appearance Cutout Embedding (GACE), that supports real-time integration of human appearance and gesture-guided control within a game. It aims to enhance immersion since it allows game users to see their personal appearance in a real-time manner. In addition, we exploit the gesture-based control to allow user interaction with other in-game characters. With the goal to make implementation easier, we address the challenges in the whole pipeline of video processing, gesture recognition, and communication. The system is successfully integrated into both entertainment and serious games. Extensive experiments show that the proposed system runs reliably and comfortably with a commodity setting. Meanwhile, the user impression study indicates our system is favored by end users.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://www.myo.com

  2. https://www.leapmotion.com

  3. https://unity3d.com

  4. https://www.oculus.com

  5. Accuracy is (T r u e P o s i t i v e + T r u e N e g a t i v e)/T o t a l.

References

  1. Aggarwal J, Ryoo M (2011) Human activity analysis: A review. ACM Comput Surv 43(3):16

    Article  Google Scholar 

  2. Cavazza M, Charles F, Mead SJ, Martin O., Marichal X., Nandi A. (2004) Multimodal acting in mixed reality interactive storytelling. IEEE MultiMedia 11(3):30–39

    Article  Google Scholar 

  3. Chambel T, Viana P, Bove VM, Strover S, Thomas G (2014) Immersiveme’14: 2nd ACM international workshop on immersive media experiences. In: ACM Multimedia, pp 1255–1256

  4. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE Conference on computer vision and pattern recognition, pp 886–893

  5. Dede C (2009) Immersive interfaces for engagement and learning. Science 323 (5910):66–69

    Article  Google Scholar 

  6. Dondlinger MJ (2007) Educational video game design: A review of the literature. Journal of Applied Educational Technology 4(1):21–31

    Google Scholar 

  7. Henry P, Krainin M, Herbst E, Ren X, Fox D (2010) Rgb-d mapping: Using depth cameras for dense 3d modeling of indoor environments. In: International symposium on experimental robotics, pp 477–491

  8. Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe RA, Kohli P, Shotton J, Hodges S, Freeman D, Davison AJ, Fitzgibbon AW (2011) Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In: ACM Symposium on User Interface Software and Technology, pp 559–568

  9. Jörg S, Normoyle A, Safonova A (2012) How responsiveness affects players’ perception in digital games. In: Proceedings of the ACM Symposium on Applied Perception. ACM, pp 33–38

  10. Lai K, Bo L, Ren X, Fox D (2011) Sparse distance learning for object recognition combining rgb and depth information. In: IEEE International Conference on Robotics and Automation, pp 4007–4013

  11. Lang C, Nguyen TV, Katti H, Yadati K., Kankanhalli MS, Yan S (2012) Depth matters: Influence of depth cues on visual saliency. In: European conference on computer vision (2), pp 101–115

  12. Lange B, Suma EA, Newman B, Phan T, Chang C-Y, Rizzo A, Bolas M (2011) Leveraging unencumbered full body control of animated virtual characters for game-based rehabilitation. In: International conference, virtual and mixed reality, pp 243–252

  13. Likert R (1932) A technique for the measurement of attitudes. Archives of Psychology

  14. Liu S, Nguyen TV, Feng J, Wang M, Yan S (2012) Hi, magic closet, tell me what to wear!. In: ACM Multimedia, pp 1333–1334

  15. Lu J, Nguyen V, Niu Z, Singh B, Luo Z, Do M (2011) Cutechat: a lightweight tele-immersive video chat system. In: ACM Multimedia, pp 1309–1312

  16. Nguyen TV, Feng J, Yan S (2014) Seeing human weight from a single rgb-d image. J Comput Sci Technol 29(5):777–784

    Article  Google Scholar 

  17. Nguyen TV, Li L, Tan J, Yan S (2012) 3DME: 3d media express from rgb-d images. In: ACM Multimedia, pp 1331–1332

  18. Nguyen TV, Tan YH, Sepulveda J (2014) GACE: Gesture and appearance cutout embedding for gaming applications. In: Proceedings of the 2nd ACM International Workshop on Immersive Media Experiences, ImmersiveMe ’14, pp 41–44. ACM, New York

    Google Scholar 

  19. Shotton J, Fitzgibbon AW, Cook M, Sharp T, Finocchio M, Moore R, Kipman A, Blake A (2011) Real-time human pose recognition in parts from single depth images. In: IEEE Conference on computer vision and pattern recognition, pp 1297–1304

  20. Slater M, Spanlang B, Sanchez-Vives MV, Blanke O et al (2010) First person experience of body transfer in virtual reality. PloS one 5(5):e10564

    Article  Google Scholar 

  21. Soltani F, Eskandari F, Golestan S (2012) Developing a gesture-based game for deaf/mute people using microsoft kinect. In: International Conference on Complex, Intelligent, and Software Intensive Systems, pp 491–495

  22. Tang S, Wang X, Lv X, Han TX, Keller JM, He Z, Skubic M, Lao S (2012) Histogram of oriented normal vectors for object recognition with a depth sensor. In: Asian conference on computer vision, pp 525–538

  23. Wang J, Liu Z, Wu Y, Yuan J (2012) Mining actionlet ensemble for action recognition with depth cameras. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 1290–1297

Download references

Acknowledgments

This work was supported by Singapore Ministry of Education under research Grant MOE2012-TIF-2-G-016.

Author information

Authors and Affiliations

  1. ARTIC Centre, Singapore Polytechnic, Singapore, 139651, Singapore

    Tam V. Nguyen & Jose Sepulveda

Authors
  1. Tam V. Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jose Sepulveda
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tam V. Nguyen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nguyen, T.V., Sepulveda, J. Augmented immersion: video cutout and gesture-guided embedding for gaming applications. Multimed Tools Appl 75, 12351–12363 (2016). https://doi.org/10.1007/s11042-016-3435-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3435-x

Keywords

Search

Navigation