Remo Pillat
ABSTRACT
This paper describes the development of an MR environment that can be used in teaching STEM (Science, Technology, Engineering, and Mathematics) topics. Specifically we seek to create a space for facilitating whole-body metaphors where learners use the physical movement and positioning of their entire bodies to enact their understanding of complex concepts.
A rigorous technical approach comprised of virtual elements, real users, spatial audio, and an integrated sensor network is presented that fulfills the requirements of an embodied learning environment. An algorithm that uses homography-based multi-projector blending is used to create a large, seamless projection on the floor that affords a human-scale interaction environment. To further improve the immersive quality, projectors are strategically overlapped to minimize user shadows on the projected surface. A hybrid sensor solution using a Kinect and a laser scanner is developed that tracks users' physical movements and extracts relevant game parameters such as position and velocity. Requiring no pre-training or props, this tracking setup is adaptable and shows high performance over a wide range of users, from children to adults. An exhibit employing this MR system was field-tested at the Museum of Science and Industry in Tampa, FL.
- Birchfield, D., and Johnson-Glenberg, M. 2010. A Next Gen Interface for Embodied Learning: SMALLab and the Geological Layer Cake. International Journal of Gaming and Computer-Mediated Simulations 2, 1, 49--58.Google Scholar
Cross Ref
- Birchfield, D., Ciufo, T., and Minyard, G. 2006. SMALLab: A Mediated Platform for Education. In ACM SIGGRAPH - Educators Program, ACM Press, New York, New York, USA, 1--7. Google ScholarDigital Library
- Cameron, L. 2002. Metaphors in the Learning of Science: A Discourse Focus. British Educational Research Journal 28, 5, 673--688.Google ScholarCross Ref
- Chang, C.-W., Lee, J.-H., Wang, C.-Y., and Chen, G.-D. 2010. Improving the authentic learning experience by integrating robots into the mixed-reality environment. Computers & Education 55<, 4 (Dec.), 1572--1578. Google ScholarDigital Library
- Chen, H., Sukthankar, R., and Wallace, G. 2002. Scalable alignment of large-format multi-projector displays using camera homography trees. In IEEE Visualization (VIS2002), IEEE, 339--346. Google ScholarDigital Library
- Christidou, V., Koulaidis, V., and Christidis, T. 1997. Children's Use of Metaphors in Relation to their Mental Models: The Case of the Ozone Layer and its Depletion. Research in Science Education 27, 4, 541--552.Google ScholarCross Ref
- Cotting, D., Nebel, I., Gross, M. H., and Fuchs, H. 2011. Towards a Continuous, Unified Calibration of Projectors and Cameras. Tech. rep., ETH Zuerich, Department of Computer Science.Google Scholar
- Cruz-Neira, C., Sandin, D. J., and DeFanti, T. A. 1993. Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE. In ACM Conference on Computer Graphics and Interactive Techniques (SIGGRAPH). 135--142. Google ScholarDigital Library
- Gallagher, S. 2005. How the Body Shapes the Mind. Oxford University Press, USA.Google Scholar
- GestureTek, 2012. Gesture Recognition & Computer Vision Control Technology & Motion Sensing Systems for Presentation & Entertainment.Google Scholar
- Goldin-Meadow, S., Cook, S. W., and Mitchell, Z. a. 2009. Gesturing Gives Children new Ideas about Math. Psychological Science 20, 3 (Mar.), 267--272.Google ScholarCross Ref
- Hartley, R., and Zisserman, A. 2003. Multiple View Geometry, 2nd ed. Cambridge University Press. Google ScholarDigital Library
- Honey, M. A., and Hilton, M. L., Eds. 2011. Learning Science Through Computer Games and Simulations. The National Academies Press.Google Scholar
- Hughes, C., Stapleton, C., Hughes, D., and Smith, E. 2005. Mixed Reality in Education, Entertainment, and Training. IEEE Computer Graphics and Applications 25, 6 (Nov.), 24--30. Google ScholarDigital Library
- Johnson-Glenberg, M., Koziupa, T., Birchfield, D., and Li, K. 2011. Games for Learning in Embodied Mixed-Reality Environments: Principles and Results. In International Conference on Games + Learning + Society Conference (GLS), 129--137. Google ScholarDigital Library
- Khoo, E. T., Cheok, A. D., Nguyen, T. H. D., and Pan, Z. 2008. Age Invaders: Social and Physical Inter-Generational Mixed Reality Family Entertainment. Virtual Reality 12, 1 (Mar.), 3--16. Google ScholarDigital Library
- Khoshelham, K., and Elberink, S. O. 2012. Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications. Sensors 12, 2 (Feb.), 1437--1454.Google ScholarCross Ref
- Kirkley, S. E., and Kirkley, J. R. 2002. Creating Next Generation Blended Learning Environments Using Mixed Reality, Video Games and Simulations. TechTrends 49, 3, 42--54.Google ScholarCross Ref
- Lindgren, R., and Moshell, J. M. 2011. Supporting Children's Learning with Body-Based Metaphors in a Mixed Reality Environment. In International Conference on Interaction Design and Children (IDC), ACM Press, New York, New York, USA, 177--180. Google ScholarDigital Library
- Nagase, M., Iwai, D., and Sato, K. 2011. Dynamic Defocus and Occlusion Compensation of Projected Imagery by Model-Based Optimal Projector Selection in Multi-Projection Environment. Virtual Reality 15, 2--3 (Aug.), 119--132. Google ScholarDigital Library
- National Science Board. 2010. Preparing the Next Generation of STEM Innovators: Identifying and Developing our Nations Human Capital. Tech. rep., National Science Foundation.Google Scholar
- Pan, Z., Cheok, A. D., Yang, H., Zhu, J., and Shi, J. 2006. Virtual Reality and Mixed Reality for Virtual Learning Environments. Computers & Graphics 30<, 1 (Feb.), 20--28. Google ScholarDigital Library
- Raskar, R., Brown, M., Welch, G., Towles, H., Scales, B., and Fuchs, H. 1999. Multi-projector displays using camera-based registration. In IEEE Visualization (VIS1999), IEEE, 161--522. Google ScholarDigital Library
- Raskar, R., Baar, J. V., and Chai, J. 2002. A low-cost projector mosaic with fast registration. In Asian Conference on Computer Vision (ACCV), 114--119.Google Scholar
- Sajadi, B., and Majumder, A. 2012. Autocalibration of Multiprojector CAVE-Like Immersive Environments. IEEE Transactions on Visualization and Computer Graphics 18, 3 (Mar.), 381--393. Google ScholarDigital Library
- Snibbe, S. S., and Raffle, H. S. 2009. Social Immersive Media: Pursuing Best Practices for Multi-User Interactive Camera/Projector Exhibits. In International Conference on Human Factors in Computing Systems (CHI), 1447--1456. Google ScholarDigital Library
- Ye, C., and Borenstein, J. 2002. Characterization of a 2D laser scanner for mobile robot obstacle negotiation. In International Conference on Robotics and Automation (ICRA), IEEE, Washington, DC, 2512--2518.Google Scholar
Index Terms
- A mixed reality system for teaching STEM content using embodied learning and whole-body metaphors
Recommendations
Enhancing learning and engagement through embodied interaction within a mixed reality simulation
Computer simulations have been shown to be effective instruments for teaching students about difficult concepts, particularly in the STEM disciplines. Emerging interface technologies are expanding the modalities with which learners can interact with ...
Virtual reality and mixed reality for virtual learning environments
This paper explores educational uses of virtual learning environment (VLE) concerned with issues of learning, training and entertainment. We analyze the state-of-art research of VLE based on virtual reality and augmented reality. Some examples for the ...
Mixed Reality MIDI Keyboard Demonstration
AM '17: Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music ExperiencesThe Mixed Reality MIDI Keyboard is a prototype designed to augment virtual reality experiences through the inclusion of a physical interface which aligns the user's senses with the virtual environment. It also serves as a platform on which the uses of ...
Comments