Keywords

1 Introduction

Virtual Reality (VR) has been in the laboratory for years, the realistic experiences that these computer simulations have provided to researches was incredibly exciting in the 1980s, but due to low resolutions, and large screens it was difficult to bring virtual reality out of the laboratory and into the hands of consumers. Recent advances in virtual reality, powered mainly by the advent of small, high resolution screens, with high enough refresh rates to make VR glasses light enough to be comfortable to the average consumer. Additionally, the cost has come down to below $1000.00 plus the cost of a new PC. Surprisingly, while VR has become consumer friendly, Augmented Reality (AR) has in many ways surpassed it, utilizing similar technology but with see through, or video backed lenses.

As these technologies have matured their use in consumers lives, as well as in the research laboratory is growing. Especially given the new-found likelihood of applications being adopted by users. This exciting opportunity has led to the design of VR and AR experiences to potentially help train limb deficient children to use their prosthetic arms, as well as provide an opportunity for others to experience what like is like for them today.

2 Background

The history of immersive technologies is a long and storied one. Though the actual term “Virtual Reality” was termed by Jaron Lanier in 1987 [1], the idea behind immersive experience dates back over 100 years. As early back as the nineteenth century painters were painting 360-degree panoramic art pieces [2].

The late nineteenth century also brought the advent of stereoscopic technology in the form of viewers and toys. This would lead to the popular 20th century toy the View-Master, a stereoscope (patented 1939), was used for “virtual tourism” [3].

Even the Link Trainer, designed in the 1930’s first found popularity as a carnival ride before being accepted as the first use of immersive technology incorporated into training. As the first foray in flight simulators, the “Link Trainer”, developed by Edward Link, was a fully electromechanical simulator that would eventually be used to train pilots for World War II. At first users were skeptical, which left the device only really useful as a toy, but during the war pilots were having problems landing due to heavy fog on the eastern front, the need to train for instrument only landing became apparent and the Link Trainers were a huge success for training pilots to land [4] (Figs. 1 and 2).

Fig. 1.
figure 1

Sevastopol Panorama (created in 1905), depicting the Siege of Sevastopol [2]

Full size image
Fig. 2.
figure 2

Morton Heilig’s Sensorama [5]

Full size image

During the 1950’s, Morton Heilig’s Sensorama that provided the illusion of reality with smell, stereo sound, simulated real world effects like wind and vibration for up to four people at a time. Heilig also developed the first head mounted displays Heilig’s development and research has earned him the moniker the “father of virtual reality” [5].

The mid to late 1960’s Ivan Sutherland created a concept that would drive the future of the VR industry, known as the “Ultimate Display.” The idea behind the ultimate display is a room in which a computer could control matter, if it exists, where, and how. In such a space a virtual object like a chair could be sat on, chains could imprison, bullets could be fatal. At the time such a room was described as similar Wonderland from the Alice in Wonderland Books. The most obvious comparison today would be the Holodeck from Star Trek [6].

It was not until the late 1960s that Myron Kruegere developed interactive art with VR in which he termed “Artificial Reality.” [7] It was not until around 20 years later during the late 1980’s that a term would be coined to encompass all the past and future research in this field. Jaron Lanier, founder of the visual programming lab not only coined the term “Virtual Reality” but also played a major role in the development of haptic feedback [1].

From this point to the early 2000’s Virtual and Augmented reality truly had its revolution. VR and AR products were being developed, not only for simulation, but also for entertainment value. The 1990’s saw commercial VR developments from video game powerhouses SEGA and Nintendo. It also brought substantial price drops in the hardware factor of these devices. Both Nintendo and Sega provided products, though flawed, under the $200 price point [8].

The Nintendo Virtual Boy, was Nintendo’s Virtual Reality follow up to the Amazingly popular Game Boy. Designed by Gunpei Yokoi, the designer of most of Nintendo’s major hardware releases from the Game & Watch series of mobile games until the ill-fated Virtual Boy. Gunpei Yokoi’s design philosophy was to use withered technology in new ways, that is to say he would find tested and market ready technology. The Virtual Boy, did not follow this philosophy. It used a red and black only display that was not ready for the consumer market. The screen gave users headaches, and the Virtual Boy is known as one of Nintendo’s first major failures in the hardware arena. It also led to Gunpei Yokoi leaving the company shortly after [8].

The rise in augmented reality (AR) has really stemmed from the recent mass availability of AR ready technology. From AR Smartphones, Automotive HUD Displays, and the next generation of AR Headsets, users are being introduced to AR in their everyday routines. AR was first introduced via military applications in the early 1990’s [9].

This brings us to today’s VR and AR market, “Global virtual reality revenues will reach $7.17 billion by the end of this year, according to a new report by Greenlight Insights”. They also estimate that the industry will rise to over 75 billion dollars in 2021. Companies such as Samsung, Google, Apple Oculus and HTC have all put their hats into the VR and AR market. The advent of mobile technology has allowed everyday consumers to have easier access to VR/AR capable hardware to utilize the tech in their own home [10].

3 The Current State of the Art in Training and Rehabilitation

VR and AR have a long tradition of being used in both training and rehabilitation. The current state of the art in these fields is laying the ground work for future applications to support prosthetics training.

3.1 Augmented Reality

AR technology offers many advantages in both training and learning environments [11]. AR’s allows users to interact, collaborate and communicate in hybrid, digital and physical spaces. Current trends in tech actual state that augmented reality is set to take over the virtual reality markets in years to come. Mark Zuckerberg, founder of FB states, “The phone is probably going to be the mainstream consumer platform [where] a lot of these AR features become mainstream, rather than a glasses form factor that people will wear on their face.” [12] This downsizing of equipment hardware allows AR to be instituted into more non-commercial fields such as research and rehabilitation.

AR is commonly used in the support of maintenance in industrial applications. This commonly is implemented as a world overlay generated by looking through a see translucent screen [13]. The technology is becoming smaller and less expensive. Microsoft entered this market with the Hololens which has been used in many applications including similar assembly applications, data visualization, and entertainment as well [14]. AR hardware has been found to be used in laparoscopic surgical training, neurosurgical procedure training, and echocardiography, however, there is less data available concerning transfer of learning, despite promising results existing [15].

AR technology is currently being researched in the medical field in conjunction with the treatment of multiple conditions, such as, extremity pain, phantom limb pain, and post stroke motor-rehabilitation. A recent study conducted by Brazilian physiotherapists, an augmented reality environment was utilized where participants could see themselves and their surroundings, as in mirror. They were asked to perform a series of tasks with and without the VR-mirror. The survey scores suggested greater motor improvement in the augmented reality group than in the control. These studies are intriguing from a training and education standpoint due the rising availability and lowering price points of AR tech. This should allow participants to have faster access to AR in home, as well as in hospital environments [16].

AR has been applied to post stroke finger rehabilitation. An AR training environment was used in conjunction with an assistive device to train finger extension. After 6 weeks of training there was a trend towards the reduction of assistive force in the impaired hand. There is encouraging evidence to continue this type of training [17].

3.2 Virtual Reality

VR is also being used in a variety of training and rehabilitation applications across military, education, healthcare, and industry. Virtual Reality has experienced a recent boom in the consumer level that has resulted in a resurgence of interest in the research lab as well.

Researchers at Aalborg University found that Virtual Reality could be used to reduce phantom limb pain. This happens when the VR environment tricks the amputee’s brain into thinking that it is still able to control the missing limb. Phantom limb pain is thought to occur with the brain no longer receives feedback from a part of the body it expects to. The VR environment made the participant brain think it was still receiving feedback even when it wasn’t [18].

In the treatment of phantom limb pain, a condition where amputees feel body parts that no longer exist, a team at the University of Tokyo Hospital has been using VR with much success. Masahiko Sumitani at University of Tokyo Hospital, has succeeded in using VR to treat phantom limb pain, a condition in which amputees or people with damaged nerves still feel pain from body parts that no longer exist. “We’ve found a mechanism through which pain can be reduced when patients are able to get an image of moving (their absent limbs) in their minds. This is something new,” Sumitani said. VR technology allows amputees to see and manipulate virtual images of their lost limbs. The virtual limbs move accordingly with the intact parts of the arm, allowing the images to take root in patients’ minds [19].

VR has also been used for stroke rehabilitation. Patients who obtained VR rehabilitation had a significant increase in upper limb motor impairments and motor relation functional abilities then patients that were rehabilitated using traditional methods [20].

Embodied Lab is an immersive project stemmed from a daughter’s firsthand experience taking care of her ailing mother who was suffering from early-onset Alzheimer’s disease. The mission of this project is to allow health care providers step into the perspectives of the patient and other members of the patients care team. This same patient-centric approach could be ported to those with missing limbs. Understanding the daily task of a limb deficient person would allow clinicians, developers and caregivers to better create accessible platforms and/or adapt to an accessible lifestyle [21].

Researchers in Seoul found VR to be useful to provide training for repetitive tasks and developed a mobile game-based virtual reality rehabilitation program for upper limb dysfunction after ischemic stroke. They found improvement from the VR intervention, but the results were not statistically significant. This is different from the AR similar AR research in that the difference was not significant. This is most likely a product of VR recreating reality while AR enhances it [22].

VR has also been found useful in rehabilitation in Parkinson’s disease, specifically on tasks that could improve balance in users. Over a 12 week intervention participants using Wii remote based training reported more confidence in their ability to balance.

3.3 Current Uses of VR/AR in Gaming

As previously mentioned, the Microsoft Hololense is bringing the AR experience to the masses. This product has fueled the low-cost AR market by providing an attainable solution. Unfortunately, the Hololense suffers from an anemic field of view. This makes the most interesting applications out of reach. There is still solid support for the Unity game engine and the ability to play games. The perceived value of these applications is much higher given the mixed reality demonstrations that show a person interacting with Minecraft, a popular video game, while it covers their coffee table. In reality the field of view would not allow the user to view the whole coffee table at the same time.

A more common use of AR in games is through the use of AR cards. These are markers that a camera on a device like a phone, or Nintendo 3DS can pick up on their camera and use to add three-dimensional imagery to the scene. AR cards came with both the 3DS and PlayStation Vita. They also power the card game version of Skylanders for cell phones. One technology that powers this type of interaction Vufloria has evolved to use actual toys and other objects as AR markers.

The AR overlay glasses have become less used, as Google Glass has ended production. There are however competitors entering the marketplace, and the company Magic Leap, who received hundreds of millions in investment to build the next generation of AR for consumers has yet to release a product. AR is on the cusp of becoming the next big thing, but it is still hiding behind the major and more visible strides of VR.

What is becoming even more popular in AR is the use of AR stickers or graphics being added to pictures and photos. Popular services like SnapChat and Facebook are allowing users to change their faces to look dressed up like popular movies, of funny stretch and squished faces. These lenses are changing the way users share moments with each other, by being able to be silly or physically manifest emotion.

There are many commercially available systems for VR ranging from professional-level which can only be attained with large budget expenditures to the consumer level where some products can be had for as little as 20 dollars. On the stratospheric end of the professional level, providing an experience that is fully immersive consisting of a room of 360-degree projections and use 3D glasses to emphasize the virtual environment.

Another high-end experience is the VOID which includes VR headsets and haptic wearables worn in an environment that is mapped within the VR so that doors and hallways are replicated within the experience. This experience is location based and utilizes hardware that is far beyond what consumers can have at home, but the differentiation may be reduced as the next generation of consumer technology is released.

Already the consumer level are headsets like the PlayStation VR, Oculus Rift and HTC VIVE that are extremely affordable and can give a relatively immersive VR experience. The HTC VIVE’s room scale VR is the closest thing consumers have to the embodied experience provided by the VOID. The other consumer experiences use front facing hand tracking making it difficult to turn around in the environment. The next generation will utilize inside out tracking, allowing the glasses to track the hands reducing this issue.

There are also attachments for cell phones like the Google Cardboard that, when coupled with apps, can convert the cell phone into a headset for VR use. The experiences provided by these devices are not nearly as full featured, mostly due to the quality of a phone compared to a $2,000.00 PC, but the line is blurring. The next generation of VR technology will be even more portable and affordable than ever before.

4 Design of a VR Environment for Prosthetic Training

Here are the University of Central Florida our research team is using AR and VR in various ways to create interactive training for underserved populations. This research utilizes gamification, custom video game controllers, AR and VR tech to aid in disability training and rehabilitation. This multi-modal approach allows our research team to develop broad and engaging interactions to keep user interest but still accomplish training. We are working with Limbitless Solutions Inc, a company that provides 3D printed prosthetics for children for no cost. As mentioned above the team developed a multitude of training tools to aid in the training associated with the prosthetics.

The team is incorporating AR stickers in the form of an interactive application that allows the user to select their customized prosthetic arm. This app allows children to customize and visually “test drive” their prosthetic before they receive it. Using mobile based tracking, researchers can track the virtual prosthetic on the video of the child utilizing a cell phone camera. This is a custom solution but works similarly to a SnapChat filter.

Our VR project comes from two vantage points that of the prosthetic users and users with two organic limbs. Our team is working on a VR game experience that will convey the concept of limb loss to those who have use of all of their limbs. The experience will immerse the player into a world where they may have an arm missing or no arms at all. They will have to perform tasks without the use of a limb and learn how to overcome the difficulty within the virtual game environment.

Concerning the prosthetic users, the VR training simulates exercises that fine tune aspects of timing and motor skills such that of a catch and release situation while tossing a ball. Utilizing custom created EMG video game controllers and trackers, researchers use the HTC VIVE to create a virtual space in which the user can use auditory and visual cues in efforts to improve their hand eye coordination. Our research goal is in conjunction with this training, prosthetic users will be able to more quickly improve their hand to eye coordination using their artificial limb.

Finally, a benefit in VR is role immersion. Developing for empathy and education is another way our research team is using VR. The team has created an interactive experience in which the player will only have access to one fully developed arm. The user has to perform a series of daily tasks that are timed. After the challenge the user is given full access both their limbs in efforts for them to related to the challenges limb deficient children face. The team is planning to provide, The Empathy Quotient (EQ), a pre and post assessment to determine a change in users empathy.

5 Conclusions

In conclusion, the authors have determined that VR and AR technology are current methods to enhance training for motor skill development. The researchers suggest a multi-modal approach when developing for motor skill training. The technology should drive the research never hinder it. The mixed mode approach in VR/AR is an ideal supplement to the current traditional methods of rehabilitation training.

Combining a solid foundation of research and prior art with new advances and commercial off the shelf technologies is allowing for better tools to train and rehabilitate users of prosthetic arms. The ability to bring prosthetics into the virtual world and allow users to train how to use there arm, while at the same time providing opportunities for empathy through having others experience what having a prosthetic arm would be like, is changing the way the world will think about prosthetics. It is clear that as VR and AR applications are built for prosthetic users new avenues will open to be explored. The future for this technology is bright.

5.1 Future Work

Another challenge arising in the near future is how to train for more advanced prosthetic gestures. The current iteration of the Limbitless arm simply opens and closes. The Limbitless team is in development of gestural controls and finger movements for their next release. This has posed a challenge for our games research team: How do you train a child to learn multiple, pattern based, electromyographic (EMG) inputs? Our team has developed a proprietary calibration system to simulate an analog input from a single EMG sensor. In addition, by using virtual reality and a 3D virtual limb, we are able to test feedback and EMG controlled gestures and simulate them in a virtual space. This allows our team to test various methods of input and output from the EMG mechanics. All doing so before porting the method of choice to the children user population.