Using Virtual Reality to Create an Inclusive Virtual Drumming Environment

Abstract

Upon doing research in the realm of physical disability, a common occurrence that was found was people missing the ability to take part in passions they had when they were without any disability. Two areas that were targeted in particular were those affected by paralysis by unnatural causes and those dealing with multiple sclerosis. Accounts of paralysis stem from injuries such as car accidents or sports injuries, while multiple sclerosis is a disease in which the immune system attacks the protective covering of the nerves. In order to find solutions to these lost abilities we attempted to find a way allow people to play music again. The research presented in this paper describes an interactive virtual drumming environment for people who do not have the use of their arms and it also contains a visual representation of the drum rhythms for people who are not able to hear. Players play the drums by moving their head and the audio is played through speakers and through a DMX lighting system. Twenty-five people played the virtual drums and their feedback was recorded and used to make suggestions about improvements.

1 Background

Drums come in a variety of types and sizes. For example, Japanese Taiko drums are played on very large drums that are meant to be played in a group setting. Similarly, African drums such as the djembe or conga are also played in a group setting. Each drummer is playing a specific drum and a specific tone. These types of drums have only one basic tone and could be simulated by a foot pedal. In this case a foot pedal could be hit to generate the sound, and the intensity of the sound could follow the amount of pressure the foot hit the pedal with. A drummer in these types is rarely playing two different drums at the same time.

Looking at more complicated drum arrangements like a modern day rock drum set, the simulation of these sounds and rhythms without the use of arms and hands becomes more difficult. A simple drum kit contains a snare drum, two toms, a floor tom, a hihat cymbal and a crash symbol as well as a bass drum. Simulating all of these with foot movements is difficult as the hihat and bass drum typically are played using two different foot pedals leaving the arms to do the rest of the work.

Rick Allen is one of the most famous drummers who does not use both arms. He is the drummer for the rock band Def Leppard and due to a car accident had his left arm amputated in 1984. Despite this setback, he has remained the band’s drummer and has played with a modified drum kit. He has a mostly electronic kit with additional pedals used to generate the sounds and tones previously done by his left arm. He has four pedals for his left foot that play the hihat, bass drum, snare drum, and floor tom. These modifications allow him to continue to play drums at a very high level of skill and consistency [13].

Another drummer, Cornel Hrisca Munn, was born without forearms. He is able to play the drums by attaching the sticks to his arm with a band, moving his arm at the shoulder to create the motion necessary to hit the drum heads while also using his feet to hit the pedals [14]. Alvin Law, is a drummer who was born without arms. Throughout his life he has learned to use his feet as arms in a variety of situations, including playing the drums [15].

As described above, playing the drums without arms is possible, however playing the drums also requires the neurological capability to quickly play within the desired rhythm. Some people may not be able to do that. Music has shown to be useful in the treatment of Parkinson’s disease. However playing music can prove to be difficult for someone with a disease such as Parkinson’s. Parkinson’s Disease whose symptoms include tremors and bradykinesia. Bradykinesia is the slowness of the body to react physically to mental commands. Tasks such as finger tapping, or buttoning shirts become difficult. Foot movement issues like shuffling or dragging feet when walking are common. These types of issues make it difficult for someone with Parkinson’s Disease to keep a rhythm by using physical drums even if foot pedals are used. A person with Parkinson’s Disease might also have tremors, which are involuntary muscle movements. A Parkinson’s tremor is typically a resting tremor as it occurs when the body is at rest. A person may attempt to hide the tremors by moving limbs on purpose and that could also have an effect on a person attempting to drum.

A virtual reality based drumming system has been created before [2], however the participants in that study played a real physical drum while in virtual reality. The study was examining the users interpretation of the virtual world. Virtual avatar based drummers have also been created [3]. Virtual reality has been shown as an effective method for rehabilitation [4, 8, 9], and in this case we seek not to rehabilitate the body, but to rehabilitate the mind and bring back an enjoyable experience to someone who can no longer play the drums. As shown in [5], a virtual drumming type of environment can be measured by watching for accelerometer values, however attaching controllers to a person’s arms or limbs would not be an effective method when a person no longer has functioning arms. Making accessible instruments has been done for things such as virtual guitars [12].

2 Implementation

The development technology used to create a solution to losing the ability to play the drums with a user’s arms was the Oculus Rift and the Unity game development program. The Oculus Rift is used in this application by taking advantage of its positional tracker and headset. The positional tracker is set in a position that faces the headset. Then the tracker monitors a series of infrared LEDs embedded in the headset. This allows the Oculus to know the position of the user’s head which is essential for locating where the user is looking in the application.

Target acquisition in virtual reality can be done a variety of ways [10, 11]. In this case the gaze detection was used as found in similar approaches [1, 7]. Unity’s Raycast function was used which triggers and returns a true boolean whenever it intersects with a game object collider. The Raycast casts a ray from an origin point, the user’s first-person camera, for a set distance in a direction straight out from the user’s perspective. Each drum in the environment has an individual collider which will return the game object’s name when the user’s Raycast intersects it.

The application itself, utilizes two different modes in an attempt to find the most effective environment for someone without mobility below the neck to play the drums with head movements. In both environments the user is placed in a small virtual soundproof room and is placed in front of a drum set.

The first environment created involves a free play mode in which the drum that is looked at will play automatically. In order for the user to know which drum is being played, the piece that is being intersected by the Raycast will be highlighted with a blue circle. After being highlighted, a script will play the corresponding sound. In order to play the next beat the user must look away from the current piece and either look back at it or to a different piece. The user could choose between a crash or ride cymbal, bass drum, snare drum, and 3 types of toms Floor, High and Middle (Fig. 1).

Fig. 1.

Open play.

Upon initial tests with drummers it was found that this limitation, although easy to understand how to play, was limiting. Many drum patterns involve hitting multiple drums at the same time. And using the glance method did not allow for multiple drums to be played at the same time. Several design options were considered, but it was decided to implement a time line type of a feature where a user would select one or more drums, and then increment a time pointer.

The second application mode is a more involved interaction as the user composes a sheet of music by looking at the drum set and then plays it all upon completion. For clarity, a music sheet is placed in front of the user in the interface. As the user selects the beats by looking at the different drum pieces, notes will appear on the music sheet. Within the environment, there is also a button placed in the center of the room for the user to select the pacing of the music by looking at it. A run through of this environment would involve the user looking at drum pieces to be played together, then setting the pacing, then looking at the next button to the user’s left, and then repeating this process until the user has queued up the desired music sheet. Then upon looking at the play button to the user’s right the entire sheet will play (Fig. 2).

Fig. 2.

Incrementing the time pointer.

This method was found to allow users to create somewhat complex drum patterns and play them back. It was found that different types of drummers preferred different methods of interactions.

To properly display the graphics in this environment the user needs to have a computer with at least 8 Gigs of RAM, an Nvidia 1060, and an i5-4590. Player interaction was done completely through head movements using the Oculus Rift Virtual Reality Headset. The rift as 2 OLED displays each with a resolution of 1080 × 1200, a 90 Hz refresh rate, contains sensors for positional and rotational tracking, and has support for 3 dimensional audio. The rift ships with a Constellation sensor that tracks the position of the head and any motion controllers through infrared sensors. Oculus provides motion controlled touch controllers. These controllers’ motion are sensed with an additional infrared Constellation sensor. In the specific usage here, the motion controllers were not used and neither was the second Constellation sensor.

3 Public Demonstration and Feedback

In the public area of a University, students and faculty were invited to test out the VR Drummer. As this was in a public area, no formal data was collected, however many items were observed. As users walked up to the demonstration table, a VR drummer would give a quick explanation of how things worked and start the application by using a mouse. Then the user was given the virtual reality headset and all further controls were done using head movements.

It was found that the glance method was much easier for people to understand. When a person would first start to play with the system, he was shown how to use the glance method. This gave immediate satisfaction to the person as wherever the head was moved a corresponding drum sound was being played. The display gained a lot of attention as the user was doing a lot of head bobbing and the drum set was on display on a large screen. After a few minutes the player would be tapped in the shoulder and informed of the second method of play – composition.

Players understood the composition mode, but through anecdotal evidence, only people with previous drumming experience had any interest in continuing on with this method. The composition method required planning and some sort of knowledge of what sounds good when as well as the patience to finish off a complete piece of drumming music before playing it back. When shown this method, many users requested to go back to the glance method and just have fun making noise.

Discovered during this public play was the lack of additional audio cues. Observers could see what the player was doing by watching the external screen, however the audio was only available to the player through the virtual reality headset. The built-in head phones provided the required amount of the audio for the user, however for the observers they were left to guess what was happening with the audio. This also could have been a good thing because for people to experience the full atmosphere it was required to be in virtual reality. If the observers were provided with both audio and video it may have prevented people from actually participating in the virtual reality experience. However with the audio missing, observers understood that they were missing a large component of the experience and wanted to try it out.

Overall feedback on the system was good. Suggestions included allowing for the user to import other audio such as guitar backing tracks or vocals to play along with. Also users requested a method to export the drumming track audio. As this was a proof of concept demonstration, there was no way to export anything created within the experience (Fig. 3).

Fig. 3.

Press play.

As an added feature to this application we have added a light system that changes colors upon the user playing different beats [6]. This is meant to add inclusiveness for the deaf community. Utilizing the drums in VR provided an environment for someone to play the drums but it also prevented others from hearing the sounds of the drums. This can be fixed by splitting the audio from the headset to also go into an external amplifier, but it also brought up the question about how a person who is deaf would get any enjoyment out of this application. Without seeing arms going up and down or any vibration or motion from the cymbals or the kick drum it would be very difficult for someone who is deaf to understand what is going on.

As a common solution to these types of issues, sensory substitution can be used. That is converting the items that would normally be perceived by a specific sense into another sense that can perceive the same data. For example a person who is blind may have visuals converted to audio. In this case the audio information would need to be transposed into a different sensory modality. Vision is an alternative, however vision requires focus from the user. Unlike audio that can be heard from multiple directions at once, vision requires the focus of the user on the signals, for example looking at a teleprompter or video display. In order to overcome this, a much larger scale lighting can be used. For example lighting up an entire room to different colors can create an ambient secondary visual cue.

An example of a beat pattern to be represented as video could be bright white flashes for the kick drum sound, blue for the toms, and red for the cymbals. Although only one color could be visible at a time (assuming 1 light source) the order of priority for controlling the light was used in order to optimize the information space. For most drum patterns, the bass or kick drum is relatively constant as is the cymbal pattern. As a result, the kick drum was given the lowest priority. A user could get a feel for the overall beat of the music through this cue. Next up on the priority list was the cymbals. Much like the kick drum cymbals keep the beat with occasional accents. The highest priority was given to the toms as they add the characteristics to the songs.

Two different lighting styles were investigated: consumer grade smart bulbs and professional stage lighting. The interface was implemented in Unity as that is where the logic for the hit detection was done. It was then translated into either of the two supported lighting options.

There are many different smart bulb models and manufacturers, here the Philips Hue bulb was investigated. From a smart phone app a user can change the brightness and the color of a bulb. In order to do this through software, open source libraries can be used. It is required to have a Philips Hue Hub. This hub contains a physical network interface and communicates to the smart bulbs. All communication to the light bulbs from any kind of device must travel through this hub. The hub will be assigned an ip address on the network and all communication is through TCP/IP. The open source library phue was utilized to send commands. The protocol using phue is pretty straight forward. The developer is required to connect to the hub, then query for the number of bulbs connected to the hub, then send instructions that set the brightness or color to the desired bulb (Fig. 4).

Fig. 4.

Philips Hue smart bulb architecture.

After running this first application, it was found that the latency was unusable. It was in the order of seconds for the lights to respond to any commands. This may be due to the involvement of the Hue Bridge, or some underlying issues within the phue python library, but whatever the reason it was essentially unusable. The delay not only misrepresented what was going on to observers but confused the active player as indirect light might leak into the headset and not be in beat with what was going on causing confusion for everyone. Because of this a more responsive lighting solution was sought out (Fig. 5).

Fig. 5.

DMX controller architecture.

DMX512 is the standard for communication between a controller and lighting equipment. In order to control DMX light sources from a computer, a USB interface is required. For this experiment, the ENTTEC DMXUSB PRO 70304 was used. It is supported on Windows and on OSX, and is supported by open source software. Using this setup, the response time dropped to be sub-second and the system worked as desired. For the light itself, a SlimPAR 64 RGBA was used. The light was pointed at a white painted wall in order to make a bigger and softer visual response.

4 Conclusion and Future Work

The VR Drummer shows promise as a tool for someone to play the drums without the use of the arms. The two methods, glance and composition were preferred by different types of drummers with different experience. Future revisions will include the ability to import tracks to play along with and to export the drumming track audio for use in external utilities like Garage Band. All of the people who played through the game had the use of their arms. The VR Drummer should have a study done with drummers who have lost the use of their arms to get some direct feedback on how they would like the application to be modified.