Keywords

1 Introduction

As one of most popular musical instruments, piano is frequently used in jazz and classical music which consists of 88 keys (52 white keys, 36 black keys) and a metal chord pad. However, people meet many difficulties in the process of learning piano because they are unfamiliar to the sheet music notation and its mapping to respective piano keys [1], especially for the older adults.

In recent years, many technologies were invented for piano – aided learning. Among them, augmented reality has been proved to be efficient as it could set the virtual scene on the screen and interact with the real world. Huang et al. [2] proposed a markerless augmented reality based piano teaching system by detecting and calculating the transformation matrix from keyboard coordinate to camera coordinate, which tracks the real keyboard of piano naturally. Chow et al. [3] explored whether the use of augmented reality could create an immersive experience and improve the beginner piano students’ learning efficiency, and the user study shows that he system initially puts some pressure on users and improved their learning. Rogers et al. [1] proposed a piano learning system with interactive projection that facilitates a fast learning process, in which music notation information is directly projected onto an electronic piano, and their user study shows this system supports faster learning, requires significantly less cognitive load and provides better user experience. Das et al. [4] implements of an augmented reality piano learning tool by using a Microsoft HoloLens, and users could play their own solos by watching virtual hand demonstrations.

However, the above papers don’t consider older adults’ circumstances. The human’s physical and cognitive function would decline as the growth of age, so it’s an important issue to improve older adults’ leisure and mental health. Castro et al. [5] figured out that the older adults hold a positive attitude to leisure activities, however it’s hard to transform motivation to behavior. Kim et al. [6] pointed out that the following three themes associated with the benefits of serious engagement in leisure activities mainly: the experience of psychological benefits, the creation of social support and the enhancement of physical health. Uptampohtiwat et al. [7] shows that music could be taken as a therapy to improve people’s mental health.

In our pilot study, we developed an app operated on the AR smart glasses to help novices learning on a real piano. In our evaluation experiment, the elderly people are invited as learners and both the single play and the paired play are introduced.

2 Methodology

2.1 Hardware and Software

The AR assisted system for piano playing is composed of a pair of commercially available AR smart glasses, i.e., Epson Moverio BT-300, and a self-developed app. The Moverio BT-300 shown in Fig. 1 weighs 69 g with binocular see-through viewing and running Android 5.1. By replacing with the supplied nose pad for glasses, the headset can be worn over glasses that are less than 147 mm wide. The glasses can be operated by using the keys and the trackpad on a controller which is connected with the headset by a wire.

Fig. 1.
figure 1

Epson Moverio BT-300 smart glasses

Full size image

The software was created as an Android app by using the Android studio development environment. The app mainly includes two windows. As shown in the Fig. 2, the first window shows the main menu to select songs to play, either for practicing or for testing in the experiment, to select the desired tempo and to select playing the right-hand melody or the left-hand harmony. Three songs are prepared in the pilot study. The “Twinkle twinkle little star” which is a popular English lullaby is chosen as the song for practicing. In play mode, the “Farewell” originally from “Dreaming of home and mother” by Jone Pond Ordway is a testing song for the single play and the “My sun” which is a simplified version of “O Sole Mio” sung by Luciano Pavarotti for the paired play.

Fig. 2.
figure 2

Main menu of the AR based piano learning app

Full size image

After clicking the “start” button, a new window will open and an octave of piano keys appears on the screen as shown in the Fig. 3. There are 7 white with 5 black keys for twelve different notes corresponding to the octave marked on the piano to be played. In our pilot study for either a practicing or a testing song, all the notes are within the range of no more than 9 consecutive keys. Once the “Start” key on the controller of the smart glasses is pressed, a red square appearing onto a key on screen indicates the key currently that needs to be pressed. If the red mark keeps staying on the same key, the finger is supposed to hold the key down lightly until the red mark changes the position or disappears. A blinking red mark, i.e. quickly reappearing after disappearing, indicates the finger needs to be raised and then pressed on the same key again. Current version of the app does not supply any information about the correct fingering due to the simplicity of the songs selected. Therefore in this paper the accuracy measurement does not count any fingering mistakes and the users can press the piano keys in the way they feel comfortable. In the Fig. 3 the background is set to black in the drawing on the projection because for Moverio BT-300, the black color in the app generates a transparent background effect so that the visual elements on screen overlap with the actual objects and stand out vividly. Therefore the user can see the images in the app with the view of the surroundings, or in another word the user can see both the keyboard on screen and the piano keys.

Fig. 3.
figure 3

AR keyboard screen (the red square indicates the key that needs to be pressed and the duration of the red mark on the key is consistent with the duration of the note) (Color figure online)

Full size image

2.2 Experimental Process

In order to examine the impact of AR based piano learning on the novices’ performance, we conducted an experimental study of a short-term learning process. The volunteers who are considered qualified for current experimental assessment satisfy the following two conditions, i.e., the novices did not learn piano before and they have no knowledge of the five-line staff based sheet music notation. Normally the volunteers participate as a pair since the arrangement of the paired play is involved and generally the process lasts more than 40 min but less than an hour.

The Fig. 4 shows the process diagram of the experiment. At the beginning of the session, the participants listen to the background information of the experiment, learn about the process and answer a couple of questions in the questionnaire regarding the task difficulty estimation. And then the first participant puts on the smart glasses, selects the proper nose pad and makes some adjustment in the position of the nose pad to see the AR screen clearly while the second participant starts the special ability test. In practice mode, the first learner play the “Twinkle twinkle little star” alone and the learner will decide how many times they play. Since each learner in the study is familiar with the rhythm and melody of this song, the music sound supports a live feedback in finding correct keys. Normally after the practice is repeated two or three times, the users will feel they are accustomed to mapping the key to be pressed on screen to the respective piano key and decide to enter into the single play mode. The learner is allowed to play either the single-play or the paired-play song, i.e. “Farewell” and “My sun”, only once. In our pilot study, the “slow” tempo is selected for all the learners since the reduced speed ease the mapping between the AR screen and the piano keys. As the speed is slower than the normal speed that the song demands the perceived enjoyment of the melody has to be sacrificed. The two learners swap tasks after the first learner finishes the single play recital.

Fig. 4.
figure 4

Process diagram of the evaluation experiment

Full size image

In the next step, the two participants will work together to complete a paired-play song with one playing the left-hand harmony and the other playing the right-hand melody. Left or right hand play is randomly selected by two learners. And finally the participants will fill out the questionnaire and a small interview will be conducted with one person at a time. The questions for the interview are listed as follows:

  1. (1)

    Describe the degree of familiarity with the other participant.

  2. (2)

    Compare different feelings between the single play and the paired play.

  3. (3)

    Have you ever thought of playing these 3 songs in such a short period of time?

  4. (4)

    Have you used a similar app or software to learn any musical instruments before?

  5. (5)

    Do you have any discomfort during the playing process and any suggestions to improve the app which can assist you better for piano learning?

3 Results and Discussion

A total of 13 volunteers were invited to the evaluation experiment, 12 of whom came in pairs. One volunteer came alone, for her paired play part one of our lab staff members played with her and the rest of the process remained the same. The age of 11 participants is between 20 and 30 and all of them are students. And another two participants are aged 60–69. 13 participants were fairly balanced in gender (8 male and 5 female).

During the evaluation experiment, the spatial ability score of each participant was collected. With the permission of each participant, their single play and paired play testing songs were recorded for accuracy calculation as the quantitative analysis of the participants’ short-term learning performances. Currently the accuracy calculation only focuses on the keys pressed correctly and the duration of each note will not be evaluated. Therefore the accuracy rating is equal to the number of keys pressed correctly divided by the total number of notes to be played in a song. For the single play song “Farewell”, 28 keys need to be pressed and for the paired play song “My sun” there are 34 notes in the right-hand melody part and another 34 in the left-hand harmony part.

The questionnaire mainly includes three variables, i.e. the expectation rating [8] change before and after the play, the perceived enjoyment [9] and the behavioral intention. In results analysis, we tried to find the statistical significance of the relationship among those three variables in 7-point scale, accuracy rating and the special ability score. Figure 5 shows the expectation rating change for single and paired play. The single play has the values evenly distributed around zero (before the play: mean value = 4.54 and standard deviation = 1.44; after the play: mean value = 4.54 and standard deviation = 1.1) which means the participants do not realize the change in difficulty before and after the single play. However, the paired play has the average value of the expectation rating change above zero (before the play: mean value = 4.08 and standard deviation = 2.08; after the play: mean value = 4.92 and standard deviation = 0.91). Therefore the participants do recognize that the paired play is easier than they expected.

Fig. 5.
figure 5

Expectation rating change for single and paired play

Full size image

The second relationship which has some statistical significance is between the perceived enjoyment and the spatial ability score as shown in the Fig. 6. The perceived enjoyment of the paired play is slight higher than that of the single play. However, the slope has the opposite direction than we expected. Before the evaluation experiment was conducted we assumed that the participants with higher score of spatial ability will enjoy the playing more than those with lower score. In fact through our communication experiences with the participants during the evaluation it is found that the perceived enjoyment was influenced by many factors, such as the degree of comfort of wearing the smart glasses, the perceived enjoyment of the melody of the songs, feelings about the software design, the degree of practice, spatial ability, other personal experiences, etc. Those factors are supported from the participants’ answers to the designed interview questions. First, 85% of the participants did not use any similar systems for learning musical instruments and 15% had tried some tutoring software for Guitar learning or the piano learning app on the cell phone. 77% of the participants thought they were able to play these three songs on a real piano in such a short period of time, showing varying degrees of surprise in a positive way.

Fig. 6.
figure 6

Linear regression between the perceived enjoyment and the spatial ability

Full size image

For the single play and the paired play, 31% of the participants did not express any preference. 38% expressed that the paired play requires being more focused and more patient in collaboration. 8% appreciated the music more by adding the left hand harmony part. The rest of 23% expressed the paired play is more enjoyable due to the companionship of the other player. Interestingly the only two elderly people both belong to this 23% category while only one young person enjoyed the paired play in this manner. According to our understanding, younger people pay more attention to their performances in the collaboration while the elderly people do not show too much emphasis on results so that they can experience more pleasure brought by the partner.

As far as the discomfort during the playing, 30% of the participants complained about the discomfort caused by the smart glasses for its unadjustable size. 46% found it difficult for mapping the keys to be pressed on the AR screen and the real piano keys. 15% mentioned they cannot enjoy the melody for the tempo in the setup is slower than the normal speed. Otherwise, no complain of headaches caused by the hardware is reported by any participant including the elderly, which proves the advantages of AR smart glasses over VR head mounted Display (HMD). Therefore the opposite slope in the Fig. indicated that the participants with the higher score of the spatial ability might identify more drawbacks of the current learning system, while those with lower score pay more attention on the help of the system for their piano learning. Meanwhile the distribution of the spatial ability score is concentrated between 4 and 5 due to the young age of most of the participants. However, the spatial ability of the elderly people varies widely and our future studies will involve more elderly learners which might lead to more interesting conclusions.

There is also another point worth mentioning, which is that the highest accuracy rating of 98% in the paired play belongs to a pair of the most familiar friends. They showed very ordinary performance in their single play with both of the accuracy ratings of 85% while the range of the accuracy rating is between 72%–100% in the single play mode. However, they surpassed all other pairs in collaboration.

4 Conclusion and Future Work

In conclusion, first the main contributions of this pilot study are summarized: (1) to the best of our knowledge, our work is the first study on involving the paired play in piano learning to improve the interest of novices. (2) The elderly were invited as volunteers for the evaluation of an AR based piano learning system for the first time. They show special interests in the paired play which becomes a promising research direction for our future work. (3) Interesting findings in our results are that the paired play is easier than the participants expected; the perceived enjoyment of the paired play is slightly higher than the single play and the elderly people enjoy more the companionship of the partner during the paired play; the best performance in the paired play was generated by a pair of the most familiar friends even though they showed very ordinary performance in their single play among all the participants; the perceived enjoyment is influenced by many factors such as the degree of comfort of wearing the smart glasses, the perceived enjoyment of the melody of the songs, mapping process, the degree of practice, spatial ability, other personal experiences, etc.

Our future work will add in the recognition techniques in AR so that the note information can be projected directly on the piano keyboard and the learning process will become even easier and attractive especially for the elderly people. Meanwhile, we will focus more on the elderly participants to keep exploring the learning interest fostered by the paired play.