Abstract
This paper studies the realization of the virtual piano system based on a monocular inverted camera. Compared to the traditional physical piano with the high price and large size, the virtual piano based on computer vision is economical and portable for novices to learn playing piano. However, while reducing learning cost, the virtual piano has to make compromises in terms of user experience and interactivity. On one hand, the software virtual piano only uses a computer keyboard, which cannot meet the players’ cognition of the positions of physical piano keys. On the other hand, the virtual piano system which uses external binocular cameras is still at the theoretical stage. In practice, it is greatly limited by the problems of mutual coverage by left and right hands, large depth detection error, poor real-time performance, and so on. Therefore, in order to simulate the real experience of physical piano playing, this paper proposes the realization method of the virtual piano system based on a monocular inverted camera and a transparent plate with piano keys pattern. In detail, we use the offline modeling method to get the position of the piano keys and use the color threshold segmentation algorithm to realize the fingertips detection. Then, we detect the keys’ position and depth according to fingertips pressing the keys, in turn, realize the virtual piano functions and complete the human-computer interaction. The experiments verify the practicality of the virtual piano system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Conklin Jr., H.A.: Design and tone in the mechanoacoustic piano. Part I. Piano hammers and tonal effects. J. Acoust. Soc. Am. 99(6), 3286–3296 (1996)
Marrington, M., et al.: Composing with the digital audio workstation. In: The Singer-Songwriter Handbook. Bloomsbury Publishing, London (2017)
Desnoyers-Stewart, J., Gerhard, D., Smith, M.: Mixed reality MIDI keyboard. In: Proceedings of the 13th International Symposium on CMMR, pp. 376–386 (2017)
Shapiro, L.: Computer Vision and Image Processing. Academic Press, Cambridge (1992)
DePra, Y., Fontana, F., Tao, L.: Infrared vs. ultrasonic finger detection on a virtual piano keyboard. In: International Computer Music Conference (2014)
Corso, J.J.: Techniques for vision-based human-computer interaction. Ph.D. thesis, Johns Hopkins University (2005)
Yeh, C.H., Tseng, W.Y., Bai, J.C., Yeh, R.N., Wang, S.C., Sung, P.Y.: Virtual piano design via single-view video based on multi finger actions recognition. In: 2010 3rd International Conference on Human-Centric Computing, pp. 1–5 (2010)
van Wyk, S., van der Haar, D.: A multimodal gesture-based virtual interactive piano system using computer vision and a motion controller. In: 2017 2nd Asia-Pacific Conference on Intelligent Robot Systems, pp. 1–6. IEEE (2017)
Begum, H., Shaheen, S., Moetesum, M., Siddiqi, I.: Digital beethoven—an android based virtual piano. In: 2017 13th International Conference on Emerging Technologies, pp. 1–5. IEEE (2017)
Qiao, W., Wei, R., Zhao, S., Huo, D., Li, F.: A real-time virtual piano based on gesture capture data. In: 2017 12th International Conference on Computer Science and Education, pp. 740–743. IEEE (2017)
Guo, R., Cui, J., Zhao, W., Li, S., Hao, A.: Hand-by-hand mentor: an AR based training system for piano performance. In: 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, pp. 436–437. IEEE (2021)
Zhao, L.: Research on virtual piano based on computer binocular stereo vision. J. Phys.: Conf. Ser. 1533, 032006 (2020)
Du, H., Oggier, T., Lustenberger, F., Charbon, E.: A virtual keyboard based on true-3d optical ranging. In: Proceedings of the British Machine Vision Conference, vol. 1, pp. 220-229 (2005)
Malik, S., Laszlo, J.: Visual touchpad: a two-handed gestural input device. In: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 289–296 (2004)
Hernanto, S., Suwardi, I.S.: Webcam virtual keyboard. In: Proceedings of the 2011 International Conference on Electrical Engineering and Informatics, pp. 1–5. IEEE (2011)
Goodwin, A., Green, R.: Key detection for a virtual piano teacher. In: 2013 28th International Conference on Image and Vision Computing New Zealand, pp. 282–287. IEEE (2013)
Torralba, A., Oliva, A.: Depth estimation from image structure. IEEE Trans. Pattern Anal. Mach. Intell. 24(9), 1226–1238 (2002)
Kuehni, R.G.: Color space and its divisions. Color Res. Appl. 26(3), 209–222 (2001). Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Francais de la Couleur
Lim, Y.W., Lee, S.U.: On the color image segmentation algorithm based on the thresholding and the fuzzy c-means techniques. Pattern Recogn. 23(9), 935–952 (1990)
Gong, J., Jiang, Y., Xiong, G., Guan, C., Tao, G., Chen, H.: The recognition and tracking of traffic lights based on color segmentation and CAMSHIFT for intelligent vehicles. In: 2010 IEEE Intelligent Vehicles Symposium, pp. 431–435. IEEE (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, Y., Song, L. (2021). Virtual Piano System Based on Monocular Camera. In: Huang, DS., Jo, KH., Li, J., Gribova, V., Hussain, A. (eds) Intelligent Computing Theories and Application. ICIC 2021. Lecture Notes in Computer Science(), vol 12837. Springer, Cham. https://doi.org/10.1007/978-3-030-84529-2_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-84529-2_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-84528-5
Online ISBN: 978-3-030-84529-2
eBook Packages: Computer ScienceComputer Science (R0)