Development of Fingering Learning Support System Using Fingertip Tracking from Monocular Camera

Abstract

In recent years, as smartphones have become a part of life of young generation, opportunities to enter text using keyboards have decreased, and the speed of keyboard text entry has declined. However, it is still necessary for young children to use keyboards. In typing using a keyboard, the correspondence between keys and fingers is defined; however, there is no practical system for learning fingering to press keys with the correct fingers. A general typing learning software can only judge whether the correct input has been given or not and does not consider the correct fingering of the keys. If the system can evaluate fingering when learning typing, the user can concentrate on typing without having to judge the correctness of their own fingering. In previous research, there have been methods to acquire fingertip coordinates by attaching color stickers to fingers as markers or by using a distance camera, but these methods lack versatility. In this study, we used a monocular camera to acquire fingertip images on a keyboard. After obtaining images, we used MediaPipe to perform hand tracking and obtain the position of the fingertip. For each frame, the system calculates the distance between each key coordinate and each fingertip coordinate, determines that the closest finger hits the key, and feeds back the fingering information.