TARS Mobile App with Deep Fingertip Detector for the Visually Impaired

Abstract

Tactile graphics with an Audio Response System (TARS) allows visually impaired people to learn maps, diagrams, and figures by synthesizing audio descriptions of tactile graphics. The current TARS is cumbersome since it relies on a computer and touch panel to detect fingertip coordinates. Here, we improve portability by implementing a TARS mobile app endowed with a camera-based deep fingertip detector. We implemented the hand keypoint detector proposed in Simon et al. 2017, and restricted our attention on the fingertips. Performance benchmarking was done over different tactile graphics types, participants and experimental conditions (i.e. dark/light environment, partial occlusion). In normal conditions, accuracy is 98% on average, but in the case of a partially occluded hand, the detection accuracy of the thumb decreases. These results indicate that the proposed mobile TARS app would perform similar to the standard TARS while being more portable and intuitive.