Abstract
Visual impairment refers to the partial or complete loss of one’s ability to see. It is estimated that there are 1.3 billion people in the world with some form of vision loss. In this work, we present Viva, an Android-based virtual assistant aiming to help people with visual impairment. The application provides haptic and voice navigation assistance by detecting obstacles in the user’s surroundings and calculating the potential risk. We present the architecture, as well as a proof-of-concept prototype intended to demonstrate a potential use-case for a commercial embedded product that can be integrated into a walking stick or any wearable gadget. This Android application has features such as navigation assistant, object detection, voice-controlled UI and emergency assistant. The navigation assistant analyzes a user’s surroundings by detecting and estimating distances from the user to the object. Object recognition mode includes a pre-built object recognition model that can recognize over 100 different common objects. Data collected is then processed by a risk-prediction algorithm to calculate the risk of collision. Feedback is provided to the user whenever there is a potential risk observed. The UI of the virtual assistant is uniquely designed from the ground-up to be intuitive, without the need for any usual aids via voice commands or single point touch control – where the entire screen acts as a soft button. Viva operates in a low-power mode with the screen turned off to efficiently utilize the limited battery resources on mobile phones. Viva is a prototype intended to demonstrate the potential use-cases of this idea. It can be integrated into other IoT devices such as smart walking sticks or wearable gadgets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
WHO World Report on Vision 2020. https://www.who.int/publications/i/item/world-report-on-vision. Retrieved on Feb 2021)
Bourne, R.R.A., et al.: Global prevalence of blindness and distance and near vision impairment in 2020: progress towards the vision 2020 targets and what the future holds. Invest. Ophthalmol. Vis. Sci. 61(7), 2317 (2020)
WHO fact sheet on blindness and visual impairment. https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment. Retrieved on Feb 2021
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., Berg, Alexander C.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2
Zhao, L., Li, S.: Object detection algorithm based on improved YOLOv3. Electronics 9, 537 (2020). https://doi.org/10.3390/electronics9030537
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.Lawrence: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
COCO dataset. https://cocodataset.org/#download. https://github.com/cocodataset/cocoapi. Retrieved May 2020
Pachodiwale, Z.A., Brahmankar, Y., Parakh, N., Patel, D.: Virtual Assistant for Visually Impaired Video (2020). https://youtu.be/fDmqeYOlYWU
Smart band, Sunu band. https://www.sunu.com/en/index. Retrieved on Feb 2021
Ambutech. iGlasses Ultrasonic Mobility Aid. https://ambutech.com/products/iglasses%E2%84%A2-ultrasonic-mobility-aid. Retrieved on Feb 2021
AFB: LowViz Guide: Indoor Navigation for People who are Blind or Visually Impaired. AccessWorld, July (2015). https://www.afb.org/aw/16/7/15437. Retrieved on Feb 2021
Extended Visual Assistant “Eva”. https://www.eva.vision. Retrieved on Feb 2021
Aljahdali, M., Abokhamees, R., Bensenouci, A., Brahimi, T., Bensenouci, M.: IoT based assistive walker device for frail &visually impaired people. In: 2018 15th Learning and Technology Conference (L&T), Jeddah, pp. 171–177 (2018). https://doi.org/10.1109/lt.2018.8368503
Hung, D.N., Minh-Thanh, V., Minh-Triet, N., Huy, Q.L., Cuong, V.T.: Design and implementation of smart cane for visually impaired people. BME 2017. IP, vol. 63, pp. 249–254. Springer, Singapore (2018). https://doi.org/10.1007/978-981-10-4361-1_41
Nisha, K.K., Pruthvi, H.R., Ashwini, T.S., Hadimani, S.N., Domanal, S., Ram Mohana Reddy, G.: An android GPS-based navigation application for blind. In: The 7th International Symposium on Visual Information Communication and Interaction, pp. 240–241 (2014). https://doi.org/10.1145/2636240.2636878
Harish Kumar, N., Deepak, G., Nagaraja, J.: An IoT based obstacle detection and alerting system in vehicles using ultrasonic sensor. In: International Journal of Engineering Research & Technology (IJERT) NCETEIT, vol. 5, Issue 20 (2017)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, pp. 580–587 (2014). https://doi.org/10.1109/cvpr.2014.81
Girshick, R.: Fast R-CNN. In: IEEE International Conference on Computer Vision (ICCV), Santiago, pp. 1440–1448 (2015). https://doi.org/10.1109/iccv.2015.169
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017). https://doi.org/10.1109/TPAMI.2016.2577031
Lee, J., Wang, J., Crandall, D., Sabanovic, S., Fox, G.: Real-time, cloud-based object detection for unmanned aerial vehicles. In: First IEEE International Conference on Robotic Computing (IRC), pp. 36–43 (2017). https://doi.org/10.1109/irc.2017.77
Beksi, W.J., Spruth, J., Papanikolopoulos, N.: CORE: a cloud-based object recognition engine for robotics. In: International Conference on Intelligent Robots and Systems, IROS 2015, Hamburg, Germany, pp. 4512–4517 (2015). https://doi.org/10.1109/iros.2015.7354018
Meliones, A., Filios, C.: BlindHelper: a pedestrian navigation system for blinds and visually impaired. In: Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments (PETRA 2016). ACM, New York, NY, USA, Article 26, pp. 1–4 (2016). https://doi.org/10.1145/2910674.2910721
Transfer Learning in Keras with Computer Vision Models. https://machinelearningmastery.com/how-to-use-transfer-learning-when-developing-convolutional-neural-network-models/. Retrieved on Feb 2021
Test-To-Speech API. https://developer.android.com/reference/android/speech/tts/TextToSpeech. Retrieved on Feb 2021
Speech Recognizer API. https://developer.android.com/reference/android/speech/SpeechRecognizer. Retrieved on Feb 2021
Camera2API package. https://developer.android.com/reference/android/hardware/camera2/package-summary. Retrieved on Feb 2021
TensorFlow Lite API. https://www.tensorflow.org/lite/guide/build_android. Retrieved on Feb 2021
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, ICLR (2015). https://arxiv.org/abs/1409.1556
Inception Resnet version2. https://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1. Retrieved on May 2020
ImageNet dataset. http://www.image-net.org/. Retrieved on May 2020
OpenAI. https://openai.com/. Retrieved on Feb 2021
Huang, J.: Speed/accuracy trade-offs for modern convolutional object detectors. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 3296–3297 (2017). https://doi.org/10.1109/cvpr.2017.351
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Pachodiwale, Z.A., Brahmankar, Y., Parakh, N., Patel, D., Eirinaki, M. (2021). Viva: A Virtual Assistant for the Visually Impaired. In: Antona, M., Stephanidis, C. (eds) Universal Access in Human-Computer Interaction. Design Methods and User Experience. HCII 2021. Lecture Notes in Computer Science(), vol 12768. Springer, Cham. https://doi.org/10.1007/978-3-030-78092-0_30
Download citation
DOI: https://doi.org/10.1007/978-3-030-78092-0_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78091-3
Online ISBN: 978-3-030-78092-0
eBook Packages: Computer ScienceComputer Science (R0)