Abstract
Scene query is an important problem for the visually impaired population. While existing systems are able to recognize objects surrounding a person, one of their significant shortcomings is that they typically rely on the phone camera with a finite field of view. Therefore, if the object is situated behind the user, it will go undetected unless the user spins around and takes a series of pictures. The recent introduction of affordable, panoramic cameras solves this problem. In addition, most existing systems report all “significant” objects in a given scene to the user, rather than respond to a specific user-generated query as to where an object located. The recent introduction of text-to-speech and speech recognition capabilities on mobile phones paves the way for such user-generated queries, and for audio response generation to the user. In this paper, we exploit the above advancements to develop a query system for the visually impaired utilizing a panoramic camera and a smartphone. We propose three designs for such a system: the first is a handheld device, and the second and third are wearable backpack and ring. In all three cases, the user interacts with our systems verbally regarding whereabouts of objects of interest. We exploit deep learning methods to train our system to recognize objects of interest. Accuracy of our system for the disjoint test data from the same buildings in the training set is 99%, and for test data from new buildings not present in the training data set is 53%.
This project was funded by Microsoft AI for Accessibility program.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bai, J., Liu, Z., Lin, Y., Li, Y., Lian, S., Liu, D.: Wearable travel aid for environment perception and navigation of visually impaired people. CoRR abs/1904.13037 (2019). http://arxiv.org/abs/1904.13037
Balamurugan, A., Zakhor, A.: Online learning for indoor asset detection. In: 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1–6 (2019)
Bentley, J.L.: Multidimensional binary search trees used for associative searching. Commun. ACM 18(9), 509–517 (1975). https://doi.org/10.1145/361002.361007
CloudSight: Taptapsee (2012)
Dai, J., Li, Y., He, K., Sun, J.: R-FCN: object detection via region-based fully convolutional networks. CoRR abs/1605.06409 (2016). http://arxiv.org/abs/1605.06409
Eyes, B.M.: Be my eyes (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385
Huang, J., et al.: Speed/accuracy trade-offs for modern convolutional object detectors. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3296–3297 (2016)
Intel: Realsense d415 (2018)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv e-prints arXiv:1412.6980, December 2014
Kotyan, S., Kumar, N., Sahu, P.K., Udutalapally, V.: Drishtikon: an advanced navigational aid system for visually impaired people. CoRR abs/1904.10351 (2019). http://arxiv.org/abs/1904.10351
Lin, T., Dollár, P., Girshick, R.B., He, K., Hariharan, B., Belongie, S.J.: Feature pyramid networks for object detection. CoRR abs/1612.03144 (2016). http://arxiv.org/abs/1612.03144
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. arXiv e-prints arXiv:1708.02002, August 2017
Lin, T., et al.: Microsoft COCO: common objects in context. CoRR abs/1405.0312 (2014). http://arxiv.org/abs/1405.0312
Microsoft: Seeing AI (2017)
Moulon, P., Monasse, P., Perrot, R., Marlet, R.: OpenMVG: open multiple view geometry. In: Kerautret, B., Colom, M., Monasse, P. (eds.) RRPR 2016. LNCS, vol. 10214, pp. 60–74. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-56414-2_5
World Health Organization: World report on vision. World Health Organization (2019)
Povey, D., et al.: The Kaldi speech recognition toolkit (2011)
Ricoh: Theta v (2017)
Sandler, M., Howard, A.G., Zhu, M., Zhmoginov, A., Chen, L.: Inverted residuals and linear bottlenecks: mobile networks for classification, detection and segmentation. CoRR abs/1801.04381 (2018). http://arxiv.org/abs/1801.04381
Toyota: Project blaid (2016)
Tzutalin: Labelimg (2015)
Yelamarthi, K., Haas, D., Nielsen, D., Mothersell, S.: RFID and GPS integrated navigation system for the visually impaired, pp. 1149–1152, August 2010. https://doi.org/10.1109/MWSCAS.2010.5548863
Yi, C., Flores, R., Chincha, R., Tian, Y.: Finding objects for assisting blind people. Netw. Model. Anal. Health Inform. Bioinform. 2, 71–79 (2013). https://doi.org/10.1007/s13721-013-0026-x
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, L., Herzi, I., Zakhor, A., Hiremath, A., Bazargan, S., Tames-Gadam, R. (2020). Indoor Query System for the Visually Impaired. In: Miesenberger, K., Manduchi, R., Covarrubias Rodriguez, M., Peňáz, P. (eds) Computers Helping People with Special Needs. ICCHP 2020. Lecture Notes in Computer Science(), vol 12376. Springer, Cham. https://doi.org/10.1007/978-3-030-58796-3_59
Download citation
DOI: https://doi.org/10.1007/978-3-030-58796-3_59
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58795-6
Online ISBN: 978-3-030-58796-3
eBook Packages: Computer ScienceComputer Science (R0)