Abstract
With the rise of AI and Deep Learning technologies, it is now possible to give the visually impaired a sense of sight. This work intends to propose a system, which helps the blind to perceive their surrounding without any extra hand. The system harnesses the power of revolutionary cloud technology, cutting-edge artificial intelligence systems and state of the art language translation technologies for the inevitable cause of assisting the blind. This work mainly focusses on developing a simple gesture-controlled cloud based mobile application, which would allow them to capture their surroundings and help them to navigate through their surroundings in real-time. In this work a real case system architecture is proposed which would analyse the spatial reference of objects in the image and also the custom trained NLP engine generates a description that is narrated in their own native language which stands the unique aspect of the work. The proposed application proves to be a one-stop solution for the visually impaired with real case analysis. To strengthen the analysis of the work, results pertaining to the system architecture emphasising its real-time performance and accessibility are done.
Similar content being viewed by others
Data availability
The manuscript has no associated data. Data sharing not applicable to this article as no datasets were generated or analysed during the current study.
References
Ahmed F, Mahmud MS, Al-Fahad R, Alam S, Yeasin M (2018) Image captioning for ambient awareness on a sidewalk. In: 2018 1st International Conference on Data Intelligence and Security (ICDIS). IEEE, pp 85–91
Arora A, Grover A, Chugh R, Reka SS (2019) Real time multi object detection for blind using single shot multibox detector. Wirel Pers Commun 107(1):651–661
Bagwan SMR, Sankpal LJ (2015) VisualPal: a mobile app for object recognition for the visually impaired. In: 2015 International Conference on Computer, Communication and Control (IC4). IEEE, pp 1–6
Bai J, Liu D, Su G, Fu Z (2017) A cloud and vision-based navigation system used for blind people. In: Proceedings of the 2017 international conference on artificial intelligence, automation and control technologies, pp 1–6
Bakshi AM, Simson J, de Castro C, Yu CC, Dias A (2019) Bright: an augmented reality assistive platform for visual impairment. In: 2019 IEEE Games, Entertainment, Media Conference (GEM). IEEE, pp 1–4
Berger A, Maly F (2019) Smart google glass solution used as education support tool. In: 2019 International Symposium on Educational Technology (ISET). IEEE, pp 265–267
Chaudhuri A, Mandaviya K, Badelia P, Ghosh SK (2017) Optical character recognition systems. In: Optical character recognition systems for different languages with soft computing. Springer, Cham, pp 9–41
Cutter M, Manduchi R (2017) Improving the accessibility of mobile OCR apps via interactive modalities. ACM Trans Accessible Comput (TACCESS) 10(4):1–27
Dong J, Li X, Lan W, Huo Y, Snoek CG (2016) Early embedding and late reranking for video captioning. In: Proceedings of the 24th ACM international conference on multimedia, pp 1082–1086
Evans G, Miller J, Pena MI, MacAllister A, Winer E (2017) Evaluating the Microsoft HoloLens through an augmented reality assembly application. In: Degraded environments: sensing, processing, and display 2017, vol 10197. International Society for Optics and Photonics, pp 101970
Fiannaca A, Apostolopoulous I, Folmer E (2014) Headlock: a wearable navigation aid that helps blind cane users traverse large open spaces. In: Proceedings of the 16th international ACM SIGACCESS conference on computers & accessibility, pp 19–26
Fusco G, Tekin E, Ladner RE, Coughlan JM (2014) Using computer vision to access appliance displays. In: Proceedings of the 16th international ACM SIGACCESS conference on computers & accessibility, pp 281–282
Gaudissart V, Ferreira S, Thillou C, Gosselin B (2004) SYPOLE: mobile reading assistant for blind people. In: 9th Conference Speech and Computer
Gupta P, Shukla M, Arya N, Singh U, Mishra K (2022) Let the blind see: an AIIoT-based device for real-time object recognition with the voice conversion. In: Machine learning for critical internet of medical things. Springer, Cham, pp 177–198
Gurari D, Zhao Y, Zhang M, Bhattacharya N (2020) Captioning images taken by people who are blind. arXiv preprint arXiv:2002.08565
Hammami A, Ben Hamida A, Ben Amar C (2021) Blind semi-fragile watermarking scheme for video authentication in video surveillance context. Multimed Tools Appl 80(5):7479–7513
Hasnine MN, Flanagan B, Akcapinar G, Ogata H, Mouri K, Uosaki N (2019) Vocabulary learning support system based on automatic image captioning technology. In: International conference on human-computer interaction. Springer, Cham, pp 346–358
Hu M, Chen Y, Zhai G, Gao Z, Fan L (2019) An overview of assistive devices for blind and visually impaired people. Int J Robot Autom 34(5):580–598
Huang J, Lin S, Wang N, Dai G, Xie Y, Zhou J (2019) Tse-cnn: a two-stage end-to-end cnn for human activity recognition. IEEE J biomedical health Inf 24(1):292–299
Jain A, Dubey A, Gupta R, Jain N, Tripathi P (2013) Fundamental challenges to mobile based ocr. vol, 2(5), 86–101
Jasmine GS, Marry DM, Lakshmi SS, Rishiwanth R, Sreehariprasath K, Surendhar J (2021) Camera based text and product lable reading for blind people. In: 2021 7th International Conference on Advanced Computing and Communication Systems (ICACCS), vol 1. IEEE, pp 1122–1126
Khan MA, Paul P, Rashid M, Hossain M, Ahad MAR (2020) An AI-based visual aid with integrated reading assistant for the completely blind. IEEE Trans Human-Machine Syst 50(6):507–517
Kumar S, Mathew S, Anumula N, Chandra KS (2020) Portable camera-based assistive device for real-time text recognition on various products and speech using android for blind people. In: Innovations in electronics and communication engineering. Springer, Singapore, pp 437–448
Kuriakose B, Shrestha R, Sandnes FE (2020) Smartphone navigation support for blind and visually impaired people-a comprehensive analysis of potentials and opportunities. In: International conference on human-computer interaction. Springer, Cham, pp 568–583
Landa Y (2011) Videation assistant for blind and cognitively-impaired users
Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D … Zitnick CL (2014) Microsoft coco: Common objects in context. In: European conference on computer vision. Springer, Cham, pp 740–755
Lu D, Fang Y (2021) Audi-exchange: AI-guided hand-based actions to assist human-human interactions for the blind and the visually impaired. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 1718–1726
Maćkowski M, Brzoza P, Żabka M, Spinczyk D (2018) Multimedia platform for mathematics’ interactive learning accessible to blind people. Multimed Tools Appl 77(5):6191–6208
Maheshan MS, Harish BS, Nagadarshan N (2020) A convolution neural network engine for sclera recognition. Int J Interact Multimed Artif Intell 6(1)
Martinez Gutierrez MF (2019) Automated image captioning: exploring the potential of microsoft computer vision for English and Spanish (Doctoral dissertation, University of Geneva)
Matusiak K, Skulimowski P, Strurniłło P (2013) Object recognition in a mobile phone application for visually impaired users. In: 2013 6th International Conference on Human System Interactions (HSI). IEEE, pp 479–484
Murali M, Sharma S, Nagansure N (2020) Reader and object detector for blind. In: 2020 International Conference on Communication and Signal Processing (ICCSP). IEEE, pp 0795–0798
Neto R, Fonseca N (2014) Camera reading for blind people. Procedia Technol 16:1200–1209
Onyejegbu LN, Ikechukwu OA (2016) Optical character recognition as a cloud service in azure architecture. Int J Comput Appl 146(13):14–20
Pamparău C, Vatavu RD (2021) FlexiSee: flexible configuration, customization, and control of mediated and augmented vision for users of smart eyewear devices. Multimed Tools Appl 80(20):30943–30968
Peters JP, Thillou C, Ferreira S (2004) Embedded reading device for blind people: a user-centered design. 33rd Applied Imagery pattern recognition workshop (AIPR’04). IEEE, pp 217–222
Price LC, Chen J, Park J, Cho YK (2021) Multisensor-driven real-time crane monitoring system for blind lift operations: lessons learned from a case study. Autom Constr 124:103552
Qureshi TA, Rajbhar M, Pisat Y, Bhosale V (2021) AI based app for blind people. Int Res J Eng Technol 8(3):2883–2887
Rahman MW, Tashfia SS, Islam R, Hasan MM, Sultan SI, Mia S, Rahman MM (2021) The architectural design of smart blind assistant using IoT with deep learning paradigm. Internet Things 13:100344
Raj VG, Vigneswaran EE, Deshnaa M, RajPrasanth K (2022) Virtual smart glass for blind using object detection. In: 2022 4th international conference on smart systems and inventive technology (ICSSIT). IEEE, pp 1419–1424
Sandhya BR, Sahana S, Sneha S (2021) Third eye for blind. Perspectives in Communication, Embedded-systems and Signal-processing-PiCES 4(11):280–283
Schauerte B, Martinez M, Constantinescu A, Stiefelhagen R (2012) An assistive vision system for the blind that helps find lost things. In: International conference on computers for handicapped persons. Springer, Berlin, pp 566–572
Singh N, Ji G (2021) Computer vision assisted, real time blind spot detection based collision warning system for two wheelers. In: 2021 5th International Conference on Electronics, Communication and Aerospace Technology (ICECA), pp 1179–1184. IEEE
Smith R (2007) An overview of the Tesseract OCR engine. In: Ninth international conference on document analysis and recognition (ICDAR 2007), vol 2. IEEE, pp 629–633
Tamimi AA, Hammad AAA, Abdalla AM, Al-Allaf ON (2019) A system for the detection and identification of objects and their distances to aid blind people. In: Proceedings of the International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV). The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp), pp 98–104
Vaithiyanathan D, Muniraj M (2019) Cloud based text extraction using google cloud vison for visually impaired applications. In: 2019 11th International Conference on Advanced Computing (ICoAC). IEEE, pp 90–96
Vázquez SR, Fitzpatrick D, O’Brien S (2018) Is web-based Computer-Aided Translation (CAT) software usable for blind translators?. In: International Conference on Computers Helping People with Special Needs. Springer, Cham, pp 31–34
Verma KK, Singh BM, Mandoria HL, Chauhan P (2020) Two-stage human activity recognition using 2D-ConvNet. Int J Interact Multimed Artif Intell 6(2):125–135
Wang Q, Chan AB (2018) Cnn + cnn: convolutional decoders for image captioning. arXiv preprint arXiv:1805.09019
Wang M, Song L, Yang X, Luo C (2016) A parallel-fusion RNN-LSTM architecture for image caption generation. In: 2016 IEEE International Conference on Image Processing (ICIP). IEEE, pp 4448–4452
Yang CS, Yang YH (2017) Improved local binary pattern for real scene optical character recognition. Pattern Recognit Lett 100:14–21
Yin XC, Zuo ZY, Tian S, Liu CL (2016) Text detection, tracking and recognition in video: a comprehensive survey. IEEE Trans Image Process 25(6):2752–2773
Funding
There is no funding involved for the work
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that there is no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Venkat Ragavan, S., Tarun, A.H., Yogeeshwar, S. et al. A realtime portable and accessible aiding system for the blind – a cloud based approach. Multimed Tools Appl 82, 20641–20654 (2023). https://doi.org/10.1007/s11042-023-14419-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-14419-9