Skip to main content
SpringerLink
Log in
Book cover

European Conference on Computer Vision

ECCV 2020: Computer Vision – ECCV 2020 Workshops pp 422–439Cite as

Active Crowd Analysis for Pandemic Risk Mitigation for Blind or Visually Impaired Persons

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12538))

Abstract

During pandemics like COVID-19, social distancing is essential to combat the rise of infections. However, it is challenging for the visually impaired to practice social distancing as their low vision hinders them from maintaining a safe physical distance from other humans. In this paper, we propose a smartphone-based computationally-efficient deep neural network to detect crowds and relay the associated risks to the Blind or Visually Impaired (BVI) user through directional audio alerts. The system first detects humans and estimates their distances from the smartphone’s monocular camera feed. Then, the system clusters humans into crowds to generate density and distance maps from the crowd centers. Finally, the system tracks detections in previous frames creating motion maps predicting the motion of crowds to generate an appropriate audio alert. Active Crowd Analysis is designed for real-time smartphone use, utilizing the phone’s native hardware to ensure the BVI can safely maintain social distancing.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: the clear mot metrics. EURASIP J. Image Video Process. 2008, 1–10 (2008). https://doi.org/10.1155/2008/246309

    Article  Google Scholar 

  2. Bhowmick, A., Hazarika, S.M.: An insight into assistive technology for the visually impaired and blind people: state-of-the-art and future trends. J. Multimodal User Interfaces 11(2), 149–172 (2017). https://doi.org/10.1007/s12193-016-0235-6

    Article  Google Scholar 

  3. BlindSquare: Pioneering accessible navigation - indoors and outdoors, May 2020. https://www.blindsquare.com/

  4. Bologna, G., Deville, B., Pun, T., Vinckenbosch, M.: Transforming 3D coloured pixels into musical instrument notes for vision substitution applications. EURASIP J. Image Video Process. 2007(1), 1–14 (2007). https://doi.org/10.1155/2007/76204

    Article  Google Scholar 

  5. Bourne, R.R., et al.: Causes of vision loss worldwide, 1990–2010: a systematic analysis. Lancet Glob. Health 1(6), e339–e349 (2013)

    Article  Google Scholar 

  6. Brainport: Disabilities technology: Brainport technologies: United states. https://www.wicab.com/

  7. CDC: Social distancing, quarantine, and isolation, May 2020. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/social-distancing.html

  8. Chahyati, D., Fanany, M.I., Arymurthy, A.M.: Tracking people by detection using CNN features. Procedia Comput. Sci. 124, 167–172 (2017). https://doi.org/10.1016/j.procs.2017.12.143

    Article  Google Scholar 

  9. Coniglio, C., Meurie, C., Lézoray, O., Berbineau, M.: People silhouette extraction from people detection bounding boxes in images. Pattern Recogn. Lett. 93, 182–191 (2017). https://doi.org/10.1016/j.patrec.2016.12.014

    Article  Google Scholar 

  10. Corp, A.T.: Flattening the inaccessibility curve. https://flatteninaccessibility.com/

  11. Coughlan, J.M., Miele, J.: AR4VI: AR as an accessibility tool for people with visual impairments. In: 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), pp. 288–292. IEEE (2017)

    Google Scholar 

  12. Dakopoulos, D., Bourbakis, N.G.: Wearable obstacle avoidance electronic travel aids for blind: a survey. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 40(1), 25–35 (2009)

    Article  Google Scholar 

  13. Dale, Z.: Experiences of deafblind persons during the covid-19 outbreak. International Disability Alliance (2020). http://www.internationaldisabilityalliance.org/content/experiences-deafblind-amid-covid-19-outbreak

  14. Deveopers, A.: Android NDK. https://developer.android.com/ndk/guides/audio

  15. Dewhurst, D.C.: Audiotactile vision substitution system, 7 August 2012, US Patent 8,239,032

    Google Scholar 

  16. Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. arXiv:1406.2283 [cs], June 2014

  17. Erhan, D., Szegedy, C., Toshev, A., Anguelov, D.: Scalable object detection using deep neural networks. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2155–2162 (2014)

    Google Scholar 

  18. Ess, A., Leibe, B., Van Gool, L.: Depth and appearance for mobile scene analysis. In: 2007 IEEE 11th International Conference on Computer Vision, pp. 1–8 (2007)

    Google Scholar 

  19. Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining. KDD 1996, pp. 226–231. AAAI Press (1996)

    Google Scholar 

  20. Fox, D., Kar, R., Li, A., Pandey, A.: Augmented reality for visually impaired people. https://www.ischool.berkeley.edu/projects/2019/augmented-reality-visually-impaired-people

  21. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)

    Google Scholar 

  22. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587. IEEE, June 2014. https://doi.org/10.1109/CVPR.2014.81

  23. Giudice, N.A., Legge, G.E.: Blind Navigation and the Role of Technology (chap. 25), pp. 479–500. Wiley (2008). https://doi.org/10.1002/9780470379424.ch25

  24. Gökçe, F., Üçoluk, G., Sahin, E., Kalkan, S.: Vision-based detection and distance estimation of micro unmanned aerial vehicles. Sensors (Basel, Switzerland) 15, 23805–23846 (2015)

    Article  Google Scholar 

  25. Google: Google glass. https://www.google.com/glass/start/

  26. Gordon, K.D.: Survey: the impact of the COVID-19 pandemic on Canadians who are blind deaf-blind, and partially-sighted (2020). http://ccbnational.net/shaggy/wp-content/uploads/2020/05/COVID-19-Survey-Report-Final-wb.pdf

  27. Haseeb, M., Guan, J., Ristić-Durrant, D., Gräser, A.: DisNet: a novel method for distance estimation from monocular camera. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS, Spain 2018). IEEE (2018)

    Google Scholar 

  28. Haymes, S.A., Johnston, A.W., Heyes, A.D.: Relationship between vision impairment and ability to perform activities of daily living. Ophthalmic Physiol. Opt. 22(2), 79–91 (2002)

    Article  Google Scholar 

  29. Helal, A., Moore, S.E., Ramachandran, B.: Drishti: an integrated navigation system for visually impaired and disabled. In: Proceedings Fifth International Symposium on Wearable Computers, pp. 149–156. IEEE (2001)

    Google Scholar 

  30. HTC: HTC VIVE. https://www.vive.com/us/product/

  31. Ignatov, A., Timofte, R., Chou, W., Wang, K., Wu, M., Hartley, T., Van Gool, L.: AI benchmark: running deep neural networks on android smartphones. arXiv:1810.01109 [cs], October 2018

  32. IrisVision: Wearable low vision glasses for visually impaired, May 2020. https://irisvision.com/

  33. Jackson, A.: The hidden struggles America’s disabled are facing during the coronavirus pandemic. CNBC News (2020). https://www.cnbc.com/2020/05/10/the-struggles-americas-disabled-are-facing-during-coronavirus-pandemic.html

  34. Kajabad, E.N., Ivanov, S.V.: People detection and finding attractive areas by the use of movement detection analysis and deep learning approach. Procedia Comput. Sci. 156, 327–337 (2019). https://doi.org/10.1016/j.procs.2019.08.209

    Article  Google Scholar 

  35. Kajimoto, H., Kanno, Y., Tachi, S.: Forehead electro-tactile display for vision substitution. In: Proceedings of the EuroHaptics (2006)

    Google Scholar 

  36. Kempen, G.I., Ballemans, J., Ranchor, A.V., van Rens, G.H., Zijlstra, G.R.: The impact of low vision on activities of daily living, symptoms of depression, feelings of anxiety and social support in community-living older adults seeking vision rehabilitation services. Qual. Life Res. 21(8), 1405–1411 (2012). https://doi.org/10.1007/s11136-011-0061-y

    Article  Google Scholar 

  37. Kulyukin, V., Gharpure, C., Nicholson, J., Pavithran, S.: RFID in robot-assisted indoor navigation for the visually impaired. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No. 04CH37566), vol. 2, pp. 1979–1984. IEEE (2004)

    Google Scholar 

  38. Lamoureux, E.L., Hassell, J.B., Keeffe, J.E.: The determinants of participation in activities of daily living in people with impaired vision. Am. J. Ophthalmol. 137(2), 265–270 (2004)

    Article  Google Scholar 

  39. Lee, J., et al.: On-device neural net inference with mobile GPUs. arXiv:1907.01989 [cs, stat], July 2019

  40. Liu, W., et al.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2

    Chapter  Google Scholar 

  41. Loomis, J.M., Golledge, R.G., Klatzky, R.L.: GPS-based navigation systems for the visually impaired. In: Fundamentals of Wearable Computers and Augmented Reality, p. 429, 46 (2001)

    Google Scholar 

  42. Loomis, J.M., Golledge, R.G., Klatzky, R.L., Speigle, J.M., Tietz, J.: Personal guidance system for the visually impaired. In: Proceedings of the First Annual ACM Conference on Assistive Technologies, pp. 85–91 (1994)

    Google Scholar 

  43. MacDonald, J.A., Henry, P.P., Letowski, T.R.: Spatial audio through a bone conduction interface: Audición espacial a través de una interfase de conducción ósea. Int. J. Audiol. 45(10), 595–599 (2006). https://doi.org/10.1080/14992020600876519

    Article  Google Scholar 

  44. Maidenbaum, S., Abboud, S., Amedi, A.: Sensory substitution: Closing the gap between basic research and widespread practical visual rehabilitation. Neurosci. Biobehav. Rev. 41, 3–15 (2014)

    Article  Google Scholar 

  45. Microsoft: Microsoft Hololens. https://www.microsoft.com/en-us/hololens

  46. Microsoft: Seeing AI. https://www.microsoft.com/en-us/ai/seeing-ai

  47. Milan, A., Leal-Taixe, L., Reid, I., Roth, S., Schindler, K.: MOT16: a benchmark for multi-object tracking. arXiv:1603.00831 [cs], May 2016

  48. Nilsson, M.E., Schenkman, B.N.: Blind people are more sensitive than sighted people to binaural sound-location cues, particularly inter-aural level differences. Hear. Res. 332, 223–232 (2016).https://doi.org/10.1016/j.heares.2015.09.012, https://linkinghub.elsevier.com/retrieve/pii/S0378595515300174

  49. Okonji, P.E., Ogwezzy, D.C.: Awareness and barriers to adoption of assistive technologies among visually impaired people in Nigeria. Assist. Technol. 31(4), 209–219 (2019)

    Article  Google Scholar 

  50. OpenAL: Open audio library (2020). https://openal.org/. Accessed 20 July 2020

  51. Praveen, S.: Efficient depth estimation using sparse stereo-vision with other perception techniques (chap. 7). In: Radhakrishnan, S., Sarfraz, M. (eds.) Coding Theory. IntechOpen, Rijeka (2020). https://doi.org/10.5772/intechopen.86303

  52. Qiu, S., Han, T., Osawa, H., Rauterberg, M., Hu, J.: HCI design for people with visual disability in social interaction. In: Streitz, N., Konomi, S. (eds.) DAPI 2018. LNCS, vol. 10921, pp. 124–134. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91125-0_10

    Chapter  Google Scholar 

  53. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)

    Google Scholar 

  54. Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. arXiv:1804.02767 [cs], April 2018

  55. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)

    Google Scholar 

  56. Rezaei, M., Terauchi, M., Klette, R.: Robust vehicle detection and distance estimation under challenging lighting conditions. IEEE Trans. Intell. Transp. Syst. 16(5), 2723–2743 (2015)

    Article  Google Scholar 

  57. Roentgen, U.R., Gelderblom, G.J., Soede, M., De Witte, L.P.: Inventory of electronic mobility aids for persons with visual impairments: a literature review. J. Vis. Impair. Blindness 102(11), 702–724 (2008)

    Article  Google Scholar 

  58. Kanagamalliga, S., Vasuki, S.: Contour-based object tracking in video scenes through optical flow and Gabor features. Optik 157, 787–797 (2018). https://doi.org/10.1016/j.ijleo.2017.11.181

    Article  Google Scholar 

  59. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: inverted residuals and linear bottlenecks (2018)

    Google Scholar 

  60. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: inverted residuals and linear bottlenecks. arXiv:1801.04381 [cs], March 2019

  61. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 [cs], April 2015

  62. Singh, P., Deepak, B., Sethi, T., Murthy, M.D.P.: Real-time object detection and tracking using color feature and motion. In: 2015 International Conference on Communications and Signal Processing (ICCSP), pp. 1236–1241. IEEE, April 2015. https://doi.org/10.1109/ICCSP.2015.7322705

  63. Stelmack, J.: Quality of life of low-vision patients and outcomes of low-vision rehabilitation. Optom. Vis. Sci. 78(5), 335–342 (2001)

    Article  Google Scholar 

  64. Tuohy, S., O’Cualain, D., Jones, E., Glavin, M.: Distance determination for an automobile environment using inverse perspective mapping in OpenCV. In: IET Irish Signals and Systems Conference (ISSC 2010), pp. 100–105. IET (2010). https://doi.org/10.1049/cp.2010.0495

  65. Valve: Valve index. https://store.steampowered.com/valveindex

  66. WHO: Global data on visual impairment 2010 (2010). https://www.who.int/blindness/GLOBALDATAFINALforweb.pdf

  67. Zhang, R., Li, Y., Zhang, A.L., Wang, Y., Molina, M.J.: Identifying airborne transmission as the dominant route for the spread of COVID-19. Proc. Natl. Acad. Sci. 117(26), 14857–14863 (2020). https://doi.org/10.1073/pnas.2009637117

    Article  Google Scholar 

  68. Zhu, J., Fang, Y.: Learning object-specific distance from a monocular image. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3838–3847. IEEE, Seoul, Korea, October 2019. https://doi.org/10.1109/ICCV.2019.00394

Download references

Acknowledgement

The authors gratefully acknowledge the financial support from the NYUAD Institute (Research Enhancement Fund - RE132).

Author information

Authors and Affiliations

  1. NYU Multimedia and Visual Computing Lab, Abu Dhabi and New York, UAE

    Samridha Shrestha, Daohan Lu, Hanlin Tian, Qiming Cao, Julie Liu & Yi Fang

  2. New York University Abu Dhabi, Abu Dhabi, 129188, UAE

    Samridha Shrestha, Julie Liu & Yi Fang

  3. New York University, New York, NY, 10012, USA

    Daohan Lu, Hanlin Tian, Qiming Cao, John-Ross Rizzo, Maurizio Porfiri & Yi Fang

  4. Lighthouse Guild, New York, NY, 10023, USA

    William H. Seiple

Authors
  1. Samridha Shrestha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daohan Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hanlin Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qiming Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Julie Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. John-Ross Rizzo
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. William H. Seiple
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Maurizio Porfiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Yi Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yi Fang .

Editor information

Editors and Affiliations

  1. University of Clermont Auvergne, Clermont Ferrand, France

    Adrien Bartoli

  2. Università degli Studi di Udine, Udine, Italy

    Andrea Fusiello

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 600 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shrestha, S. et al. (2020). Active Crowd Analysis for Pandemic Risk Mitigation for Blind or Visually Impaired Persons. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science(), vol 12538. Springer, Cham. https://doi.org/10.1007/978-3-030-66823-5_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-66823-5_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-66822-8

  • Online ISBN: 978-3-030-66823-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation