Skip to main content
Springer Nature Link
Log in

Neuromorphic sensory computing

  • Review
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

The number of sensory nodes in the Internet of everything continues to increase rapidly and generate massive data. The generated information from sensory nodes is much larger than the total collective human sensory throughput. It is quite challenging to send all of the data produced at sensory terminals to the “Cloud” computation center, especially for those time-delay sensitive applications. This situation demands a dramatic increase in the computation near or inside sensory networks. Inspired by biological sensory systems with a high data compression ratio, neuromorphic sensory computing provides a way to efficiently acquire and process a large volume of data from complex environments. Researchers have been investigating emerging materials, devices, circuits, and computing architectures to implement an artificial sensory system with high energy efficiency, speed, and density. Here we summarize the important features of biological systems and their hardware implementations. Electrons and photons are two representative information carriers, in which electron carrier allows high integration density for complex computing and photon carrier has high connectivity, high speed, wide bandwidth, and low power consumption. We overview the electronic and optical neuromorphic sensory computing and hybrid opto-electronic sensory computing, and present advances on multimodal sensory computing and their potential challenges.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ji X L, Zhao X Y, Tan M C, et al. Artificial perception built on memristive system: visual, auditory, and tactile sensations. Adv Intell Syst, 2020, 2: 1900118

    Article  Google Scholar 

  2. Wan C L, Cai P Q, Wang M, et al. Artificial sensory memory. Adv Mater, 2020, 32: 1902434

    Article  Google Scholar 

  3. Chai Y. In-sensor computing for machine vision. Nature, 2020, 579: 32–33

    Article  Google Scholar 

  4. Zhou F C, Chai Y. Near-sensor and in-sensor computing. Nat Electron, 2020, 3: 664–671

    Article  Google Scholar 

  5. Liao F Y, Zhou F C, Chai Y. Neuromorphic vision sensors: principle, progress and perspectives. J Semicond, 2021, 42: 013105

    Article  Google Scholar 

  6. Cottini N, Gottardi M, Massari N, et al. A 33 µW 64×64 pixel vision sensor embedding robust dynamic background subtraction for event detection and scene interpretation. IEEE J Solid-State Circ, 2013, 48: 850–863

    Article  Google Scholar 

  7. Lichtsteiner P, Posch C, Delbruck T. A 128×128 120 dB 15 µs latency asynchronous temporal contrast vision sensor. IEEE J Solid-State Circ, 2008, 43: 566–576

    Article  Google Scholar 

  8. Moini A. Vision Chips2000. Berlin: Springer, 2000

    Book  Google Scholar 

  9. Ruedi P F, Heim P, Kaess F, et al. A 128×128 pixel 120-dB dynamic-range vision-sensor chip for image contrast and orientation extraction. IEEE J Solid-State Circ, 2003, 38: 2325–2333

    Article  Google Scholar 

  10. Dittman J S, Kreitzer A C, Regehr W G. Interplay between facilitation, depression, and residual calcium at three presynaptic terminals. J Neurosci, 2000, 20: 1374–1385

    Article  Google Scholar 

  11. Abbott L F, Varela J A, Sen K, et al. Synaptic depression and cortical gain control. Science, 1997, 275: 221–224

    Article  Google Scholar 

  12. Rothman J S, Cathala L, Steuber V, et al. Synaptic depression enables neuronal gain control. Nature, 2009, 457: 1015–1018

    Article  Google Scholar 

  13. Shastri B J, Tait A N, de Lima T F, et al. Photonics for artificial intelligence and neuromorphic computing. Nat Photonics, 2021, 15: 102–114

    Article  Google Scholar 

  14. Chen D G, Matolin D, Bermak A, et al. Pulse-modulation imaging-review and performance analysis. IEEE Trans Biomed Circ Syst, 2011, 5: 64–82

    Article  Google Scholar 

  15. Posch C, Serrano-Gotarredona T, Linares-Barranco B, et al. Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc IEEE, 2014, 102: 1470–1484

    Article  Google Scholar 

  16. Finateu T, Niwa A, Matolin D, et al. A 1280×720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86 µm pixels, 1.066GEPS readout, programmable event-rate controller and compressive data-formatting pipeline. In: Proceedings of IEEE International Solid-State Circuits Conference, San Francisco, 2020

  17. Hasler P, Smith P D, Graham D, et al. Analog floating-gate, on-chip auditory sensing system interfaces. IEEE Sens J, 2005, 5: 1027–1034

    Article  Google Scholar 

  18. Hsu T H, Chen Y K, Wu J S, et al. A 0.8 V multimode vision sensor for motion and saliency detection with ping-pong PWM pixel. In: Proceedings of IEEE International Solid-State Circuits Conference, San Francisco, 2020

  19. Jimenez-Fernandez A, Cerezuela-Escudero E, Miro-Amarante L, et al. A binaural neuromorphic auditory sensor for FPGA: a spike signal processing approach. IEEE Trans Neural Netw Learn Syst, 2017, 28: 804–818

    Article  Google Scholar 

  20. Kyuma K, Lange E, Ohta J, et al. Artificial retinas — fast, versatile image processors. Nature, 1994, 372: 197–198

    Article  Google Scholar 

  21. Lichtsteiner P, Posch C, Delbruck T. A 128×128 120 dB 30 mW asynchronous vision sensor that responds to relative intensity change. In: Proceedings of IEEE International Solid-State Circuits Conference, San Francisco, 2006

  22. Lyon R F, Mead C. An analog electronic cochlea. IEEE Trans Acoust Speech Signal Process, 1988, 36: 1119–1134

    Article  MATH  Google Scholar 

  23. Song Y M, Xie Y, Malyarchuk V, et al. Digital cameras with designs inspired by the arthropod eye. Nature, 2013, 497: 95–99

    Article  Google Scholar 

  24. Wen B, Boahen K. A 360-channel speech preprocessor that emulates the cochlear amplifier. In: Proceedings of IEEE International Solid State Circuits Conference, San Francisco, 2006

  25. Wang Z R, Joshi S, Savel’ev S E, et al. Memristors with diffusive dynamics as synaptic emulators for neuromorphic computing. Nat Mater, 2017, 16: 101–108

    Article  Google Scholar 

  26. Ling H F, Koutsouras D A, Kazemzadeh S, et al. Electrolyte-gated transistors for synaptic electronics, neuromorphic computing, and adaptable biointerfacing. Appl Phys Rev, 2020, 7: 011307

    Article  Google Scholar 

  27. Zang Y P, Shen H G, Huang D Z, et al. A dual-organic-transistor-based tactile-perception system with signal-processing functionality. Adv Mater, 2017, 29: 1606088

    Article  Google Scholar 

  28. Yao P, Wu H Q, Gao B, et al. Fully hardware-implemented memristor convolutional neural network. Nature, 2020, 577: 641–646

    Article  Google Scholar 

  29. Wan C J, Cai P Q, Guo X T, et al. An artificial sensory neuron with visual-haptic fusion. Nat Commun, 2020, 11: 4602

    Article  Google Scholar 

  30. He Y L, Nie S, Liu R, et al. Spatiotemporal information processing emulated by multiterminal neuro-transistor networks. Adv Mater, 2019, 31: 1900903

    Article  Google Scholar 

  31. Das S, Dodda A, Das S. A biomimetic 2D transistor for audiomorphic computing. Nat Commun, 2019, 10: 3450

    Article  Google Scholar 

  32. Sheridan P M, Cai F X, Du C, et al. Sparse coding with memristor networks. Nat Nanotech, 2017, 12: 784–789

    Article  Google Scholar 

  33. Li C, Belkin D, Li Y N, et al. Efficient and self-adaptive in-situ learning in multilayer memristor neural networks. Nat Commun, 2018, 9: 2385

    Article  Google Scholar 

  34. Prezioso M, Merrikh-Bayat F, Hoskins B D, et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature, 2015, 521: 61–64

    Article  Google Scholar 

  35. Wang Z R, Joshi S, Savel’ev S, et al. Fully memristive neural networks for pattern classification with unsupervised learning. Nat Electron, 2018, 1: 137–145

    Article  Google Scholar 

  36. Serb A, Bill J, Khiat A, et al. Unsupervised learning in probabilistic neural networks with multi-state metal-oxide memristive synapses. Nat Commun, 2016, 7: 12611

    Article  Google Scholar 

  37. Eryilmaz S B, Kuzum D, Jeyasingh R, et al. Brain-like associative learning using a nanoscale non-volatile phase change synaptic device array. Front Neurosci, 2014, 8: 205

    Article  Google Scholar 

  38. Hu S G, Liu Y, Liu Z, et al. Associative memory realized by a reconfigurable memristive Hopfield neural network. Nat Commun, 2015, 6: 7522

    Article  Google Scholar 

  39. Pershin Y V, Di Ventra M. Experimental demonstration of associative memory with memristive neural networks. Neural Netw, 2010, 23: 881–886

    Article  Google Scholar 

  40. Moon J, Ma W, Shin J H, et al. Temporal data classification and forecasting using a memristor-based reservoir computing system. Nat Electron, 2019, 2: 480–487

    Article  Google Scholar 

  41. Marinella M J, Agarwal S. Efficient reservoir computing with memristors. Nat Electron, 2019, 2: 437–438

    Article  Google Scholar 

  42. Zhong Y N, Tang J S, Li X Y, et al. Dynamic memristor-based reservoir computing for high-efficiency temporal signal processing. Nat Commun, 2021, 12: 408

    Article  Google Scholar 

  43. Xu R Q, Lv P, Xu F J, et al. A survey of approaches for implementing optical neural networks. Opt Laser Tech, 2021, 136: 106787

    Article  Google Scholar 

  44. Chen H G, Jayasuriya S, Yang J Y, et al. ASP vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 27–30

  45. Hirsch M, Sivaramakrishnan S, Jayasuriya S, et al. A switchable light field camera architecture with angle sensitive pixels and dictionary-based sparse coding. In: Proceedings of IEEE International Conference on Computational Photography (ICCP), 2014

  46. Wang A, Sivaramakrishnan S, Molnar A. A 180 nm CMOS image sensor with on-chip optoelectronic image compression. In: Proceedings of IEEE Custom Integrated Circuits Conference, 2012

  47. Weaver C S, Goodman J W. A technique for optically convolving two functions. Appl Opt, 1966, 5: 1248

    Article  Google Scholar 

  48. Chang J L, Sitzmann V, Dun X, et al. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci Rep, 2018, 8: 12324

    Article  Google Scholar 

  49. LeCun Y, Bengio Y. Convolutional networks for images, speech, and time series. In: Handbook of Brain Theory and Neural Networks. Cambridge: MIT Press, 1995

    Google Scholar 

  50. Shen Y C, Harris N C, Skirlo S, et al. Deep learning with coherent nanophotonic circuits. Nat Photon, 2017, 11: 441–446

    Article  Google Scholar 

  51. Bueno S, Salmeron J L. Benchmarking main activation functions in fuzzy cognitive maps. Expert Syst Appl, 2009, 36: 5221–5229

    Article  Google Scholar 

  52. Goodman J W. Fan-in and fan-out with optical interconnections. Opt Acta-Int J Opt, 1985, 32: 1489–1496

    Article  MathSciNet  Google Scholar 

  53. Hill M T, Frietman E E E, de Waardt H, et al. All fiber-optic neural network using coupled SOA based ring lasers. IEEE Trans Neural Netw, 2002, 13: 1504–1513

    Article  Google Scholar 

  54. Vandoorne K, Dierckx W, Schrauwen B, et al. Toward optical signal processing using photonic reservoir computing. Opt Express, 2008, 16: 11182

    Article  Google Scholar 

  55. Mesaritakis C, Papataxiarhis V, Syvridis D. Micro ring resonators as building blocks for an all-optical high-speed reservoircomputing bit-pattern-recognition system. J Opt Soc Am B, 2013, 30: 3048

    Article  Google Scholar 

  56. Rosenbluth D, Kravtsov K, Fok M P, et al. A high performance photonic pulse processing device. Opt Express, 2009, 17: 22767

    Article  Google Scholar 

  57. Yan T, Wu J M, Zhou T K, et al. Fourier-space diffractive deep neural network. Phys Rev Lett, 2019, 123: 023901

    Article  Google Scholar 

  58. Zuo Y, Li B H, Zhao Y J, et al. All-optical neural network with nonlinear activation functions. Optica, 2019, 6: 1132

    Article  Google Scholar 

  59. Chakraborty I, Saha G, Sengupta A, et al. Toward fast neural computing using all-photonic phase change spiking neurons. Sci Rep, 2018, 8: 12980

    Article  Google Scholar 

  60. Feldmann J, Youngblood N, Wright C D, et al. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature, 2019, 569: 208–214

    Article  Google Scholar 

  61. Khoram E, Chen A, Liu D J, et al. Nanophotonic media for artificial neural inference. Photon Res, 2019, 7: 823

    Article  Google Scholar 

  62. Zhou F C, Zhou Z, Chen J W, et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat Nanotechnol, 2019, 14: 776–782

    Article  Google Scholar 

  63. Guo Y B, Zhu L Q. Recent progress in optoelectronic neuromorphic devices. Chin Phys B, 2020, 29: 078502

    Article  Google Scholar 

  64. Zhu Q B, Li B, Yang D D, et al. A flexible ultrasensitive optoelectronic sensor array for neuromorphic vision systems. Nat Commun, 2021, 12: 1798

    Article  Google Scholar 

  65. Islam M M, Dev D, Krishnaprasad A, et al. Optoelectronic synapse using monolayer MoS2 field effect transistors. Sci Rep, 2020, 10: 21870

    Article  Google Scholar 

  66. Choi C, Leem J, Kim M S, et al. Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system. Nat Commun, 2020, 11: 5934

    Article  Google Scholar 

  67. Seo S, Jo S H, Kim S, et al. Artificial optic-neural synapse for colored and color-mixed pattern recognition. Nat Commun, 2018, 9: 5106

    Article  Google Scholar 

  68. Stein B E, Stanford T R. Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci, 2008, 9: 255–266

    Article  Google Scholar 

  69. Holmes N P. The law of inverse effectiveness in neurons and behaviour: multisensory integration versus normal variability. Neuropsychologia, 2007, 45: 3340–3345

    Article  Google Scholar 

  70. Colonius H, Diederich A. Multisensory interaction in saccadic reaction time: a time-window-of-integration model. J Cogn Neurosci, 2004, 16: 1000–1009

    Article  Google Scholar 

  71. Ohshiro T, Angelaki D E, DeAngelis G C. A normalization model of multisensory integration. Nat Neurosci, 2011, 14: 775–782

    Article  Google Scholar 

  72. Fetsch C R, DeAngelis G C, Angelaki D E. Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons. Nat Rev Neurosci, 2013, 14: 429–442

    Article  Google Scholar 

  73. Zhang J Y, Xue Y Y, Sun Q Y, et al. A miniaturized electronic nose with artificial neural network for anti-interference detection of mixed indoor hazardous gases. Sens Actuat B-Chem, 2021, 326: 128822

    Article  Google Scholar 

  74. Hua Q L, Sun J L, Liu H T, et al. Skin-inspired highly stretchable and conformable matrix networks for multifunctional sensing. Nat Commun, 2018, 9: 244

    Article  Google Scholar 

  75. Lu Y Y, Xu K C, Zhang L S, et al. Multimodal plant healthcare flexible sensor system. ACS Nano, 2020, 14: 10966–10975

    Article  Google Scholar 

  76. You I, Mackanic D G, Matsuhisa N, et al. Artificial multimodal receptors based on ion relaxation dynamics. Science, 2020, 370: 961–965

    Article  Google Scholar 

  77. Yu J R, Yang X X, Gao G Y, et al. Bioinspired mechano-photonic artificial synapse based on graphene/MoS2 heterostructure. Sci Adv, 2021, 7: 9117

    Article  Google Scholar 

  78. Wu X M, Li E L, Liu Y Q, et al. Artificial multisensory integration nervous system with haptic and iconic perception behaviors. Nano Energy, 2021, 85: 106000

    Article  Google Scholar 

  79. Imam N, Cleland T A. Rapid online learning and robust recall in a neuromorphic olfactory circuit. Nat Mach Intell, 2020, 2: 181–191

    Article  Google Scholar 

  80. Pei J, Deng L, Song S, et al. Towards artificial general intelligence with hybrid Tianjic chip architecture. Nature, 2019, 572: 106–111

    Article  Google Scholar 

  81. Koscielniak W C, Pelouard J L, Littlejohn M A. Dynamic behavior of photocarriers in a GaAs metal-semiconductor-metal photodetector with sub-half-micron electrode pattern. Appl Phys Lett, 1989, 54: 567–569

    Article  Google Scholar 

  82. Beling A, Campbell J C. InP-based high-speed photodetectors. J Lightwave Technol, 2009, 27: 343–355

    Article  Google Scholar 

  83. Mueller T, Xia F N, Avouris P. Graphene photodetectors for high-speed optical communications. Nat Photon, 2010, 4: 297–301

    Article  Google Scholar 

  84. Urich A, Unterrainer K, Mueller T. Intrinsic response time of graphene photodetectors. Nano Lett, 2011, 11: 2804–2808

    Article  Google Scholar 

  85. Xia F N, Mueller T, Lin Y M, et al. Ultrafast graphene photodetector. Nat Nanotech, 2009, 4: 839–843

    Article  Google Scholar 

  86. Youngblood N, Li M. Ultrafast photocurrent measurements of a black phosphorus photodetector. Appl Phys Lett, 2017, 110: 051102

    Article  Google Scholar 

  87. Park H L, Kim H, Lim D, et al. Retina-inspired carbon nitride-based photonic synapses for selective detection of UV light. Adv Mater, 2020, 32: 1906899

    Article  Google Scholar 

  88. Kim S G, Kim S H, Park J, et al. Infrared detectable MoS2 phototransistor and its application to artificial multilevel optic-neural synapse. ACS Nano, 2019, 13: 10294–10300

    Article  Google Scholar 

  89. Qian C, Choi Y, Choi Y J, et al. Oxygen-detecting synaptic device for realization of artificial autonomic nervous system for maintaining oxygen homeostasis. Adv Mater, 2020, 32: 2002653

    Article  Google Scholar 

  90. Yang J, Chen J, Su Y J, et al. Eardrum-inspired active sensors for self-powered cardiovascular system characterization and throat-attached anti-interference voice recognition. Adv Mater, 2015, 27: 1316–1326

    Article  Google Scholar 

  91. Zhu B W, Wang H, Liu Y Q, et al. Skin-inspired haptic memory arrays with an electrically reconfigurable architecture. Adv Mater, 2016, 28: 1559–1566

    Article  Google Scholar 

  92. Ledda P, Santos L P, Chalmers A. A local model of eye adaptation for high dynamic range images. In: Proceedings of the 3rd International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa, 2004. 151–160

  93. Pattanaik S N, Ferwerda J A, Fairchild M D, et al. A multiscale model of adaptation and spatial vision for realistic image display. In: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, 1998. 287–298

Download references

Acknowledgements

This work was supported by Research Grant Council of Hong Kong (Grant No. PolyU 15205619), Science, Technology and Innovation Commission of Shenzhen (Grant No. JCYJ20180507183424383), and Hong Kong Polytechnic University (Grant No. 1-ZE1T).

Author information

Authors and Affiliations

  1. Department of Applied Physics, The Hong Kong Polytechnic University, Hong Kong, China

    Tianqing Wan, Sijie Ma, Fuyou Liao, Lingwei Fan & Yang Chai

  2. The Hong Kong Polytechnic University Shenzhen Research Institute, Shenzhen, 515100, China

    Fuyou Liao & Yang Chai

Authors
  1. Tianqing Wan
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Sijie Ma
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Fuyou Liao
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Lingwei Fan
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Yang Chai
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Yang Chai.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wan, T., Ma, S., Liao, F. et al. Neuromorphic sensory computing. Sci. China Inf. Sci. 65, 141401 (2022). https://doi.org/10.1007/s11432-021-3336-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-021-3336-8

Keywords