Skip to main content
SpringerLink
Log in

MENet: A Memory-Based Network with Dual-Branch for Efficient Event Stream Processing

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13684))

Included in the following conference series:

  • 2425 Accesses

Abstract

Event cameras are bio-inspired sensors that asynchronously capture per-pixel brightness change and trigger a stream of events instead of frame-based images. Each event stream is generally split into multiple sliding windows for subsequent processing. However, most existing event-based methods ignore the motion continuity between adjacent spatiotemporal windows, which will result in the loss of dynamic information and additional computational costs. To efficiently extract strong features for event streams containing dynamic information, this paper proposes a novel memory-based network with dual-branch, namely MENet. It contains a base branch with a full-sized event point-wise processing structure to extract the base features and an incremental branch equipped with a light-weighted network to capture the temporal dynamics between two adjacent spatiotemporal windows. For enhancing the features, especially in the incremental branch, a point-wise memory bank is designed, which sketches the representative information of event feature space. Compared with the base branch, the incremental branch reduces the computational complexity up to 5 times and improves the speed by 19 times. Experiments show that MENet significantly reduces the computational complexity compared with previous methods while achieving state-of-the-art performance on gesture recognition and object recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amir, A., et al.: A low power, fully event-based gesture recognition system. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 7388–7397. IEEE Computer Society (2017). https://doi.org/10.1109/CVPR.2017.781

  2. Bardow, P., Davison, A.J., Leutenegger, S.: Simultaneous optical flow and intensity estimation from an event camera. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 884–892. IEEE Computer Society (2016). https://doi.org/10.1109/CVPR.2016.102

  3. Bi, Y., Chadha, A., Abbas, A., Bourtsoulatze, E., Andreopoulos, Y.: Graph-based spatial-temporal feature learning for neuromorphic vision sensing. CoRR abs/1910.03579 (2019). http://arxiv.org/abs/1910.03579

  4. Brandli, C., Berner, R., Yang, M., Liu, S., Delbrück, T.: A 240 \({\times }\) 180 130 db 3 \({\mu }s\) latency global shutter spatiotemporal vision sensor. IEEE J. Solid State Circuits 49(10), 2333–2341 (2014). https://doi.org/10.1109/JSSC.2014.2342715

    Article  Google Scholar 

  5. Cai, Q., Pan, Y., Yao, T., Yan, C., Mei, T.: Memory matching networks for one-shot image recognition. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018, pp. 4080–4088. Computer Vision Foundation/IEEE Computer Society (2018). https://doi.org/10.1109/CVPR.2018.00429

  6. Cannici, M., Ciccone, M., Romanoni, A., Matteucci, M.: Asynchronous convolutional networks for object detection in neuromorphic cameras. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 1656–1665. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPRW.2019.00209

  7. Cannici, M., Ciccone, M., Romanoni, A., Matteucci, M.: Attention mechanisms for object recognition with event-based cameras. In: IEEE Winter Conference on Applications of Computer Vision, WACV 2019, Waikoloa Village, HI, USA, 7–11 January 2019, pp. 1127–1136. IEEE (2019). https://doi.org/10.1109/WACV.2019.00125

  8. Cannici, M., Ciccone, M., Romanoni, A., Matteucci, M.: A differentiable recurrent surface for asynchronous event-based data. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12365, pp. 136–152. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58565-5_9

    Chapter  Google Scholar 

  9. Chen, J., Meng, J., Wang, X., Yuan, J.: Dynamic graph CNN for event-camera based gesture recognition. In: IEEE International Symposium on Circuits and Systems, ISCAS 2020, Sevilla, Spain, 10–21 October 2020, pp. 1–5. IEEE (2020). https://doi.org/10.1109/ISCAS45731.2020.9181247

  10. Cheng, W., Luo, H., Yang, W., Yu, L., Chen, S., Li, W.: DET: a high-resolution DVS dataset for lane extraction. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 1666–1675. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPRW.2019.00210

  11. Gallego, G., Rebecq, H., Scaramuzza, D.: A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018, pp. 3867–3876. Computer Vision Foundation/IEEE Computer Society (2018). https://doi.org/10.1109/CVPR.2018.00407

  12. Gehrig, D., Loquercio, A., Derpanis, K.G., Scaramuzza, D.: End-to-end learning of representations for asynchronous event-based data. In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), 27 October–2 November 2019, pp. 5632–5642. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00573

  13. Gehrig, D., Rebecq, H., Gallego, G., Scaramuzza, D.: Eklt: asynchronous photometric feature tracking using events and frames. Int. J. Comput. Vision 128, 601–618 (2019)

    Article  Google Scholar 

  14. Gong, D., Liu, L., Le, V., Saha, B., Mansour, M.R., Venkatesh, S., van den Hengel, A.: Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection. In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), 27 October–2 November 2019, pp. 1705–1714. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00179

  15. Li, H., Liu, H., Ji, X., Li, G., Shi, L.: Cifar10-dvs: an event-stream dataset for object classification. Front. Neurosci. 11, 309 (2017)

    Article  Google Scholar 

  16. He, W., et al.: Comparing snns and rnns on neuromorphic vision datasets: similarities and differences. CoRR abs/2005.02183 (2020). https://arxiv.org/abs/2005.02183

  17. Huang, H., Yu, A., He, R.: Memory oriented transfer learning for semi-supervised image deraining. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, 19–25 June 2021, pp. 7732–7741. Computer Vision Foundation/IEEE (2021)

    Google Scholar 

  18. Jack, D., Maire, F., Denman, S., Eriksson, A.: Sparse convolutions on continuous domains for point cloud and event stream networks. In: Ishikawa, H., Liu, C.-L., Pajdla, T., Shi, J. (eds.) ACCV 2020. LNCS, vol. 12622, pp. 400–416. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-69525-5_24

    Chapter  Google Scholar 

  19. Jiang, Z., Zhang, Y., Zou, D., Ren, J.S.J., Lv, J., Liu, Y.: Learning event-based motion deblurring. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 3317–3326. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00338, https://openaccess.thecvf.com/content_CVPR_2020/html/Jiang_Learning_Event-Based_Motion_Deblurring_CVPR_2020_paper.html

  20. Kaiser, L., Nachum, O., Roy, A., Bengio, S.: Learning to remember rare events. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 24–26 April 2017, Conference Track Proceedings. OpenReview.net (2017). https://openreview.net/forum?id=SJTQLdqlg

  21. Khairallah, M.Z., Bonardi, F., Roussel, D., Bouchafa, S.: PCA event-based optical flow for visual odometry. CoRR abs/2105.03760 (2021). https://arxiv.org/abs/2105.03760

  22. Khoei, M.A., Yousefzadeh, A., Pourtaherian, A., Moreira, O., Tapson, J.: Sparnet: sparse asynchronous neural network execution for energy efficient inference. In: 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems, AICAS 2020, Genova, Italy, 31 August–2 September 2020, pp. 256–260. IEEE (2020). https://doi.org/10.1109/AICAS48895.2020.9073827

  23. Kim, H., Leutenegger, S., Davison, A.J.: Real-time 3D reconstruction and 6-DoF tracking with an event camera. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 349–364. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_21

    Chapter  Google Scholar 

  24. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1412.6980

  25. Krizhevsky, A.: Learning multiple layers of features from tiny images, pp. 32–33 (2009). https://www.cs.toronto.edu/kriz/learning-features-2009-TR.pdf

  26. Kugele, A., Pfeil, T., Pfeiffer, M., Chicca, E.: Efficient processing of spatio-temporal data streams with spiking neural networks. Front. Neuroscie. 14, 439 (2020). https://doi.org/10.3389/fnins.2020.00439, https://www.frontiersin.org/article/10.3389/fnins.2020.00439

  27. Lagorce, X., Orchard, G., Galluppi, F., Shi, B.E., Benosman, R.: HOTS: a hierarchy of event-based time-surfaces for pattern recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(7), 1346–1359 (2017). https://doi.org/10.1109/TPAMI.2016.2574707

    Article  Google Scholar 

  28. Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998). https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  29. Lee, S., Kim, H.G., Choi, D.H., Kim, H., Ro, Y.M.: Video prediction recalling long-term motion context via memory alignment learning. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, 19–25 June 2021, pp. 3054–3063. Computer Vision Foundation/IEEE (2021)

    Google Scholar 

  30. Lichtsteiner, P., Posch, C., Delbrück, T.: A 128\({\times }\)128 120 db 15 \({\mu }s\) latency asynchronous temporal contrast vision sensor. IEEE J. Solid State Circuits 43(2), 566–576 (2008). https://doi.org/10.1109/JSSC.2007.914337

    Article  Google Scholar 

  31. Liu, Q., Ruan, H., Xing, D., Tang, H., Pan, G.: Effective AER object classification using segmented probability-maximization learning in spiking neural networks. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, 7–12 February 2020, pp. 1308–1315. AAAI Press (2020). https://aaai.org/ojs/index.php/AAAI/article/view/5486

  32. Manderscheid, J., Sironi, A., Bourdis, N., Migliore, D., Lepetit, V.: Speed invariant time surface for learning to detect corner points with event-based cameras. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 10245–10254. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPR.2019.01049

  33. Maqueda, A.I., Loquercio, A., Gallego, G., García, N., Scaramuzza, D.: Event-based vision meets deep learning on steering prediction for self-driving cars. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018, pp. 5419–5427. Computer Vision Foundation/IEEE Computer Society (2018). https://doi.org/10.1109/CVPR.2018.00568, http://openaccess.thecvf.com/content_cvpr_2018/html/Maqueda_Event-Based_Vision_Meets_CVPR_2018_paper.html

  34. Massa, R., Marchisio, A., Martina, M., Shafique, M.: An efficient spiking neural network for recognizing gestures with a DVS camera on the loihi neuromorphic processor. CoRR abs/2006.09985 (2020). https://arxiv.org/abs/2006.09985

  35. Messikommer, N., Gehrig, D., Loquercio, A., Scaramuzza, D.: Event-based asynchronous sparse convolutional networks. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12353, pp. 415–431. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58598-3_25

    Chapter  Google Scholar 

  36. Miller, A.H., Fisch, A., Dodge, J., Karimi, A., Bordes, A., Weston, J.: Key-value memory networks for directly reading documents. In: Su, J., Carreras, X., Duh, K. (eds.) Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, 1–4 November 2016, pp. 1400–1409. The Association for Computational Linguistics (2016). https://doi.org/10.18653/v1/d16-1147

  37. Mitrokhin, A., Hua, Z., Fermüller, C., Aloimonos, Y.: Learning visual motion segmentation using event surfaces. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 14402–14411. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.01442

  38. Mueggler, E., Bartolozzi, C., Scaramuzza, D.: Fast event-based corner detection. In: British Machine Vision Conference 2017, BMVC 2017, London, UK, 4–7 September 2017. BMVA Press (2017). https://www.dropbox.com/s/vicqrsz0yicq65c/0070.pdf?dl=1

  39. Munda, G., Reinbacher, C., Pock, T.: Real-time intensity-image reconstruction for event cameras using manifold regularisation. Int. J. Comput. Vision 126(12), 1381–1393 (2018). https://doi.org/10.1007/s11263-018-1106-2

    Article  Google Scholar 

  40. Nguyen, A., Do, T., Caldwell, D.G., Tsagarakis, N.G.: Real-time 6dof pose relocalization for event cameras with stacked spatial LSTM networks. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 1638–1645. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPRW.2019.00207

  41. Orchard, G., Benosman, R., Etienne-Cummings, R., Thakor, N.V.: A spiking neural network architecture for visual motion estimation. In: 2013 IEEE Biomedical Circuits and Systems Conference (BioCAS), Rotterdam, The Netherlands, 31 October–2 November 2013, pp. 298–301. IEEE (2013). https://doi.org/10.1109/BioCAS.2013.6679698

  42. Orchard, G., Jayawant, A., Cohen, G., Thakor, N.: Converting static image datasets to spiking neuromorphic datasets using saccades (2015)

    Google Scholar 

  43. Orchard, G., Meyer, C., Etienne-Cummings, R., Posch, C., Thakor, N.V., Benosman, R.: Hfirst: a temporal approach to object recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(10), 2028–2040 (2015). https://doi.org/10.1109/TPAMI.2015.2392947

    Article  Google Scholar 

  44. Pan, L., Liu, M., Hartley, R.: Single image optical flow estimation with an event camera. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 1669–1678. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00174

  45. Pan, L., Scheerlinck, C., Yu, X., Hartley, R., Liu, M., Dai, Y.: Bringing a blurry frame alive at high frame-rate with an event camera. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 6820–6829. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPR.2019.00698

  46. Paredes-Vallés, F., Scheper, K.Y.W., de Croon, G.C.H.E.: Unsupervised learning of a hierarchical spiking neural network for optical flow estimation: from events to global motion perception. IEEE Trans. Pattern Anal. Mach. Intell. 42(8), 2051–2064 (2020). https://doi.org/10.1109/TPAMI.2019.2903179

    Article  Google Scholar 

  47. Park, H., Noh, J., Ham, B.: Learning memory-guided normality for anomaly detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 14360–14369. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.01438, https://openaccess.thecvf.com/content_CVPR_2020/html/Park_Learning_Memory-Guided_Normality_for_Anomaly_Detection_CVPR_2020_paper.html

  48. Pei, W., Zhang, J., Wang, X., Ke, L., Shen, X., Tai, Y.: Memory-attended recurrent network for video captioning. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 8347–8356. Computer Vision Foundation / IEEE (2019). https://doi.org/10.1109/CVPR.2019.00854, http://openaccess.thecvf.com/content_CVPR_2019/html/Pei_Memory-Attended_Recurrent_Network_for_Video_Captioning_CVPR_2019_paper.html

  49. Posch, C., Matolin, D., Wohlgenannt, R.: A QVGA 143 db dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS. IEEE J. Solid State Circuits 46(1), 259–275 (2011). https://doi.org/10.1109/JSSC.2010.2085952

    Article  Google Scholar 

  50. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In: Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA, 4–9 December 2017, pp. 5099–5108 (2017)

    Google Scholar 

  51. Ramesh, B., Yang, H., Orchard, G., Thi, N.A.L., Zhang, S., Xiang, C.: DART: distribution aware retinal transform for event-based cameras. IEEE Trans. Pattern Anal. Mach. Intell. 42(11), 2767–2780 (2020). https://doi.org/10.1109/TPAMI.2019.2919301

    Article  Google Scholar 

  52. Rebecq, H., Gallego, G., Mueggler, E., Scaramuzza, D.: EMVS: event-based multi-view stereo—3D reconstruction with an event camera in real-time. Int. J. Comput. Vision 126(12), 1394–1414 (2017). https://doi.org/10.1007/s11263-017-1050-6

    Article  Google Scholar 

  53. Rebecq, H., Ranftl, R., Koltun, V., Scaramuzza, D.: Events-to-video: bringing modern computer vision to event cameras. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 3857–3866. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPR.2019.00398

  54. Rebecq, H., Ranftl, R., Koltun, V., Scaramuzza, D.: High speed and high dynamic range video with an event camera. IEEE Trans. Pattern Anal. Mach. Intell. 43(6), 1964–1980 (2021). https://doi.org/10.1109/TPAMI.2019.2963386

    Article  Google Scholar 

  55. , Sekikawa, Y., Hara, K., Saito, H.: Eventnet: asynchronous recursive event processing. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 3887–3896. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPR.2019.00401

  56. Shi, C., Li, J., Wang, Y., Luo, G.: Exploiting lightweight statistical learning for event-based vision processing. IEEE Access 6, 19396–19406 (2018). https://doi.org/10.1109/ACCESS.2018.2823260

    Article  Google Scholar 

  57. Shrestha, S.B., Orchard, G.: SLAYER: spike layer error reassignment in time. CoRR abs/1810.08646 (2018). http://arxiv.org/abs/1810.08646

  58. Sironi, A., Brambilla, M., Bourdis, N., Lagorce, X., Benosman, R.: HATS: histograms of averaged time surfaces for robust event-based object classification. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018, pp. 1731–1740. Computer Vision Foundation/IEEE Computer Society (2018). https://doi.org/10.1109/CVPR.2018.00186

  59. Wang, Q., Zhang, Y., Yuan, J., Lu, Y.: Space-time event clouds for gesture recognition: From RGB cameras to event cameras. In: IEEE Winter Conference on Applications of Computer Vision, WACV 2019, Waikoloa Village, HI, USA, 7–11 January 2019, pp. 1826–1835. IEEE (2019). https://doi.org/10.1109/WACV.2019.00199

  60. Weston, J., Chopra, S., Bordes, A.: Memory networks. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1410.3916

  61. Wu, Z., Zhang, H., Lin, Y., Li, G., Wang, M., Tang, Y.: Liaf-net: leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. CoRR abs/2011.06176 (2020). https://arxiv.org/abs/2011.06176

  62. Li, Y., Zhou, H., Yang, B.: Graph-based asynchronous event processing for rapid object recognition. In: ICCV, pp. 934–943 (2021)

    Google Scholar 

  63. Yang, J., Zhang, Q., Ni, B., Li, L., Liu, J., Zhou, M., Tian, Q.: Modeling point clouds with self-attention and gumbel subset sampling. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 3323–3332. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPR.2019.00344

  64. Yao, M., et al.: Temporal-wise attention spiking neural networks for event streams classification. CoRR abs/2107.11711 (2021). https://arxiv.org/abs/2107.11711

  65. Zheng, H., Wu, Y., Deng, L., Hu, Y., Li, G.: Going deeper with directly-trained larger spiking neural networks. CoRR abs/2011.05280 (2020). https://arxiv.org/abs/2011.05280

  66. Zhou, Y., Gallego, G., Shen, S.: Event-based stereo visual odometry. IEEE Trans. Rob. 37(5), 1433–1450 (2021). https://doi.org/10.1109/TRO.2021.3062252

    Article  Google Scholar 

  67. Zhu, A.Z., Yuan, L., Chaney, K., Daniilidis, K.: Unsupervised event-based learning of optical flow, depth, and egomotion. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 989–997. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPR.2019.00108, http://openaccess.thecvf.com/content_CVPR_2019/html/Zhu_Unsupervised_Event-Based_Learning_of_Optical_Flow_Depth_and_Egomotion_CVPR_2019_paper.html

  68. Zhu, L., Yang, Y.: Inflated episodic memory with region self-attention for long-tailed visual recognition. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 4343–4352. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00440

Download references

Acknowledgments

This work was supported in part by the National Key Research and Development Program of China under Grant 2020AAA0103402, Jiangsu Key Research and Development Plan (No. BE2021012-2), and NSFC 61876182, 61906195.

Author information

Authors and Affiliations

  1. Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Linhui Sun, Yifan Zhang, Ke Cheng, Jian Cheng & Hanqing Lu

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100049, China

    Linhui Sun, Yifan Zhang, Ke Cheng, Jian Cheng & Hanqing Lu

Authors
  1. Linhui Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yifan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ke Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jian Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hanqing Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yifan Zhang .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sun, L., Zhang, Y., Cheng, K., Cheng, J., Lu, H. (2022). MENet: A Memory-Based Network with Dual-Branch for Efficient Event Stream Processing. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13684. Springer, Cham. https://doi.org/10.1007/978-3-031-20053-3_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20053-3_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20052-6

  • Online ISBN: 978-3-031-20053-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation