Skip to main content
Springer Nature Link
Log in

BKDSNN: Enhancing the Performance of Learning-Based Spiking Neural Networks Training with Blurred Knowledge Distillation

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15108))

Included in the following conference series:

  • 437 Accesses

Abstract

Spiking neural networks (SNNs), which mimic biological neural systems to convey information via discrete spikes, are well-known as brain-inspired models with excellent computing efficiency. By utilizing the surrogate gradient estimation for discrete spikes, learning-based SNN training methods that can achieve ultra-low inference latency (i.e., number of time-step) have emerged recently. Nevertheless, due to the difficulty of deriving precise gradient for discrete spikes in learning-based methods, a distinct accuracy gap persists between SNNs and their artificial neural networks (ANNs) counterparts. To address the aforementioned issue, we propose a blurred knowledge distillation (BKD) technique, which leverages randomly blurred SNN features to restore and imitate the ANN features. Note that, our BKD is applied upon the feature map right before the last layer of SNNs, which can also mix with prior logits-based knowledge distillation for maximal accuracy boost. In the category of learning-based methods, our work achieves state-of-the-art performance for training SNNs on both static and neuromorphic datasets. On the ImageNet dataset, BKDSNN outperforms prior best results by 4.51% and 0.93% with the network topology of CNN and Transformer, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Amir, A., et al.: A low power, fully event-based gesture recognition system. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7243–7252 (2017)

    Google Scholar 

  2. Bu, T., Fang, W., Ding, J., Dai, P., Yu, Z., Huang, T.: Optimal ann-snn conversion for high-accuracy and ultra-low-latency spiking neural networks. arXiv preprint arXiv:2303.04347 (2023)

  3. Cao, Y., Chen, Y., Khosla, D.: Spiking deep convolutional neural networks for energy-efficient object recognition. Int. J. Comput. Vision 113, 54–66 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  4. Cermelli, F., Mancini, M., Bulo, S.R., Ricci, E., Caputo, B.: Modeling the background for incremental learning in semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9233–9242 (2020)

    Google Scholar 

  5. Chen, L., Yu, C., Chen, L.: A new knowledge distillation for incremental object detection. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–7. IEEE (2019)

    Google Scholar 

  6. Davies, M., et al.: Loihi: a neuromorphic manycore processor with on-chip learning. IEEE Micro 38(1), 82–99 (2018)

    Article  MATH  Google Scholar 

  7. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. Ieee (2009)

    Google Scholar 

  8. Deng, S., Gu, S.: Optimal conversion of conventional artificial neural networks to spiking neural networks. arXiv preprint arXiv:2103.00476 (2021)

  9. Deng, S., Li, Y., Zhang, S., Gu, S.: Temporal efficient training of spiking neural network via gradient re-weighting. arXiv preprint arXiv:2202.11946 (2022)

  10. Diehl, P., Neil, D., Binas, J., Cook, M., Liu, S.C., Pfeiffer, M.: Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing (July 2015). https://doi.org/10.1109/IJCNN.2015.7280696

  11. Ding, J., Bu, T., Yu, Z., Huang, T., Liu, J.: Snn-rat: robustness-enhanced spiking neural network through regularized adversarial training. Adv. Neural. Inf. Process. Syst. 35, 24780–24793 (2022)

    Google Scholar 

  12. Dosovitskiy, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  13. Fang, W., et al.: Spikingjelly: an open-source machine learning infrastructure platform for spike-based intelligence. Science Advances 9(40), eadi1480 (2023). https://doi.org/10.1126/sciadv.adi1480, https://www.science.org/doi/abs/10.1126/sciadv.adi1480

  14. Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., Tian, Y.: Deep residual learning in spiking neural networks. Adv. Neural. Inf. Process. Syst. 34, 21056–21069 (2021)

    MATH  Google Scholar 

  15. Garg, I., Chowdhury, S.S., Roy, K.: Dct-snn: using dct to distribute spatial information over time for low-latency spiking neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4671–4680 (2021)

    Google Scholar 

  16. Gerstner, W., Kistler, W.M.: Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press (2002)

    Google Scholar 

  17. Han, B., Roy, K.: Deep spiking neural network: energy efficiency through time based coding. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12355, pp. 388–404. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58607-2_23

    Chapter  MATH  Google Scholar 

  18. Han, B., Srinivasan, G., Roy, K.: Rmp-snn: residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13558–13567 (2020)

    Google Scholar 

  19. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  20. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

  21. Hong, D., Shen, J., Qi, Y., Wang, Y.: Lasnn: Layer-wise ann-to-snn distillation for effective and efficient training in deep spiking neural networks. arXiv preprint arXiv:2304.09101 (2023)

  22. Hongmin, L., Hanchao, L., Xiangyang, J., Guoqi, L., Luping, S.: Cifar10-dvs: An event-stream dataset for object classification. Front. Neurosci. 11 (2017)

    Google Scholar 

  23. Hu, Y., Tang, H., Pan, G.: Spiking deep residual network (2020)

    Google Scholar 

  24. Hu, Y., Zheng, Q., Jiang, X., Pan, G.: Fast-snn: Fast spiking neural network by converting quantized ann. arXiv preprint arXiv:2305.19868 (2023)

  25. Hu, Y., Deng, L., Wu, Y., Yao, M., Li, G.: Advancing spiking neural networks towards deep residual learning (2023)

    Google Scholar 

  26. Khan, A., Sohail, A., Zahoora, U., Qureshi, A.S.: A survey of the recent architectures of deep convolutional neural networks. Artif. Intell. Rev. 53, 5455–5516 (2020)

    Article  MATH  Google Scholar 

  27. Kheradpisheh, S.R., Ganjtabesh, M., Thorpe, S.J., Masquelier, T.: Stdp-based spiking deep convolutional neural networks for object recognition. Neural Netw. 99, 56–67 (2018)

    Article  MATH  Google Scholar 

  28. Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

    Google Scholar 

  29. Kundu, S., Pedram, M., Beerel, P.A.: Hire-snn: Harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise (2021)

    Google Scholar 

  30. Kushawaha, R.K., Kumar, S., Banerjee, B., Velmurugan, R.: Distilling spikes: knowledge distillation in spiking neural networks. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 4536–4543. IEEE (2021)

    Google Scholar 

  31. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  MATH  Google Scholar 

  32. Lee, D., Park, S., Kim, J., Doh, W., Yoon, S.: Energy-efficient knowledge distillation for spiking neural networks. arXiv preprint arXiv:2106.07172 (2021)

  33. Li, C., Ma, L., Furber, S.: Quantization framework for fast spiking neural networks. Front. Neurosci. 16, 918793 (2022)

    Article  MATH  Google Scholar 

  34. Li, Y., Deng, S., Dong, X., Gong, R., Gu, S.: A free lunch from ann: towards efficient, accurate spiking neural networks calibration. In: International Conference on Machine Learning, pp. 6316–6325. PMLR (2021)

    Google Scholar 

  35. Liu, F., Zhao, W., Chen, Y., Wang, Z., Yang, T., Li, J.: Sstdp: supervised spike timing dependent plasticity for efficient spiking neural network training. Front. Neurosci., 1413 (2021)

    Google Scholar 

  36. Liu, Y., Chen, K., Liu, C., Qin, Z., Luo, Z., Wang, J.: Structured knowledge distillation for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2604–2613 (2019)

    Google Scholar 

  37. Maass, W.: Networks of spiking neurons: the third generation of neural network models. Neural Netw. 10(9), 1659–1671 (1997)

    Article  MATH  Google Scholar 

  38. Merolla, P.A., et al.: A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345(6197), 668–673 (2014)

    Article  MATH  Google Scholar 

  39. Neftci, E.O., Mostafa, H., Zenke, F.: Surrogate gradient learning in spiking neural networks: bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Process. Mag. 36(6), 51–63 (2019)

    Article  MATH  Google Scholar 

  40. Orchard, G., Jayawant, A., Cohen, G.K., Thakor, N.: Converting static image datasets to spiking neuromorphic datasets using saccades. Front. Neurosci. 9, 437 (2015)

    Article  Google Scholar 

  41. Ostojic, S.: Two types of asynchronous activity in networks of excitatory and inhibitory spiking neurons. Nat. Neurosci. 17(4), 594–600 (2014)

    Article  MATH  Google Scholar 

  42. Peng, C., Zhao, K., Maksoud, S., Li, M., Lovell, B.C.: Sid: Incremental learning for anchor-free object detection via selective and inter-related distillation. Comput. Vis. Image Underst. 210, 103229 (2021)

    Article  Google Scholar 

  43. Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12179–12188 (2021)

    Google Scholar 

  44. Roy, K., Jaiswal, A., Panda, P.: Towards spike-based machine intelligence with neuromorphic computing. Nature 575(7784), 607–617 (2019)

    Article  MATH  Google Scholar 

  45. Rueckauer, B., Lungu, I.A., Hu, Y., Pfeiffer, M., Liu, S.C.: Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Front. Neurosci. 11, 682 (2017)

    Article  MATH  Google Scholar 

  46. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. Inter. J. Comput. Vis. 128(2), 336–359 (Oct 2019). https://doi.org/10.1007/s11263-019-01228-7

  47. Sengupta, A., Ye, Y., Wang, R., Liu, C., Roy, K.: Going deeper in spiking neural networks: Vgg and residual architectures. Front. Neurosci. 13, 95 (2019)

    Article  Google Scholar 

  48. Sironi, A., Brambilla, M., Bourdis, N., Lagorce, X., Benosman, R.: Hats: histograms of averaged time surfaces for robust event-based object classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1731–1740 (2018)

    Google Scholar 

  49. Sun, S., Cheng, Y., Gan, Z., Liu, J.: Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355 (2019)

  50. Takuya, S., Zhang, R., Nakashima, Y.: Training low-latency spiking neural network through knowledge distillation. In: 2021 IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS), pp. 1–3. IEEE (2021)

    Google Scholar 

  51. Tan, G., Wang, Y., Han, H., Cao, Y., Wu, F., Zha, Z.J.: Multi-grained spatio-temporal features perceived network for event-based lip-reading. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20094–20103 (2022)

    Google Scholar 

  52. Wu, Y., Deng, L., Li, G., Zhu, J., Xie, Y., Shi, L.: Direct training for spiking neural networks: Faster, larger, better. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1311–1318 (2019)

    Google Scholar 

  53. Xu, Q., Li, Y., Shen, J., Liu, J.K., Tang, H., Pan, G.: Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7886–7895 (2023)

    Google Scholar 

  54. Xu, Q., et al.: Hierarchical spiking-based model for efficient image classification with enhanced feature extraction and encoding. IEEE Trans. Neural Netw. Learn. Syst. (2022)

    Google Scholar 

  55. Xu, Z., et al.: Delving into transformer for incremental semantic segmentation (2022)

    Google Scholar 

  56. Yang, Z., et al.: Focal and global knowledge distillation for detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4643–4652 (2022)

    Google Scholar 

  57. Yao, M., et al.: Spike-driven transformer v2: meta spiking neural network architecture inspiring the design of next-generation neuromorphic chips. In: The Twelfth International Conference on Learning Representations (2023)

    Google Scholar 

  58. Zenke, F., Agnes, E.J., Gerstner, W.: Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks. Nat. Commun. 6(1), 6922 (2015)

    Article  MATH  Google Scholar 

  59. Zenke, F., Vogels, T.P.: The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Comput. 33(4), 899–925 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  60. Zheng, H., Wu, Y., Deng, L., Hu, Y., Li, G.: Going deeper with directly-trained larger spiking neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 11062–11070 (2021)

    Google Scholar 

  61. Zhou, C., et al.: Spikingformer: Spike-driven residual learning for transformer-based spiking neural network. arXiv preprint arXiv:2304.11954 (2023)

  62. Zhou, C., et al.: Enhancing the performance of transformer-based spiking neural networks by improved downsampling with precise gradient backpropagation. arXiv preprint arXiv:2305.05954 (2023)

  63. Zhou, D., et al.: Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886 (2021)

  64. Zhou, H., et al.: Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint arXiv:2102.00650 (2021)

  65. Zhou, Z., et al.: Spikformer: When spiking neural network meets transformer (2022)

    Google Scholar 

Download references

Acknowledgements

This work is partially supported by National Key R&D Program of China (2022YFB4500200), National Natural Science Foundation of China (No. 62102257), Biren Technology-Shanghai Jiao Tong University Joint Laboratory Open Research Fund, Microsoft Research Asia Gift Fund, Lingang Laboratory Open Research Fund (No.LGQS-202202-11).

Author information

Authors and Affiliations

  1. School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China

    Zekai Xu, Kang You & Zhezhi He

  2. Huawei Technologies, Shenzhen, China

    Qinghai Guo & Xiang Wang

Authors
  1. Zekai Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kang You
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qinghai Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhezhi He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhezhi He .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1068 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xu, Z., You, K., Guo, Q., Wang, X., He, Z. (2025). BKDSNN: Enhancing the Performance of Learning-Based Spiking Neural Networks Training with Blurred Knowledge Distillation. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15108. Springer, Cham. https://doi.org/10.1007/978-3-031-72973-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72973-7_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72972-0

  • Online ISBN: 978-3-031-72973-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics