Skip to main content
Springer Nature Link
Log in

The Analog Layer: Simulating Imperfect Computations in Neural Networks to Improve Robustness and Generalization Ability

  • Conference paper
  • First Online:
Pattern Recognition (ICPR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15326))

Included in the following conference series:

  • 182 Accesses

Abstract

Usage of noise is common at the input level of neural networks as a means of data augmentation. This study examines the impact of incorporating stochastic noise deeply into the activation signals between layers of neural networks, simulating analog circuit computation. We introduce the “Analog Layer” model, which embeds inherent stochasticity in the computation of activations and develop an algorithm to dynamically adjust noise levels during training, thus creating a noisy yet controlled curriculum learning training environment. We evaluate our approach on Fully Connected and Convolutional Networks using the MNIST, FashionMNIST, CIFAR10, and CIFAR100 datasets. The proposed framework is assessed considering accuracy, robustness to input and state perturbations, resistance to FSGM adversarial attacks and feature map entropy. We show that our method can improve the network’s base accuracy, as well as its resilience to input and state perturbations and adversarial attacks. The proposed approach allows to compute representations which have a lower distribution entropy across its neurons, allowing to achieve improved robustness. We finally give an interpretation of the proposed technique as both a regularization method and a consensus mechanism.

G. M. Manduca—This research has been supported by the project Future Artificial Intelligence Research (FAIR) – PNRR MUR Cod. PE0000013 - CUP: E63C22001940006.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Jastrzȩbski, S., Kenton, Z., Arpit, D., Ballas, N., Fischer, A., Bengio, Y., Storkey, A. (2017). Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623

  2. Smith, L. N. (2017, March). Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV) (pp. 464-472). IEEE

    Google Scholar 

  3. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R. (2013). Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199

  4. Xu, H., Ma, Y., Liu, H.C., Deb, D., Liu, H., Tang, J.L., Jain, A.K.: Adversarial attacks and defenses in images, graphs and text: A review. Int. J. Autom. Comput. 17, 151–178 (2020)

    Article  MATH  Google Scholar 

  5. Goodfellow, I. J., Shlens, J., Szegedy, C. (2015). Explaining and Harnessing Adversarial Examples. arXiv preprint arXiv:1412.6572

  6. Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R. R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580

  7. Krizhevsky, A., Sutskever, I., Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems (pp. 1097-1105)

    Google Scholar 

  8. Yang, S., Xiao, W., Zhang, M., Guo, S., Zhao, J., Shen, F. (2022). Image data augmentation for deep learning: A survey. arXiv preprint arXiv:2204.08610

  9. Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. Journal of big data 6(1), 1–48 (2019)

    Article  MATH  Google Scholar 

  10. Antoniou, A., Storkey, A., Edwards, H. (2017). Data augmentation generative adversarial networks. arXiv preprint arXiv:1711.04340

  11. Chlap, P., Min, H., Vandenberg, N., Dowling, J., Holloway, L., Haworth, A.: A review of medical image data augmentation techniques for deep learning applications. J. Med. Imaging Radiat. Oncol. 65(5), 545–563 (2021)

    Article  Google Scholar 

  12. Chen, H., Cao, P. (2019, July). Deep learning based data augmentation and classification for limited medical data learning. In 2019 IEEE international conference on power, intelligent computing and systems (ICPICS) (pp. 300-303). IEEE

    Google Scholar 

  13. Loshchilov, I., Hutter, F. (2016). Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983

  14. Smith, S. L., Kindermans, P. J., Ying, C., Le, Q. V. (2017). Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489

  15. Ba, J., Frey, B. (2013). Adaptive dropout for training deep neural networks. Advances in neural information processing systems, 26

    Google Scholar 

  16. Li, Z., Gong, B., Yang, T. (2016). Improved dropout for shallow and deep learning. Advances in neural information processing systems, 29

    Google Scholar 

  17. Achille, A., Soatto, S.: Information dropout: Learning optimal representations through noisy computation. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2897–2905 (2018)

    Article  MATH  Google Scholar 

  18. Kingma, D. P., Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

  19. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818-2826)

    Google Scholar 

  20. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P. A. (2008, July). Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning (pp. 1096-1103)

    Google Scholar 

  21. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S. (2015, June). Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning (pp. 2256-2265). PMLR

    Google Scholar 

  22. Huang, G., Sun, Yu., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep Networks with Stochastic Depth. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 646–661. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_39

    Chapter  Google Scholar 

  23. Han, B., Sim, J., Adam, H. (2017). Branchout: Regularization for online ensemble tracking with convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3356-3365)

    Google Scholar 

  24. Noh, H., You, T., Mun, J., Han, B. (2017). Regularizing deep neural networks by noise: Its interpretation and optimization. Advances in neural information processing systems, 30

    Google Scholar 

  25. French, R.M.: Catastrophic forgetting in connectionist networks. Trends Cogn. Sci. 3(4), 128–135 (1999)

    Article  MATH  Google Scholar 

  26. Eddy, W., Ed., “Transmission Control Protocol (TCP)”, STD 7, RFC 9293,https://doi.org/10.17487/RFC9293, August 2022, <https://www.rfc-editor.org/info/rfc9293>

  27. Lecun, Y., Cortes, C., Burges, C. J. (1998). Mnist. The MNIST Database of handwritten digits

    Google Scholar 

  28. Xiao, H., Rasul, K., Vollgraf, R. (2017). Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747

  29. Krizhevsky, A., Hinton, G. (2009). Learning multiple layers of features from tiny images

    Google Scholar 

  30. Carlini, N., Wagner, D. (2017, May). Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) (pp. 39-57). Ieee

    Google Scholar 

  31. Athalye, A., Carlini, N., Wagner, D. (2018, July). Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning (pp. 274-283). PMLR

    Google Scholar 

  32. Dhillon, G. S., Azizzadenesheli, K., Lipton, Z. C., Bernstein, J., Kossaifi, J., Khanna, A., Anandkumar, A. (2018). Stochastic activation pruning for robust adversarial defense. arXiv preprint arXiv:1803.01442

  33. Moosavi-Dezfooli, S. M., Fawzi, A., Frossard, P. (2016). Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2574-2582)

    Google Scholar 

  34. Bengio, Y., Louradour, J., Collobert, R., Weston, J. (2009, June). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning (pp. 41-48)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics and Computer Science, University of Catania, Catania, Italy

    Giovanni Maria Manduca, Antonino Furnari & Giovanni Maria Farinella

Authors
  1. Giovanni Maria Manduca
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonino Furnari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Giovanni Maria Farinella
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Antonino Furnari .

Editor information

Editors and Affiliations

  1. University of Salford, Salford, Lancashire, UK

    Apostolos Antonacopoulos

  2. Indian Institute of Technology Bombay, Mumbai, Maharashtra, India

    Subhasis Chaudhuri

  3. Johns Hopkins University, Baltimore, MD, USA

    Rama Chellappa

  4. Chinese Academy of Sciences, Beijing, China

    Cheng-Lin Liu

  5. IIT Kharagpur, Kharagpur, West Bengal, India

    Saumik Bhattacharya

  6. Indian Statistical Institute, kolkata, West Bengal, India

    Umapada Pal

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Manduca, G.M., Furnari, A., Farinella, G.M. (2025). The Analog Layer: Simulating Imperfect Computations in Neural Networks to Improve Robustness and Generalization Ability. In: Antonacopoulos, A., Chaudhuri, S., Chellappa, R., Liu, CL., Bhattacharya, S., Pal, U. (eds) Pattern Recognition. ICPR 2024. Lecture Notes in Computer Science, vol 15326. Springer, Cham. https://doi.org/10.1007/978-3-031-78395-1_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-78395-1_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-78394-4

  • Online ISBN: 978-3-031-78395-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships