Skip to main content
SpringerLink
Log in

Symmetrical filters in convolutional neural networks

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Symmetry is present in nature and science. In image processing, kernels for spatial filtering possess some symmetry (e.g. Sobel operators, Gaussian, Laplacian). Convolutional layers in artificial feed-forward neural networks have typically considered the kernel weights without any constraint. We propose to investigate the impact of a symmetry constraint in convolutional layers for image classification tasks, taking our inspiration from the processes involved in the primary visual cortex and common image processing techniques. The goal is to determine if it is necessary to learn each weight of the filters independently, and the extent to which it is possible to enforce symmetrical constraints on the filters throughout the training process of a convolutional neural network by modifying the weight update preformed during the backpropagation algorithm and to evaluate the change in performance. The symmetrical constraint reduces the number of free parameters in the network, and it is able to achieve near identical performance. We address the following cases: x/y-axis symmetry, point reflection, and anti-point reflection. The performance is evaluated on four databases of images representing handwritten digits. The results support the conclusion that while random weights offer more freedom to the model, the symmetry constraint provides a similar level of performance while decreasing substantially the number of free parameters in the model. Such an approach can be valuable in phase-sensitive applications that require a linear phase property throughout the feature extraction process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Basu M (2002) Gaussian-based edge-detection methods—a survey. IEEE Trans Syst Man Cybern Part C Appl Rev 32(3):252–260

  2. Bhattacharya U, Chaudhuri B (2005) Databases for research on recognition of handwritten characters of Indian scripts. In: Proceedings of the 8th international conference on document analysis and recognition (ICDAR’05), pp 789–793

  3. Bhowmick T, Parui S, Bhattacharya U, Shaw B (2006) An HMM based recognition scheme for handwritten oriya numerals. In: Proceedings of the 9th international conference on information technology (ICIT 2006), pp 105–110

  4. Cecotti H, Gräser A (2011) Convolutional neural networks for P300 detection with application to brain-computer interfaces. IEEE Trans Pattern Anal Mach Intell 33(3):433–445

    Article  Google Scholar 

  5. Chaudhuri B (2006) A complete handwritten numeral database of Bangla—a major Indic script. In: Proceedings of the 10th international workshop on frontiers in handwriting recognition (IWFHR’10), pp 1–6

  6. Chaudhuri BB, Pal U (1998) A complete printed Bangla OCR system. Pattern Recognit 31:531–549

    Article  Google Scholar 

  7. Cheng Y, Wang D, Zhou P, Zhang T (2018) Model compression and acceleration for deep neural networks: the principles, progress, and challenges. IEEE Signal Process Mag 35(1):126–136

    Article  Google Scholar 

  8. Cireşan D, Meier U, Schmidhuber J (2012) Multi-column deep neural networks for image classification. In: Computer vision and pattern recognition (CVPR), pp 3642–3649

  9. Cohen TS, Welling M (2016) Group equivalent convolutional networks. In: Proceedings of the 33rd international conference on machine learning, 48:2990–2999

  10. Dieleman S, De Fauw J, Kavukcuoglu K (2016) Exploiting cyclic symmetry in convolutional neural networks. In: Proceedings of the 33rd international conference on machine learning, vol 48

  11. Dieleman S, Willett KW, Dambre J (2015) Rotation-invariant convolutional neural networks for galaxy morphology prediction. Mon Not R Astron Soc 450:1441–1459

    Article  Google Scholar 

  12. Erhan D, Bengio Y, Courville A, Manzagol PA, Vincent P, Bengio S (2010) Why does unsupervised pre-training help deep learning? J Mach Learn Res 11:625–660

    MathSciNet  MATH  Google Scholar 

  13. Fukushima K (1980) A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybern 36(4):193–202

    Article  Google Scholar 

  14. Gonzalez RC, Woods RE (2008) Digital image processing. Pearson, London

    Google Scholar 

  15. Goodfellow I, Bengio Y, Courville A (2017) Deep learning. The MIT Press, Cambridge

    MATH  Google Scholar 

  16. Grill-Spector K, Malach R (2004) The human visual cortex. Annu Rev Neurosci 27:649–77

    Article  Google Scholar 

  17. Hagiwara K, Hayashi T, Kawasaki S, Arakawa F, Endo O, Nomura H, Tsukamoto A, Nguyen D, Nguyen B, Tran A, Hyunh H, Kudoh I, Pham C (2018) A two-stage-pipeline CPU of sh-2 architecture implemented on FPGA and SOC for IOT, edge AI and robotic applications. In: 2018 IEEE symposium in low-power and high-speed chips (COOL CHIPS), pp 1–3

  18. Hassibi B, Stork DG (1993) Second order derivatives for network pruning: optimal brain surgeon. In: Hanson SJ, Cowan JD, Giles CL (eds) Advances in neural information processing systems, vol 5, pp 164–171

  19. Huang G, Bai Z, Kasun LLC, Vong CM (2015) Local receptive fields based extreme learning machine. IEEE Comput Intell Mag 10(2):18–29

    Article  Google Scholar 

  20. Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70:489–501

    Article  Google Scholar 

  21. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and \(<\)0.5 mb model size

  22. Jain AK, Ratha NK, Lakshmanan S (1997) Object detection using Gabor filters. Pattern Recognit 30:295–309

    Article  Google Scholar 

  23. Jarrett K, Kavukcuoglu K, Ranzato M, LeCun Y (2009) What is the best multi-stage architecture for object recognition? In: Proceedings of the 12th international conference on computer vision (ICCV’09), pp 2146–2153

  24. Kamarainen JK, Kyrki V, Kalviainen H (2006) Invariance properties of Gabor filter-based features-overview and applications. IEEE Trans Image Process 15:1088–1099

    Article  Google Scholar 

  25. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural information processing systems 25. Curran Associates, Inc, pp 1097–1105

  26. LeCun Y, Boser B, Denker J, Henderson D, Howard R, Hubbard W, Jackel L (1990) Handwritten digit recognition with a back-propagation network. Adv Neural Inf Process Syst 2:396–404

    Google Scholar 

  27. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  28. LeCun YA, Bottou L, Orr GB, Müller KR (2012) Efficient backprop. In: Montavon G, Orr GB, Müller KR (eds) Neural networks: tricks of the trade, 2nd edn. Springer, Berlin, pp 9–48

  29. Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the international conference on computer vision, vol 2, pp 1150–1157

  30. Roberts P (2011) Signals and systems: analysis using transform methods & MATLAB. McGraw-Hill Education

  31. Ma J, Sheridan RP, Liaw A, Dahl GE, Svetnik V (2015) Deep neural nets as a method for quantitative structure-activity relationships. J Chem Inf Model 55:263–274

    Article  Google Scholar 

  32. Oppenheim AV, Schafer RW (1989) Discrete-time signal processing. Prentice-Hall, Englewood Cliffs

    MATH  Google Scholar 

  33. Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 9(1):62–66

    Article  MathSciNet  Google Scholar 

  34. Pal U, Chaudhuri BB (2004) Indian script character recognition: a survey. Pattern Recognit 37(9):1887–1899

    Article  Google Scholar 

  35. Redmon J, Angelova A (2015) Real-time grasp detection using convolutional neural networks. In: Proceedings of IEEE international conference on robotics and automation (ICRA), pp 1316–1322

  36. Riesenhuber M, Poggio T (1999) Hierarchical models of object recognition in cortex. Nat Neurosci 2:1019 EP

  37. Russell S, Norvig P (2009) Artificial intelligence: a modern approach, 3rd edn. Prentice Hall Press, Upper Saddle River

    MATH  Google Scholar 

  38. Scherbaum F, Bouin MP (1997) FIR filter effects and nucleation phases. Geophys J Int 130:661–668

    Article  Google Scholar 

  39. Sethi IK, Chatterjee B (1977) Machine recognition of constrained hand printed Devanagari. Pattern Recognit 9:69–75

    Article  Google Scholar 

  40. Seto K, Nejatollahi H, An J, Kang S, Dutt N (2019) Small memory footprint neural network accelerators. In: 20th international symposium on quality electronic design (ISQED), pp 253–258

  41. Shang W, Sohn K, Almeida D, Lee H (2016) Understanding and improving convolutional neural networks via concatenated rectified linear units

  42. Shen LL, Ji Z (2009) Gabor wavelet selection and SVM classification for object recognition. Acta Automatica Sinica 35:350–355

    Article  Google Scholar 

  43. Simard P, Steinkraus D, Platt J (2003) Best practices for convolutional neural networks applied to visual document analysis. In: Proceedings of the 7th international conference on document analysis and recognition (ICDAR), pp 958–962

  44. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. CoRR. arXiv:abs/1409.1556

  45. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958

    MathSciNet  MATH  Google Scholar 

  46. Sundararajan D (2003) Digital signal processing: theory and practice. World Scientific, Singapore

    Book  Google Scholar 

  47. Szegedy C, Liu W, Jia Y, Sermanet P, Reed SE, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2014) Going deeper with convolutions. CoRR. arXiv:abs/1409.4842

  48. Vajda S, Roy K, Pal U, Chaudhuri B, Belaïd A (2009) Automation of Indian postal documents written in Bangla and English. Int J Pattern Recognit Artif Intell 23(8):1599–1632

    Article  Google Scholar 

  49. Wu J, Leng C, Wang Y, Hu Q, Cheng J (2016) Quantized convolutional neural networks for mobile devices. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR), pp 4820–4828

  50. Zee A (2007) Fearful symmetry. Princeton University Press, Princeton

    Book  Google Scholar 

  51. Zhai S, Cheng Y, Zhang ZM (2016) Doubly convolutional neural networks. In: Advances in neural information processing systems, pp 1082–1090

  52. Zheng L, Yang Y, Tian Q (2018) SIFT meets CNN: a decade survey of instance retrieval. IEEE Trans PAMI 40(5):1224–1244

    Article  Google Scholar 

  53. Ziou D, Tabbone S (1998) Edge detection techniques: an overview. Int J Pattern Recognit Image Anal 8(4):537–559

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, College of Science and Mathematics, Fresno State, Fresno, CA, USA

    Gregory Dzhezyan & Hubert Cecotti

Authors
  1. Gregory Dzhezyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hubert Cecotti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hubert Cecotti.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dzhezyan, G., Cecotti, H. Symmetrical filters in convolutional neural networks. Int. J. Mach. Learn. & Cyber. 12, 2027–2039 (2021). https://doi.org/10.1007/s13042-021-01290-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-021-01290-z

Keywords

Search

Navigation