Abstract
Following recent advances in morphological neural networks, we propose to study in more depth how Max-plus operators can be exploited to define morphological units and how they behave when incorporated in layers of conventional neural networks. Besides showing that they can be easily implemented with modern machine learning frameworks, we confirm and extend the observation that a Max-plus layer can be used to select important filters and reduce redundancy in its previous layer, without incurring performance loss. Experimental results demonstrate that the filter selection strategy enabled by a Max-plus layer is highly efficient and robust, through which we successfully performed model pruning on two neural network architectures. We also point out that there is a close connection between Maxout networks and our pruned Max-plus networks by comparing their respective characteristics. The code for reproducing our experiments is available online (for code release, please visit https://github.com/yunxiangzhang.).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
This formulation can be easily generalized to the case of convolutional layers.
- 2.
Note that the classical universal approximation theorems for neural networks (see for example [10]) do not hold for networks containing max-plus units.
- 3.
For example, if a training example has label i, then we compare its activation vector \(\mathbf {y} \in \mathbb {R}_{max}^{J}\) with the weight vector \(\mathbf {w}_{\cdot (i+1)}^{m} \in \mathbb {R}_{max}^{J}\).
References
Abadi, M., Barham, P., Chen, J., Chen, Z., et al.: TensorFlow: a system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation, vol. 16, pp. 265–283 (2016)
Charisopoulos, V., Maragos, P.: Morphological perceptrons: geometry and training algorithms. In: Angulo, J., Velasco-Forero, S., Meyer, F. (eds.) ISMM 2017. LNCS, vol. 10225, pp. 3–15. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-57240-6_1
Chen, W., Wilson, J., Tyree, S., Weinberger, K., Chen, Y.: Compressing neural networks with the hashing trick. In: International Conference on Machine Learning, pp. 2285–2294 (2015)
Davidson, J.L., Ritter, G.X.: Theory of morphological neural networks. In: Digital Optical Computing II, vol. 1215, pp. 378–389 (1990)
Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: 14th International Conference on Artificial Intelligence and Statistics, pp. 315–323 (2011)
Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. arXiv preprint arXiv:1302.4389 (2013)
Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: International Conference on Learning Representations (2015)
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, pp. 1135–1143 (2015)
Hassibi, B., Stork, D.G.: Second order derivatives for network pruning: optimal brain surgeon. In: Advances in Neural Information Processing Systems, pp. 164–171 (1993)
Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. University of Toronto, Technical report (2009)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Advances in Neural Information Processing Systems, pp. 598–605 (1990)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient ConvNets. In: International Conference on Learning Representations (2016)
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: ICCV, pp. 2755–2763 (2017)
Luo, J., Wu, J., Lin, W.: ThiNet: a filter level pruning method for deep neural network compression. In: ICCV, pp. 5068–5076 (2017)
Mondal, R., Santra, S., Chanda, B.: Dense morphological network: an universal function approximator. arXiv preprint arXiv:1901.00109 (2019)
Pessoa, L.F., Maragos, P.: MRL-filters: a general class of nonlinear systems and their optimal design for image processing. IEEE Trans. Image Process. 7(7), 966–978 (1998)
Pessoa, L.F., Maragos, P.: Neural networks with hybrid morphological/rank/linear nodes: a unifying framework with applications to handwritten character recognition. Pattern Recogn. 33(6), 945–960 (2000)
Ritter, G.X., Sussner, P.: An introduction to morphological neural networks. In: 13th International Conference on Pattern Recognition, vol. 4, pp. 709–717 (1996)
Ritter, G., Urcid, G.: Lattice algebra approach to single-neuron computation. IEEE Trans. Neural Netw. 14(2), 282–295 (2003)
Rosenblatt, F.: Principles of neurodynamics: perceptrons and the theory of brain mechanisms. Technical report, Cornell Aeronautical Lab Inc., Buffalo, NY (1961)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Sussner, P., Esmi, E.L.: Morphological perceptrons with competitive learning: lattice-theoretical framework and constructive learning algorithm. Inf. Sci. 181(10), 1929–1950 (2011)
Wang, S.: General constructive representations for continuous piecewise-linear functions. IEEE Trans. Circ. Syst. I Regul. Pap. 51(9), 1889–1896 (2004)
Wen, W., Wu, C., et al.: Learning structured sparsity in deep neural networks. In: Advances in Neural Information Processing Systems, pp. 2074–2082 (2016)
Wilson, S.S.: Morphological networks. In: Visual Communications and Image Processing IV, vol. 1199, pp. 483–496 (1989)
Zamora, E., Sossa, H.: Dendrite morphological neurons trained by stochastic gradient descent. Neurocomputing 260, 420–431 (2017)
Acknowledgements
This work was partially funded by a grant from Institut Mines Telecom.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, Y., Blusseau, S., Velasco-Forero, S., Bloch, I., Angulo, J. (2019). Max-Plus Operators Applied to Filter Selection and Model Pruning in Neural Networks. In: Burgeth, B., Kleefeld, A., Naegel, B., Passat, N., Perret, B. (eds) Mathematical Morphology and Its Applications to Signal and Image Processing. ISMM 2019. Lecture Notes in Computer Science(), vol 11564. Springer, Cham. https://doi.org/10.1007/978-3-030-20867-7_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-20867-7_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20866-0
Online ISBN: 978-3-030-20867-7
eBook Packages: Computer ScienceComputer Science (R0)