Abstract
In this paper, we present the power of max switch locations in convolutional neural networks (CNNs) with two experiments: image reconstruction and classification. First, we realize image reconstruction via a convolutional auto-encoder (CAE) that includes max pooling/unpooling operations in an encoder and decoder, respectively. During decoder operation, we alternate max switch locations extracted from another image, which was chosen from among the real images and noise images. Meanwhile, we set up a classification experiment in a teacher-student manner, allowing the transmission of max switch locations from the teacher network to the student network. During both the training and test phases, we let the student network receive max switch locations from the teacher network, and we observe prediction similarity for both networks while the input to the student network is either randomly shuffled test data or a noise image. Based on the results of both experiments, we conjecture that max switch locations could be another form of distilled knowledge. In a teacher-student scheme, therefore, we present a new max pool method whereby the distilled knowledge improves the performance of the student network in terms of training speed. We plan to implement this method in future work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Boureau, Y.L., Ponce, J., LeCun, Y.: A theoretical analysis of feature pooling in visual recognition. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 111–118 (2010)
Boureau, Y.L., Bach, F., LeCun, Y., Ponce, J.: Learning mid-level features for recognition. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2559–2566. IEEE (2010)
Boureau, Y.L., Le Roux, N., Bach, F., Ponce, J., LeCun, Y.: Ask the locals: multi-way local pooling for image recognition. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2651–2658. IEEE (2011)
Zeiler, M.D., Taylor, G.W., Fergus, R.: Adaptive deconvolutional networks for mid and high level feature learning. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2018–2025. IEEE (2011)
Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1520–1528 (2015)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: European Conference on Computer Vision, pp. 818–833. Springer (2014)
Yang, J., Price, B., Cohen, S., Lee, H., Yang, M.H.: Object contour detection with a fully convolutional encoder-decoder network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 193–202 (2016)
de La Roche Saint Andre, M., Rieger, L., Hannemose, M., Kim, J.: Tunnel effect in CNNS: image reconstruction from max switch locations. IEEE Signal Process. Lett. 24(3), 254–258 (2017)
Zhang, Y., Lee, K., Lee, H., EDU, U.: Augmenting supervised neural networks with unsupervised objectives for large-scale image classification. In: International Conference on Machine Learning (ICML) (2016)
Zhao, J., Mathieu, M., Goroshin, R., LeCun, Y.: Stacked what-where auto-encoders (2015). arXiv:1506.02351
Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: Hints for thin deep nets (2014). arXiv:1412.6550
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Ju, J., Yim, J., Lee, S., Kim, J. (2019). An Exploration of the Power of Max Switch Locations in CNNs. In: Kim, JH., et al. Robot Intelligence Technology and Applications 5. RiTA 2017. Advances in Intelligent Systems and Computing, vol 751. Springer, Cham. https://doi.org/10.1007/978-3-319-78452-6_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-78452-6_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-78451-9
Online ISBN: 978-3-319-78452-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)